Parallel Processing and Applied Mathematics, Part I: 8th International Conference, PPAM 2009, Wroclaw, Poland, September 13-16, 2009 (Lecture Notes in Computer Science, 6067) 364214389X, 9783642143892

Citation preview

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

6067

Roman Wyrzykowski Jack Dongarra Konrad Karczewski Jerzy Wasniewski (Eds.)

Parallel Processing and Applied Mathematics 8th International Conference, PPAM 2009 Wroclaw, Poland, September 13-16, 2009 Revised Selected Papers, Part I

13

Volume Editors Roman Wyrzykowski Konrad Karczewski Czestochowa University of Technology Institute of Computational and Information Sciences, Poland E-mail:{roman, xeno}@icis.pcz.pl Jack Dongarra University of Tennessee, Department of Electrical Engineering and Computer Science, Knoxville, TN 37996-3450, USA E-mail: [email protected] Jerzy Wasniewski Technical University of Denmark, Department of Informatics and Mathematical Modeling, 2800 Kongens Lyngby, Denmark E-mail: [email protected]

Library of Congress Control Number: 2010930174 CR Subject Classification (1998): D.2, H.4, D.4, C.2.4, D.1.3, F.2 LNCS Sublibrary: SL 1 – Theoretical Computer Science and General Issues ISSN ISBN-10 ISBN-13

0302-9743 3-642-14389-X Springer Berlin Heidelberg New York 978-3-642-14389-2 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2010 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper 06/3180

Preface

We are pleased to present the proceedings of the 8th International Conference on Parallel Processing and Applied Mathematics – PPAM 2009, which was held in Wroclaw, Poland, September 13–16, 2009. It was organized by the Department of Computer and Information Sciences of the Czestochowa University of Techno logy, with the help of the Wroclaw University of Technology, Faculty of Computer Science and Management. The main organizer was Roman Wyrzykowski. PPAM is a biennial conference. Seven previous events have been held in different places in Poland since 1994. The proceedings of the last four conferences have been published by Springer in the Lecture Notes in Computer Science series 2003, vol.3019; Pozna´ n, 2005, vol.3911; (Nalecz´  ow, 2001, vol.2328; Czestochowa,  Gda´ nsk, 2007, vol. 4967). The PPAM conferences have become an international forum for exchanging ideas between researchers involved in parallel and distributed computing, including theory and applications, as well as applied and computational mathematics. The focus of PPAM 2009 was on models, algorithms, and software tools which facilitate efficient and convenient utilization of modern parallel and distributed computing architectures, as well as on large-scale applications. This meeting gathered more than 210 participants from 32 countries. A strict refereeing process resulted in the acceptance of 129 contributed presentations, while approximately 46% of the submissions were rejected. Regular tracks of the conference covered such important fields of parallel/distributed/grid computing and applied mathematics as: – – – – – –

Parallel/distributed architectures and mobile computing Numerical algorithms and parallel numerics Parallel and distributed non-numerical algorithms Tools and environments for parallel/distributed/grid computing Applications of parallel/distributed computing Applied mathematics and neural networks

Plenary and Invited Speakers The plenary and invited talks were presented by: – – – –

Srinivas Aluru from the Iowa State University (USA) Dominik Behr from AMD (USA) Ewa Deelman from the University of Southern California (USA) Jack Dongarra from the University of Tennessee and Oak Ridge National Laboratory (USA) – Iain Duff from the Rutherford Appleton Laboratory (UK) – Anne C. Elster from NTNU, Trondheim (Norway)

VI

– – – – – – – – –

Preface

Wolfgang Gentzsch from the DEISA Project Michael Gschwind from the IBM T.J. Watson Research Center (USA) Fred Gustavson from the IBM T.J. Watson Research Center (USA) Simon Holland from Intel (UK) Vladik Kreinovich from the University of Texas at El Paso (USA) Magnus Peterson from the Synective Labs (Sweden) Armin Seyfried from the Juelich Supercomputing Centre (Germany) Boleslaw Szyma´ nski from the Rensselaer Polytechnic Institute (USA) Jerzy Wa´sniewski from the Technical University of Denmark (Denmark)

Workshops and Minisymposia Important and integral parts of the PPAM 2009 conference were the workshops: – Minisymposium on GPU Computing organized by Jos´e R. Herrero from the Universitat Politecnica de Catalunya (Spain), Enrique S. Quintana-Ort´ı from the Universitat Jaime I (Spain), and Robert Strzodka from the Max-PlanckInstitut f¨ ur Informatik (Germany) – The Second Minisymposium on Cell/B.E. Technologies organized by Roman University of Technology (Poland), and Wyrzykowski from the Czestochowa  David A. Bader from the Georgia Institute of Technology (USA) – Workshop on Memory Issues on Multi- and Manycore Platforms organized by Michael Bader and Carsten Trinitis from the TU M¨ unchen (Germany) – Workshop on Novel Data Formats and Algorithms for High-Performance Computing organized by Fred Gustavson from the IBM T.J. Watson Research Center (USA), and Jerzy Wa´sniewski from the Technical University of Denmark (Denmark) – Workshop on Scheduling for Parallel Computing - SPC 2009 organized by Maciej Drozdowski from the Pozna´ n University of Technology (Poland) – The Third Workshop on Language-Based Parallel Programming Models WLPP 2009 organized by Ami Marowka from the Shenkar College of Engineering and Design in Ramat-Gan (Israel) – The Second Workshop on Performance Evaluation of Parallel Applications on Large-Scale Systems organized by Jan Kwiatkowski, Dariusz Konieczny and Marcin Pawlik from the Wroclaw University of Technology (Poland) – The 4th Grid Application and Middleware Workshop - GAMW 2009 organized by Ewa Deelman from the University of Southern California (USA), and Norbert Meyer from the Pozna´ n Supercomputing and Networking Center (Poland) – The 4th Workshop on Large Scale Computations on Grids - LaSCoG 2009 organized by Marcin Paprzycki from IBS PAN and SWPS in Warsaw (Poland), and Dana Petcu from the Western University of Timisoara (Romania) – Workshop on Parallel Computational Biology - PBC 2009 organized by David A. Bader from the Georgia Institute of Technology in Atlanta (USA), Denis Trystram from ID-IMAG in Grenoble (France), Alexandros Stamatakis ˙ from the TU M¨ unchen (Germany), and Jaroslaw Zola from the Iowa State University (USA)

Preface

VII

– Minisymposium on Applications of Parallel Computations in Industry and ˇ Engineering organized by Raimondas Ciegis from the Vilnius Gediminas ˇ Technical University (Lithuania), and Julius Zilinskas from the Institute of Mathematics and Informatics in Vilnius (Lithuania) – The Second Minisymposium on Interval Analysis organized by Vladik Kreinovich from the University of Texas at El Paso (USA), Pawel University of Technology (Poland), Sewastjanow from the Czestochowa  Bartlomiej J. Kubica from the Warsaw University of Technology (Poland), and Jerzy Wa´sniewski from the Technical University of Denmark (Denmark) – Workshop on Complex Collective Systems organized by Pawel Topa and Jaroslaw Was  from the AGH University of Science and Technology in Cracow (Poland)

Tutorials The PPAM 2009 meeting began with four tutorials: – GPUs, OpenCL and Scientific Computing, by Robert Strzodka from the Max-Planck-Institut f¨ ur Informatik (Germany), Dominik Behr from AMD (USA), and Dominik G¨ oddeke from the University of Dortmund (Germany) – FPGA Programming for Scientific Computing, by Magnus Peterson from the Synective Labs (Sweden) – Programming the Cell Broadband Engine, by Maciej Remiszewski from IBM (Poland), and Maciej Cytowski from the University of Warsaw (Poland) – New Data Structures Are Necessary and Sufficient for Dense Linear Algebra Factorization Algorithms, by Fred Gustavson from the the IBM T.J. Watson Research Center (USA), and Jerzy Wa´sniewski from the Technical University of Denmark (Denmark)

Best Poster Award The PPAM Best Poster Award is given to the best poster on display at the PPAM conferences, and was first awarded at PPAM 2009. This award is bestowed by the Program Committee members to the presenting author(s) of the best poster. The selection criteria are based on the scientific content and on the quality of the poster presentation. The PPAM 2009 winner was Tomasz Olas University of Technology, who presented the poster “Parfrom the Czestochowa  allel Adaptive Finite Element Package with Dynamic Load Balancing for 3D Thermomechanical Problems.”

New Topics at PPAM 2009 GPU Computing: The recent advances in the hardware, functionality, and programmability of graphics processors (GPUs) have greatly increased their appeal

VIII

Preface

as add-on co-processors for general-purpose computing. With the involvement of the largest processor manufacturers and the strong interest from researchers of various disciplines, this approach has moved from a research niche to a forwardlooking technique for heterogeneous parallel computing. Scientific and industry researchers are constantly finding new applications for GPUs in a wide variety of areas, including image and video processing, molecular dynamics, seismic simulation, computational biology and chemistry, fluid dynamics, weather forecast, computational finance, and many others. GPU hardware has evolved over many years from graphics pipelines with many heterogeneous fixed-function components over partially programmable architectures towards a more and more homogeneous general purpose design, although some fixed-function hardware has remained because of its efficiency. The general-purpose computing on GPU (GPGPU) revolution started with programmable shaders; later, NVIDIA Compute Unified Device Architecture (CUDA) and to a smaller extent AMD Brook+ brought GPUs into the mainstream of parallel computing. The great advantage of CUDA is that it defines an abstraction which presents the underlying hardware architecture as a sea of hundreds of fine-grained computational units with synchronization primitives on multiple levels. With OpenCL there is now also a vendor-independent high-level parallel programming language and an API that offers the same type of hardware abstraction. GPU are very versatile accelerators because besides the high hardware parallelism they also feature a high bandwidth connection to dedicated device memory. The latency problem of DRAM is tackled via a sophisticated thread scheduling and switching mechanism on-chip that continues the processing of the next thread as soon as the previous stalls on a data read. These characteristics make GPUs suitable for both compute- and data-intensive parallel processing. The PPAM 2009 conference recognized the great impact of GPUs by including in its scientific program two major related events: a minisymposium on GPU Computing, and a full day tutorial on “GPUs, OpenCL and Scientific Computing.” The minisymposium received 18 submissions, of which 10 were accepted (55%). The contributions were organized in three sessions. The first group was related to Numerics, and comprised the following papers: “Finite Element Numerical Integration on GPUs,”“ Reduction to Condensed Forms for Symmetric Eigenvalue Problems on Multi-core Architectures,”“On Parallelizing the MRRR Algorithm for Data-Parallel Coprocessors,” and “A Fast GPU Implementation for Solving Sparse Ill-Posed Linear Equation Systems.” The second session dealt with Applications. The papers presented were: “Simulations of the Electrical Activity in the Heart with Graphic Processing Units,”“Stream Processing on GPUs Using Distributed Multimedia Middleware,” and “A GPU Approach to the Simulation of Spatio–temporal Dynamics in Ultrasonic Resonators.” Finally, a third session about General GPU Computing included presentations of three papers: “Fast In-Place Sorting with CUDA Based on Bitonic Sort,” “Parallel Minimax

Preface

IX

Tree Searching on GPU,” and “Modeling and Optimizing the Power Performance of Large Matrices Multiplication on Multi-core and GPU Platform with CUDA.” The tutorial covered a wide variety of GPU topics and also offered handson examples of OpenCL programming that any paticipant could experiment with on their laptop. The morning sessions discussed the basics of GPU architecture, ready-to-use libraries and OpenCL. The afternoon session went in depth on OpenCL and scientific computing on GPUs. All slides are available at http://gpgpu.org/ppam2009. Complex Collective Systems: Collective aspects of complex systems are attracting an increasing community of researchers working in different fields and dealing with theoretical aspects as well as practical applications. In particular, analyzing local interactions and simple rules makes it possible to model complex phenomena efficiently. Collective systems approaches show great promise in establishing scientific methods that could successfully be applied across a variety of application fields. Many studies in complex collective systems science follow either a cellular automata (CA) method or an agent-based approach. Hybridization between these two complementary approaches gives a promising perspective. The majority of work presented during the workshop on complex collective systems represents the hybrid approach. We can distinguish four groups of subjects presented during the workshop. The first group was modeling of pedestrian dynamics: Armin Seyfried from the Juelich Supercomputing Center presented actual challenges in pedestrian dynamics modeling. Another important issue of crowd modeling was also taken into account during the workshop: modeling of stop-and-go waves (Andrea Portz and Armin Seyfried), calibration of pedestrian stream models (Wolfram Klein, Gerta K¨ oster and Andreas Meister), parallel design patterns in a pedestrian simulation (Sarah Clayton), floor fields models based on CA (Ekaterina Kirik, Tat’yana Yurgel’yan and Dmitriy Krouglov), and discrete potential field construction (Konrad Kulakowski and Jaroslaw Was).  The second group dealt with models of car traffic: a fuzzy cellular model of traffic (Bartlomiej Placzek), and an adaptive time gap car-following model (Antoine Tordeux and Pascal Bouvry). The third group included work connected with cryptography based on cellular automata: weakness analysis of a key stream generator (Frederic Pinel and Pascal Bouvry), and properties of safe CA-based S-Boxes (Miroslaw Szaban and Franciszek Seredy´ nski). The fourth group dealt with various applications in a field of complex collective systems: frustration and collectivity in spatial networks (Anna Ma´ nkaKraso´ n, Krzysztof Kulakowski), lava flow hazard modeling (Maria Vittoria Avolio, Donato D’Ambrosio, Valeria Lupiano, Rocco Rongo and William Spataro), FPGA realization of a CA-based epidemic processor (Pavlos Progias, Emmanouela Vardaki and Georgios Sirakoulis)

X

Preface

Acknowledgements The organizers are indebted to the PPAM 2009 sponsors, whose support was vital to the success of the conference. The main sponsor was the Intel Corporation. The other sponsors were: Hewlett-Packard Company, Microsoft Corporation, IBM Corporation, Action S.A., and AMD. We thank to all members of the International Program Committee and additional reviewers for their diligent work in refereeing the submitted papers. Finally, we thank all of the local organizers from the Czestochowa University of Technology and Wroclaw University  of Technology who helped us to run the event very smoothly. We are especially indebted to Gra˙zyna Kolakowska, Urszula Kroczewska, L  ukasz Kuczy´ nski, and University of Technology; and to Jerzy Marcin Wo´zniak from the Czestochowa  ´ atek, and Jan Kwiatkowski from the Wroclaw University of Technology. Swi 

PPAM 2011 We hope that this volume will be useful to you. We would like everyone who reads it to feel invited to the next conference, PPAM 2011, which will be held September 11–14, 2011, in Toru´ n, a city in northern Poland where the great astronomer Nicolaus Copernicus was born. February 2010

Roman Wyrzykowski Jack Dongarra Konrad Karczewski Jerzy Wa´sniewski

Organization

Program Committee Jan Weglarz  Roman Wyrzykowski Boleslaw Szyma´ nski Peter Arbenz Piotr Bala David A. Bader Michael Bader Mark Baker Radim Blaheta Jacek Bla˙zewicz Leszek Borzemski Pascal Bouvry Tadeusz Burczy´ nski Jerzy Brzezi´ nski Marian Bubak ˇ Raimondas Ciegis Andrea Clematis Zbigniew Czech Jack Dongarra Maciej Drozdowski Erik Elmroth Anne C. Elster Mariusz Flasi´ nski Maria Ganzha Jacek Gondzio Andrzej Go´sci´ nski Laura Grigori Frederic Guinand Jos´e R. Herrero Ladislav Hluchy Ondrej Jakl Emmanuel Jeannot Grzegorz Kamieniarz Alexey Kalinov Ayse Kiper

Pozna´ n University of Technology, Poland Honorary Chair University of Technology, Poland Czestochowa  Chair Rensselaer Polytechnic Institute, USA Vice-Chair ETH, Zurich, Switzerland N. Copernicus University, Poland Georgia Institute of Technology, USA TU M¨ unchen, Germany University of Reading, UK Institute of Geonics, Czech Academy of Sciences Pozna´ n University of Technology, Poland Wroclaw University of Technology, Poland University of Luxembourg Silesia University of Technology, Poland Pozna´ n University of Technology, Poland Institute of Computer Science, AGH, Poland Vilnius Gediminas Tech. University, Lithuania IMATI-CNR, Italy Silesia University of Technology, Poland University of Tennessee and ORNL, USA Pozna´ n University of Technology, Poland Umea University, Sweden NTNU, Trondheim, Norway Jagiellonian University, Poland IBS PAN, Warsaw, Poland University of Edinburgh, Scotland, UK Deakin University, Australia INRIA, France Universit´e du Havre, France Universitat Politecnica de Catalunya, Barcelona, Spain Slovak Academy of Sciences, Bratislava Institute of Geonics, Czech Academy of Sciences INRIA, France A. Mickiewicz University, Pozna´ n, Poland Cadence Design System, Russia Middle East Technical University, Turkey

XII

Organization

Jacek Kitowski Jozef Korbicz Stanislaw Kozielski Dieter Kranzlmueller

Institute of Computer Science, AGH, Poland University of Zielona G´ ora, Poland Silesia University of Technology, Poland Ludwig Maximillian University, Munich, and Leibniz Supercomputing Centre, Germany Henryk Krawczyk Gda´ nsk University of Technology, Poland Piotr Krzy˙zanowski University of Warsaw, Poland Jan Kwiatkowski Wroclaw University of Technology, Poland Giulliano Laccetti University of Naples, Italy Marco Lapegna University of Naples, Italy Alexey Lastovetsky University College Dublin, Ireland Vyacheslav I. Maksimov Ural Branch, Russian Academy of Sciences Victor E. Malyshkin Siberian Branch, Russian Academy of Sciences Tomas Margalef Universitat Autonoma de Barcelona, Spain Ami Marowka Shenkar College of Engineering and Design, Israel Norbert Meyer PSNC, Pozna´ n, Poland Jarek Nabrzyski University of Notre Dame, USA Marcin Paprzycki IBS PAN and SWPS, Warsaw, Poland Dana Petcu Western University of Timisoara, Romania Enrique S. Quintana-Ort´ı Universitat Jaime I, Spain Yves Robert Ecole Normale Superieure de Lyon, France Jacek Rokicki Warsaw University of Technology, Poland University of Technology, Poland Leszek Rutkowski Czestochowa  Franciszek Seredy´ nski Polish Academy of Sciences and Polish-Japanese Institute of Information Technology, Warsaw, Poland Robert Schaefer Institute of Computer Science, AGH, Poland Jurij Silc Jozef Stefan Institute, Slovenia Peter M.A. Sloot University of Amsterdam, The Netherlands Masha Sosonkina Ames Laboratory and Iowa State University, USA Leonel Sousa Technical University Lisbon, Portugal Maciej Stroi´ nski PSNC, Pozna´ n, Poland Domenico Talia University of Calabria, Italy Andrei Tchernykh CICESE, Ensenada, Mexico Carsten Trinitis TU M¨ unchen, Germany Roman Trobec Jozef Stefan Institute, Slovenia Denis Trystram ID-IMAG, Grenoble, France Marek Tudruj Polish Academy of Sciences and Polish-Japanese Institute of Information Technology, Warsaw, Poland Pavel Tvrdik Czech Technical University, Prague Jens Volkert Johannes Kepler University, Linz, Austria Jerzy Wa´sniewski Technical University of Denmark Bogdan Wiszniewski Gda´ nsk University of Technology, Poland Ramin Yahyapour University of Dortmund, Germany Jianping Zhu University of Texas at Arlington, USA

Table of Contents – Part I

Parallel/Distributed Architectures and Mobile Computing R R Evaluating Performance of New Quad-Core IntelXeon 5500 Family Processors for HPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pawel Gepner, David L. Fraser, and Michal F. Kowalik

Interval Wavelength Assignmentin All-Optical Star Networks . . . . . . . . . . Robert Janczewski, Anna Malafiejska, and Michal Malafiejski Graphs Partitioning: An Optimal MIMD Queueless Routing for BPC-Permutations on Hypercubes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jean-Pierre Jung and Ibrahima Sakho Probabilistic Packet Relaying in Wireless Mobile Ad Hoc Networks . . . . . Marcin Seredynski, Tomasz Ignac, and Pascal Bouvry

1 11

21 31

Numerical Algorithms and Parallel Numerics On the Performance of a New Parallel Algorithm for Large-Scale Simulations of Nonlinear Partial Differential Equations . . . . . . . . . . . . . . . ´ Juan A. Acebr´ on, Angel Rodr´ıguez-Rozas, and Renato Spigler Partial Data Replication as a Strategy for Parallel Computing of the Multilevel Discrete Wavelet Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Liesner Acevedo, Victor M. Garcia, Antonio M. Vidal, and Pedro Alonso Dynamic Load Balancing for Adaptive Parallel Flow Problems . . . . . . . . . Stanislaw Gepner, Jerzy Majewski, and Jacek Rokicki A Balancing Domain Decomposition Method for a Discretization of a Plate Problem on Nonmatching Grids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Leszek Marcinkowski Application Specific Processors for the Autoregressive Signal Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anatolij Sergiyenko, Oleg Maslennikow, Piotr Ratuszniak, Natalia Maslennikowa, and Adam Tomas A Parallel Non-square Tiled Algorithm for Solving a Kind of BVP for Second-Order ODEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Przemyslaw Stpiczy´ nski

41

51

61

70

80

87

XIV

Table of Contents – Part I

Graph Grammar Based Petri Nets Model of Concurrency for Self-adaptive hp-Finite Element Method with Rectangular Elements . . . . Arkadiusz Szymczak and Maciej Paszy´ nski

95

Numerical Solution of the Time and Rigidity Dependent Three Dimensional Second Order Partial Differential Equation . . . . . . . . . . . . . . . Anna Wawrzynczak and Michael V. Alania

105

Hardware Implementation of the Exponent Based Computational Core for an Exchange-Correlation Potential Matrix Generation . . . . . . . . . . . . . Maciej Wielgosz, Ernest Jamro, and Kazimierz Wiatr

115

Parallel Implementation of Conjugate Gradient Method on Graphics Processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcin Wozniak, Tomasz Olas, and Roman Wyrzykowski

125

Iterative Solution of Linear and Nonlinear Boundary Problems Using PIES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Eugeniusz Zieniuk and Agnieszka Boltuc

136

Paralel and Distributed Non-numerical Algorithms Implementing a Parallel Simulated Annealing Algorithm . . . . . . . . . . . . . . Zbigniew J. Czech, Wojciech Mikanik, and Rafal Skinderowicz

146

Parallel Computing Scheme for Graph Grammar-Based Syntactic Pattern Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mariusz Flasi´ nski, Janusz Jurek, and Szymon My´sli´ nski

156

Extended Cascaded Star Schema for Distributed Spatial Data Warehouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcin Gorawski

166

Parallel Longest Increasing Subsequences in Scalable Time and Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Peter Krusche and Alexander Tiskin

176

A Scalable Parallel Union-Find Algorithm for Distributed Memory Computers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fredrik Manne and Md. Mostofa Ali Patwary

186

Tools and Environments for Parallel/Distributed/Grid Computing Extracting Both Affine and Non-linear Synchronization-Free Slices in Program Loops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wlodzimierz Bielecki and Marek Palkowski

196

Table of Contents – Part I

XV

A Flexible Checkpoint/Restart Model in Distributed Systems . . . . . . . . . . Mohamed-Slim Bouguerra, Thierry Gautier, Denis Trystram, and Jean-Marc Vincent

206

A Formal Approach to Replica Consistency in Directory Service . . . . . . . Jerzy Brzezi´ nski, Cezary Sobaniec, and Dariusz Wawrzyniak

216

Software Security in the Model for Service Oriented Architecture Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Grzegorz Kolaczek and Adam Wasilewski

226

Automatic Program Parallelization for Multicore Processors . . . . . . . . . . . Jan Kwiatkowski and Radoslaw Iwaszyn

236

Request Distribution in Hybrid Processing Environments . . . . . . . . . . . . . Jan Kwiatkowski, Mariusz Fras, Marcin Pawlik, and Dariusz Konieczny

246

Vine Toolkit - Grid-Enabled Portal Solution for Community Driven Computing Workflows with Meta-Scheduling Capabilities . . . . . . . . . . . . . Dawid Szejnfeld, Piotr Domagalski, Piotr Dziubecki, Piotr Kopta, Michal Krysinski, Tomasz Kuczynski, Krzysztof Kurowski, Bogdan Ludwiczak, Jaroslaw Nabrzyski, Tomasz Piontek, Dominik Tarnawczyk, Krzysztof Witkowski, and Malgorzata Wolniewicz

256

Applications of Parallel/Distributed Computing GEM – A Platform for Advanced Mathematical Geosimulations . . . . . . . . Radim Blaheta, Ondˇrej Jakl, Roman Kohut, and Jiˇr´ı Star´y Accelerating the MilkyWay@Home Volunteer Computing Project with GPUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Travis Desell, Anthony Waters, Malik Magdon-Ismail, Boleslaw K. Szymanski, Carlos A. Varela, Matthew Newby, Heidi Newberg, Andreas Przystawik, and David Anderson Vascular Network Modeling - Improved Parallel Implementation on Computing Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Krzysztof Jurczuk, Marek Kr¸etowski, and Johanne B´ezy-Wendling Parallel Adaptive Finite Element Package with Dynamic Load Balancing for 3D Thermo-Mechanical Problems . . . . . . . . . . . . . . . . . . . . . . Tomasz Olas, Robert Le´sniak, Roman Wyrzykowski, and Pawel Gepner Parallel Implementation of Multidimensional Scaling Algorithm Based on Particle Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Piotr Pawliczek and Witold Dzwinel

266

276

289

299

312

XVI

Table of Contents – Part I

Particle Model of Tumor Growth and Its Parallel Implementation . . . . . . Rafal Wcislo and Witold Dzwinel

322

Applied Mathematics and Neural Networks Modular Neuro-Fuzzy Systems Based on Generalized Parametric Triangular Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcin Korytkowski and Rafal Scherer Application of Stacked Methods to Part-of-Speech Tagging of Polish . . . . Marcin Kuta, Wojciech W´ ojcik, Michal Wrzeszcz, and Jacek Kitowski Computationally Efficient Nonlinear Predictive Control Based on State-Space Neural Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maciej L  awry´ nczuk Relational Type-2 Interval Fuzzy Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . Rafal Scherer and Janusz T. Starczewski Properties of Polynomial Bases Used in a Line-Surface Intersection Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gun Srijuntongsiri and Stephen A. Vavasis

332 340

350 360

369

Minisymposium on GPU Computing A GPU Approach to the Simulation of Spatio–temporal Dynamics in Ultrasonic Resonators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pedro Alonso–Jord´ a, Isabel P´erez–Arjona, and Victor J. S´ anchez–Morcillo Reduction to Condensed Forms for Symmetric Eigenvalue Problems on Multi-core Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Paolo Bientinesi, Francisco D. Igual, Daniel Kressner, and Enrique S. Quintana-Ort´ı On Parallelizing the MRRR Algorithm for Data-Parallel Coprocessors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christian Lessig and Paolo Bientinesi

379

387

396

Fast In-Place Sorting with CUDA Based on Bitonic Sort . . . . . . . . . . . . . . Hagen Peters, Ole Schulz-Hildebrandt, and Norbert Luttenberger

403

Finite Element Numerical Integration on GPUs . . . . . . . . . . . . . . . . . . . . . . Przemyslaw Plaszewski, Pawel Maciol, and Krzysztof Bana´s

411

Modeling and Optimizing the Power Performance of Large Matrices Multiplication on Multi-core and GPU Platform with CUDA . . . . . . . . . . Da Qi Ren and Reiji Suda

421

Table of Contents – Part I

Stream Processing on GPUs Using Distributed Multimedia Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Michael Repplinger and Philipp Slusallek Simulations of the Electrical Activity in the Heart with Graphic Processing Units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bernardo M. Rocha, Fernando O. Campos, Gernot Plank, Rodrigo W. dos Santos, Manfred Liebmann, and Gundolf Haase Parallel Minimax Tree Searching on GPU . . . . . . . . . . . . . . . . . . . . . . . . . . . Kamil Rocki and Reiji Suda A Fast GPU Implementation for Solving Sparse Ill-Posed Linear Equation Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Florian Stock and Andreas Koch

XVII

429

439

449

457

The Second Minisymposium on Cell/B.E. Technologies Monte Carlo Simulations of Spin Glass Systems on the Cell Broadband Engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Francesco Belletti, Marco Guidetti, Andrea Maiorano, Filippo Mantovani, Sebastiano Fabio Schifano, and Raffaele Tripiccione

467

Montgomery Multiplication on the Cell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Joppe W. Bos and Marcelo E. Kaihara

477

An Exploration of CUDA and CBEA for Einstein@Home . . . . . . . . . . . . . Jens Breitbart and Gaurav Khanna

486

Introducing the Semi-stencil Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ra´ ul de la Cruz, Mauricio Araya-Polo, and Jos´e Mar´ıa Cela

496

Astronomical Period Searching on the Cell Broadband Engine . . . . . . . . . Maciej Cytowski, Maciej Remiszewski, and Igor Soszy´ nski

507

Finite Element Numerical Integration on PowerXCell Processors . . . . . . . Filip Kru˙zel and Krzysztof Bana´s

517

The Implementation of Regional Atmospheric Model Numerical Algorithms for CBEA-Based Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dmitry Mikushin and Victor Stepanenko

525

Adaptation of Double-Precision Matrix Multiplication to the Cell Broadband Engine Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Krzysztof Rojek and L  ukasz Szustak

535

XVIII

Table of Contents – Part I

Optimization of FDTD Computations in a Streaming Model Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adam Smyk and Marek Tudruj

547

Workshop on Memory Issues on Multi- and Manycore Platforms An Orthogonal Matching Pursuit Algorithm for Image Denoising on the Cell Broadband Engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dominik Bartuschat, Markus St¨ urmer, and Harald K¨ ostler

557

A Blocking Strategy on Multicore Architectures for Dynamically Adaptive PDE Solvers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wolfgang Eckhardt and Tobias Weinzierl

567

Affinity-On-Next-Touch: An Extension to the Linux Kernel for NUMA Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stefan Lankes, Boris Bierbaum, and Thomas Bemmerl

576

Multi–CMP Module System Based on a Look-Ahead Configured Global Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Eryk Laskowski, L  ukasz Ma´sko, and Marek Tudruj

586

Empirical Analysis of Parallelism Overheads on CMPs . . . . . . . . . . . . . . . . Ami Marowka

596

An Implementation of Parallel 3-D FFT with 2-D Decomposition on a Massively Parallel Cluster of Multi-Core Processors . . . . . . . . . . . . . . . . . . Daisuke Takahashi

606

Introducing a Performance Model for Bandwidth-Limited Loop Kernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jan Treibig and Georg Hager

615

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

625

Table of Contents – Part II

Workshop on Scheduling for Parallel Computing (SPC 2009) Fully Polynomial Time Approximation Schemes for Scheduling Divisible Loads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Joanna Berli´ nska Semi-online Preemptive Scheduling: Study of Special Cases . . . . . . . . . . . . Tom´ aˇs Ebenlendr Fast Multi-objective Reschulding of Grid Jobs by Heuristics and Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wilfried Jakob, Alexander Quinte, Karl-Uwe Stucky, and Wolfgang S¨ uß

1

11

21

Comparison of Program Task Scheduling Algorithms for Dynamic SMP Clusters with Communication on the Fly . . . . . . . . . . . . . . . . . . . . . . . . . . . . L  ukasz Ma´sko, Marek Tudruj, Gregory Mounie, and Denis Trystram

31

Study on GEO Metaheuristic for Solving Multiprocessor Scheduling Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Piotr Switalski and Franciszek Seredynski

42

Online Scheduling of Parallel Jobs on Hypercubes: Maximizing the Throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ondˇrej Zaj´ıˇcek, Jiˇr´ı Sgall, and Tom´ aˇs Ebenlendr

52

The Third Workshop on Language-Based Parallel Programming Models (WLPP 2009) Verification of Causality Requirements in Java Memory Model Is Undecidable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Matko Botinˇcan, Paola Glavan, and Davor Runje

62

A Team Object for CoArray Fortran . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Robert W. Numrich

68

On the Definition of Service Abstractions for Parallel Computing . . . . . . Herv´e Paulino

74

XX

Table of Contents – Part II

The Second Workshop on Performance Evaluation of Parallel Applications on Large-Scale Systems Performance Debugging of Parallel Compression on Multicore Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Janusz Borkowski Energy Considerations for Divisible Load Processing . . . . . . . . . . . . . . . . . . Maciej Drozdowski Deskilling HPL: Using an Evolutionary Algorithm to Automate Cluster Benchmarking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dominic Dunlop, S´ebastien Varrette, and Pascal Bouvry Monitoring of SLA Parameters within VO for the SOA Paradigm . . . . . . Wlodzimierz Funika, Bartosz Kryza, Renata Slota, Jacek Kitowski, Kornel Skalkowski, Jakub Sendor, and Dariusz Krol

82 92

102 115

A Role-Based Approach to Self-healing in Autonomous Monitoring Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wlodzimierz Funika and Piotr P¸egiel

125

Parallel Performance Evaluation of MIC(0) Preconditioning Algorithm for Voxel μFE Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ivan Lirkov, Yavor Vutov, Marcin Paprzycki, and Maria Ganzha

135

Parallel HAVEGE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alin Suciu, Tudor Carean, Andre Seznec, and Kinga Marton

145

The Fourth Grid Applications and Middleware Workshop (GAMW 2009) UNICORE Virtual Organizations System . . . . . . . . . . . . . . . . . . . . . . . . . . . Krzysztof Benedyczak, Marcin Lewandowski, Aleksander Nowi´ nski, and Piotr Bala Application of ADMIRE Data Mining and Integration Technologies in Environmental Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marek Ciglan, Ondrej Habala, Viet Tran, Ladislav Hluchy, Martin Kremler, and Martin Gera

155

165

Performance Based Matchmaking on Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . Andrea Clematis, Angelo Corana, Daniele D’Agostino, Antonella Galizia, and Alfonso Quarati

174

Replica Management for National Data Storage . . . . . . . . . . . . . . . . . . . . . . Renata Slota, Darin Nikolow, Marcin Kuta, Mariusz Kapanowski, Kornel Skalkowski, Marek Pogoda, and Jacek Kitowski

184

Table of Contents – Part II

Churn Tolerant Virtual Organization File System for Grids . . . . . . . . . . . . Leif Lindb¨ ack, Vladimir Vlassov, Shahab Mokarizadeh, and Gabriele Violino

XXI

194

The Fourth Workshop on Large Scale Computations on Grids (LaSCoG 2009) Quasi-random Approach in the Grid Application SALUTE . . . . . . . . . . . . Emanouil Atanassov, Aneta Karaivanova, and Todor Gurov

204

Mobile Agents for Management of Native Applications in GRID . . . . . . . Rocco Aversa, Beniamino Di Martino, Renato Donini, and Salvatore Venticinque

214

Leveraging Complex Event Processing for Grid Monitoring . . . . . . . . . . . . Bartosz Balis, Bartosz Kowalewski, and Marian Bubak

224

Designing Execution Control in Programs with Global Application States Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Janusz Borkowski and Marek Tudruj

234

Distributed MIND - A New Processing Model Based on Mobile Interactive Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Magdalena Godlewska and Bogdan Wiszniewski

244

A Framework for Observing Dynamics of Agent-Based Computations . . . Jaroslaw Kawecki and Maciej Smolka HyCube: A DHT Routing System Based on a Hierarchical Hypercube Geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Artur Olszak

250

260

Workshop on Parallel Computational Biology (PBC 2009) Accuracy and Performance of Single versus Double Precision Arithmetics for Maximum Likelihood Phylogeny Reconstruction . . . . . . . Simon A. Berger and Alexandros Stamatakis

270

Automated Design of Assemblable, Modular, Synthetic Chromosomes . . . Sarah M. Richardson, Brian S. Olson, Jessica S. Dymond, Randal Burns, Srinivasan Chandrasegaran, Jef D. Boeke, Amarda Shehu, and Joel S. Bader

280

GPU Parallelization of Algebraic Dynamic Programming . . . . . . . . . . . . . . Peter Steffen, Robert Giegerich, and Mathieu Giraud

290

Parallel Extreme Ray and Pathway Computation . . . . . . . . . . . . . . . . . . . . Marco Terzer and J¨ org Stelling

300

XXII

Table of Contents – Part II

Minisymposium on Applications of Parallel Computation in Industry and Engineering Parallelized Transient Elastic Wave Propagation in Orthotropic Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Peter Arbenz, J¨ urg Bryner, and Christine Tobler

310

Parallel Numerical Solver for Modelling of Electromagnetic Properties of Thin Conductive Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ˇ ˇ ˇ Raimondas Ciegis, Zilvinas Kancleris, and Gediminas Slekas

320

Numerical Health Check of Industrial Simulation Codes from HPC Environments to New Hardware Technologies . . . . . . . . . . . . . . . . . . . . . . . . Christophe Denis

330

Application of Parallel Technologies to Modeling Lithosphere Dynamics and Seismicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Boris Digas, Lidiya Melnikova, and Valerii Rozenberg

340

AMG for Linear Systems in Engine Flow Simulations . . . . . . . . . . . . . . . . . Maximilian Emans Parallel Implementation of a Steady State Thermal and Hydraulic Analysis of Pipe Networks in OpenMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mykhaylo Fedorov High-Performance Ocean Color Monte Carlo Simulation in the Geo-info Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tamito Kajiyama, Davide D’Alimonte, Jos´e C. Cunha, and Giuseppe Zibordi EULAG Model for Multiscale Flows – Towards the Petascale Generation of Mesoscale Numerical Weather Prediction . . . . . . . . . . . . . . . Zbigniew P. Piotrowski, Marcin J. Kurowski, Bogdan Rosa, and Michal Z. Ziemianski

350

360

370

380

Parallel Implementation of Particle Tracking and Collision in a Turbulent Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bogdan Rosa and Lian-Ping Wang

388

A Distributed Multilevel Ant-Colony Approach for Finite Element Mesh Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ˇ Katerina Taˇskova, Peter Koroˇsec, and Jurij Silc

398

Minisymposium on Interval Analysis Toward Definition of Systematic Criteria for the Comparison of Verified Solvers for Initial Value Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ekaterina Auer and Andreas Rauh

408

Table of Contents – Part II

Fuzzy Solution of Interval Nonlinear Equations . . . . . . . . . . . . . . . . . . . . . . Ludmila Dymova

XXIII

418

Solving Systems of Interval Linear Equations with Use of Modified Interval Division Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ludmila Dymova, Mariusz Pilarek, and Roman Wyrzykowski

427

Remarks on Algorithms Implemented in Some C++ Libraries for Floating-Point Conversions and Interval Arithmetic . . . . . . . . . . . . . . . . . . Malgorzata A. Jankowska

436

An Interval Method for Seeking the Nash Equilibria of Non-Cooperative Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bartlomiej Jacek Kubica and Adam Wo´zniak

446

From Gauging Accuracy of Quantity Estimates to Gauging Accuracy and Resolution of Measuring Physical Fields . . . . . . . . . . . . . . . . . . . . . . . . . Vladik Kreinovich and Irina Perfilieva

456

A New Method for Normalization of Interval Weights . . . . . . . . . . . . . . . . . Pavel Sevastjanov, Pavel Bartosiewicz, and Kamil Tkacz

466

A Global Optimization Method for Solving Parametric Linear Systems Whose Input Data Are Rational Functions of Interval Parameters . . . . . . Iwona Skalna

475

Direct Method for Solving Parametric Interval Linear Systems with Non-affine Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Iwona Skalna

485

Workshop on Complex Collective Systems Evaluating Lava Flow Hazard at Mount Etna (Italy) by a Cellular Automata Based Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maria Vittoria Avolio, Donato D’Ambrosio, Valeria Lupiano, Rocco Rongo, and William Spataro

495

Application of CoSMoS Parallel Design Patterns to a Pedestrian Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sarah Clayton, Neil Urquhard, and Jon Kerridge

505

Artificial Intelligence of Virtual People in CA FF Pedestrian Dynamics Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ekaterina Kirik, Tat’yana Yurgel’yan, and Dmitriy Krouglov

513

Towards the Calibration of Pedestrian Stream Models . . . . . . . . . . . . . . . . Wolfram Klein, Gerta K¨ oster, and Andreas Meister

521

XXIV

Table of Contents – Part II

Two Concurrent Algorithms of Discrete Potential Field Construction . . . Konrad Kulakowski and Jaroslaw Was 

529

Frustration and Collectivity in Spatial Networks . . . . . . . . . . . . . . . . . . . . . Anna Ma´ nka-Kraso´ n and Krzysztof Kulakowski

539

Weakness Analysis of a Key Stream Generator Based on Cellular Automata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fr´ed´eric Pinel and Pascal Bouvry

547

Fuzzy Cellular Model for On-line Traffic Simulation . . . . . . . . . . . . . . . . . . Bartlomiej Placzek

553

Modeling Stop-and-Go Waves in Pedestrian Dynamics . . . . . . . . . . . . . . . . Andrea Portz and Armin Seyfried

561

FPGA Realization of a Cellular Automata Based Epidemic Processor . . . Pavlos Progias, Emmanouela Vardaki, and Georgios Ch. Sirakoulis

569

Empirical Results for Pedestrian Dynamics at Bottlenecks . . . . . . . . . . . . . Armin Seyfried and Andreas Schadschneider

575

Properties of Safe Cellular Automata-Based S-Boxes . . . . . . . . . . . . . . . . . . Miroslaw Szaban and Franciszek Seredynski

585

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

593

Evaluating Performance of New Quad-Core R R 5500 Family Processors for HPC IntelXeon Pawel Gepner, David L. Fraser, and Michal F. Kowalik Intel Corporation {pawel.gepner,david.l.fraser,michal.f.kowalik}@intel.com

Abstract. In this paper we take a look at what the new Quad-Core Intel Xeon Processor code name Nehalem brings to high performance computing. We compare Intel Xeon 5400 series based system with a server utilizing his successor the new Intel Xeon X5560. We compare both CPU generations utilizing dual socket platforms using a number of HPC benchmarks. The results clearly prove that the new Intel Xeon processor 5500 family provide significant performance advantage on typical HPC workloads and demonstrate to be a right choice for many of HPC installations. Keywords: HPC, multi-core processors, quad-core processors, parallel processing, benchmarks.

1

Introduction

The new 5500 family Quad-Core Intel Xeon processor is the first Intel server CPU with an integrated memory controller (IMC) as well as the first Xeon processor with a Quick Path Interconnect (QPI) interface. Nehalem based products are different from the previous generation products and it was clearly intended not only to scale across all the different product lines, but to be optimized for all the different product segments and market needs from mobile via desktop to server. This new modular design strategy is called at Intel core and uncore approach. Basically Nehalem products are defined in two vectors: one specifies the explicit core functionality universal for all the Nehalem family members. The second is focused on uncore elements and is highly optimized to best fit the specific market needs and segment specifics: this includes number of cores, number of QPI links or memory interfaces and many others uncore components which might be completely different form one family member to another. Core and unncore components can be designed and validated separately. The Quad-Core Intel Xeon processor 5500 family redefines not only every feature of the microprocessor micro-architecture but requires fundamental modification in the system architecture. Systems based on Xeon 5500 family introduce a completely new model of memory hierarchy as well interconnect for connecting processors and other components. In addition to all of the micro-architecture and system innovations Intel Xeon 5500 processors have been designed with all the R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 1–10, 2010. c Springer-Verlag Berlin Heidelberg 2010 

2

P. Gepner, D.L. Fraser, and M.F. Kowalik

advantages of 45-nanometer (nm) Hi-k metal gate silicon technology. This process technology uses a new combination of Hi-k gate dielectrics and conductive materials to improve transistor properties such as reducing electrical leakage, chip size, power consumption, and manufacturing costs[1].

2

Processor Microarchitecture and System Changes

The microarchitecture (or computer organization) is mainly a lower level structure that governs a large number of details, hidden in the programming model. It describes the constituent parts of the processor and how these interconnect and interoperate to implement the architectural specification. This means that the new microarchitecture enhances the end-user benefits when compared to the previous generations, achieving a new level of performance and changes the Instruction Set Architecture in silicon, including cache memory design, execution units, and pipelining[2]. Modifications made inside the Nehalem core increase compute resources and deliver higher performance, enhanced parallelism, an improved macrofiusion mechanism, modified Loop Stream Detector (LSD), a new branch prediction mechanism, add a new set of SSE 4.2 instructions and we bring in a new cache hierarchy. In addition to all of the above we re-introduce the Simultaneous MultiThreading (SMT) techniques pioneered on previous generations of the NetBurst R microarchitecture based Xeonprocessor. Increasing parallelism at a core level and extending the microperation concurrency requires a change in size and structure of the out of order window. Nehalem’s out of order buffer (ROB) grew from 96 entries in previous generation of the Xeon family to 128 entries on Nehalem. The Reservation Station, which is used to dispatch operations for execution units, grew from 32 to 36 entries. To track all allocations of load and store, the load buffer extended in size from 32 to 48 entries and the store buffer from 20 to 32. A new macrofusion mechanism identifies more macrofusion opportunities to further increase performance and power efficiency. Nehalem decodes CMP or TEST and conditional branch Jcc as a single microperation (CMP+Jcc), the same as it preprocessor did plus it extends the case for the following branch conditions: —JL/JNGE —JGE/JNL —JLE/JNG —JG/JNLE This increases the decode bandwidth and reduces the microperation count and effectively makes the machine wider. Intel Xeon 5400 family only support macrofusion in 32-bit mode, the Intel Xeon 5500 family supports macrofusion in both 32-bit and 64-bit modes. The Loop Stream Detector identifies software loops and takes advantage of this behavior by removing instruction fetch limitations and disabling unneeded blocks of logic. During a loop the processor decodes the same instructions over

R R Evaluating Performance of New Quad-Core IntelXeon 5500

3

and over and makes the same branch predictions over and over, these units can be disabled to increase higher performance and saving power. The Loop R R Stream Detector was first introduced in the IntelXeon 5300 family and it also operates as a cache for the instruction fetch unit. The Nehalem architecture improves on this concept by moving the Loop Stream Detector closer to the decode stage and keeps macroperations instead of instructions, theoretically this R family. is a very similar approach to the trace cache we used on the Pentium4 Nehalem has two levels of branch target buffer (BTB) these are designed to increase performance and power efficiency. The first and second level BTBs actually use different prediction algorithms and also different history files. These new and improved branch predictors are optimized not only for the best possible branch prediction accuracy but also to work with SMT. Simultaneous Multi-Threading returns to Intel Xeon family after first being implemented in the NetBurst microarchitecture based Xeon. Because an SMTenabled CPU requires substantial memory bandwidth it is likely that SMT will bring even more end-user benefit than Hyper- Threading did. However even for some workloads, including HPC applications SMT may need to be disabled to get the best possible performance[3]. Nehalem supports the same SSE 4.2 instructions as the Xeon 5400 family but in addition to the 47 instructions previously supported, 7 new instructions have been added. The SSE 4.2 instruction set includes instructions for the Vectorizing Compiler and Media Accelerator. The new instructions should improve the performance of audio, video, image editing applications, video encoders, 3-D applications, and games. The last major change to the core microarchitecture for Nehalem is its new cache hierarchy. Nehalem’s cache hierarchy has been extended to three levels, L1 and L2 staying fairly small and private to each core, while the L3 cache is much larger and shared between all cores. Each core in Nehalem has a private 32KB first level L1 Instruction and 32kB Data Cache. In addition the unified 256KB L2 cache, is 8 way associative and provides extremely fast access to data and instructions, the latency is typically smaller than 11 clock cycles. The first members of the Intel Xeon 5500 family have 8MB L3 cache, however additional products may be introduced with different cache sizes depending on the market segment requirements and platform specifics. The L3 Cache is organised as a 16 way set associative, inclusive and shared cache. The cache size can also be different, as a result of the uncore design philosophy. The L3 cache clock is decoupled from the cores frequency, so theoretically the clock of the cache can be different from the frequency of the cores. Nehalem’s L3 load to use latency is less than 35 cycles and is implement as an inclusive cache model, this approach guaranties coherency in the most efficient way. Any cache access misses in the L3 has a guarantee that data is not present it in any of the L2 or L1 caches of the cores. The core valid bits mechanism limits unnecessary snoops of the cores during hit data and only checks the indentified core where it has a possibility to find a private copy of this cache line which might be modified in L1 or L2.

4

P. Gepner, D.L. Fraser, and M.F. Kowalik

This method allows the L3 cache to protect each of the cores from expensive coherency traffic. The Nehalem microarchitecture changes all these pieces and also redefines system architecture. The most ground-breaking changes are the integrated memory controller and Quick Path Interconnect. The memory controller is optimized per R R market segment, the same analogy as the L3 cache. The initial IntelXeon 5500 family products support 3 memory channels per socket of DDR3-1333Mhz both registered and un-registered DIMMs. Each channel of memory can operate independently and handles request in an out-of-order fashion. The local memory latency for Nehalem is 60 ns. The memory controller has been designed for low latency and optimized for both local and remote memory latency. Nehalem delivers huge reduction in local memory latency comparing to previous generation Front Side Bus based Xeon processors, even remote memory latency is a fast 93 ns. Of course effective memory latency is dependent on the application and Operating System implementation of the Non Uniform Memory Architecture. Linux is a NUMA ready operating system, so too is Windows Vista. The biggest improvement comparing to previous generation based system is memory bandwidth. Total peak bandwidth for a single socket is 32GB/s and typical HPC two socket configuration gives 64GB/s. The QuickPath Interconnect (QPI) is a packet-based, high bandwidth, low latency point to point interconnect that operates at up to 6.4GT/s. This new point-to-point interconnect provides socket to socket connections and socket to chipset connections thus enabling system architects to build scalable solutions. Each link is a 20 bit wide interface using differential signaling. The QPI packet is 80 bits long but, only 64 bits are dedicated for data, the rest is used for flow control. This means that each link provides a transfer rate of 12.8GB/s in each direction and 25.6GB/s in total as links are bi-directional. The QPI links are uncore components and depending on the CPU implementation might have a different number of links. The first occurrence of Nehalem dedicated for DP servers (Intel Xeon 5500 family) has two links but the desktop version e.g. (Intel Core i7) has only one. All the above microarchitecture innovations as well as system enhancements increase the system performance and should benefit high performance computing. The exact detail of those innovations is described further in this paper. During all tests we have used single system performance on standard HPC workloads. For all of the tests where we have been comparing two systems: Intel Xeon processor 5400 (Harpertown) family based platform versus Intel Xeon processor 5500 family (Nehalem-EP) family based system. Intel Xeon processor 5400 family based platform: Platform configured with two Quad-Core Intel Xeon processors E5472 3.00GHz, 2x6MB L2 cache, 1600MHz FSB, 16GB memory (8x2GB FBDIMM 800MHz), RedHat Enterprise Linux Server 5 on x86 64. Intel Xeon processor 5500 family based platform: Pre-production platform Green City (Tylersburg B0 stepping) with two Quad-Core Intel Xeon processors X5560 (Nehalem-EP) 2.80GHz with 8M L3 cache, QPI -6.4 GT/s, 24

R R Evaluating Performance of New Quad-Core IntelXeon 5500

5

GB memory (6x4GB DDR3, 1066 MHz), RedHat Enterprise Linux Server 5 on x86 64. Those configurations are the typical ones suggested by vendors as typical HPC high end platform. Difference of memory size is mainly driven by dissimilar memory type (FBDIMM vs. DDR3) and different memory architecture subsystem (FSB vs. Integrated Memory Controller).

3

Processor Performance

The main focus of this section is to present a comparison of two generations R R processors. A popular benchmark well-suited of the Quad-Core IntelXeon for parallel, core-limited workloads is the Linpack HPL benchmark. Linpack is a floating-point benchmark that solves a dense system of linear equations in parallel. The metric produced is Giga-FLOPS or billions of floating point operations per second. Linpack performs operations called LU Factorization. These are highly parallel and store most of their working data set on processor cache. It makes relatively few references to memory for the amount of computation it performs. The processor operations it does perform are predominantly 64-bit floating-point vector operations and these use SSE instructions [4]. This benchmark is used to determine the world’s fastest computers published at the website http://www.top500.org/.

Fig. 1. LINPACK: Dense Floating-Point Operations

In both cases each processor core has 3 functional units that are capable of generating 128-bit results per clock. In this case we may assume that a single processor core does two 64 bit floating-point ADD instructions and two 64 bit floating-point MUL instructions per clock. The theoretical performance, calculated is the product of MUL and ADD executed in each clock, multiplied by frequency giving the following results. For Quad-Core Intel Xeon processor E5472 this gives 3GHz x 4 operations per clock x 4 cores = 48 GFLOPS. For

6

P. Gepner, D.L. Fraser, and M.F. Kowalik

Intel Xeon X5560 theoretical performance we have = 2.8GHz x 4 operations per clock x 4 cores = 44.8 GFLOPS. This is theoretical performance only, and does not fully reflect the real life scenario of running Linpack, in fact this level of theoretical performance disadvantage has also been observed in the Linpack benchmarking scenario. All the innovations of Nehalem do not compensate for the slower clock it has at these initial launch frequencies when purely considering theoretical performance or CPU bounded workloads. The Intel Xeon processor E5472 benefits from a higher CPU clock on the Linpack application. Using LINPACK HPL we see 6% performance disadvantage between the sysR R processor E5472 and new Quad-Core tem based on the Quad-Core IntelXeon Intel Xeon processor X5560 as Fig. 1 shows. It indicates that as we have expected based on the theoretical performance the Intel Xeon 5500 family will not realize a performance improvement on tasks that are heavily dependent on CPU clock frequency.

4

Memory Performance

Theory describes memory performance as a combination of the two following elements, latency and throughput. These components can have an impact on the different workloads, and they are applicable to different usage models. Latency explains how long it takes to chase a chain of pointers through memory. Only a single chain is tracked at a time. Each chain stores only one pointer in a cache line and each cache line is randomly selected from a pool of memory. The pool of memory simulates the working environment of an application. When the memory pool is small enough to be placed inside cache, the benchmark measures the latency required to fetch data from cache. By changing the size of the memory pool we can measure the latency of any specific level of cache, or to main memory, by simply creating the pool bigger than all levels of cache [5]. We measured latency using a 3.0 GHz Quad-Core Intel Xeon processor E5472 and a 2.8 GHz Quad-Core Intel Xeon processor X5560. The results of this experiment are shown in Figure 2. The Intel Xeon processor X5560 with its new cache structure and integrated memory controller significantly improves latency and data movement. The L1 cache stays on the same level even taking slightly longer to fetch data because of the lower clock frequency of the Intel Xeon X5560 vs. Intel Xeon E5472. Also the new hierarchy of cache with a smaller dedicated L2 256KB and shared L3 8MB makes the picture less clear. Based on the preliminary results we have achieved it seems that LMBENCH does not count L3 at all or hides it in the memory latency. Such a huge reduction in memory latency of 48% has been achieved simply by the local memory allocation as the LMBENCH does not measure the NUMA aspects of the processor. The remote memory allocation latency will not be so greatly improved in fact it only increases by 5% versus Xeon E5472. The improvement made on latency reduction for local memory access significantly helps random memory access where the NUMA element is partly considered and overall reduction is 22%.

R R Evaluating Performance of New Quad-Core IntelXeon 5500

7

Fig. 2. Memory Latency

Fig. 3. Stream benchmark - memory throughput improvement

The second component of the memory performance is the throughput for sequential memory accesses. The benchmark we have used to measure the throughput is the Stream benchmark. The Stream benchmark is a synthetic benchmark program, written in standard Fortran 77 developed by John McCalpin. It estimates, both memory reads and memory writes (in contrast to the standard usage for bcopy). It measures the performance of four long vector operations. These operations are: COPY: a(i) = b(i) SCALE: a(i) = q*b(i) SUM: a(i) = b(i) + c(i) TRIAD: a(i) = b(i) + q*c(i)

8

P. Gepner, D.L. Fraser, and M.F. Kowalik

These operations are representative of long vector operations and the array sizes are defined in that way so that each array is larger than the cache of the processors that are going to be tested. This gives us an indication of how effective the memory subsystem is excluding caches. As Fig. 3 shows we see significant R R memory bandwidth improvement with the Quad-Core IntelXeon processor X5560 and this is mainly due to the Integrated Memory controller, different memory type and memory arrangement. This enormous throughput improvement versus the previous generation Quad Core Intel Xeon based system will be reflected in all memory intensive applications not only Stream but also other HPC workloads.

5

Application Performance

Linpack and Stream, which are synthetic benchmarks, used during memory performance tests, measure the performance of specific subsystems and do not explain the full spectrum of system capability. Typical HPC applications use by nature much more than a single subsystem and their behavior is much more sophisticated. So to get a better understanding of how the Quad-Core R R IntelXeon processor X5560 based platform benefits real HPC applications we have selected several real examples. These application and benchmarks represent a broad spectrum of HPC workloads and seem to be a typical representation of a testing suite for this class of calculation. Similar benchmarks were used in publication [1] therefore benchmark descriptions are similar. Abaqus from Simula is a commercial software package for finite element analysis. This is general-purpose solver using a traditional implicit integration scheme to solve finite element analyses. The product is popular with academic and re search institutions due to the wide material modeling capability, and the ability to be easy customized. Amber is a package of molecular simulation programs. The workload measures the number of problems solved per day (PS) using eight standard molecular dynamic simulations. See http://amber.ch.ic.ac.uk/amber8.bench1.html for more information. Fluent is a commercial engineering application used to model computational fluid dynamics. The benchmark consists of 9 standard workloads organized into small, medium and large models. These comparisons use all but the largest of the models which do not fit into the 8GB of memory available on the platforms. The Rating, the default Fluent metric, was used in calculating the ratio of the platforms by taking a geometric mean of the 8 workload ratings measured. GAMESS from Iowa State University, Inc - a quantum chemistry program widely used to calculate energies, geometries, frequencies and properties of molecular systems http://www.msg.ameslab.gov/GAMESS/GAMESS.html. Gaussian from Gaussian, Inc - a quantum chemistry program widely used to calculate energies, geometries, frequencies and properties of molecular systems http://www.gaussian.com.

R R Evaluating Performance of New Quad-Core IntelXeon 5500

9

GROMACS (Groningen Machine for Chemical Simulations) from Groningen University is molecular dynamics program used to calculate energies, geometries, frequencies and properties of molecular systems http://www.gromacs.org. LS-DYNA is a commercial engineering application used in finite element analysis such as a car collision. The workload used in these comparisons is called 3 Vehicle Collision and is publicly available from http://www.topcrunch.org/. The metric for the benchmark is elapsed time in seconds. Monte Carlo from Oxford Center for Computational Finance (OCCF) - financial simulation engine using Monte Carlo technique http://www.occf.ox.ac.uk/. PamCrash from ESI Group - an explicit finite-element program well suited for crash simulations. For more information go to http://www.esi-group.com/ /SimulationSoftware/NumericalSimulation/index.html. Star-CD is a suite of test cases, selected to demonstrate the versatility and robustness of STAR-CD in computational fluid dynamic solutions. The metric produced is elapsed seconds converted to jobs per day. For more information go to http://www.cd-adapco.com/products/STAR-CD.

Fig. 4. Platform comparison across HPC selected workloads

All these selected workloads have been tested on two dual socket HPC optiR R mized platforms. The Quad-Core IntelXeon processor E5472 based platform has been used as the baseline to illustrate the improvement that the new platform is going to bring in different HPC workloads. As we can see the Quad-Core Intel Xeon processor X5560 based platform shows significant performance improvement up to 133%. The new miroarchitecture helps especially in the data intensive applications and workloads where data movement plays an important role. If the task is more CPU intensive the difference is around 5-15% for most of these workloads. For reference we have also placed the best published results of the AMD two sockets platform utilizing two Quad-Core AMD Opteron 2354 (Barcelona 2.2GHz) processors with 16 GB RAM (8x2 GB, 667MHz DDR2). When looking at a broad range of highly applicable workloads the Quad-Core Intel Xeon processor E5560 based platform demonstrates clear leadership.

10

6

P. Gepner, D.L. Fraser, and M.F. Kowalik

Conclusion

R R The new Quad-Core IntelXeon X5560 processor brings a new level of performance to HPC, improving significantly on applications where its preprocessors were not the most powerful. The new microarchitecture enhancements and system level modifications benefit data intensive types of HPC applications by up to as much as 130%. Processor intensive tasks will not achieve the same performance improvement as memory banded workloads, but they continue to stay on the same high level. Both generations of tested CPUs deliver the same theoretical performance when running the same clock and keep within the same thermal envelope but as shown deliver an improvement in performance of HPC workloads. Also the platform level modifications like QPI make the new platform more balanced from an I/O perspective and open new possibilities for more superior interconnect solutions. The Quad-Core Intel Xeon 5500 series is the first instance of the Nehalem generation and new additions to this family will bring even more functionality, scalability and performance and will become a compelling choice for many of the new HPC installations.

References 1. Gepner, P., Fraser, D.L., Kowalik, M.F.: Multi-Core Processors: Second generation Quad-Core Intel Xeon processors bring 45nm technology and a new level of performance to HPC applications. In: ICCS 2008, pp. 417–426 (2008) 2. Smith, J.E., Sohi, G.S.: The Microarchitecture of superscalar processors. Proc. IEEE 83, 1609–1624 (1995) 3. Eggers, S.J., Emer, J.S., Levy, H.M., Lo, J.L., Stamm, R.L., Tullsen, D.M.: Simultaneous multithreading: A platform for next generation processors. Proc. IEEE 17, 12–19 (1997) 4. Dongarra, J., Luszczek, P., Petitet, A.: Linpack Benchmark: Past, Present, and Future, http://www.cs.utk.edu/luszczek/pubs/hplpaper.pdf 5. Gepner, P., Kowalik, M.F.: Multi-Core Processors: New Way to Achieve High System Performance. PARELEC 2006, pp. 9–13 (2006)

Interval Wavelength Assignment in All-Optical Star Networks Robert Janczewski, Anna Małafiejska, and Michał Małafiejski Gdańsk University of Technology, Poland Algorithms and System Modeling Department {robert,aniam,mima}@kaims.pl

Abstract. In the paper we consider a new problem of wavelength assigment for multicasts in the optical star networks. We are given a star network in which nodes from a set V are connected to the central node with optical fibres. The central node redirects the incoming signal from a single node on a particular wavelength from a given set of wavelengths to some of the other nodes. The aim is to minimize the total number of used wavelengths, which means that the overall cost of the transmission is minimized (i.e. wavelength conversion or optoelectronic conversion is minimized). This problem can be modelled by a p-fiber coloring of some labelled digraph, where colors assigned to arcs of the digraph correspond to the wavelengths. In the paper we assume the set of all wavelengths of the incoming signals to a particular node forms an interval, i.e. a consecutive set of numbers. We analysed the problem of one-multicast transmission (per node). We constructed polynomial time algorithms for some special classes of graphs: complete k-partite graphs, trees and subcubic bipartite graphs. Keywords: WDM optical networks, multicasting, interval coloring, star arboricity.

1

Introduction

The considerations in this paper are motivated by the multicasting communication in a multifiber WDM (wavelength-division multiplexing) all-optical networks, which was recently considered in papers [3,6,7,22]. By the multifiber WDM network we mean the network where nodes are connected with parallel fibers. The problem is formulated as follows: we are given an all-optical star network with a set of nodes V , which are connected to the central node by optical fibers. Each node v ∈ V sends a set of at most q multicasts to some other nodes S1 (v), . . . , Sq (v), where Si (v) ⊂ V . The transmission through the central node is using WDM, i.e. different signals may be sent at the same time through the same fiber but on different wavelengths. First step of every multicast transmission from a node v is sending a message to the central node on the particular wavelength from given set of wavelengths. In the next step, the central node redirects the message to each node of Si (v) using one of these wavelengths. The R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 11–20, 2010. c Springer-Verlag Berlin Heidelberg 2010 

12

R. Janczewski, A. Małafiejska, and M. Małafiejski

goal is to minimize the total number of wavelengths during the simultaneous transmission of all multicasts in the network. This problem can be modelled by arc coloring of labelled (multi)digraph with some special requirements on the colors [3,7]. If we assume there is only one fiber between the central node and each node from V , and there is only one multicast per node (we define such a network as one-multicast one-fiber network) the problem reduces to the (directed) star arboricity (coloring) problem [1,2,15] and incidence coloring problem [8]. In this paper we assume the set of wavelengths of incoming signals forms an interval, i.e. a consecutive set of numbers, which may mean the multiplies of some base wavelength. Model and problem definition. In the general case, every node from set V of n nodes is connected to the central node with p optical fibres, such a network is sometimes called the star network. The multicast transmission in this network can be modelled by a (multi)digraph D with vertex set V and with labelled arc sets leaving v corresponding to multicasts S1 (v), . . . , Sq (v), i.e. every set Ai (v) = {vw : w ∈ Si (v)} of arcs leaving v is labelled with i (Fig. 1).

Fig. 1. S1 (v) = {a, b}, S2 (v) = {a, c}

We define a p-fiber coloring as a function assigning positive integers (colors, wavelengths) to the arcs such that at each vertex v and for every color c, inarc(v, c) + outlab(v, c) ≤ p, where inarc(v, c) is the number of arcs entering v and colored with c, and outlab(v, c) = |{i : ∃e ∈ Ai (v) and e is colored with c}|. In other words, outlab(v, c) is the number of labels of arcs leaving v and colored with c. For a given p-fiber coloring of digraph D we can easily construct a proper assignment of fibers in a network corresponding to D as follows: we assign a different fiber corresponding to every arc entering v and colored with c, and to each set of arcs leaving v, colored with c and of the same label. The correspondence between colors and wavelengths is shown in Fig. 1. Hence, the problem of simultaneous transmission of all multicasts in the p-fiber network with minimum number of wavelengths is equivalent to the p-fiber coloring of arcs of the digraph D with a minimum number of colors. Now, we formulate a decision version of the problem of p-fiber wavelength assigment for q-multicasts in optical star networks:

Interval Wavelength Assignment in All-Optical Star Networks

13

The (p, q)-WAM problem Instance: A digraph D with at most q labels on arcs, a positive integer k. Question: Is there a p-fiber coloring of D with at most k colors? A comprehensive survey of the (p, q)-WAM problem one can find in [3]. The problem (p, 1)-WAM was considered in [16] (i.e. for multitrees). In the paper we consider the (1, 1)-WAM problem, i.e. the fundamental case, where the fiber is unique and each node sends only one multicast. In this case, the p-fiber coloring of multidigraph reduces to the coloring of arcs of digraph satysfying only two conditions: (c1) any two arcs entering the same vertex have different colors, (c2) any arc entering and any arc leaving the same vertex have different colors. This coloring is well-studied [3,7] and arises from the problem of partitioning of a set of arcs into the smallest number of forests of directed stars, this problem is called the directed star arboricity problem [1,2,15]. Moreover, we focus our attention to the symmetrical communication, i.e. every transmission from v to w implies the communication between w and v, in this case the digraph modeling the communication between nodes is a simple graph. This problem can be described as (1, 1)-WAM problem with symmetric multicasts and can be reduced to the incidence coloring of graphs [8,10,11,20,23] (for details see Section 2). Previous results and our contribution. In this paper we study the WAM problem under the assumption that the set of colors assinged to arcs entering a vertex forms an interval, i.e. a consecutive set of numbers. This corresponds to having a consecutive wavelengths on the link between the central node and the destination node. This problem is important for traffic grooming in WDM networks, where wavelengths could be groomed into wavebands [22]. The interval (p, q)-WAM problem is a new concept arising from well-studied model of interval-edge coloring [4,12,13,14]. In [17] the authors introduced the problem of interval incidence coloring modeling message passing in networks. The authors proposed some bounds on the interval incidence coloring number (IIC) and studied this problem for some classes of graphs (paths, cycles, stars, fans, wheels and complete graphs). In this paper we construct polynomial time algorithms for trees, subcubic bipartite graphs, i.e. bipartite graphs with the degree of vertex at most three, and complete k-partite graphs. Outline of the paper. In Section 2 we introduce definitions, prove some useful bounds for the interval incidence chromatic number, and construct polynomial time algorithm for complete k-partite graphs. In Section 3 we construct polynomial time algorithms for biparite graphs with maximum degree at most 3, i.e.

14

R. Janczewski, A. Małafiejska, and M. Małafiejski

subcubic bipartite graphs. In Section 4 we construct polynomial time algorithm for trees using dynamic methods.

2

Definitions and Simple Properties

In the following we consider connected simple graphs only, and use standard notations in graph theory. Incidence coloring. For a given simple graph G = (V, E), we define an incidence as a pair (v, e), where vertex v ∈ V (G) is one of the ends of edge e ∈ E(G) (we say v is incident with e). Let us define a set of incidences I(G) = {(v, e) : v ∈ V (G) ∧ e ∈ E(G) ∧ v ∈ e}. We say that two incidences (v, e) and (w, f ) are adjacent if one of the following holds: (i) v = w, e = f , (ii) e = f , v = w, (iii) e = {v, w}, f = {w, u} and v = u. By an incidence coloring of G we mean a function c : I(G) → N such that c((v, e)) = c((w, f )) for any adjacent incidences (v, e) and (w, f ). In the following we use simplified notation c(v, e) instead of formal c((v, e)). Interval incidence coloring. A finite subset A of N is an interval if and only if it contains all numbers between min A and max A. For a given incidence coloring c of graph G let Ac (v) = {c(v, e) : v ∈ e ∧ e ∈ E(G)}. By an interval incidence coloring of graph G we mean an incidence coloring c of G such that for each vertex v ∈ V (G) set Ac (v) is an interval. The interval incidence coloring number of G, denoted by IIC(G), is the smallest number of colors in an interval incidence coloring. The interval incidence coloring of G using IIC(G) colors is said to be minimal. Symmetric (1, 1)-WAM problem. For a given symmetric digraph D consider (1, 1)-WAM problem and observe that we can ommit the unique label on arcs. Every two symmetric arcs uv and vu between vertices u and v we can replace by one edge e = {u, v} and define two incidences (v, e) and (u, e), which corresponds to arcs uv and vu, respectively. Hence, the coloring of arcs of digraphs satisfying conditions (c1) and (c2) is equivalent to the incidence coloring of graphs. 2.1

Bounds for the Interval Incidence Coloring Number

The IIC number can be bounded by maximum degree Δ(G) and its chromatic number χ(G) as follows. Theorem 1. For a given non-empty graph G we have Δ(G) + 1 ≤ IIC(G) ≤ χ(G) · Δ(G).

Interval Wavelength Assignment in All-Optical Star Networks

15

Proof. The lefthand inequality holds because for any interval incidence coloring c, each vertex v ∈ V (G) and its neighbour u we have c(u, {u, v}) = c(v, {u, v}). To prove the righthand inequality, we divide vertex set into χ(G) indepedent sets denoted by I1 , I2 , . . . , Iχ(G) . For any v ∈ Ii we can assign {c(v, e) : (v, e) ∈ I(G)} ⊆ {1 + (i − 1) · Δ, . . . , i · Δ} in such a way that Ac (v) is an interval. It is easy to see that any two adjacent incidences (v, {v, u}) and (u, e) will receive different colors. Theorem 2. For every k ≥ 2 and for every complete k-partite graph G = Kp1 ,p2 ,...,pk holds the following lower bound IIC(G) ≥ 2n − max{pi + pj : i, j = 1, . . . , k ∧ i = j}. Proof. Let c be an interval incidence coloring of G that uses IIC(G) colors. Let v be a vertex of G such that min Ac (v) = min c(I(G)). Let u be a neighbour of v such that c(v, {u, v}) = max Ac (v). Then Ac (u) ∩ Ac (v) = ∅ and IIC(G) ≥ |Ac (u)| + |Ac (v)| = deg(u) + deg(v) ≥ 2n − max{pi + pj : i, j = 1, . . . , k ∧ i = j}. 2.2

IIC Number for Some Class of Graphs

In the Table 1 we give exact values of the interval incidence coloring number for some known classes of graphs. Table 1. The value of IIC for some classes of graphs Graph’s family paths Pn , n ≥ 5 cycles Cn stars Sn 2-stars Sn2 k-stars Snk , k ≥ 3 wheels and fans complete graphs complete bipartite Kp,q complete k-partite Kp1 ,p2 ,...,pk 2n − max{pq bipartite torus regular bipartite

2.3

IIC(G)

Ref.

Δ+2 Δ+2 Δ+1 Δ+1 2Δ Δ+3 2Δ p+q + pr : 1 ≤ q < r ≤ k} 2Δ 2Δ

[17] [17] [17] [17] [21] [17] [17] [21] [21] [18] [18]

IIC Number for Complete k-Partite Graphs

Theorem 3. A complete k-partite graph Kp1 ,p2 ,...,pk can be colored optimally with 2n − max{pi + pj : i, j = 1, . . . , k ∧ i = j} colors in O(m + n) time. Proof. By Theorem 2, it suffices to construct interval incidence coloring of G = Kp1 ,p2 ,...,pk that uses exactly 2n − max{pi + pj : i, j = 1, . . . , k ∧ i = j} colors.

16

R. Janczewski, A. Małafiejska, and M. Małafiejski

Let I1 , I2 , . . . , Ik be partitions of G. Without loss of generality we can assume that pi = |Ii | (1 ≤ i ≤ k), p1 ≤ p2 ≤ . . . ≤ pk and Ii = {v1i , v2i , . . . , vpi i }. We construct the desired interval incidence coloring of G as follows:  p1 + p2 + . . . + pk−1 + l k > i, i i k c(vj , {vj , vl }) = n + p1 + p2 + . . . + pk−1 + l otherwise. We leave it to the reader to verify that the above formula defines interval incidence coloring that uses 2n − max{pi + pj : i, j = 1, . . . , k ∧ i = j} colors.

3

Polynomial Time Algorithms for Subcubic Bipartite Graphs

Cycles and paths were considered in Section 2.2, and there are only a few connected graphs with IIC ≤ 3. Let G be a subcubic bipartite graph G with Δ(G) = 3. By Theorem 1 the IIC(G) is between 4 and 6. In this section we will construct different algorithms for interval incidence coloring of subcubic bipartite graphs using 4, 5 and 6 colors. 3.1

Interval Incidence Coloring with 4 Colors

Suppose IIC(G) = 4. We prove that graph G satisfies the following properties: (i) each vertex v of degree 3 has at most one adjacent vertex of degree 3, (ii) each vertex v of degree 3 has at least one adjacent vertex of degree 1, (iii) on every path between two non-adjacent vertices of degree 3 there are at least two vertices of degree 2. The property (i) follows from the fact, that in {1, 2, 3, 4} one can find only two intervals of length 3: {1, 2, 3} and {2, 3, 4}, and in each of them there is only one element not belonging to the second one. To prove the property (ii) observe that if Ac (v) = {1, 2, 3} (Ac (v) = {2, 3, 4}), then the vertex u for which the incidence (v, {v, u}) is colored by 3 (by 2, respectively) must be a leaf. Suppose, contrary to the property (iii), that on the path between 3-degree vertices v1 , v2 there is only one vertex u with degree 2. Assume that c(u, {u, v1 }) = 1, c(u, {u, v2 }) = 2. This cleary implies 2 ∈ / Ac (v2 ) and max Ac (v2 ) ≥ 5, a contradiction. The same reasoning applies to the cases where Ac (u) = {2, 3} or Ac (u) = {3, 4}, so the property (iii) holds. We propose the algorithm using 4 colors for interval incidence coloring of graph G which satisfies properties (i) − (iii). First, color vertices of G using two colors a and b, which represent intervals containing 1 and 4, respectively. Next, for every 3-degree vertex v colored with a (or b, respectively), let • c(v, {v, u}) = 1 (or 4 respectively), if deg(u) = 3, • c(v, {v, u}) = 2 (or 3 respectively), if deg(u) = 2, or if deg(u) = 1 and there is no u ∈ N (v) such that deg(u) = 2,

Interval Wavelength Assignment in All-Optical Star Networks

17

• c(v, {v, u}) = 3 (or 2 respectively), if deg(u) = 1. For every 2-degree vertex v colored by a which represents an interval {1, 2} (or b, respectively, which represents an interval {3, 4}), we have to assign c(v, {v, u}) = 1 (respectively 4) if deg(u) = 3. Finally, we color all pending incidences (i.e. at leafs). By the above, the problem of interval incidence coloring of a graph using 4 colors is equivalent to veryfing properties (i) − (iii), hence we get Theorem 4. The problem of interval incidence coloring with 4 colors can be solved for subcubic bipartite graphs in linear time. 3.2

Interval Incidence Coloring with 5 Colors

For a given subcubic bipartite graph G, if IIC(G) ≤ 5 then the following algorithm for interval incidence coloring problem uses 5 colors, otherwise returns F ALSE: (i) Color vertices of G using a and b. (ii) For every v ∈ V (G) color incidences (v, {v, u}), for all u ∈ N (v) as follows: • if deg(v) = 3 and v is colored with a (b, respectively), then let c(v, {v, u}) ∈ {1, 2, 3} (c(v, {v, u}) ∈ {3, 4, 5}, respectively) in such a way that if deg(u) = 3 then c(v, {v, u}) = 3. If for every u ∈ N (v) deg(u) = 3, then return F ALSE, • if deg(v) = 2 and v is colored by a (b, respectively), then let c(v, {v, u}) ∈ {1, 2} (c(v, {v, u}) ∈ {4, 5}, respectively) in any way, • if deg(v) = 1 and v is colored by a (b respectively), then let c(v, {v, u}) = 1 (c(v, {v, u}) = 5 respectively).

Fig. 2. Interval incidence coloring of subcubic bipartite graphs with 5 colors

Consider any v ∈ V (G), if deg(v) ≤ 2, then because G is bipartite, we have Ac (v) ∩ Ac (u) = ∅, for any u ∈ N (v). The same observation holds, if deg(v) = 3 and there exists u ∈ N (v) such that deg(u) = 3, as showed in Figure 2. If for all u ∈ N (v), deg(u) = 3, then there is no interval incidence coloring of G with 5

18

R. Janczewski, A. Małafiejska, and M. Małafiejski

colors and algorithm returns F ALSE. In the other case we get proper incidence coloring c of G with 5 colors. It is easy to see that c is interval coloring, hence holds the following Theorem 5. The problem of interval incidence coloring with 5 colors can be solved for subcubic bipartite graphs in linear time. 3.3

Interval Incidence Coloring with 6 Colors

Suppose G is a subcubic bipartite graph such that IIC(G) > 5. Then we use the same algorithm as in the proof of Theorem 1 for the upper bound, which is exactly 6 in this case. Hence combining all the algorithms we have Theorem 6. Finding a minimal interval incidence coloring of subcubic bipartite graph can be done in linear time.

4

Polynomial Time Dynamic Algorithm for Trees

In this chapter one can find a polynomial-time algorithm for interval incidence coloring of trees. Given a tree T , we take any leaf r as a root of T and direct tree T from the leaves different from r to the root r. It means that we direct every edge e ∈ E(T ) in such a way that for every v ∈ V (T ), v = r, there is exactly one edge leaving v. We will color the incidences of T using the bottom-up technique in accordance to the defined direction. For every edge e = (v, u) ∈ E(T ) we build a matrix A of size 2Δ × 2Δ in such a way that A(i, j) is the maximum of already used colors of incidences in some minimal coloring of subtree Tv with the root v, assuming that the incidence (v, e) is colored by i and the incidence (u, e) is colored by j. The main idea of the algorithm for finding IIC(T ) is as follows: (i) For each edge e ∈ E(T ), beginning from the bottom, find a matrix A. (ii) Return IIC(T ) = mini,j=1,2,...,2Δ A(i, j), where A is a matrix for r. Constructing consecutive incidence coloring of T with IIC(T ) colors is possible by using additional structures for remembering chosen pair of colors for adjacent incidences while building a matrix A. 4.1

Constructing a Matrix A

We will assume that A(i, i) = ∞ for every i = 1, 2, . . . , 2Δ. Let’s consider an edge e = (v, u) directed from v to u. If v is a leaf, then A(i, j) = max{i, j} for i = j. In the other case, let v1 , v2 , . . . , vp be the neighbours of v different from u, p = deg(v) − 1. According to our assumption, there is a matrix Ai built for every edge (vi , v), i = 1, . . . , p. We use the following algorithm for constructing a matrix A for e = (v, u):

Interval Wavelength Assignment in All-Optical Star Networks

19

Algorithm MATRIX(e = (v, u)) for each pair of colors of incidences (v, e), (u, e), denoted by a, b, a = b for each k = 0, 1, . . . , deg(v) − 1 let mk = ∞ let Ik = {a − deg(v) + 1 + k, . . . , a + k} if b ∈ / Ik then construct bipartite graph G with partitions V1 , V2 and with weights of edges as follows: V1 = {v, v1 , v2 , . . . , vp }, V2 = Ik E(G) = {{vi , j} : i = 1, . . . , p ∧ j ∈ Ik } ∪ {v, a} w : E(G) → N w({vi , j}) = minl∈{1,...,2Δ}\Ik Ai (l, j) w({v, a}) = max{a, b} remember mk = MATCHING(G) end for A(a, b) = mink mk end for The procedure MATCHING is used for finding a perfect matching M of G that minimalizes the maximum of w(e), e ∈ E(M ). Algorithm MATCHING(G) for each i = 1, 2, . . . , 2Δ G = G \ {e ∈ E(G) : w(e) > i} if there is a perfect matching of G then return i end for To speed up the algorithm MATCHING, we can use bisection method instead of the loop for i = 1, 2, . . . , 2Δ. Using the Hopcroft-Karp algorithm for finding perfect matching in a graph G we obtain time complexity of MATCHING es5 timated by O(log Δ · Δ 2 ). The time complexity √ of the MATRIX algoritm for a given edge e ∈ E(T ) is estimated by O(log Δ ΔΔ5 ). Using the MATRIX algorithm O(n) times leads to the resulting algorithm for finding IIC(T ) with time 1 complexity estimated by O(nΔ5 2 log Δ).

5

Final Remarks

Theorem 7. Interval incidence coloring of planar graphs of degree 4 is NPC. The proof of this theorem can be found in [19]. An interesting question is the complexity of the interval incidence coloring of bipartite graphs.

20

R. Janczewski, A. Małafiejska, and M. Małafiejski

References 1. Algor, I., Alon, N.: The star arboricity of graph. Discrete Mathematics 75, 11–22 (1989) 2. Alon, N., McDiarmid, C., Reed, B.: Star arboricity. Combinatorica 12, 375–380 (1992) 3. Amini, O., Havet, F., Huc, F., Thomasse, S.: WDM and directed star arboricity. In: INRIA, France, pp. 1–20 (2007) 4. Asratian, A., Kamalian, R.: Investigation on interval edge-colorings of graphs. J. Combin. Theory. Ser. B 62, 34–43 (1994) 5. Beaquier, B., Bermond, J.C., Gargano, L., Hell, P., Perennes, S., Vacarro, U.: Graph problems arising from wavelength-routing in all-optical networks. In: Proc. of WOCS 1997. IEEE, Los Alamitos (1997) 6. Brandt, R., Gonzalez, T.F.: Multicasting using WDM in Multifiber Optical Star Networks. In: Proc. of the 15th IASTED International Conference on Parallel and Distributed Computing and Systems PDCS 2003, Canada, pp. 56–61 (2003) 7. Brandt, R., Gonzalez, T.F.: Wavelength assignment in multifiber optical star networks under the multicasting communication mode. Journal of Interconnection Networks 6, 383–405 (2005) 8. Brualdi, R.A., Massey, J.Q.: Incidence and strong edge colorings of graphs. Discrete Mathematics 122, 51–58 (1993) 9. Chen, D.L., Liu, X.K., Wang, S.D.: The incidence chromatic number and the incidence coloring conjecture of graphs. Math. Econom. 15(3), 47–51 (1998) 10. Chen, D.L., Pang, S.C., Wang, S.D.: The incidence coloring number of Halin graphs and outerplanar graphs. Discrete Mathematics 256, 397–405 (2002) 11. Dolama, M.H., Sopena, E., Zhu, X.: Incidence coloring of k-degenerated graphs. Discrete Mathematics 283, 121–128 (2004) 12. Giaro, K.: Interval edge-coloring Contemporary Mathematics. In: Kubale, M. (ed.) Graph Colorings, pp. 105–121. AMS (2004) 13. Giaro, K., Kubale, M., Małafiejski, M.: Compact scheduling in open shop with zero-one time operations. INFOR 37, 37–47 (1999) 14. Giaro, K., Kubale, M., Małafiejski, M.: Consecutive colorings of the edges of general graphs. Discrete Mathemathics 236, 131–143 (2001) 15. Guiduli, B.: On incidence coloring and star arboricity of graphs. Discrete Mathematics 163, 275–278 (1997) 16. Janczewski, R., Małafiejski, M.: Incidence coloring of multitrees. Zesz. Nauk. Pol. Gd. 13, 465–472 (2007) (in Polish) 17. Janczewski, R., Małafiejska, A., Małafiejski, M.: Interval incidence coloring of graphs. Zesz. Nauk. Pol. Gd. 13, 481–488 (2007) (in Polish) 18. Janczewski, R., Małafiejska, A., Małafiejski, M.: Interval incidence coloring of graphs. Rap. Tech. WETI 12, 1–16 (2008) (in Polish) 19. Janczewski, R., Małafiejska, A., Małafiejski, M.: The complexity of interval incidence coloring of graphs. Rap. Tech. WETI 16, 1–17 (2008) (in Polish) 20. Li, X., Tu, J.: NP-completeness of 4-incidence colorability of semi-cubic graphs. Discrete Mathematics 308, 1334–1340 (2008) 21. Małafiejska, A.: Interval coloring of graphs MSc Thesis, University of Gdańsk, pp. 1–70 (2008) 22. Modiano, E., Lin, P.J.: Traffic grooming in WDM networks. IEEE Communications Magazine 39(7), 124–129 (2001) 23. Shiu, W.C., Lam, P.C.B., Chen, D.-L.: Note on incidence coloring for some cubic graphs. Discrete Mathematics 252, 259–266 (2002)

Graphs Partitioning: An Optimal MIMD Queueless Routing for BPC-Permutations on Hypercubes Jean-Pierre Jung and Ibrahima Sakho Universit´e de Metz - UFR MIM - Dpt. d’Informatique Ile du Saulcy BP 80794 - 57012 Metz Cedex 01, France {jpjung,sakho}@univ-metz.fr

Abstract. Bit-Permute-Complement permutations (BPC) constitute the subclass of particular permutations which have gained the more attention in the search of optimal routing of permutations on hypercubes. The reason of this attention comes from the fact that they care permutations for general-purpose computing like matrix transposing, vector reversal, bit shuffling and perfect shuffling. In this paper we revisit the optimal routing of BPC problem on hypercubes under MIMD queueless communication model through a new paradigm which takes advantage of their topology: the so-called graphs partitioning. We prove that BPC are partitionable in any dimension of the hypercube and that the resulting permutations are also BPC. It follows that any BPC on n-dimensional hypercube is routable in at most n steps of data exchanges, each one realizing the partition of the hypercube. Keywords: Interconnection network, hypercube, BPC permutations, MIMD queueless routing, perfect matching of bipartite graph, graph partitioning.

1

Introduction

The processors interconnection network (IN) is the heart of the message passing parallel computers. Indeed, the performance of such computers depends greatly on the performance of their IN whose essential criteria are the scalability for massive parallelism, the capability of deadlock-free routing on shortest paths, the capability of simulating others IN and the management facility. In the research for IN which fulfil these criteria, hypercubes constitute a very attractive alternative. The incremental construction of hypercubes confers to them interesting mathematical properties which allow meeting most of performance criteria. On the contrary of most IN, they allow exponential increase of the number of the nodes while the diameter increases linearly. They admit implicit routing functions which alleviate the effort that requires their management in avoiding the computation of huge routing tables and which route on shortest paths. Finally most of the popular IN, rings, 2D wrap-around grids, trees, can be almost isometrically embed in hypercubes so guaranteeing their simulation capability. R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 21–30, 2010. c Springer-Verlag Berlin Heidelberg 2010 

22

J.-P. Jung and I. Sakho

For all these reasons, several commercial parallel machines have been built over the years and several theoretical and practical research works have also been done on different aspects of their use as IN. Among the theoretical research, one of the most challenging is the optimal rearrangeability of hypercubes that is, their capability for optimal routing any permutation such that the routes of each routing step constitute a one-to-one correspondence of adjacent nodes. BPC constitute the subclass of particular permutations which have gained the more attention in search of optimal permutations routing in hypercubes because they care permutations for general-purpose computing like matrix transposing, vector reversal, bit shuffling and perfect shuffling. In this paper we address the problem of the optimal routing of BPC. We revisit the problem through a new paradigm the so called graphs partitioning to take advantage of the recursive structure of the hypercubes topology. The remainder of the paper is organised in five sections. Section 2 gives the problem formulation and some basic definitions relative to hypercubes, permutations and routing. Section 3 presents the related works. Section 4 introduces the mathematical foundation used to develop the proposed routing algorithm. Section 5 characterizes partitionable BPC and proposes a routing algorithm. Section 6 concludes the paper and presents some perspectives.

2 2.1

Problem Formulation Definitions

Definition 1. An n-dimensional hypercube, nD-hypercube for short, is a graph H (n) = (V, E) where V is a set of 2n nodes u = (un−1 , un−2 , . . . , u0 ) such that ui ∈ {0, 1} and E is the set of arcs (u, v) ∈ V × V such that there is only one dimension i for which ui = vi . It is well known that for n > 0 an nD hypercube is obtained by interconnecting two (n− 1)D-hypercubes in any dimension 0 ≤ i ≤ n− 1. So any hypercube H (n) (n) (n) can be viewed as anyone of the n couples (H0,i , H1,i ) of (n − 1)D-hypercubes obtained by restricting H (n) to its nodes u such that ui = 0 (resp. 1). Figure 1 illustrates such a view.

5

4

6 2

14

7 3

13 9

8

1

0

12

15

10 11

Fig. 1. 4D-hypercube viewed as the interconnection of two 3D-hypercubes in the dimension 3

Optimal Queueless Routing for BPC on Hypercubes

23

Definition 2. A permutation on a nD-hypercube H (n) is an automorphism on V ; that is a one-to-one application π which associates to each node u = (un−1 , un−2 , . . ., u0 ) of H (n) one and only one node π(u) = (πn−1 (u),πn−2 (u),. . ., π0 (u)). It is represented by the sequence π = (π(u); u = 0, 1, . . ., 2n − 1). Definition 3. A permutation π in a nD-hypercube is a BPC if for any i, there is j such that for any u, πi (u) = uj or πi (u) = uj , the complement of uj . 2.2

MIMD Queueless Routing

Let π be a permutation on an nD-hypercube network with bidirectional links. Let’s consider a set of 2n messages of the same size each one located on one node u and destined for the node π(u). Routing the permutation π on the hypercube under MIMD queueless communication model consists in conveying all the messages to their destination such that each node holds one and only one message. Clearly, such a routing consists in a sequence of global and synchronous exchanges of messages between neighbourhood nodes such that no more than one message is located at each node after each exchange step.

3

Related Works

Optimal routing of permutations on nD-hypercubes is one of the most challenging open problems in the theory of IN. For an arbitrary permutation, it is well known, from e-cube routing [1], that such a routing requires at least n exchange steps. To specify its complexity, it has been extensively studied under several communication models and routing paradigms. Szimansky in [2] considers the offline routing in circuit-switched and packet switched commutation models under all-port MIMD communication model. Under the circuit-switched hypothesis he proves that, for n ≤ 3, any hypercube is rearrangeable. He also conjectured that routing can be made on the shortest paths, conjecture for which a counterexample has been given by Lubiw in [3]. Under packet-switched hypothesis he also shows that routing can be made in 2n − 1 steps, result which has been then improved to 2n − 3 by Shen et al in [4] in assuming that each link is used at most twice. Under the single port MIMD communication model, Zhang in [5] proposes a routing in O(n) steps on a spanning tree of the hypercube. While the above results considered offline routing and clearly established communication models, others works like the ones by Hwang et al [6,7] considered online oblivious routing under buffered all port MIMD communication models. They prove that in using local information, n steps routing is possible for n ≤ 12. The better routings under the previous models are due to V¨ ockling. In [8], he √ proves on one side that deterministic offline routing can be done in n + O( n log n) steps on the other side that online oblivious randomized routing can be made in n + O(n/ log n) steps but in buffered all port MIMD model.

24

J.-P. Jung and I. Sakho

For the more restrictive single-port, MIMD queueless communication model the works can be classified in routing of arbitrary and particular permutations among which the BPC. For arbitrary permutations, the personal communication of Coperman to Ramras and the works of Ramras [9] constitute certainly the leading ones. Indeed, while Coperman gives the computational proof that any arbitrary permutation in the 3D-hypercube can be routed in 3 steps. Ramras proves that if a permutation can be routed in r steps in rD-hypercubes, then for n ≥ r arbitrary permutations can be routed in 2n − r steps. Thus, it follows that arbitrary permutations can be routed in 2n − 3 steps improving so the 2n − 1 routing steps of Gu and Tamaki [10]. Recently, Laing and Krumme in [11] have introduced an approach based on the concept of k-separability that is the possibility to partition a permutation after k routing steps into 2k permutations on disjoints (n − k)D-hypercubes. For BPC, Nassimi and Sahni in [12] prove that under SIMD communication models BPC can be routed in n steps. Johnson and Ho present algorithms for matrix transposition in all-port MIMD [13] and all-to-all personalized [14] communication models. Recently, M´elatiaga and Tchuent´e have developed an n-steps parallel routing algorithm.

4

Mathematical Foundation

Hypercubes partitioning is the foundation of our routing algorithm. It is based on the computation of bipartite graphs perfect matching. 4.1

Definitions

Definition 4. A bipartite graph is a graph whose the nodes set V = V1 ∪ V2 and the edges set E is constituted of edges (u, v) such that u ∈ V1 and v ∈ V2 . Definition 5. The adjacency matrix of a bipartite graph is the |V1 |×|V2 | matrix M whose each component M [u, v] = 1 (resp. 0) if (u, v) ∈ (resp. ∈) / E. Let’s observe that, as an nD-hypercube can be viewed as the interconnection of two (n−1)D-hypercubes in anyone of its n dimensions, its adjacency matrix M is anyone of the n 2 × 2 block matrix M (i) whose block M (i) [x, x] (resp. M (i) [x, x]) (n) is the adjacency matrix of the (n − 1)D-hypercube Hi,x (resp. interconnection (n)

(n)

graph of the nodes of Hi,x to the ones of Hi,x ). Table 1 illustrates such a matrix. Definition 6. A matching of a bipartite graph G is an one-to-one correspondence Γ which associates to each node u of a subset of V1 a node Γ (u) of V2 such that the edges (u, Γ (u)) belong to E and are two-by-two non adjacent. A matching Γ is said to be maximum (resp. perfect, maximal) if its cardinal |Γ | is maximum (resp. = |V1 | = |V2 |, maximal). The main result about the computation of maximum and perfect matching is due to C. Berge [15].

Optimal Queueless Routing for BPC on Hypercubes

25

Table 1. 4D-hypercube adjacency matrix view according to dimension i = 2 2 0 1 2 3 8 9 10 11 4 5 6 7 12 13 14 15

0 1 1 1

1 1 1 1

2 1 1 1

3

8 1

1 1 1

1 1

10 11

4 1

1

5

6

1

1

12 13 14 15

1 1

1 1

7

1 1

1 1 1

1

9

1

1 1 1

1 1 1 1

1

1 1 1 1 1 1

1 1 1

1 1 1

1

1 1 1

1 1 1 1

1 1

1 1

1 1 1 1 1

1 1

1

1

1 1 1

1 1 1

1 1 1

Theorem 1 (of C. Berge). A matching Γ of a bipartite graph G is maximum if and only if there is no path with extremities nodes not saturated by Γ which alternates edges belonging and edges not belonging to Γ . The implementations of this theorem have lead to several algorithms. We will use the Neiman algorithm [16] which proceeds by distinguishing one and only one “1” by row and by column in the adjacency matrix of the bipartite graph. In the sequel, the non distinguished “1” will be double crossed. 4.2

Characterization of Partitionable Permutations

Given a permutation π on H (n) , for x = 0, 1 and 0 ≤ i ≤ n − 1, let n min be its minimal number of routing steps, Gx,i the bipartite graph (Sx,i , Dx,i , E), where Sx,i = {u ∈ H (n) : πi (u) = x} and Dx,i = {u ∈ H (n) : ui = x} and Γ x,i a maximum matching of Gx,i . Definition 7. A permutation π on H (n) is partitionable in a dimension i if there is a permutation Γ = (Γ 0,i , Γ 1,i ) on H (n) routable in one step and which leads to the distinct permutation α (resp. β) on a distinct (n − 1)D-hypercube, routable in n min − 1 steps and which associates π(u) to each Γ (u) such that Γi (u) = 0 (resp. 1). Actually, the partionability of a permutation on H (n) is a guaranty that it can be decomposed, after at most one exchange step, in two independent permutations on two disjoint (n − 1)D hypercubes. Now let’s characterize the partitionable permutations. Lemma 1. A necessary and sufficient condition for a permutation on an nDhypercube to be partitionable is that there is a dimension i such that for any x, the bipartite graph Gx,i = (Sx,i , Dx,i , E) admits a perfect matching.

26

J.-P. Jung and I. Sakho

 

Proof. It is straightforward. It comes from Definition 4.

Lemma 2. A necessary and sufficient condition for the bipartite graph Gx,i = (Sx,i , Dx,i , E) to admit no perfect matching is that there is v ∈ Dx,i which cannot match with any node u ∈ Sx,i . Proof. The condition is obviously sufficient. To prove that it is also necessary it suffices to prove that the adjacency matrix, say N (i,x) , of Gx,i has at least one null column. To do this let’s first observe that this adjacency matrix is a mix of rows of M (i) [x, x] and M (i) [x, x]. Then by suited rows and columns permutations it can be rewritten in the form of table 2a where m = 2n−1 , A and B, I and O are respectively, non null, identity and null matrices. Secondly let’s observe that, by construction: (a) there is a perfect matching between the rows of I and the columns of B, (b) there is a matching of the rows of A with the columns of A and B which saturates the rows of A, (c) any perfect matching of Gx,i contains the perfect matching between the rows of I and the columns of B. Now let’s suppose that Gx,i does admit no perfect matching and consider any of its maximum matching. From the above properties, necessarily the rows and the columns of A are not perfectly matched, but they are maximally matched. Then from the Neiman algorithm, by permuting the rows and the columns of A, it can be decomposed as shown in table 2b where B’ results from a permutation of the rows of B, A1 contains in its diagonal a maximal matching of the rows and the columns of A, while A2 and A3 are non null matrices which do contain no matching. From the theorem of Berge, there is no alternated path whose initial extremity, a row of A3 , and the terminal extremity, a column of A2 are not saturated. In others words any attempt of building an alternated path from a row of A3 to a column of A2 whose extremities are not saturated will fail while, according to (b) it should be a success. Then necessarily A2 is null and N (i,x) contains at least one null column.   Table 2. Block decomposition of N (i,x) according to a maximum matching a u

v1

…

vk

vk

vm

…

1

…

A

B

u uk+1 um

y1…

…yk

yk+1

…

ym

1

k

…

b

O

I

x … … xk xk+1 …

A1

A2

A3

O

B’

O

I

xm

Proposition 1. A necessary and sufficient condition for a permutation on a nD-hypercube to be partitionable is that there is a dimension i such that for any x, the adjacency matrix N (i,x) of the bipartite graph Gx,i does contain no null column. Proof. It is straightforward. It comes from the two previous lemmas.

 

Optimal Queueless Routing for BPC on Hypercubes

5

27

Optimal Routing of BPC in nD-Hypercubes

In this section we deal with the declination of the above characterization of partitionable permutations for BPC. 5.1

Characterization of Partitionable BPC

Lemma 3. If π is a BPC on H (n) then for any dimension i, Sx,i = Dx,j (resp. Dx,j ) if πi (u) = uj (resp. uj ) for any u ∈ H (n) . Proof. By definition, for any i, there is a dimension j such that πi (u) = uj or πi (u) = uj for any u. As Sx,i = {u ∈ H (n) : πi (u) = x}, if πi (u) = uj then Sx,i = {u ∈ H (n) : uj = x} = Dx,j and if πi (u) = uj then Sx,i = {u ∈ H (n) : uj = x} = {u ∈ H (n) : uj = x} = Dx,j .   Proposition 2. Any BPC on H (n) is partitionable in any dimension. Proof. According to proposition 1, we have to prove that for any dimension i, the adjacency matrices N (i,x) contains no null column. Let v be a component of Dx,i . By definition, v = (vn−1 , . . . , vi+1 , x, vi−1 , . . ., vj+1 , vj , vj−1 , . . ., v0 ). According to Lemma 3, we distinguish two cases. Case 1. Sx,i = Dx,j . As Dx,j = {u ∈ H (n) : u = (un−1 , . . . , ui+1 , ui , ui−1 , . . ., uj+1 , x, uj−1 , . . ., u0 )}, then u = (vn−1 , . . . , vi+1 , x, vi−1 , . . ., vj+1 , x, vj−1 , . . ., v0 ) ∈ Dx,i . If vj = x then u = v and N (i,x) [u, v] = 1. If vj = x then u and v differ on only one bit, so they are adjacent nodes and N (i,x) [u, v] = 1. Case 2. Sx,i = Dx,j . In this case too, by a similar reasoning we prove that N (i,x) [u, v] = 1. 5.2

Partitioning Strategy of BPC

In order to determine a partitioning strategy, let’s examine the structure of N (i,x) . Before, let’s remark that for given i and x, N (i,x) is constituted of the adjacency matrices of the bipartite graphs ((Sx,i ∩ Dy,i , Dx,i , E), y = x, x). Let π be a BPC on H (n) , i and j such that πi (u) = uj or πi (u) = uj for any u and k = j (resp. n − 1, j − 1) if j < (resp. =, >)i. We have the following results. Lemma 4. Sx,i ∩ Dy,i is the sequence of the odd (resp. even) rank subsequences if πi (u) = uj (resp. uj ) of the longest sequence of disjoint 2k -terms subsequences of Dy,i in ascending (resp. descending) order if x = 0 (resp. 1) of their addresses. Proof. It comes from the fact that in the binary code table of the first 2n natural integers taken in the ascending order, the ith bit alternates 2i -length sequences of 0 and 1. Indeed, as Sx,i ∩Dy,i = Dz,j ∩Dy,i = {u ∈ H (n) : uj = z and ui = y}, from Lemma 3, the result follows in comparing the ith and j th bits of the binary code table.  

28

J.-P. Jung and I. Sakho

Lemma 5. N (i,x) is the 2n−k × 2n−k block matrix of 2k × 2k blocks whose each block [I, J], for 0 ≤ I, J ≤ 2n−(k+1) − 1 is such that: N (i,x) [I, J] = M (i) [(2n−(k+1) −1)x+(−1)x 2I, (2n−(k+1) −1)x+(−1)x J] (resp. M (i) [(2n−(k+1) − 1)x + (−1)x (2I + 1), (2n−(k+1) − 1)x + (−1)xJ]) if πi (u) = uj (resp. πi (u) = uj ). Proof. It is straightforward. Indeed it is a corollary of Lemma 4. Table 3 illustrates from table 1 such adjacency matrices.   Table 3. N (2,x) matrices for x = 0 and 1 of a BPC π on a 4D-hypercube such that π2 (u) = u1

0 1 8 9 4 5 12 13

0 1 1 1

1 1 1 1

2 1

3

8 1

1 1 1

9 1 1 1

10 11

1 1

1 1 1 1

Proposition 3. Let Γ be a partition ⎧ ⎨u Γ (u) = u ⊕ 2j ⎩ u ⊕ 2i

15 14 7 6 11 10 3 2

15 14 13 12 1 1 1 1 1 1 1 1 1 1

7 1 1 1

6

5

1 1 1

1

4

1

1 1

of π in dimension i. if (πi (u) = ui , u ∈ H (n) ) else if πi (u) = uj otherwise.

Proof. Let Γ be a perfect matching of Gx,i . We distinguish the following three cases. Case 1. πi (u) = ui , for any u ∈ H (n) . From Lemma 5, N (i,x) coincides with M (i) [x, x] and then admits the one-to-one correspondence (Γ (u) = u, u ∈ Dx,i ) as a perfect matching. Case 2. πi (u) = ui , for any u ∈ H (n) . From Lemma 5, N (i,x) coincides with M (i) [x, x] which is an identity matrix and then admits only the one-to-one correspondence (Γ (u) = u ⊕ 2i , u ∈ Dx,i ) as a perfect matching. Case 3. πi (u) = uj or πi (u) = uj , for any u ∈ H (n) . From Lemma 5, any Γ consists in a perfect matching of N (i,x) blocks belonging to M (i) [x, x] and M (i) [x, x]. Then from Case 2, (Γ (u) = u ⊕ 2i , u ∈ Dx,i ). According to Neiman algorithm, the only “1” residual components of N (i,x) are then the ones of the 2k × 2k identity matrix of M (i) [x, x]. As each of these residual “1” corresponds to the interconnection between the nodes whose addresses differ only on their j th bit, it follows that (Γ (u) = u ⊕ 2j , u ∈ Dx,i ). Γ is illustrated in Table 3 by the non-double-crossed “1”, the double-crossed ones being forbidden according to the Neiman algorithm.  

Optimal Queueless Routing for BPC on Hypercubes

5.3

29

Optimal Routing of BPC

The previous subsection specified a way for the first step routing of a BPC on an nD-hypercube: partition the BPC in two distinct permutations α and β on two disjoint (n − 1)D-hypercubes. Now, we have to decide how to route each one of these permutations. In this order, let’s recall that α (resp. β) is the permutation which associates π(u) to node Γ (u) such that Γi (u) = 0 (resp. 1) the node. Proposition 4. Any permutation which results from a partition of a BPC on an nD-hypercube is also a BPC on an (n − 1)D-hypercube. Proof. We have to prove that for any λ ∈ {α, β}, ∀r, ∃s such that λr (Γ (u)) = Γs (u) or Γs (u) for any Γ (u). This is the case for r = i, the partition dimension. Thus let’s consider the permutation α and a dimension r = i. From the definition of α, ∀Γ (u) such that Γi (u) = 0, αr (Γ (u)) = πr (u). As π is a BPC, ∃s such that πr (u) = us or πr (u) = us . In others words, αr (Γ (u)) = us or αr (Γ (u)) = us . We then distinguish the following three cases. Case 1. πi (u) = ui . From Proposition 3, Γ (u) = u for any u. Then αr (Γ (u)) = us = Γs (u) or αr (Γ (u)) = us = Γs (u). Case 2. πi (u) = uj . From Proposition 3, Γ (u) = u ⊕ 2k , where k = i or j for any u. Then αr (Γ (u)) = (Γ (u) ⊕ 2k )s = Γs (u) (resp. Γs (u)) if s = i and s = j (resp. s = i or s = j) or αr (Γ (u)) = (Γ (u) ⊕ 2k )s = Γs (u) (resp. Γs (u)) if s = i and s = j (resp. s = i or s = j).   Case 3. πi (u) = uj . The reasoning is similar to the Case 2 one. From the above study, routing a BPC π on an nD-hypercube under queueless communication model consists in recursive partitioning in two different BPC on two disjoint (n − 1)D-hypercubes . At each partition step, each node has either to conserve or to move its message in one of the dimensions i or j for which πi (u) = uj or πi (u) = uj .

6

Conclusion and Perspectives

This paper has addressed the problem of the optimal MIMD queueless routing of BPC on nD-hypercubes. The partitioning paradigm is the framework of the proposed routing algorithm. It consists in decomposing recursively a permutation in independent permutations on disjoint hypercubes. We have proved that any BPC can be partitioned in any dimension of the hypercube and that the resulting permutations are also BPC. Then any BPC can be routed in at most n data exchange steps. The resulting routing algorithm, for each message, consists in a simple comparison of two bits of its source and destination addresses. Clearly, the proposed routing algorithm looks like a self routing one. Unfortunately, these bits must have the same weight for all the nodes u and then require a consensual choice. Thus one of our future works will concern the BPC self routing. Beyond BPC there are others subclasses of permutations which are also subject to great attention. It is for instance the Linearly Transformed and Complemented, Omega and Inverse Omega permutations. So we also plan to test the applicability of the partitioning paradigm to these classes of permutations.

30

J.-P. Jung and I. Sakho

Acknowledgement. The conversion of this work to LATEX according to the LNCS format has been kindly realized with help of Frank Holzwarth, Springer DE.

References 1. Draper, J.T., Ghosh, J.: Multipath e-cube algorithms (MECA) for adaptive wormhole routing and broadcasting in k-ary n-cubes. In: Proceedings of International Parallel Processing Symposium, pp. 407–410 (1992) 2. Szymanski, T.: On the permutation capability of a circuit switched hypercube. In: Proceedings of the 1989 International Conference on Parallel Processing, pp. I-103–I-110 (1989) 3. Lubiw, A.: Counter example to a conjecture of Szymanski on hypercube routing. Informations Processing Letters 35, 57–61 (1990) 4. Shen, X., Hu, Q., Liang, W.: Realization of arbitrary permutations on a hypercube. Informations Processing Letters 51(5), 237–243 (1994) 5. Zhang, L.: Optimal bounds for matching routing on trees. In: Proceedings of the 8th Annual ACM-SIAM Symposium on Discrete Algorithms, New Orleans, pp. 445–453 (1997) 6. Hwang, F., Yao, Y., Grammatikakis, M.: A d-move local permutation routing for d-cube. Discrete Applied Mathematics 72, 199–207 (1997) 7. Hwang, F., Yao, Y., Dasgupta, B.: Some permutation routing algorithms for low dimensional hypercubes. Theoretical Computer Science 270, 111–124 (2002) 8. V¨ ocking, B.: Almost optimal permutation routing on hypercubes. In: Proceedings of the 33rd Annual ACM- Symposium on Theory of Computing, pp. 530–539. ACM Press, New York (2001) 9. Ramras, M.: Routing permutations on a graph. Networks 23, 391–398 (1993) 10. Gu, Q.-P., Tamaki, H.: Routing a permutation in the hypercube by two sets of edge disjoint paths. J.of Parallel and Distr. Computing 44, 147–152 (1997) 11. Laing, A.K., Krumme, D.W.: Optimal Permutation Routing for Low-dimensional Hypercubes. EEECS Tufts University (July 11, 2003) 12. Nassimi, D., Sahni, S.: Optimal BPC permutations on a cube connected SIMD computer. IEEE Trans. Comput. C-31(4), 338–341 (1982) 13. Johnsson, S.L., Ho, C.T.: Algorithms for matrix transposition for boolean n-cube configured ensemble architectures. SIAM J. Matrix Appl. 9(3), 419–454 (1988) 14. Johnsson, S.L., Ho, C.T.: Optimal communication channel utilization for matrix transpose and related permutations on boolean cubes. Disc. Appl. Math. 53(1-3), 251–274 (1994) 15. Berge, C.: Graphes, 3`eme ´edition, Dunod, Paris (1983) 16. Neiman, V.I.: Structures et commandes des r´eseau18 sans blocage, Annales des T´el´ecom (1969)

Probabilistic Packet Relaying in Wireless Mobile Ad Hoc Networks Marcin Seredynski1 , Tomasz Ignac2 , and Pascal Bouvry2 1

University of Luxembourg, Interdisciplinary Centre for Security, Reliability and Trust, 6, rue Coudenhove Kalergi, L-1359, Luxembourg, Luxembourg [email protected] 2 University of Luxembourg, Faculty of Sciences, Technology and Communication [email protected]

Abstract. A wireless mobile ad hoc network consists of a number of devices that form a temporary network that operates without a support of any fixed infrastructure. Its distributed nature, lack of a single authority, and limited battery resources may lead participating devices to be reluctant to packet forwarding, which is a key assumption of such a network. This paper demonstrates how cooperation can be reached by means of a new trust-based probabilistic packet relaying scheme. Its most significant properties are relativity of the evaluation of trust and the use of activity as one of the trust measures. Computational experiments demonstrated that the more selfish nodes were present in the network, the more reasonable was to use strict strategies by the remaining nodes, which in turns led to a good protection of the network from free-riders. Keywords: Wireless ad hoc networks, trust-based cooperation.

1

Introduction

A correct operation of wireless mobile ad hoc network (MANET) requires its participants to cooperate on packet relaying. Nodes reluctant to this duty are referred to as selfish nodes or free-riders. Most of devices in MANET rely on battery, therefore the risk of a selfish behaviour driven by the battery conservation need is very high. As shown in the literature, MANET composed of nodes that belong to different authorities can suffer from free-riding [1,2,3]. The problem of selfish behaviour can be solved if nodes relay packets only on behalf of cooperative nodes. This requires network participants to use a local trust or reputation systems in order to collect information regarding behaviour of other nodes. The main difference between the two systems is that in the former a node evaluates a subjective view of the entities trustworthiness, while in the latter the view of the whole community is included [4]. Several packet relaying approaches have been proposed in the literature [5,6,7,8,9]. However, in these works it was typically assumed that all nodes follow the same deterministic relaying approach. Since in R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 31–40, 2010. c Springer-Verlag Berlin Heidelberg 2010 

32

M. Seredynski, T. Ignac, and P. Bouvry

civilian MANET nodes belong to different authorities such an assumption might be difficult to attain. In this paper a different approach is introduced: firstly a new probabilistic relaying scheme is defined and next it is examined in various scenarios, in which nodes choose the relaying strategies that differ in its level of cooperation. Similarly to the approach introduced in [10] an activity trust measure is used as a key component to discourage nodes from strategically re-joining the network with a new identity (whitewashing action [11]). A new approach for the trust system is introduced: network participants are not classified as selfish or cooperative by some fixed standards, but rather they are compared against each other, so that one can view the system as an evaluation of subjective ranking of nodes. The next section introduces the new model of a probabilistic packet relaying. This is followed by Section 3, which contains simulation results. Last section concludes the paper.

2

Probabilistic Relaying Model

Each time an intermediate node receives a routing request (RREQ) it checks trustworthiness of the original sender of the message (source node). The action whether to relay or discard the packet originated by a node with a given trust is specified by a probabilistic relaying strategy. 2.1

Local Trust Evaluation System

In this work it is assumed that nodes use only a trust system. A source routing protocol is assumed to be used, which means that a list of intermediate nodes is included in the packet’s header. The information regarding behaviour of other nodes (referred later to as trust data) is gathered only by nodes directly participating in the communication session. The communication session involves a sender, several forwarders (nodes that relay packets) and a destination node. The trust data collection is performed in the following way: nodes are equipped with a watchdog mechanism (WD) [2] that enables them to check if a packet was successfully delivered to the destination. A node that asks another node to relay a packet verifies by means of a passive acknowledgment [12] whether the node has actually accepted the RREQ. As an example, let us assume that node n1 sends a message to node n4 via intermediate nodes n2 and n3 , and eventually the message is discarded by node n3 . This event is recorded by the WD mechanism of node n2 , which next informs node n1 about selfish behaviour of n3 . As a result, trust system of node n1 is updated with the two events - “packet relayed by n2 ” and “packet discarded by n3 ”, while the trust system of n2 is updated with the event “packet discarded by n3 ”. The detailed analysis of the performance of the WD can be found in [2]. The trust data are classified into two types referred to as personal and general trust data. The former takes considers status of packets originated by a node itself, while in the latter the status of packets originated by other nodes is taken into account. In consequence, node in a sender role collects personal trust data, while forwarders collect general trust data. The evaluation

Probabilistic Packet Relaying in Wireless Mobile Ad Hoc Networks

33

of trust of node j (source of the packet) by node i asked to relay the packet is based on two characteristics, namely relative forwarding rate (denoted by f rj|i ) and relative activity (denoted by acj|i ). Later in the text, relative forwarding rate and relative activity will be refered to as forwarding rate and activity. The notation j|i means that node j is under the evaluation of node i. Both values result from WD-based observations made by node i during its participation in the network. Only personal trust data is considered, however if such data are unavailable then general trust data is taken into account. Forwarding rate is defined over the context of ratio of packets relayed to packets discarded by a node, while activity over node’s degree of participation to the forwarding duties. It is assumed that acj|i , f rj|i ∈ [0, 1]. In the next step node i evaluates trust of node j, which is a number from the interval [0, 1]. Formally, the trust (evj|i ) is defined as a function of forwarding rate and activity: evj|i = fi (f rj|i , acj|i ), where fi : [0, 1] ⊗ [0, 1] → [0, 1]. Nodes might have different criteria of the trust evaluation, hence every function f is indexed with an index of an evaluating node. Node i observes two characteristics of j, namely req accj|i and req dscj|i , which are respectively numbers of packets relayed and discarded by j. Then, the node computes the RREQ acceptance ratio (rarj|i ) of node j: rarj|i =

req accj|i . req accj|i + req dscj|i

The values f rj|i and acj|i are functions of rarj|i and req accj|i respectively. Obviously, such functions can be defined in many various ways. However, in this paper the following definitions are proposed:  1 · rark|i < rarj|i , Oi  − 1

(1)

 1 · req acck|i < req accj|i , Oi  − 1

(2)

f rj|i =

k∈Oi

acj|i =

k∈Oi



where Statement =

1 0

if Statement is true, if Statement is false.

Oi denotes a set of all nodes observed by node i and Oi  stands for its size. The forwarding rate value defined by the equation (1) is a fraction of nodes that have lower forwarding rate than node j. A similar interpretation holds for the activity. Finally, the evj|i is computed using the following formula: evj|i = w1 · f rj|i + w2 · acj|i ,

(3)

34

M. Seredynski, T. Ignac, and P. Bouvry

where w1 , w2 are weights summing to one. In this paper the following assignment was made w1 = w1 = 12 . However, demanding on certain factors one may want to promote nodes with higher forwarding rate value or with higher activity by changing values of w1 and w2 . Based on histograms of rarj|i and req accj|i , it is possible to approximate density functions of distributions of RREQ acceptance ratio and number of relayed packets (denoted by rari (x) and req acci (x) respectively). Please note, that the function rari (x) is defined on the interval [0, 1] and req acci (x) may be considered on the semi-axis [0, +∞]. Now, equations (1) and (2) can be replaced by:  rarj|i f rj|i = rari (x)dx, (4) 0



req accj|i

acj|i =

req acci (x)dx.

(5)

0

In large networks, it may be impossible to collect informations about all nodes. In such a case, one can take an advantage of the statistical approach. Even, if there is no information about node j, there still exists possibility to evaluate it in a reasonable way. A natural choice in this case is to compute expected values of forwarding rate and activity: 

1

x · rari (x)dx,

f rj|i = 0

 acj|i =

∞

x · req acci (x)dx.

0

2.2

Decision-Making: Probabilistic Relaying Strategy

The next step for a node that received the packet for relaying is to decide whether to accept or reject the RREQ originated by a node with a given trust. The decision is based on the relaying strategy which is a function of evj|i (trust), and is denoted by si (evj|i ). A probabilistic strategy is used, i.e., si (evj|i ) is a probability that i will relay a packet originated by j. This implies that: si : [0, 1] → [0, 1]. In the experiments presented in the next section simple strategies were used. The aim of these experiments was to demonstrate that the proposed approach efficiently enforces cooperation between nodes in a network. However, if one wants to build a stable, large network there will be a need to use more complex strategies. The reason for this is that, the choice of a strategy is motivated by many factors, such as battery level, number of RREQ or computational resource of network devices. Moreover, a high variability of conditions in MANET is assumed. Hence, nodes will be forced to adapt their strategies in a dynamic way

Probabilistic Packet Relaying in Wireless Mobile Ad Hoc Networks

35

due to changes of the environment. Experimental results suggested that when choosing a strategy one must take into account a level of cooperation of other nodes. This is another parameter that should be incorporated into the process of the strategy selection. The scope of our future work is to construct an AI module which will be responsible for the strategy management. A dynamic environment of MANET implies that one must be able to describe an evolution of a node’s strategy as a stochastic process. Hence, it would be effectively if all strategies of a node were members of the same family of probability distributions. A natural candidate is the beta distribution, which is defined on the interval [0, 1] and is characterized by two parameters, a and b. For such a distribution the density function is defined by the formula: da,b (x) = 1 0

xa−1 (1 − x)b−1 ua−1 (1 − u)b−1 du

,

for x ∈ [0, 1].

(6)

The strategy is defined as a cumulative distribution function related to da,b (x), namely:  evj|i da,b (x)dx. (7) si (evj|i ) = 0

Parameters a and b determine the shape of si (x). The advantage of the beta distribution is that changing two parameters enables to cover a large set of strategies differing in their level of cooperativeness. Such a definition needs an additional assumption that a strategy is a nondecreasing function. Let us present an example of a parameter that will strongly affect the choice of a strategy - the distribution of a RREQ sent to node i. We introduce an auxiliary function ri (x) : [0, 1] → [0, +∞]. The value ri (x) is an average number of RREQ in a time unit received from nodes with the trust equal to x. The ri (x) is computed by node i out of the past RREQ received by the node. The ri (x) may be one of the key parameters used for selecting a strategy in a situation of limited power resource or channel capacity. For example, one can compute the expected value of packets relayed by i per a unit of time (with a given strategy si (x)):  1 ri (x)si (x)dx. (8) Ki = 0

However, the choice based only on Ki may be not the optimal one. There may be a need to compute other parameters of the ri (x), for instance a standard deviation, before choosing a strategy. This problem will be considered in a forthcoming paper.

3 3.1

Computational Experiments Model of the Network

The network was composed of a finite population of N nodes, while time was divided into R rounds. In each round every node initiated a single communication

36

M. Seredynski, T. Ignac, and P. Bouvry

session exactly once (acted as a sender), thus each round was composed of N communication sessions. Since dynamics of a typical MANET expressed in terms of mobility and connectivity of the nodes are unpredictable [13], intermediate and destination nodes were chosen randomly. The trust data were also used by nodes for originating its own packets. Each network participant used a path rating mechanism in order to avoid distrusted nodes. If a source node had more than one path available to the destination it would choose the one with the best rating. The rating was calculated as an arithmetic mean of trust of all nodes belonging to the route (trust of an unknown node was set to 0.3). The simulation of the network was defined by the following algorithm: 1. Specify i (source node) as i := 1, N as a number of nodes participating in the network and R as a number of rounds; 2. Randomly select node j (destination of the packet) and intermediate nodes, forming several possible paths from node i to j; 3. For each available path calculate arithmetic mean of the trust of all nodes belonging to the path and choose the path with best rating; 4. Let node i := 1 originate a packet (initiate a communication session), which is next processed by intermediate nodes according to their strategies; 5. As soon as the communication session is completed update the trust data as described in Section 2.1; 6. If i < N , then choose the next node (i := i + 1) and go to step 2. Else go to step 7; 7. If r < R, then r := r + 1 and go to step 1 (next round). Else stop the simulation. 3.2

Description of Experiments

A general form of a relaying strategy evaluated in the experiments was as follows: ⎧ p1 for evj ∈ [0, 0.4] ⎪ ⎪ ⎪ ⎪ for evj ∈ (0.4, 0.7] ⎨ p2 for evj ∈ (0.7, 1] s(evj ) = p3 ⎪ ⎪ for an unknown node in rounds 1 − 50 p ⎪ 4 ⎪ ⎩ p5 for an unknown node in rounds 51 − 600. Three types strategies were defined: a tolerant strategy (T), a strict strategy (S), and a non-cooperative strategy (N). Their relaying probabilities are shown in Table 1. The non-cooperative strategy forwarded small amount of packets in order to make it more difficult to be discovered. The T and S-type strategies were more cooperative (probability p4 set to 1) towards unknown nodes in the initial period (between rounds 1 and 50), while after that period these nodes turned to be non-cooperative (probability p5 set to 0.1). The reason for the non-cooperative behaviour after certain period was that this discouraged nodes from changing identity in order to take advantage of cooperative approach towards unknown nodes. Four sets of scenarios differing in the number of nodes using selfish strategies (represented by N strategy) were

Probabilistic Packet Relaying in Wireless Mobile Ad Hoc Networks

37

Table 1. Forwarding probabilities of the T, S and N-type strategies

p1 p2 p3 p4 p5

Tolerant strategy (T) Strict strategy (S) Non-cooperative strategy (N) 0.4 0 0 0.9 0.7 0.1 1 1 0.15 1 1 0.1 0.3 0.1 0

analyzed. In the first set (s1.1 -s1.3 ) selfish strategies were not present at all, in the second set (s2.1 -s2.3 ) 20% of nodes used the N strategy, in the third (s3.1 -s3.3 ) 40%, while in the last one (s4.1 -s4.3 ) 60%. In each set three different distributions of the remaining T and S-types strategies were defined (s∗.1 with a dominance of T strategies, s∗.2 with equal number of both strategies and s∗.3 with dominance of S strategies). Dominance was defined as a situation, in which the dominating strategy was used by around 90% of remaining nodes. The exact settings of scenarios are shown in Table 2. Table 2. Composition of strategies used by nodes in tested scenarios s1.1 #N 0 # T 54 #S 6

s1.2 0 30 30

s1.3 0 6 54

s2.1 12 43 5

s2.2 12 24 24

s2.3 12 5 43

s3.1 24 32 4

s3.2 24 18 18

s3.3 24 4 32

s4.1 36 22 2

s4.2 36 12 12

s4.3 36 2 22

The following performance characteristics of nodes using a given type of strategy were used: a throughput (tp) defined as a ratio of successful message delivery, a number of packets relayed (npr) by a node and a relaying rate (rr) specifying a ratio of packets relayed to packets discarded. Each scenario was repeated 50 times. The values of performance characteristics of each type of strategy were calculated as an arithmetic mean over the values of all runs. The evaluation of trust (ev) was performed according to equations (1)-(3). The total number of all nodes (N ) was equal to 60, while the number of rounds in the NTS (R) was set to 600. The path length was set from 1 up to 5 hops with the following probabilities: one hop - 0.1, two hops - 0.3 and three to five hops - 0.2. The number of available paths from a source to a given destination ranged from 1 up to 4 (each value with equal probability). 3.3

Results

The results of all four sets of scenarios are shown in Table 3. The average throughput ranged from 0.19 (s4.3 ) to 0.67 (s4.1 ), meaning that 19 to 67% of messages were successfully delivered without being re-send. As one might expect, these figures were mainly related to the number of N strategies present in the network.

38

M. Seredynski, T. Ignac, and P. Bouvry

Table 3. Performance of T, S and N-type strategies in the four sets of scenarios: throughputs (tp), numbers of packets relayed (npr) and relaying rates (rr) of nodes using one type of strategy vs. nodes using other type of strategy

s1.1 s1.2 s1.3 s2.1 s2.2 s2.3 s3.1 s3.2 s3.3 s4.1 s4.2 s4.3

overall tp/npr tp by T 0.67 0.69/873 0.56 0.64/821 0.45 0.62/758 0.5 0.58/779 0.43 0.55/740 0.35 0.52/691 0.36 0.47/679 0.31 0.44/650 0.26 0.41/621 0.25 0.35/581 0.22 0.34/569 0.19 0.33/558

tp/npr by S 0.54/623 0.48/580 0.43/516 0.47/550 0.43/511 0.38/459 0.4/462 0.36/425 0.32/385 0.31/366 0.29/339 0.27/318

tp/npr rr: T vs. N, by N S vs. N 0.25/35 0.42, 0.15 0.2/28 0.43, 0.16 0.14/16 0.43, 0.16 0.21/28 0.43, 0.15 0.18/24 0.43, 0.16 0.15/20 0.44, 0.16 0.19/21 0.45, 0.16 0.16/18 0.45, 0.16 0.13/15 0.45, 0.17

rr: T vs. T T vs. S 0.84, 0.73 0.86, 0.75 0.88, 0.77 0.86, 0.75 0.89, 0.78 0.9, 0.81 0.9, 0.8 0.91, 0.82 0.91, 0.83 0.93, 0.86 0.93, 0.86 0.95, 0.87

rr: S vs. T S vs. S 0.68, 0.51 0.72, 0.54 0.76, 0.58 0.71, 0.55 0.75, 0.57 0.79, 0.61 0.76, 0.59 0.78, 0.61 0.8, 0.63 0.81, 0.67 0.82, 0.67 0.84, 0.7

In the first set of experiments, where nodes did not use non-cooperative strategies the throughput was between 0.45 and 0.67. The worse performance was observed in set 4 (0.19 to 0.25), which was due a dominance of N strategies (used by 60% of nodes). Differences between three variants of scenarios within a given set were related to the proportion of T to S strategies. As T-type strategies were generous in their contribution to packet forwarding, their presence improved the performance of nodes using all other types of strategies (including the selfish ones). Nodes using selfish strategies (N strategies) managed to send from 13% (s4.3 ) up to 25% (s2.1 ) of packets, which was far below the values obtained by nodes using other types of strategies. The T strategy scored the best performance (throughput from 0.33 to 0.69), while S came second with throughput ranging from 0.27 up to 0.54. These values were positively correlated with participation to packet relying. Nodes that used T strategy forwarded on average from 558 (s4.3 ) to 873 (s1.1 ) packets each, while nodes following S strategy relayed 318 (s4.3 ) up to 623 (s1.1 ) packets. On the other hand, selfish users almost did not participate to forwarding duties at all (forwarding 15 to 35 packets). Since all strategies were trust-dependent the relaying rates depended on type of node requesting forwarding service. Nodes using S strategies were the most demanding ones in terms of trust requirements regarding source nodes. Nodes using these strategies prevented selfish nodes from obtaining good throughput, as relaying rates concerning nodes using N strategies were around 0.15. On the other hand, these nodes were by far more cooperative towards nodes that used T strategies (relaying rates ranging from 0.68 to 0.84) and S strategies (0.45-0.63). Nodes that used T strategies were even more cooperative with relaying rates being in a range of 0.84-0.95 towards nodes using T strategies and 0.73-0.87 towards S strategies. These nodes were also quite cooperative towards selfish nodes (relaying rates around 0.43), but one has to keep in mind that as a path is typically

Probabilistic Packet Relaying in Wireless Mobile Ad Hoc Networks

39

composed of more than one intermediate hop, such rate is not enough to obtain a good throughput. This was demonstrated by scenarios s2.1 , s3.1 and s4.1 , where selfish nodes were accompanied mostly by nodes using T strategies. In these scenarios, nodes using N strategies managed to successfully send at most 25% of packets. As in MANET strategy selection process is done independently by each node the question is what strategies would be most probably chosen by network participants - the more or the less cooperative ones? Such a decision would most probably be driven by two factors: state of battery resources and a level of performance that a node would like to obtain from the network. The better the battery is and the more expectations from a network a node has, the more likely it is going to use more cooperative strategy. As demonstrated by the four sets of experiments, being more cooperative results in a better level of service received from the network (defined here as throughput). But as more nodes are cooperative, the better performance is obtained by selfish nodes, so one might expect, that nodes might be tempted to switch to selfish strategies. However, as demonstrated by the experiments this is not very likely to happen. For instance, in s1.1 (no selfish nodes present) the S nodes forwarded 623 packets and obtained a throughput equal to 0.54, while T nodes forwarded 40% more packets and obtained a throughput equal to 0.69. In such a case a decision whether to use more or less cooperative strategy was a matter of preferences and battery resources. However, in a large presence of selfish nodes (s4.1 -s4.3 ) T nodes forwarded 59-75% more packets then S nodes, and obtained only slightly higher throughput. It means, that in such a situation (large presence of selfish nodes) it is more reasonable to use a less cooperative strategy. As a result, it is very unlikely, that selfish nodes might have an occasion to free-ride.

4

Conclusions

In this paper a new trust-based probabilistic scheme for packet relaying was introduced. One of its key features is the use of the notion of activity as a basis for the relaying strategy. It promotes identity persistence as nodes are discouraged from a whitewashing action because new identity means that the activity status is reset to zero. Additionally, the relativity of the evaluation of trust enables the trust system to adapt the ratings to the actual networking conditions. Thus, one can view the output of the system as an evaluation of subjective ranking of nodes. Such an approach also helps to deal with uncertainty related to the watchdog-based data collection mechanism. False or inaccurate data occasionally provided by the mechanism affect all nodes in the same way, thus these errors will not affect the ranking of nodes. In consequence, this uncertainty, as long as remains reasonable, will not create any significant threat to the trust system. The computational experiments demonstrated a substantial degree of resistance to free-riding, as in a case of a large number of selfish nodes, the stricter strategies proved to be a better choice. However, the experiments were performed under simplified assumptions and selected strategies were far from

40

M. Seredynski, T. Ignac, and P. Bouvry

the ideal. The scope of our future work is to construct a model of MANET in which additional complex factors will be taken into account.

Acknowledgments This work has been partially founded by C08/IS/21 TITAN Project (CORE programme) financed by National Research Fund of Luxembourg.

References 1. Buttyan, L., Hubaux, J.P.: Nuglets: a virtual currency to stimulate cooperation in self-organized mobile ad hoc networks. Technical Report DSC/2001/001, Swiss Federal Institute of Technology (2001) 2. Marti, S., Giuli, T., Lai, K., Baker, M.: Mitigating routing misbehavior in mobile ad hoc networks. In: Proc. ACM/IEEE 6th International Conference on Mobile Computing and Networking (MobiCom 2000), pp. 255–265 (2000) 3. Michiardi, P., Molva, R.: Simulation-based analysis of security exposures in mobile ad hoc networks. In: Proc. European Wireless Conference (2002) 4. Josang, A., Ismail, R., Boyd, C.: A survey of trust and reputation systems for online service provision. Decision Support Systems 43(2), 618–644 (2007) 5. Buchegger, S., Boudec, J.Y.L.: Performance analysis of the confidant protocol. In: Proc. 3rd International Symposium on Mobile Ad Hoc Networking and Computing (MobiHoc 2002), pp. 226–236 (2002) 6. Buchegger, S., Boudec, J.Y.L.: The effect of rumor spreading in reputation systems for mobile ad-hoc networks. In: Proc. Workshop on Modeling and Optimization in Mobile, Ad Hoc and Wireless Networks (WiOpt 2003), pp. 131–140 (2003) 7. Buchegger, S., Boudec, J.Y.L.: Self-policing mobile ad hoc networks by reputation systems. IEEE Communications Magazine, Special Topic on Advances in SelfOrganizing Networks 43(7), 101–107 (2005) 8. Giordano, S., Urpi, A.: Self-organized and cooperative ad hoc networking. In: Basagni, S., Conti, M., Giordano, S., Stojmenovic, I. (eds.) Mobile Ad Hoc Networking, pp. 355–371. Wiley, IEEE Press (2004) 9. Michiardi, P., Molva, R.: Core: A collaborative reputation mechanism to enforce node cooperation in mobile ad hoc networks. In: Proc. 6th Conference on Security Communications, and Multimedia (CMS 2002), pp. 107–121 (2002) 10. Seredynski, M., Bouvry, P., Klopotek, M.A.: Analysis of distributed packet forwarding strategies in ad hoc networks. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., Wasniewski, J. (eds.) PPAM 2007. LNCS, vol. 4967, pp. 68–77. Springer, Heidelberg (2008) 11. Feldman, M., Papadimitriou, C., Chuang, J., Stoica, I.: Free-riding and whitewashing in peer-to-peer systems. IEEE Journal on Selected Areas in Communications 24(5), 1010–1019 (2006) 12. Jubin, J., Turnow, J.D.: The darpa packet radio network protocols. Proceedings of the IEEE 75(1), 21–32 (1987) 13. Fischer, D., Basin, D., Engel, T.: Topology dynamics and routing for predictable mobile networks. In: Proc. The 16th IEEE International Conference on Network Protocols (ICNP 2008). IEEE, Los Alamitos (2008)

On the Performance of a New Parallel Algorithm for Large-Scale Simulations of Nonlinear Partial Differential Equations ´ Juan A. Acebr´ on1 , Angel Rodr´ıguez-Rozas1, and Renato Spigler2 1

2

Center for Mathematics and its Applications, Department of Mathematics, Instituto Superior T´ecnico, Av. Rovisco Pais 1049-001 Lisboa, Portugal [email protected], [email protected] Dipartimento di Matematica, Universit` a “Roma Tre”, Largo S.L. Murialdo 1, 00146 Rome, Italy [email protected]

Abstract. A new parallel numerical algorithm based on generating suitable random trees has been developed for solving nonlinear parabolic partial differential equations. This algorithm is suited for current high performance supercomputers, showing a remarkable performance and arbitrary scalability. While classical techniques based on a deterministic domain decomposition exhibits strong limitations when increasing the size of the problem (mainly due to the intercommunication overhead), probabilistic methods allow us to exploit massively parallel architectures since the problem can be fully decoupled. Some examples have been run on a high performance computer, being scalability and performance carefully analyzed. Large-scale simulations confirmed that computational time decreases proportionally to the cube of the number of processors, whereas memory reduces quadratically.

1

Introduction

An efficient design of numerical parallel algorithms for large-scale simulations becomes crucial when solving realistic problems arising from Science and Engineering. Most of them are based on domain decomposition (DD) techniques [10], since it has been shown to be specially suited for parallel computers. Nevertheless, classical approaches based on domain decomposition usually suffer from process intercommunication and synchronization, and consequently the scalability of the algorithm turns out to be seriously degraded. Moreover, an additional excess of computation is often introduced when designing the algorithms in order to diminish the effects of the two previous issues. From the parallelism point of view, probabilistic methods based on Monte Carlo techniques offer a promising alternative to overcome these problems. The possibility of using parallel computers to solve efficiently certain partial differential equations (PDEs) based on its probabilistic representation was R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 41–50, 2010. c Springer-Verlag Berlin Heidelberg 2010 

42

´ Rodr´ıguez-Rozas, and R. Spigler J.A. Acebr´ on, A.

recently explored. In fact, in [1,2], an algorithm for numerically solving linear two-dimensional boundary-value problems for elliptic PDEs, exploiting the probabilistic representation of solutions, were introduced for the first time. This consists in a hybrid algorithm, which combines the classical domain decomposition method [12], and the probabilistic method, and was called “probabilistic domain decomposition method” (PDD for short). The probabilistic method was used merely to obtain only very few values of the solution at some points internal to the domain, and then interpolating on such points, thus obtaining continuous approximations of the sought solution on suitable interfaces. Known the solution at the interfaces, a full decoupling in arbitrarily many independent subdomains can be accomplished, and a classical local solver, arbitrarily chosen, used. This fact represents a definitely more advantageous circumstance, compared to what happens in any other existing deterministic domain decomposition method. However, in contrast with the linear PDEs, probabilistic representation for nonlinear problems only exists in very particular cases. In [5], we extended the PDD method to treat nonlinear parabolic one-dimensional problems, while in [6] it was conveniently generalized to deal with arbitrary space dimensions. In this paper, for the first time the performance of the PDD algorithm in terms of computational cost and the memory consumption will be carefully analyzed. To this purpose, large-scale simulations in a high performance supercomputer were runned, and the advantageous scalability properties of the algorithm verified. Since the local solver for the independent subproblems can be arbitrarily chosen, for this paper we chose a direct method based on LAPACK for solving the ensuing linear algebra problem. This is so because ScaLAPACK were selected to compare the performance of our PDD method, being ScaLAPACK a competitive, freely available and widely used parallel numerical library. Other alternatives based on iterative methods are worth to be investigated, and will be done in a future work. In the following, we review briefly the mathematical fundamentals underlying the probabilistic representation of the solution of nonlinear parabolic PDEs. More precisely, for simplicity let consider the following initial-boundary value problem for a nonlinear two-dimensional parabolic PDE,  ∂2u ∂2u ∂u ∂u ∂u = Dx (x, y) 2 + Dy (x, y) 2 + Gx (x, y) + Gy (x, y) − cu + αi ui , ∂t ∂x ∂y ∂x ∂y i=2 m

in Ω = [−L, L] × [−L, L], t > 0, u(x, y, 0) = f (x, y), u(x, y, t)|x∈∂Ω = g(x, y, t),

(1)

m where x = (x, y) ∈ R2 , αi ∈ [0, 1], i=2 αi = 1, being f (x, y) and g(x, y, t), the initial condition and boundary data, respectively. The probabilistic representation for the solution of (1) was explicitly derived in [6] for the n-dimensional case, and requires generating suitable random trees with so many branches as the power of the nonlinearity found in (1). Such random trees play the role that a random path plays in the linear case, (cf. Feynman-Kac formula for linear parabolic PDE). Mathematically, the solution of (1) for a purely initial value

On the Performance of a New Parallel Algorithm

43

problem at (x, t) can be represented (see [11]) as follows

k(ω)

u(x, t) = Ex [

f (xi (t, ω))].

(2)

i=1

Here k(ω) is the number of branches of the random tree starting at x, and reaching the final time t. xi (t, ω) is the position of the ith branch of the random tree, corresponding to the stochastic process solution of the stochastic differential equation: dβ = b(x, y, t) dt + σ(x, y, t) dW (t), (3) where W (t) represents the standard Brownian motion, and b(x, y, t) and σ(x, y, t), the corresponding drift and diffusion related to the coefficients of the elliptic operator in (1). The different possible realizations of the random trees are labeled by ω, and a suitable average Ex should be taken over all trees. In case of a initialboundary value problem, with the boundary data u(x, t)|x∈∂Ω = g(x, t), a similar representation holds, i.e., ⎤ ⎡ k(ω)   f (xi (t, ω))1[Si >τ∂Ωi ] + g(βi (τ∂Ωi ), τ∂Ωi )1[Si > p and Ny >> p, the computational cost of the PDD method decreases as p−3 for large p, while for ScaLAPACK it has been reported in [8,9] to decrease rather as p−1 . 3.2

Numerical Results

In this subsection we consider two numerical test problems. Example A. The problem is given by ∂u ∂2u ∂2u ∂u ∂u = (1+x4 ) 2 + (1+y 2 sin2 (y)) 2 + (2+sin(x)ey ) + (2+x2 cos(y)) ∂t ∂x ∂y ∂x ∂y 1 2 1 3 −u + u + u , in Ω = [−L, L] × [−L, L], 0 < t < T, u(x, y, t)|∂Ω = 0, (7) 2 2     2 πy with initial-condition u(x, y, 0) = cos2 πx 2L cos 2L . No analytical solution is known for this problem, hence the numerical error was controlled comparing the solution obtained with the PDD method with that given by a finite difference method, with a very fine space-time mesh. Table 1. Example A: TT OT AL denotes the computational time spent in seconds by PDD. TM C , and TINT ERP correspond to the time spent by the Monte Carlo and the interpolation part, respectively; M emory denotes the total memory consumption. No. Processors 128 256 512 1024

TM C 445” 459” 463” 461”

TINT ERP 1. An explanation of this fact could be found in the size of the problem, which is not sufficiently large, and because the optimized LAPACK is speeding up considerably the LU factorization. Therefore, the dominant asymptotic behavior is governed, rather, by the second term in (6). Only for the purpose of confirming that this is indeed the case, new simulations for a larger computational load are needed, such that now Nx /p >> 1/αopt and Ny /p >> 1/αopt . The required computational resources in terms of memory, however, largely exceed the available memory (the limits have already being reached). A simple way to circumvent this problem consists in increasing the bandwidth of the associated matrix, keeping almost constant the computational load. This can be done merely considering a rectangular domain instead, Ω = [−Lx , Lx ] × [−Ly , Ly ], with Lx > Ly . Other alternatives based on adopting an heterogeneous computational grid, or even using higher order numerical schemes are equally possible, and will be analyzed elsewhere. From the results above, it is observed that increasing the size of the problem by raising the number of nodes Nx along the x−dimension, results in a computational cost growing cubically with Nx , whereas the memory consumption grows quadratically. Nevertheless, the effect on the y-dimension is more contained, since both the computational cost and the memory consumption increase linearly with the number of nodes Ny . Therefore, to increase the computational workload appreciably it is more convenient to increase the size of the problem along the x−dimension. In fact, this is successfully illustrated in the next example.

On the Performance of a New Parallel Algorithm

49

Example B. Consider the following problem ∂u ∂2u ∂2u ∂u ∂u = (1 + x2 ) 2 + (1 + y 3 ) 2 + (2 + sin(x) ey ) + (2 + sin(x) cos(y)) ∂t ∂x ∂y ∂x ∂y 1 2 1 3 −u + u + u , in Ω = [−Lx , Lx ] × [−Ly , Ly ], 0 < t < T, u(x, y, t)|∂Ω = 0, (8) 2 2     2 πy and initial-condition u(x, y, 0) = cos2 πx 2L cos 2L . In Table 2, the results corresponding to such a rectangular domain are shown. We used the parameters: Lx = 200, Ly = 0.125, Δx = Δy = 5 × 10−3 , T = 0.5, and Δt = 10−3 . Note that the scaling factor has changed significantly, reaching even the value 5, when passing from 128 to 256 processors. Increasing further the number of processors, implies reducing the scaling factor. This is in good agreement with the theoretical estimates in (6), since, asymptotically, the dominant term changes accordingly to the number of processors involved as well. Table 2. Example B: TT OT AL denotes the computational time spent in seconds by PDD. TM C and TINT ERP correspond to the time spent by the Monte Carlo and the interpolation part, respectively; M emory denotes the total memory consumption. No. Processors 128 256 512 1024

4

TM C 67” 69” 71” 72”

TINT ERP 0, ∀k, m and t, if dkmt = 0 then ukmt = 1 and uijt = 0 for i = k and j = m. We assume that the measurement error of training data is of no importance. With this assumption, the difference between the FCM memberships and the membership functions achieved from the type-1 learning phase is caused by an

Relational Type-2 Interval Fuzzy Systems

365

inner uncertainty of a modeled process. Therefore, we expand upper and lower membership functions over the training data. The upper membership function is a normal Gaussian function,   μkm ¯kn , σ kn n (xn ) = gauss xn , x km

μ

m

m

(y) = gauss (y, y¯ , σ )

(16) (17)

being a superior limit of the function family drawn through points (xn (t) , ukmt ), i.e., 6   7 σ kn = max σt : gauss xn (t) , x ¯kn , σt = ukmt t 8 8 8xn (t) − x ¯kn 8 k σ n = max √ t − log ukmt

(18) (19)

or through points (y (t) , ukmt ), i.e., |y (t) − y¯m | σ m = max √ . t − log ukmt

(20)

Consequently, the upper memberships of antecedents and consequents of the relational system can be thought as an aggregation weighted by certainty factors, i.e.,

μAkn (xn ) =

  ckm μkm (xn ) , n (xn ) + (1 − ckm ) μAk n

(21)

m

μB m (y) =

  ckm μkm (y) + (1 − ckm ) μB m (y) .

(22)

k

The lower membership function is a scaled Gaussian function,   (xn ) = hkn gauss xn , x ¯kn , σnk μkm n km

μ

m

m

m

(y) = h gauss (y, y¯ , σ )

(23) (24)

being an inferior limits of the function family drawn through points (xn (t) , ukmt ), i.e.,  7 6  (25) ¯kn , σnk = ukmt hkn = min ht : ht gauss xn (t) , x t

hkn

= min t

ukmt gauss (xn (t) , x¯kn , σnk )

(26)

or in the consequent part hm = min t

ukmt . gauss (y (t) , y¯m , σ m )

(27)

366

R. Scherer and J.T. Starczewski

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

0

2

4

6

8

10

Fig. 2. Type-2 interval uncertainty fitting: initial type-1 membership function (dashed line), membership of data (asterisks), uncertainty bounds (solid lines)

Figure 2 presents the idea of the interval uncertainty fitting. Similarly, the upper memberships of antecedents and consequents of the relational system is again a weighted aggregation

μAk (xn ) = n

μB m (y) =

,

ckm μkm (xn ) + (1 − ckm ) μAkn (xn ) n

,

(28)

m

  ckm μkm (y) + (1 − ckm ) μB m (y) .

(29)

k

The upper and lower membership functions constitute the interval type-2 fuzzy antecedents and consequents of the relational type-2 interval fuzzy system.

5

Experimental Results

We have tested our relational type-2 interval fuzzy logic system on two standard benchmarks: Iris and Wine classification. The first, Iris flowers classification benchmark contained 150 patterns uniformly distributed among three classes. The Wine dataset contained 59 patterns for class 1, 71 patterns for class 2 and 48 for class 3. The modelling scheme was ”One-against-All”, in which each of the pattern classes was trained against all other classes in independent fuzzy systems. This means that the number of the classifying MISO systems was the same as the number of classes. All classifiers had k = 2 and m = 2. Uncertainty parameter for the second step of training by the FCM was set as p = 2. The decision function

Relational Type-2 Interval Fuzzy Systems

367

was chosen in the form of maximum. We chose Gaussian shape of membership function. The Cartesian product was realised by the algebraic product t-norm. An the relational aggregation was expressed in terms of summation. 75 training patterns were chosen randomly for the Iris problem and 89 patterns were chosen randomly for Wine classification. The rest of patterns were used to test the two systems. Table 1 presents the best, the worst as well as the average classification results of the type-1 and interval type-2 fuzzy relational systems over 50 independent runs of the training algorithm. Comparing the classification rates of these relational systems, the interval systems does not reach the worst classification rate of 50 tests, while the better rate is achieved by both systems. Nevertheless, interval type-2 systems perform better in average rates. Table 1. Classification accuracy of relational systems: type-1 and interval type-2 50 independent runs of the training algorithm problem result relational type-1 FLS relational type-2 interval FLS Iris best 97.3% 97.3% worst 68.0% 73.3% average 84.0% 85.2% Wine best 92.1% 92.1% worst 57.3% 59.6% average 76.1% 78.3%

6

Conclusion

In this paper we have merged interval type-2 fuzzy logic with a relational system framework. For such system we have proposed a new two-step learning method. The first step of this method uses ordinary gradient techniques in order to tune prototype membership functions of type-1 fuzzy sets. The second step relies on the application of fuzzy c-means partition and expands upper and lower membership functions outside of the prototype functions. As a consequence, boundary membership functions of interval type-2 fuzzy sets better fit to data and its FCM membership partitions. The use of the relational fuzzy system framework has allowed us to reduce a number of antecedent and consequent fuzzy sets without decreasing a number of rules sufficient for solving a problem.

Acknowledgements This work was partly supported by the Polish Ministry of Science and Higher Education (Habilitation Project 2007-2010 Nr N N516 1155 33 and N N516 3722 34 2008–2011, Polish-Singapore Research Project 2008-2010).

368

R. Scherer and J.T. Starczewski

References 1. Castillo, O., Aguilar, L., Cazarez-Castro, N., Boucherit, M.: Application of type-2 fuzzy logic controller to an induction motor drive with seven-level diode-clamped inverter and controlled infeed. Electrical Engineering 90(5), 347–359 (2008) 2. Castillo, O., Melin, P.: Intelligent systems with interval type-2 fuzzy logic. International Journal of Innovative Computing, Information and Control 4(4), 771–783 (2008) 3. Hagras, H.: A hierarchical type-2 fuzzy logic control architecture for autonomous robots. IEEE Transactions on Fuzzy Systems 12(4), 524–539 (2004) 4. Mendez, G.: Interval type-2 anfis. In: Advances in Soft Computing – Innovations in Hybrid Intelligent Systems, pp. 64–71 (2007) 5. Torres, P., Sez, D.: Type-2 fuzzy logic identification applied to the modeling of a robot hand. In: Proc. FUZZ-IEEE 2008, Hong Kong (June 2008) 6. Uncu, O., Turksen, I.: Discrete interval type 2 fuzzy system models using uncertainty in learning parameters. IEEE Transactions on Fuzzy Systems 15(1), 90–106 (2007) 7. Starczewski, J.: Efficient triangular type-2 fuzzy logic systems. International Journal of Approximate Reasoning 50, 799–811 (2009) 8. Nowicki, R.: Rough-neuro-fuzzy structures for classification with missing data. IEEE Transactions on Systems, Man, and Cybernetics — Part B: Cybernetics 39(6), 1334–1347 (2009) 9. Scherer, R., Rutkowski, L.: Neuro-fuzzy relational systems. In: Proc. 2002 International Conference on Fuzzy Systems and Knowledge Discovery, Singapore, pp. 18–22 (November 2002) 10. Scherer, R., Rutkowski, L.: Neuro-fuzzy relational classifiers. In: Rutkowski, L., Siekmann, J.H., Tadeusiewicz, R., Zadeh, L.A. (eds.) ICAISC 2004. LNCS (LNAI), vol. 3070, pp. 376–380. Springer, Heidelberg (2004) 11. Setness, M., Babuska, R.: Fuzzy relational classifier trained by fuzzy clustering. IEEE Transactions on Systems, Man and Cybernetics - Part B: Cybernetics 29(5), 619–625 (1999) 12. Liang, Q., Mendel, J.: Interval type-2 fuzzy logic systems: Theory and design. IEEE Transactions on Fuzzy Systems 8, 535–550 (2000) 13. Karnik, N., Mendel, J., Liang, Q.: Type-2 fuzzy logic systems. IEEE Transactions on Fuzzy Systems 7(6), 643–658 (1999) 14. Karnik, N., Mendel, J.: Centroid of a type-2 fuzzy set. Information Sciences 132, 195–220 (2001)

Properties of Polynomial Bases Used in a Line-Surface Intersection Algorithm Gun Srijuntongsiri1 and Stephen A. Vavasis2 1

2

Sirindhorn International Institute of Technology, 131 Moo 5, Tiwanont Road, Bangkadi, Muang, Pathumthani, 12000, Thailand [email protected] University of Waterloo, 200 University Avenue W., Waterloo, ON N2L 3G1, Canada [email protected]

Abstract. In [1], Srijuntongsiri and Vavasis propose the KantorovichTest Subdivision algorithm, or KTS, which is an algorithm for finding all zeros of a polynomial system in a bounded region of the plane. This algorithm can be used to find the intersections between a line and a surface. The main features of KTS are that it can operate on polynomials represented in any basis that satisfies certain conditions and that its efficiency has an upper bound that depends only on the conditioning of the problem and the choice of the basis representing the polynomial system. This article explores in detail the dependence of the efficiency of the KTS algorithm on the choice of basis. Three bases are considered: the power, the Bernstein, and the Chebyshev bases. These three bases satisfy the basis properties required by KTS. Theoretically, Chebyshev case has the smallest upper bound on its running time. The computational results, however, do not show that Chebyshev case performs better than the other two. Keywords: Line/surface intersection, polynomial basis.

1

The Line-Surface Intersection Problem and the Required Basis Properties

Let φ0 , . . . , φn denote a basis for the set of univariate polynomials of degree at most n. For example, the power basis is defined by φi (t) = ti . The line-surface intersection problem can be reduced to the problem of finding all zeros of f (u, v) ≡

m  n 

cij φi (u)φj (v), 0 ≤ u, v ≤ 1,

(1)

i=0 j=0

where cij ∈ R2 (i = 0, 1, . . . , m; j = 0, 1, . . . , n) denote the coefficients [1]. For this article, let the notation · refer specifically to infinity norm. Other norms are explicitly notated so. The Kantorovich-Test Subdivision algorithm 

Supported in part by NSF DMS 0434338 and NSF CCF 0085969.

R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 369–378, 2010. c Springer-Verlag Berlin Heidelberg 2010 

370

G. Srijuntongsiri and S.A. Vavasis

(KTS in short), proposed by Srijuntongsiri and Vavasis [1], can be used to solve (1). KTS works with any polynomial basis φi (u)φj (v) provided that the following properties hold: 1. There is a natural interval [l, h] that is the domain for the polynomial. In the case of Bernstein polynomials, this is [0, 1], and in the case of power and Chebyshev polynomials, this is [−1, 1]. 2. It is possible to compute bounding polytope P of S = {f (u, v) : l ≤ u, v ≤ m a  n h}, where f (u, v) = i=0 j=0 cij φi (u)φj (v) and cij ∈ Rd for any d ≥ 1, that satisfies the following properties: (a) Determining whether 0 ∈ P can be done efficiently (ideally in O(mn) operations). (b) The polytope P is affinely invariant. In other words, the bounding polytope of {Af (u, v) + b : l ≤ u, v ≤ h} is {Ax + b : x ∈ P } for any nonsingular matrix A ∈ Rd×d and any vector b ∈ Rd . (c) For any y ∈ P , y ≤ θ max f (u, v) , (2) l≤u,v≤h

where θ is a function of m and n. (d) If d = 1, then the endpoints of P can be computed efficiently (ideally in O(mn) time). 3. It is possible to reparametrize with [l, h]2 the surface S1 = {f (x) : x ∈ ¯ 0 , r)}, where x0 ∈ R2 and r ∈ R > 0. In other words, it is possible (and B(x efficient) to compute the polynomial fˆ represented in the same basis such that S1 = {fˆ(ˆ x) : x ˆ ∈ [l, h]2 }. 4. Constant polynomials are easy to represent. 5. Derivatives of polynomials are easy to determine in the same basis. (preferably in O(mn) operations). We are generally interested in the case where d = 2. In this case, we call P a bounding polygon. Recall that P is a bounding polygon of S if and only if x ∈ S implies x ∈ P .

2

The Kantorovich-Test Subdivision Algorithm

The description of KTS, as well as the definitions of the quantities mentioned in the description, are given below. More details can be found in [1]. For a given zero x∗ of polynomial f , let ω∗ (x∗ ) and ρ∗ (x∗ ) be quantities satisfying the conditions that, first, ω∗ (x∗ ) is the smallest Lipschitz constant for f  (x∗ )−1 f  , i.e., 2  ∗ −1  2 2f (x ) (f (x) − f  (y))2 ≤ ω∗ (x∗ ) · x − y for all x, y ∈ B(x ¯ ∗ , ρ∗ (x∗ )) (3) and, second, ρ∗ (x∗ ) =

2 . ω∗ (x∗ )

(4)

Properties of Polynomial Bases Used in Line-Surface Intersection Algorithm

371

, 9 γ(θ) = 1/ 4 θ(4θ + 1) − 8θ ,

Define

where θ is as in (2). Define ωD to be the smallest nonnegative constant ω satisfying 2  ∗ −1  2 2f (x ) (f (y) − f  (z))2 ≤ ω · y − z , y, z ∈ D , x∗ ∈ [0, 1]2 (5) satisfying f (x∗ ) = 0, where

D = [−γ(θ), 1 + γ(θ)] . 2

(6)

∗

Denote ωf as the maximum of ωD and all ω∗ (x ) ωf = max{ωD ,

max

x∗ ∈C2 :f (x∗ )=0

ω∗ (x∗ )}.

Finally, define the condition number of f to be cond(f ) = max{ωf ,

max

x∗ ∈C2 :f (x∗ )=0,y∈[0,1]2

2  ∗ −1  2 2f (x ) f (y)2}.

(7)

¯ 0 , r) as the application of We define the Kantorovich test on a region X = B(x 0 0 ¯ Kantorovich’s Theorem on the point x using B(x , 2γ(θ)r) as the domain (refer to [2,3] for the statement of Kantorovich’s Theorem). The region X passes the ¯ 0 , ρ− ) ⊆ D  . Kantorovich test if ηω ≤ 1/4 and B(x The other test KTS uses is the exclusion test. For a given region X, let fˆX be the polynomial in the basis φi (u)φj (v) that reparametrizes with [l, h]2 the surface defined by f over X. The region X passes the exclusion test if the bounding polygon of {fˆX (u, v) : l ≤ u, v ≤ h} excludes the origin. Having defined the above prerequisites, the description of KTS can now be given. Algorithm KTS: – Let Q be a queue with [0, 1]2 as its only entry. Set S = ∅. – Repeat until Q = ∅ 1. Let X be the patch at the front of Q. Remove X from Q. 2. If X ⊆ XS for all XS ∈ S, ¯ 0 , r) • Perform the exclusion test on X = B(x • If X fails the exclusion test, (a) Perform the Kantorovich test on X (b) If X passes the Kantorovich test, i. Perform Newton’s method starting from x0 to find a zero x∗ . ii. If x∗ ∈ XS for any XS ∈ S (i.e., x∗ has not been found previously), ∗ its associated ω∗ (x∗ ) by binary search. ∗ Compute ρ∗ (x 6 ) and 7 ∗ ∗ ¯ ∗ Set S = S ∪ B(x , ρ∗ (x )) . (c) Subdivide X along both u and v-axes into four equal subregions. Add these subregions to the end of Q.

372

G. Srijuntongsiri and S.A. Vavasis

The following theorem shows that the efficiency of KTS has an upper bound that depends only on the conditioning of the problem and the choice of the basis. Theorem 1. Let f (x) = f (u, v) be a polynomial system in basis φi (u)φj (v) in two dimensions with generic coefficients whose zeros are sought. Let X = ¯ 0 , r) be a patch under consideration during the course of the KTS algorithm. B(x The algorithm does not need to subdivide X if r≤

1 · min 2



1 − 1/γ(θ) 1 , ωD  2θ cond(f )2

3 .

(8)

Proof. See [1]. Remark: Both terms in the bound on the right-hand side of (8) are increasing as a function of 1/θ. Therefore, our theorem predicts that the KTS algorithm will be more efficient for θ as small as possible (close to 1).

3

Properties of the Power, Bernstein, and Chebyshev Bases

As mentioned above, the basis used to represent the polynomial system must satisfy the properties listed in Section 1 for KTS to work efficiently. Three bases, the power, Bernstein, and Chebyshev bases are examined in detail. The power basis for polynomials of n is φk (t) = tk (0 ≤ k ≤ n). The Bernstein basis  degree  n is φk (t) = Zk,n (t) = (1 − t)n−k tk (0 ≤ k ≤ n). The Chebyshev basis is k φk (t) = Tk (t) (0 ≤ k ≤ n), where Tk (t) is the Chebyshev polynomial of the first kind generated by the recurrence relation T0 (t) = 1, T1 (t) = t, Tk+1 (t) = 2tTk (t) − Tk−1 (t) for k ≥ 1.

(9)

Another way to define the Chebyshev polynomials of the first kind is through the identity Tk (cos α) = cos kα. (10) This second definition shows, in particular, that all zeros of Tk (t) lies in [−1, 1]. It also shows that −1 ≤ Tk (t) ≤ 1 for any −1 ≤ t ≤ 1. The rest of this article shows that the power, Bernstein, and Chebyshev bases all satisfy these basis properties. The values θ’s of the three bases and their corresponding bounding polygons are also derived as these values dictate the efficiency of KTS operating on such bases. The upper bound of the efficiency of KTS is lowest when it operates on the basis with the smallest θ.

Properties of Polynomial Bases Used in Line-Surface Intersection Algorithm

3.1

373

Bounding Polygons

The choices of l and h and the definitions of bounding polygons of the surface S = {f (u, v) : l ≤ u, v ≤ h}, where f (u, v) is represented by one of the three bases, that satisfy the required properties are as follows: For Bernstein basis, the convex hull of the coefficients (control points), call it P1 , satisfies the requirements for l = 0 and h = 1. The convex hull P1 can be described as ⎧ ⎫ ⎨ ⎬  P1 = cij sij : sij = 1, 0 ≤ sij ≤ 1 . (11) ⎩ ⎭ i,j

i,j

For power and Chebyshev bases, the bounding polygon ⎧ ⎫ ⎨ ⎬  P2 = c00 + cij sij : −1 ≤ sij ≤ 1 ⎩ ⎭

(12)

i+j>0

satisfies the requirements for l = −1 and h = 1. Note that P2 is a bounding polygon of S in the Chebyshev case since |Tk (t)| ≤ 1 for any k ≥ 0 and any t ∈ [−1, 1]. Determining whether 0 ∈ P2 is done by solving a small linear programming problem. To determine if 0 ∈ P1 , the convex hull is constructed by conventional method and is tested if it contains the origin. The affine and translational invariance of P1 and P2 for their respective bases can be verified as follows: Let g(u, v) = Af (u, v) + b =

m  n 

cij φi (u)φj (v).

i=0 j=0

 For the Bernstein basis, by using the property that nk=0 Zk,n (t) = 1, it is seen that cij = Acij + b for all cij ’s. Therefore, the bounding polygon of {g(u, v) : 0 ≤ u, v ≤ 1} is ⎧ ⎫ ⎨ ⎬  P1 = cij sij : sij = 1, 0 ≤ sij ≤ 1 ⎩ ⎭ i,j i,j ⎧ ⎫ ⎨ ⎬  = (Acij + b)sij : sij = 1, 0 ≤ sij ≤ 1 ⎩ ⎭ i,j i,j ⎧ ⎫ ⎨  ⎬   = A cij sij + b sij : sij = 1, 0 ≤ sij ≤ 1 ⎩ ⎭ i,j i,j i,j ⎫ ⎧ ⎬ ⎨   cij sij + b : sij = 1, 0 ≤ sij ≤ 1 = A ⎭ ⎩ i,j

= {Ax + b : x ∈ P1 } .

i,j

374

G. Srijuntongsiri and S.A. Vavasis

For the power and the Chebyshev bases, note that φ0 (u)φ0 (v) = 1 for both bases. Hence, c00 = Ac00 + b and cij = Acij for i + j > 0. The bounding polygon of {g(u, v) : 0 ≤ u, v ≤ 1} for this case is ⎧ ⎫ ⎨ ⎬  cij sij : −1 ≤ sij ≤ 1 P2 = c00 + ⎩ ⎭ i+j>0 ⎫ ⎧ ⎬ ⎨  Acij sij : −1 ≤ sij ≤ 1 = Ac00 + b + ⎭ ⎩ i+j>0 ⎧ ⎛ ⎫ ⎞ ⎨ ⎬  = A ⎝c00 + cij sij ⎠ + b : −1 ≤ sij ≤ 1 ⎩ ⎭ i+j>0

= {Ax + b : x ∈ P2 } . 3.2

The Size of the Bounding Polygons Compared to the Size of the Bounded Surface

This section describes the main technical results of this article. Due to space limitation, readers are referred to [4] for the proofs of the theorems in this section. Item 2c of the basis properties in effect ensures that the bounding polygons are not unboundedly larger than the actual surface itself lest the bounding polygons lose their usefulness. The value θ also can be used as a measure of the tightness of the bounding polygon. Recall from Theorem 1 that the efficiency of KTS depends on θ. Since the bounding polygons P1 and P2 are defined by the coefficients of f , our approach to derive θ is to first derive ξ, a function of m and n, satisfying cij  ≤ ξ max f (u, v) , l≤u,v≤h

for any coefficient cij of f . But the following lemma shows that one needs only derive the equivalent of ξ for univariate polynomial to derive ξ itself. Lemma 1. Assume there exists a function h(n) such that bi  ≤ h(n) max g(t)

(13)

l≤t≤h

for any bi (i = 0, 1, . . . , n), and any univariate polynomial g(t) = Then cij  ≤ h(m)h(n) max f (u, v) , l≤u,v≤h

n

i=0 bi φi (t).

(14)

for any c = 0, 1, . . . , m; j = 0, 1, . . . , n), and any bivariate polynomial ij (i  m n f (u, v) = i=0 j=0 cij φi (u)φj (v). Proof. See [4]. The following theorems state ξ of the three bases.

Properties of Polynomial Bases Used in Line-Surface Intersection Algorithm

375

Theorem 2. Let f (t) be a polynomial system n f (t) = i=0 ci Zi,n (t), 0 ≤ t ≤ 1, where ci ∈ Rd . The norm of the coefficients can be bounded by ci  ≤ ξB (n) max f (t) , t:0≤t≤1

(15)

where ξB (n) =

n 



max{|n − j|, |j|} = O(nn+1 ). |i − j| i=0 j=0,1,...,i−1,i+1,...,n

Remark. An inequality in the other direction, namely, that max f (t) ≤ max ci  ,

t:0≤t≤1

is a well-known consequence of the convex hull property of Bernstein polynomials [5]. Proof. See [4]. Theorem 3. Let f (t) be a polynomial system f (t) =

n 

ci Ti (t),

i=0

where ci ∈ Rd . The norm of the coefficients can be bounded by √ ci  ≤ 2 max f (t) . t:−1≤t≤1

(16)

Proof. See [4]. Theorem 4. Let f (t) be a polynomial system f (t) =

n 

c i ti ,

i=0

where ci ∈ Rd . The norm of the coefficients can be bounded by ci  ≤

3n+1 − 1 √ max f (t) . t:−1≤t≤1 2

(17)

Proof. See [4]. Having ξ for each of the three bases, the values of θ for the three bases can now be derived.

376

G. Srijuntongsiri and S.A. Vavasis

Corollary 1. Let f (u, v) =

m  n 

cij Zi,m (u)Zj,n (v),

i=0 j=0

where cij ∈ R2 (i = 0, 1, . . . , m; j = 0, 1, . . . , n). Let P1 be the convex hull of {cij }. Then, for any y ∈ P1 , y ≤ ξB (m)ξB (n) max f (u, v) . 0≤u,v≤1

Proof. By the convex hull property of Bernstein polynomials, y ≤ maxi,j cij  for any y ∈ P1 . The corollary then follows from Theorem 2 and Lemma 1. Corollary 2. Let f (u, v) =

m  n 

cij ui v j ,

i=0 j=0

where cij ∈ R2 (i = 0, 1, . . . , m; j = 0, 1, . . . , n). Let P2 be defined as in (12). Then, for any y ∈ P2 , y ≤

(m + 1)(n + 1)(3m+1 − 1)(3n+1 − 1) max f (u, v) . −1≤u,v≤1 2

Proof. For any y ∈ P2 , y ≤

m  n 

cij  .

i=0 j=0

The corollary then follows from Theorem 4 and Lemma 1. Corollary 3. Let f (u, v) =

m  n 

cij Ti (u)Tj (v),

i=0 j=0

where cij ∈ R2 (i = 0, 1, . . . , m; j = 0, 1, . . . , n). Let P2 be defined as in (12). Then, for any y ∈ P2 , y ≤ 2(m + 1)(n + 1) Proof. For any y ∈ P2 , y ≤

max

−1≤u,v≤1

m  n 

f (u, v) .

cij  .

i=0 j=0

The corollary then follows from Theorem 3 and Lemma 1.

Properties of Polynomial Bases Used in Line-Surface Intersection Algorithm

4

377

Relationship between the Bounding Polygon of the Power Basis and That of Chebyshev Basis

Let P2p denote the bounding polygon P2 computed from the power basis representation of a polynomial and P2c denote P2 computed from the Chebyshev basis representation of it. The results from previous section show that the value θ of P2c is smaller than θ of P2p . This only implies that the worst case of P2c is better than the worst case of P2p . Comparing the values of θ’s of the two does not indicate that P2c is always a better choice than P2p for every polynomial. The following theorem shows, however, that P2c is, in fact, always a better choice than P2p . Theorem 5. Let f : R2 → R2 be a bivariate polynomial. Its bounding polygon P2c is a subset of its bounding polygon P2p . Proof. See [4].

5

Computational Results

Three versions of KTS algorithms are implemented in Matlab; one operating on the polynomials in power basis, one on Bernstein basis, and one on Chebyshev basis. They are tested against a number of problem instances with varying condition numbers. Most of the test problems are created by using the normally distributed random numbers as the coefficients cij ’s of f in Chebyshev basis. For some of the test problems especially those with high condition number, some coefficients are manually entered. The resulting Chebyshev polynomial system is then transformed to the equivalent system in the power and the Bernstein bases. Hence the three versions of KTS solve the same polynomial system and the efficiency of the three are compared. The degrees of the test polynomials are between biquadratic and biquartic.

Table 1. Comparison of the efficiency of KTS algorithm operating on the power, the Bernstein, and the Chebyshev bases. The number of patches examined during the course of the algorithm and the width of the smallest patch examined are shown for each version of KTS. Power basis Bernstein basis Chebyshev basis cond(f ) Num. of Smallest Num. of Smallest Num. of Smallest patches width patches width patches width 3 3.8 × 10 29 .125 17 .0625 21 .125 1.3 × 104 13 .125 17 .0625 13 .125 2.5 × 105 49 .0625 21 .0625 45 .0625 1.1 × 106 97 .0313 65 .0313 85 .0313 3.9 × 107 89 .0313 81 .0313 89 .0313

378

G. Srijuntongsiri and S.A. Vavasis

For the experiment, we use the algorithm by J´onsson and Vavasis [6] to compute the complex zeros required to estimate the condition number. Table 1 compares the efficiency of the three versions of KTS for the test problems with differing condition numbers. The total number of subpatches examined by KTS during the entire computation and the width of the smallest patch among those examined are reported. The results do not show any one version to be particularly more efficient than the others although the Chebyshev basis has better theoretical bound than the other two.

6

Conclusion

Three common bases, the power, the Bernstein, and the Chebyshev bases, are shown to satisfy the required properties for KTS to perform efficiently. In particular, the values of θ for the three bases are derived. These values are used to calculate the time complexity of KTS when that basis is used to represent the polynomial system. The Chebyshev basis has the smallest θ among the three, which shows that using KTS with the Chebyshev basis has the smallest worstcase time complexity. The computational results, however, show no significant differences between the performances of the three versions of KTS operating on the three bases. It appears that, in average case, choosing any of the three bases do not greatly affect the efficiency of KTS.

References 1. Srijuntongsiri, G., Vavasis, S.A.: A condition number analysis of a line-surface intersection algorithm. SIAM Journal on Scientific Computing 30(2), 1064–1081 (2008) 2. Deuflhard, P., Heindl, G.: Affine invariant convergence theorems for Newton’s method and extensions to related methods. SIAM J. Numer. Anal. 16, 1–10 (1980) 3. Kantorovich, L.: On Newton’s method for functional equations (Russian). Dokl. Akad. Nauk SSSR 59, 1237–1240 (1948) 4. Srijuntongsiri, G., Vavasis, S.A.: Properties of polynomial bases used in a line-surface intersection algorithm (February 2009), http://arxiv.org/abs/0707.1515 5. Farin, G.: Curves and Surfaces for CAGD: A Practical Guide, 5th edn. Academic Press, London (2002) 6. J´ onsson, G., Vavasis, S.: Accurate solution of polynomial equations using Macaulay resultant matrices. Mathematics of Computation 74, 221–262 (2005)

A GPU Approach to the Simulation of Spatio–temporal Dynamics in Ultrasonic Resonators Pedro Alonso–Jord´ a1, Isabel P´erez–Arjona2, and Victor J. S´anchez–Morcillo2 1

2

Instituto Universitario de Aplicaciones de las Tecnolog´ıas de la Informaci´ on Universidad Polit´ecnica de Valencia, Cno. Vera s/n, 46022 Valencia, Spain [email protected] Instituto de Investigaci´ on para la Gesti´ on Integrada de Zonas Costeras, Universitat Polit`ecnica de Val`encia, Crta. Nazaret-Oliva s/n, 46730 Grau de Gandia, Spain {victorsm,iparjona}@fis.upv.es

Abstract. In this work we present the implementation of an application to simulate the evolution of pressure and temperature inside a cavity when acoustic energy is injected, a physical system currently under intensive research. The particular features of the equations of the model makes the simulation problem very stiff and time consuming. However, intrinsic parallelism makes the application suitable for implementation in GPUs providing the researchers with a very useful tool to study the problem at a very reasonable price. In our experiments the problem was solved in less than half the time required by CPUs.

1

Introduction

As it is well known among researchers in high performance computing, Graphics Processor Units (GPUs), beyond their specialized capability to process graphics, have strongly consolidated as a serious alternative to general purpose CPUs to speed up scientific computing applications (GPGPU). Implementing general purpose intensive computation applications in graphics hardware is of interest for several reasons including the high computational power of GPUs at a very good cost–performance ratio together with the availability of CUDA [1]. However, programming GPUs for general purpose applications has clear limitations. Their highly specialized architecture based on breadth parallelism using single instruction multiple data (SIMD) processing units is not suitable for all applications. Although in our experiments we used a 64-bit capable hardware (Nvidia Quadro FX 5800), it does not yet perform as well as single precision. Furthermore, the development software kit still does not fully exploit this new hardware. But despite these and other well-known limitations that can be added 

This work was financially supported by Spanish Ministerio de Ciencia e Innovaci´ on, European Union FEDER (FIS2008-06024-C03-02), Universidad Polit´ecnica de Valencia (20080009) and Generalitat Valenciana (20080811).

R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 379–386, 2010. c Springer-Verlag Berlin Heidelberg 2010 

380

P. Alonso–Jord´ a, I. P´erez–Arjona, and V.J. S´ anchez–Morcillo

to the list, the potential performance benefits sometimes makes attempting to implement the application for GPUs worthwhile as is the case here. In this paper we present a study of the spatio–temporal dynamics of an acoustic field. An intensive study of the behaviour of such a system must be done in the numerical field to validate the mathematical model, contrasting it with the physical experiment. The physical experiment is characterized by very different time scales that force an extremely small simulation time step to be taken. Thus, the problem becomes extremely time consuming, and often such numerical problems cannot be undertaken in PC systems with standard processing capabilities. Basically, the computational cost of the simulation of our model relies on intensive use of two dimensional FFTs (Fast Fourier Transforms) on small matrices (64 × 64) plus the computation of linear and non–linear terms involved in the model. We resolved the problem in half the time obtained with one of the most powerful CPUs (Intel Itanium II processor). This is achieved not only because of the computation power of the GPUs but because of the intrinsic parallelism of some computations derived from the discretization method used to solve the evolution equations of the physical system. A survey of discretization of ODEs and PDEs on GPUs can be found in [2]. A similar work to ours discretizing PDEs on GPUs can be found in [3]. The paper is organized as follows. Section 2 summarizes the physical problem. Next we describe the simulation algorithm. In Section 4 we describe the implementation details of the GPU algorithm. The experimental results are analyzed in Section 5. We close with a conclusions section.

2

The Physical Model

Many systems in nature, when driven far from equilibrium, can self–organize, giving rise to a large variety of patterns or structures. Although studied intensively for most of the last century, it has only been during the past thirty years that pattern formation has emerged as branch of science in its own right [4]. A model to study the spatio–temporal dynamics of an acoustic field has been proposed in a recent work [5]. The physical system consists of two solid walls, containing a viscous fluid, where acoustic energy is injected by vibrating one of the walls. In a viscous medium sound propagates with a speed c that depends significantly on temperature. The propagation of sound in such a medium can be described by two coupled equations for pressure, p(r, t) = P (x, y, t) cos(kz), and temperature, T (r, t) = T0 (x, y, t) + T1 (x, y, t) cos(2kz). In the case of a fluid confined in a driven resonator, it was shown in [5] that the pressure and temperature fields inside the cavity obey the evolution equations τp ∂τ P = −P + Pin I + i∇2 P + i (T0 + T1 − Δ) P, 2

∂τ T0 = −T0 + D∇2 T0 + 2 |P | , ∂τ T1 =

−τg−1 T1

(1) 2

+ D∇ T1 + |P | . 2

where P , T0 and T1 are proportional to the amplitudes of pressure field, the homogeneous and grating (modulated) temperature components. The constants

A GPU Approach to the Simulation of Spatio–temporal Dynamics

381

τp and τg are the normalized relaxation times of the pressure field and the temperature grating component, respectively. Other parameters are the detuning Δ between the driving frequency ω and the closest cavity mode, Pin the injected pressure and D the diffusion coefficient. Finally ∇2 = ∂ 2 /∂x2 + ∂ 2 /∂y 2 is the transverse Laplacian operator, necessary for pattern formation. The model parameters can be estimated for a typical experimental situation [5]. It follows that the normalized decay times are τp ∼ 10−6 , and τg ∼ 10−2 under usual conditions. So the problem is typically very stiff, since 0 < τp  τg  1. Stiff systems of differential equations, where the different variables evolve along very different time scales, are usually problematic for numerical solving, and certain numerical methods are unstable, unless the step size is taken to be extremely small yielding thus very high time consumption.

3

The Simulation Algorithm

The essential part of the simulation algorithm can be summarized as: Algorithm termo2D: ...initializations... for iteration = 1:2000000 (P, T0 , T1 ) ← terms(P, T0 , T1 , n, τp , τg , Δ, Pin , δ) P ← CP ∗ P ← CT0 ∗ T0 T0 T1 ← CT1 ∗ T1 end for ...results storage and analysis...

where symbol ∗ denotes the convolution operation. P , T0 , T1 , CP , CT0 and CT1 are matrices of size n × n, where n = 64, and δ = 10−6 is the time step size. The integration method is divided into two steps: 1) the spatial part, which corresponds to the Laplacian operators in equations (1), and 2) the linear and non–linear blocks. The spatial part is integrated using an exact split–step algorithm, which consists of transforming the spatial part of the equation into the spatial Fourier space and obtaining a differential equation with exact solution: FFT ∂t f = ∇2 f −→ ∂t f˜ = −k 2 f˜ ,

where f is a function, f˜ is the corresponding spatial Fourier transformed and k is the spatial frequency. The equation is transformed into the Fourier space, propagated multiplying by the temporal step and re–transformed into the real space (convolution). The linear block is exactly integrated from the linear matrix system in equations (1) when spatial and nonlinearities are not considered. The nonlinear part is integrated via a second order Runge–Kutta scheme. The computation of these blocks is carried out in function terms (Fig. 1). The chosen method, dividing the integration in two separated steps and using the split-step method for the spatial integration, provides some advantages over other usual methods like the finite elements technique. The linear systems obtained when the finite elements method is used can often be solved by fast

382

P. Alonso–Jord´ a, I. P´erez–Arjona, and V.J. S´ anchez–Morcillo

function (P, T0 , T1 ) = terms(P, T0 , T1 , n, τp , τg , Δ, Pin , δ) is P  = P + (0.5δ/τp ) (−(1 + iΔ)P + Pin I + i(T0 + T1 ) · P ) T0 = T0 + 0.5δ (−T0 + 2P · P ∗ ) T1 = T1 + 0.5δ (−(1/τg )T1 + P · P ∗ ) P ← P + (δ/τp ) (−(1 + iΔ)P  + Pin I + i(T0 + T1 ) · P  ) T0 ← T0 + δ (−T0 + 2P  · P ∗ ) T1 ← T1 + δ (−(1/τg )T1 + P  · P ∗ ) Fig. 1. Function terms (∗ denotes complex conjugated, · the matrix–wise product of √ matrices, I is the identity matrix and i is the complex variable (i = −1))

algorithms. Nevertheless, to achieve high accuracy a large number of points in the transverse dimension or complex schemes to reduce the sparsity of the matrix are necessary. The advantage of the split–step method (and of spectral and pseudospectral methods, in general) is that it provides high accuracy in comparison with the finite elements technique for a similar number of discretization points [6].

4

The GPU Algorithm

Algorithm termo2D is implemented in CUDA as follows: ...initializations... for (iteration = 1; iteration GP U threads then, several sequential steps are needed. That is solved, by creating a queue of tasks (nodes). GPU would load nodes from the queue until there are some tasks present. 3. Execute sequential tree search once more (like in step 1) andafter reaching leaf node, read stored results (in step 2) from the array/list and calculate main nodes score. Parallelization brings an overhead of executing the sequential part twice. Therefore, proper choice of values p and s is very important. Depth s should be minimized to decrease that cost, while producing enough leaves (value k)

Root node Sequential depth

s

Sequential K leaves

K root nodes

Parallel depth

p

Parallel L leaves

Fig. 1. Parallelization

Parallel Minimax Tree Searching on GPU

451

to keep every GPU thread busy. Hence, following condition should be met: (k ≥ threads). The value k cannot be determined beforehand due to variable branching factor of each tree. One way of solving the case is estimation and the other is unrolling the root node to the moment when the condition is met, which means searching the tree gradually deeper until k is large enough. 2.4

Modifications

Straightforward minimax port to the CUDA code and parallel execution did not show significant speedup compared to the CPU sequential execution time. One of the reasons for weak performance is the SIMD way of data processing for each warp. The main feature of considered group of trees is the variable branching factor which relies on the actual game state. The approximate average branching factor of a single tree node was calculated to be 8,7. 2.5

Warp Divergence Problem

Every thread of each warp performs the same operation at given moment. Whenever the change of control operation within one warp is present (such as if-else instructions), ’warp divergence’ occurs. That means, that all the threads have to wait until if-else conditional operations are finished. Each root node processed in parallel manner represents different board state, therefore every searched tree may have a different structure. Iterative tree searching algorithm is based on one main outer loop and 2 inner loops (traversing up and down). Depending on number of children for each node, the moment of the loop break occurrence varies, and therefore that causes the divergence in each warp. The ideal case, when no divergence is present, was estimated by calculating average time of searching same root node by each thread. The result proved that reducing divergence, could significantly improve the search performance (reduce the time needed). 2.6

Improving the Performance

A method of reducing warp divergence has been implemented. Decreasing the number of different root nodes in each warp minimizes the divergence, but also results in longer processing time. A way of achieving the parallel performance and keeping low warp divergence is to modify the algorithm, so that the parallel processing parts are placed where no conditional code exists and allow warps to process shared data (each warp should processes one root node). In analyzed case each node represents a game board. Each game board consists of 64 fields. Processing a node is:

1. Load the board. If node is not terminal, then for each field: a. If move is possible, generate a child node b. Save list of children (after processing each field) 2. Else calculate the score then for each field:

452

K. Rocki and R. Suda

a. Get value (score) b. Add or subtract from the global score depending on the value and save board score Operations performed for each field can be done in parallel. Because each warp consists of 32 threads, it is not difficult to distribute the work. Each thread in a warp searches the same tree, while analyzing different node parts. In order to implement described method, both the shared memory and the atomic operations have to be used. Each warp has common (shared) temporary memory for storing children generated by the threads and for storing the score. CUDA atomic operations are needed for synchronizing data access of each thread. I.e. adding a value to shared score memory involves reading the memory, adding value and then storing the updated score. Without such a synchronization one thread could easily overwrite a value, previously store by the other one. Warps can also be divided into smaller subgroups to dedicate fewer threads to one node, i.e. if each 16 threads processes one node, then one warp searches 2 boards in parallel. Further, the results of dividing the warps into groups of 8 and 16 are also presented. Summarizing, for each node, the modified algorithm works as follows: 1. Read node 2. If node is a leaf, calculate score in parallel - function calculateScore(). Join (synchronize) the threads, return result 3. Else generate children in parallel - function getChildren(). Join (synchronize) the threads, save children list 4. Process each child of that node Described procedure decreases thread divergence, but does not eliminate it, still, some flow control instructions are present while analyzing possible movement for each board field. Many papers describe load balancing algorithm usage in detail in order to increase the performance. Here the GPU’s Thread Execution Manager performs that task automatically. Provided that a job is divided into sufficiently many parts, an idle processor will be instantly fed with waiting jobs.

Node Part Part

Part .............

Atomic Add

Score Fig. 2. Score calculation

Part

Parallel Minimax Tree Searching on GPU

3

453

Results

Each value obtained is an average value of searching 1000 nodes, being a result of continuously playing Reversi and moving at random thus the result may be an approximation of the average case. 3.1

CPU

First, the basic parallelized algorithm was tested on a 8-way Xeon E5540 machine to check the correctness and to obtain a reference for further GPU results. Graph shows the advantage of using several CPU cores to search tree of depth 11, where 7 levels are solved sequentially and 4 levels in a parallel manner. In this case, each CPU thread processed one node from a given node queue at once.

Parallellized algorithm’s speedup on multicore Xeon machine

Speedup (comparing to single core)

8

actual result ideal speedup

7

6

5

4

3

2

1

1

2

3

4

5

6

7

Number of cores

Fig. 3. CPU results

GPU

Performance of GPU implementation without warp divergency 60 Speedup (comparing with single CPU)

3.2

50

Depth 12 − 8 levels sequential, 4 parallel Depth 11 − 8 levels sequential, 3 parallel Depth 10 − 8 levels sequential, 2 parallel

40

30

20

10

0

1 GPU

2 GPUs

Fig. 4. GPU results - no divergence

8

454

K. Rocki and R. Suda Performance of GPU implementation without warp divergency

Speedup (comparing with single CPU)

60

50

No divergency, depth 12 − 8 levels sequential, 4 parallel Reduced divergency, depth 12 − 8 levels sequential, 4 parallel No divergency, depth 11 − 8 levels sequential, 3 parallel Reduced divergency, depth 11 − 8 levels sequential, 3 parallel

40

30

20

10

0

1 GPU

2 GPUs

Fig. 5. GPU results Performance comparison of implemented algorithms (searching tree of depth 11)

Speedup (comparing to single CPU)

45 40 35 30

no divergency 32 threads per board 16 threads per board 8 threads per board 1 thread per board, 9 levels sequential, 2 parallel 1 thread per board, 8 levels sequential, 3 parallel

25 20 15 10 5 0

1 GPU

2 GPUs

Fig. 6. GPU results

GPU tests were performed on a machine equipped with a 2.83 GHz Q9550 Intel CPU and 2 GeForce 280 GTX GPUs. To ensure that GPU is busy, the configuration was set to 128 threads per block (4 warps) and 240 blocks in total which equals 128 * (240 / 30 processors) = 1024 threads and 8 blocks per MP (physical multiprocessor) - maximum that can work concurrently on GTX 280. Here: 1 core time for level 11 % 1070s, and for level 10 % 115s.

4

Analysis

CPU results show that the overhead of the parallel algorithm compared to the standard one is minimal. GPU results present significant improvement in the case when no divergence is present (average speed of searching same node by each thread). I.e., when sequential depth is set to 8 and parallel depth is set to 2 we observe 20x (20.2) speedup, then for value of 3 nearly 27x (26.7) speedup for single GTX 280 GPU and over 30x (32.6) for parallel depth of 4. This can be explained by communication cost overhead, solving more on the GPU at once saves time spend on transferring the data. Algorithm also scales well into 2 GPU giving 55x (55.2) speedup for 2 GPUs in the best case of parallel depth 4. When different nodes/board are analyzed by each thread then the results are much worse in the basic case showing only 1.4x speedup for a single GPU

Parallel Minimax Tree Searching on GPU

455

and 2.7x for 2 GTXs. These are the results for sequential/parallel depth of 8/3. The performance increases slightly when parallel depth is decreased to 2 and sequential is increased to 9. Then the observed speedup for a single GPU was 3x and 5.8 for 2 GPUs. Nevertheless, such a way of improving the effectiveness is not applicable when sequential depth is greater (too many nodes to store) and the increase in performance is still insignificant. Analyzing the reduced thread divergence version of the algorithm, we can observe that the performance increases as number of threads dedicated to solve node grows. For single GPU 5.3/7.1/7.7 and for 2 GPUs 10.5/13.6/14.8 speedups are noticed respectively. Still it is only approximately 1/3 of the ideal case, nevertheless performance is much better than the basic algorithm’s one. Moreover, increasing the parallel depth from 3 to 4 also effected in the performance increase as in the no divergence example. Second important factor observed was the data turnaround size. Tree searching is a problem, where most of the time is spent on load/store operations. Just to estimate the data size for tree of depth 11, if the 10average branching factor equals 8.7: Average number of total nodes in a tree k=0 k = (8.7)k % 2.8 ∗ 109. In the presented implementation mostly bitwise operations were used to save space for a single node representation occurring in 16B memory usage for each node/board. Therefore at least 16B ∗ 2.8 ∗ 109 = 44.8GB data had to be loaded/stored (not including the other data load while analyzing the board). While implementing the algorithm, decreasing the size of a single node from a naive 64B board to bit-represented one [3] increased the overall speed ∼10 times.

5

Possible Improvements

Further warp divergence decrease could effect in an additional speedup. A general solution for every non-uniform tree could be branch reordering, so that search process starts in a branch with largest number of children and ends in the one with the smallest number. If a tasks cannot be parallelized in the way presented in this paper, a method of improving SIMD performance with MIMD algorithms is described [8][9]. Parallel node pruning (i.e. αβ) algorithm might be implemented [2][4][5][6], however the batch way of GPU processing implies serious difficulties in synchronizing the jobs processed in parallel. Moreover, the inter-thread communication is very limited (only within the same block). In some papers, SIMD αβ implementations are presented, but they differ from this case. I.e. in [6] no particular problem is being solved, moreover the tree is synthetic with constant branching factor.

6

Conclusions

Considering the results obtained, GPU as a group of SIMD processors performs well in the task of tree searching, where branching factor is constant. Otherwise, warp divergence becomes a serious problem, and should be minimized. Either by parallelizing the parts of algorithm that are processed in a SIMD way or by

456

K. Rocki and R. Suda

dividing main task into several smaller subtasks [8][9]. Direct implementation of the algorithm did not produce any significant improvement over the CPU sequential execution or effected in even worse performance. GPU outperforms a single CPU if high level of parallelism is achieved (here many nodes have to be searched in parallel). For 4 levels searched in parallel, a single GTX 280 GPU performs the task approximately 32 times faster than single CPU core if no divergence is present. 2 GPUs give 55x speedup factor (that is 72% faster than a single GPU). Regarding the modified algorithm, the values are 7.7x speedup for a single GPU and 14.8x for 2 GTXs (92% faster). That shows that the algorithm could be scaled to even larger amount of GPUs easily. One of the main disadvantages of the GPU implementation is that modifications like αβ pruning are hard to implement. Limited communication and necessity to launch all the thread at once cause the difficulty. Another aspect of analyzed approach is possibility of concurrent GPU and CPU operation. While GPU processes bigger chunks of data (thousands) at once, each CPU searches other nodes that are waiting in a queue.

References [1] Knuth, D.E., Moore, R.W.: An analysis of alpha-beta pruning. Artificial Intelligence 6, 293–326 (1975) [2] Manohararajah, V.: Parallel Alpha-Beta Search on Shared Memory Multiprocessors, Master Thesis, Graduate Department of Electrical and Computer Engineering, University of Toronto (2001) [3] Warren, H.S.: Hacker’s Delight. Addison-Wesley, Reading (2002) [4] Schaeffer, J.: Improved Parallel Alpha Beta Search. In: Proceedings of 1986 ACM Fall Joint Computer Conference (1986) [5] Borovska, P., Lazarova, M.: Efficiency of Parallel Minimax Algorithm for Game Tree Search. In: Proceedings of the International Conference on Computer Systems and Technologies (2007) [6] Schaeffer, J., Brockington, M.G.: The APHID Parallel αβ algorithm. In: Proceedings of the 8th IEEE Symposium on Parallel and Distributed Processing, p. 428 (1996) [7] Hewett, R., Ganesan, K.: Consistent Linear speedup in Parallel Alpha-beta Search. In: Proceedings of the ICCI 1992, Fourth International Conference on Computing and Information, pp. 237–240 (1992) [8] Hopp, H., Sanders, P.: Parallel Game Tree Search on SIMD Machines. In: Ferreira, A., Rolim, J.D.P. (eds.) IRREGULAR 1995. LNCS, vol. 980, pp. 349–361. Springer, Heidelberg (1995) [9] Sanders, P.: Efficient Emulation of MIMD behavior on SIMD Machines. In: Proceedings of the International Conference on Massively Parallel Processing Applications and Development, pp. 313–321 (1995) [10] Russell, S., Norvig, P.: Artificial Intelligence: A Modern Approach, pp. 163–171. Prentice Hall, Englewood Cliffs (2003)

A Fast GPU Implementation for Solving Sparse Ill-Posed Linear Equation Systems Florian Stock and Andreas Koch Embedded Systems and Applications Group Technische Universit¨ at Darmstadt {stock,koch}@eis.cs.tu-darmstadt.de

Abstract. Image reconstruction, a very compute-intense process in general, can often be reduced to large linear equation systems represented as sparse under-determined matrices. Solvers for these equation systems (not restricted to image reconstruction) spend most of their time in sparse matrix-vector multiplications (SpMV). In this paper we will present a GPU-accelerated scheme for a Conjugate Gradient (CG) solver, with focus on the SpMV. We will discuss and quantify the optimizations employed to achieve a soft-real time constraint as well as alternative solutions relying on FPGAs, the Cell Broadband Engine, a highly optimized SSE-based software implementation, and other GPU SpMV implementations.

1

Introduction and Problem Description

In this work, we describe the solution of a practical reconstruction problem under soft-real time constraints as required by an industrial embedded computing usecase. For greater generality, we have abstracted away the individual problem details and concentrate on the solution itself, namely the reconstruction of voxel information from the sensor data. For the specific use-case, this process requires the solution of a matrix linear equation system with up to 250 × 106 elements (depending on the image size). However, due to practical limitations of the sensor system (e.g., due to unavoidable mechanical inaccuracies), the gathered data does not allow perfect reconstruction of the original image and leads to a strongly ill-posed linear equation system. To achieve acceptable image quality, a domain-specific regularization, which narrows the solution space, has to be employed. It expresses additional requirements on the solution (such as ∀i : xi ≤ 0) and is applied during each step of the conjugate gradient (CG) method, allowing the reconstruction of suitable-quality images in ≈ 400 CG iterations. It is the need for this regularization vector F as a correction term, that makes other, generally faster equation solving methods (see Section 2) unsuitable for this specific problem. The matrix A, which represents the linear equation system, has up to 855000 nonzero elements. These nonzero elements comprise 0.3 − 0.4% of all elements. If stored as single-precision numbers in a non-sparse manner, it would require more than 2 GB of memory. R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 457–466, 2010. c Springer-Verlag Berlin Heidelberg 2010 

458

F. Stock and A. Koch

Table 1. Loop body of the CG algorithm. F is the regularizing correction term.

ek = AT Adk + F (xk , dk ) fk = dTk ek αk = −(gkT ek )/f k+1 x = xk + αk dk gk+1 = AT (Axk+1 − b) + F (xk , xk+1 ) T ek )/fk βk = (gk+1 dk+1 = −gk+1 + βk dk

Fig. 1. CUDA architecture

A number of storage formats are available for expressing sparse matrices more compactly (e.g., ELLPACK, CSR, CSC, jagged diagonal format [1]). For our case, as the matrix is generated row-by-row by the sensors, the most suitable approach is the compressed sparse row format (CSR, sometimes also referred as compressed row storage CRS). For our application, this reduces the required storage for A to just 7 MB. As the CG solver demands a positive semi-definite matrix, we use the CGNR (CG Normal Residual) approach [1] and left-multiply the equation system with AT . Hence, we do not solve Ax = b, but AT Ax = b. Due to the higher condition number κ of the matrix AT A, with κ(AT A) = κ(A)2 , this will however result in a slower convergence of the iterative solver. Table 1 shows the pseudo-code for the modified CG algorithm. Computing 400 iterations of this CG (size of matrix A 320,000 × 3,000 with 1,800,000 nonzero entries) requires 15 GFLOPS and a bandwidth of 105 GB/s. The softreal time constraint of the industrial application requires the reconstruction of four different images (taken by different sensors) in 0.44 seconds. However, since these images are independent, they can be computed in parallel, allowing 0.44s for the reconstruction of a single image. The core of this work is the implementation of the modified CG algorithm on a GPU, with a focus on the matrix multiplication. The techniques shown will be applicable to the efficient handling of sparse matrix problems in general. 1.1

GPGPU Computing and the CUDA System

With continued refinement and growing flexibility, graphics processing units (GPUs) can now be applied to general-purpose computing scenarios (GPGPU), often outperforming conventional processors. Nowadays, the architecture of modern GPUs consists of arrays of hundreds of flexible general-purpose processing units supporting threaded computation models and random-access memories. Dedicated languages and programming tools to exploit this many-core paradigm include the manufacturer specific flows CUDA (described below, by NVIDIA) and Firestream [2] by ATI/AMD, as well as the hardware-independent like the new OpenCL [3]. For our work, we target NVIDIA GPUs, specifically the GTS 8800 models and GTX 280. They are programmed using the Compute Unified Device

GPU Implementation for Solving Sparse Linear Equation Systems

459

Architecture (CUDA) development flow. Figure 1 shows the block diagram of such a CUDA-supported GPU. The computation is done by multiprocessors, which consist of eight scalar processors, operating in SIMD (Single Instruction Multiple Data) mode. Each of the scalar processors can execute multiple threads, which it schedules without overhead. This thread scheduling is used to hide memory latencies in each thread. Each multiprocessor furthermore has a small shared memory (16 KB). which is like the device global memory randomly accessible. But accesses to the global memory require for good overall throughput, that consecutive threads must access consecutive memory addresses (such accesses are called coalesced). Others have a significant performance penalty. An entire computation, a so-called kernel, is instantiated on the GPU device as multiple blocks. These are then executed in arbitrary order, ideally in parallel, but may be serialized, e.g., when the number of blocks exceeds the number of multiprocessors available on the device. The block structure must be chosen carefully by the programmer, since no fast inter-block communication is possible. A block is assigned atomically for execution on a multiprocessor, which then processes the contained threads in parallel in a SIMD manner. Note that the threads within a block may communicate quickly via the shared memory and can be efficiently synchronized. At invocation time, a kernel is parametrized with the number of blocks it should execute in as well as the number of threads within each block (see [4] for more details). For high performance computing (HPC) applications the number of threads is usually much larger than the number of multiprocessors.

2

Related Work

A large number of methods can be used for solving linear equations. They are often classified into different types of algorithms, such as direct methods (e.g., LU decomposition), multi grid methods (e.g., AMG) and iterative methods (e.g., Krylov subspace; see [1]) Due to the need to compensate for sensor limitations by the correction term F (see Section 1), an appropriately modified iterative CG method fits our requirements best. The computation of the sparse matrix vector product (SpMV) has long been discovered to be the critical operations of the CG method. With the importance of matrix multiplication in general, both to this and many other important numerical problems, it is worthwhile to put effort towards fast implementations. The inherent high degree of parallelism makes it an attractive candidate for acceleration on the highly parallel GPU architecture. For the multiplication of dense matrices, a number of high-performance GPU implementations have already been developed [5,6]. A very good overview and analysis of sparse multiplications is given in [7]. Different storage formats and methods for CUDA implemented SpMV are evaluated and compared, but the focus is on much larger matrices and the different storage formats. Furthermore,

460

F. Stock and A. Koch

NVIDIA supplies the segmented scan library CUDPP with a segmented scanbased SpMV implementation (described in [8]). Another similar work, which focuses on larger matrices, is presented in [9], where two storage formats on different platforms (different GPGPUs and CPU) are compared. On a more abstract level, the CG algorithm in its entirety is best accelerated by reducing the number of iterations. This is an effective and popular method, but often requires the exploitation of application-specific properties (e.g., [10]). Beyond our application-specific improvement using the F correction term, we have also attempted to speed-up convergence by using one of the few domainindependent methods, namely an incomplete LU pre-conditioner [1]. However, with the need to apply F between iterations, this did not improve convergence.

3

Implementation

Due to the memory bottleneck between GPU and the, in our case significant, fixed-time overhead of starting a kernel on the GPU [11], we have to design our implementation carefully to actually achieve wall-clock speed-ups over conventional CPUs. 3.1

Sparse Matrix Vector Multiplications (SpMV)

As we focus in this paper on the SpMV, we implemented and evaluated a number of different ways to perform SpMVs for our application on the GPU. Variant Simple. This baseline implementation uses a separate thread to compute the scalar product of one row of the matrix and the vector. As described before, the matrix is stored in CSR format and the vector as an array. This arrangement, while obvious, has the disadvantage that for consecutive thread IDs, neither the accesses to the matrix nor to the vector are coalesced and lead to a dramatically reduced data throughput. Variant MemoryLayout. This first GPU-specific optimization uses an altered ELLPACK-like (see e.g. [7] for more details on the format) memory layout for the matrix. By using a transposed structure, we can now arrange the matrix data so that consecutive threads find their data at consecutive addresses. This is achieved by having k separate index and value arrays, with k being the maximum number of nonzero elements in row. indexi [j] and valuei[j] is the ith nonzero value/index of row j. Since in our case A has a uniform distribution of nonzero elements, this optimization has little impact on the required memory. Variant LocalMemory. The next variant attempts to achieve better coalescence of memory accesses by employing the shared memory. Since shared memory supports arbitrary accesses without a performance penalty, we will transfer chunks of data from/to global memory to/from shared memory using coalesced accesses and perform non-coalesced accesses penalty-free within shared memory. Due to

GPU Implementation for Solving Sparse Linear Equation Systems

461

the strongly under-determined nature of our equation system, the result vector of A · x is much smaller than x and fits completely into the local memory for the multiplication with AT . However, the complete vector x does not fit into shared memory for the actual multiplication with A. Thus, the operation has to be split into sub-steps. From the size of the local memory (16 KB) the maximum size mv for a vector n in local memory can be computed. Mathematically, a thread j computes i=1 aji xi . 2m  mv aji xi + i=mv v +1 aji xi + . . .. These sums can now be This is equivalent to i=1 used as separate SpMV problems, where the sub-vectors fit completely into the local memory. Variant OnTheFly. This variant trades a higher number of operations for reduced memory bandwidth. The correction term represented by the matrix F used for the image reconstruction is the result of a parametrized computation, that allows the separate computation of each of the rows of F . Thus, each thread is able to compute the indexes and values it needs and only has to fetch from memory the corresponding vector component. As this on-the-fly generation of the matrix F is only possible row by row, but not column by column, this variant can only be used for the multiplication with A, but not for the multiplication with AT . This variant can be combined with the previous one, generating only segments of the matrix row and multiplying them with a partial vector which is buffered in the local memory. Variant Backprojection. The last of our implementation variants tries to reverse the matrix multiplication. Typically, all threads read each component of the x vector multiple times, but write each component in the result vector just once. The idea is to reverse this procedure and read each component of x just once (i.e. thread i would read component xi , which could then be read coalesced) and would write multiple times to the result component (non-coalesced). While the same result is computed, the altered flow of the computation would allow a very efficient memory layout: Reconsider from the local memory variant that the result y of A · x would entirely fit into shared memory. Thus, the noncoalesced writes to the result components yj would carry no performance penalty. Instead, the expensive non-coalesced reads of the xi from main memory could be turned into coalesced ones. However, this promising approach had to be discarded due to limitations of the current CUDA hardware: Parallel threads would have to correctly update yj by accumulating their local result. To ensure correctness, this would require an atomic update operation that is simply not supported for floating point numbers at this time. Variant Prior Work. As mentioned in Section 2, only limited work on GPU accelerated SpMV is published. For comparison with our kernels, we evaluated all available implementations (i.e. CUDPP[8] and the kernels from Bell and Garland [7]) with our matrices.

462

F. Stock and A. Koch

We follow the scheme of Bell and Garland, which classifies the algorithm according to matrix storage layout and multiplication method. In this scheme, CUDPP would be a so-called COO method. As the details on the different groups and kernels is beyond this paper, we can only refer to the original works. The group of DIA kernels were left out, as they operate only on diagonal structured matrices. Variant Xeon CPU using SSE Vector Instructions. To evaluate the performance of our GPU-acceleration against a modern CPU, we also evaluated a very carefully tuned software SpMV implementation running on a 3.2 GHz Xeon CPU with 6 MB cache. The software version was written to fully exploit the SSE vector instructions and compiled with the highly optimizing Intel C compiler. Kernel Invocation Overhead. As already described in Section 2, we expected to deal with a long kernel invocation delay. We measured the overhead of starting a kernel of the GPU as 20μs for an NVIDIA GTS 8800 512 and of 40μs for an NVIDIA GTX 280. One possible explanation for the increased overhead on the larger GTX card could be the doubled number of multiprocessors (compared to the GTS) card and a correspondingly longer time to initialize all of them. When composing an iteration of the CG algorithm (see Table 1) from kernels for the operations, there will be 14 kernel invocations per iteration, translating to 5600 kernel starts over the 400 iterations required to converge. This would require a total of 0.11s on the GTS-series GPU (one fourth of our entire timing budget). To lessen this invocation overhead, we manually performed loop fusion to merge calls to the same kernels (but operating on different data) into a single kernel (e.g. performing two consecutive scalar products not with two kernel invocations, but with one invocation to a special kernel doing two products). In addition to reducing the call overhead, we gain additional performance by loading data, that is used in both fused computations, only once from memory. In this manner, the number of kernel invocations is reduced from 14 to just six per iteration, now taking a total of 0.048s. Resources. All kernels were invoked with at least as many blocks as multiprocessors were available on each GPU, and on each multiprocessor with as many threads as possible to hide memory latencies. Only the matrix-on-the-fly and the correction term kernels could not execute the maximum of 512 threads per block due to excessive register requirements. Nevertheless, even these kernels still could still run 256 threads per block. 3.2

Alternate Implementations - Different Target Technologies

Beyond the GPU, other platforms, namely FPGA and Cell, were considered, but dismissed in an early stage.

GPU Implementation for Solving Sparse Linear Equation Systems

463

The required compute performance of 15 GFLOPS could easily be achieved using an FPGA-based accelerator, which would most likely also be more energy efficient than the GPGPU solution. However, the required memory bandwidth becomes a problem here. The Virtex 5 series of FPGAs by Xilinx [12] is typical of modern devices. Even the largest packages currently available have an insufficient number of I/O pins to connect the number of memory banks required to achieve the memory bandwidth demanded by the application: A single 64b wide DDR3 DRAM bank, clocked at 400 MHz, could deliver a peak transfer rate of 6.4 GB/s and requires 112 pins on the FPGA. A large package XC5VLX110 device can connect at most to six such banks with a total throughput of < 40 GB/s, which is insufficient. Multi-chip solutions would of course be possible, but quickly become uneconomical. Similar bandwidth problems exist when considering IBM’s Cell Broadband Engine (Cell BE). The Cell BE is a streaming architecture, where eight streaming Synergistic Processing Elements are controlled by a Power Processor. Memory IO is here also limited by a theoretical maximum bandwidth of 25.6 GB/s ([13], which also gives some performance numbers on SpMV). Again, one could of course use a set of tightly-coupled Cell BEs. But since even a single Cell blade is considerably more expensive than multiple graphics cards, such a solution would also be uneconomical in a commercial setting.

4

Experimental Evaluation

We used the following hardware to experimentally evaluate our approach: – Host CPU, also used for evaluating the software version: Intel Xeon with 6 MB Cache, clocked at 3.2 GHz – NVIDIA GTX 280 GPU (16 KB shared memory, 30 multiprocessors, 1300 MHz) with GT200 chip. – NVIDIA 8800 GTS 512 (16 KB shared memory, 16 multiprocessors, 1620 MHz) with G92 chip. Due to power constraints in the embedded system this was the target platform. Table 2 shows the run times of the different variants. The measurements were taken using the NVIDIA CUDA profiler, so the data is only valid for comparison with other profiled runs. Depending on the SpMV (Ax or AT y), the variants perform differently: For the transposed matrix multiplication, the best variant is the method LocalMemory combined with MemoryLayout (not using the OnTheFly technique). The measurements include the time to transfer the vector prior to the operation into the shared memory, where it fits completely. Thus, the random accesses into the vector are not penalized (in comparison to those to global device memory). For the multiplication with the non-transposed matrix, the variant OnTheFly computing the whole row (but not using the LocalMemory technique) is most efficient. The effect of non-coalesced random accesses could be a reduced even further by utilizing the texture cache (increasing performance by 15%).

464

F. Stock and A. Koch

Table 2. Performance of the different SpMV variants, measured for Ax and AT y (A is 81, 545 × 3, 072, containing 854, 129 nonzero elements) Ax time [μs] AT y time [μs] optimization Simple 13,356 1,744 MemoryLayout 1,726 n/a n/a 160 MemoryLayout & LocalMemory OnTheFly 400 11,292,080 807 n/a OnTheFly & LocalMemory Prior Work DIA n/a n/a 606 198 Prior Work ELL 1,183 1,439 Prior Work CSR 750 549 Prior Work COO/CUDPP 793 289 Prior Work HYB

The variant LocalMemory in itself performs poorly for our application, due to the very specific structure of the matrix A: Although the number of nonzero elements is uniform, i.e. approximately the same in each row, their distribution is different. If A was subdivided into s sub matrices with at most as many columns as elements in the partial vector in the shared memory, the number of nonzero elements in the rows of the sub matrices would not be uniform. The longest path for our SIMD computation on a multiprocessor is determined by the largest number of nonzero values in a row processed by one of the threads. Since this may be a different row in each of sub matrices, the worst case execution time of this variant may be s times as long than without the LocalMemory modification. The last block of results in Table 2 shows the different runtimes of the CUDPP kernel and the kernels from [7]. For each group, we show the best time from all kernels of this group. As the numbers indicate, our best implementations are faster than these others kernels. This is due to our algorithm being specialized for the given matrix structure. In addition to the profiler-based measurements reported above, we also evaluated the performance of our GPU implementation of the complete CG on the system level, comparing to the SSE-optimized software version. As further contender, the GTX 280 GPU was used. These tests also encompass the time to transfer the data/matrix from host memory to the GPU device memory. Since only limited amounts of data have to be exchanged during an iteration, these transfers are relatively short. Figure 2 shows the system-level computation time for different image sizes (expressed as number of nonzero elements). For very small matrices, the SSE CPU does actually outperform the GPU (due to data transfer and kernel invocation overheads). However, as soon as the data size exceeds the CPU caches (the knee in the SSE-line at ca. 450K nonzero elements), the performance of the CPU deteriorates significantly. Remarkably, the G92-class GPU actually outperforms its more modern GT200class sibling. Only when the matrices get much larger (and exceed the size of the

GPU Implementation for Solving Sparse Linear Equation Systems

465

Fig. 2. Execution time of different implementations as function of the number of nonzero elements

images required by our application) does the GT200 solve the problem faster. We trace this to two causes: First, the G92 is simply clocked higher than the GT200, and the larger number of multiprocessors and increased memory bandwidth on the GT200 come into play only for much larger images. Second, the G92 spends just 10% of its total execution time in kernel invocation overhead, while the GT200 requires 20%. Going back to the requirements of our industrial application: Our aim of reconstructing images of the given size in just 0.44s was not fully achievable on current generation GPUs. However, our implementation managed to restore images at 70% of the initially specified resolution in time, which proved sufficient for practical use. In this setting, the GPU achieves a peak performance of 1 GFLOPS and 43 GB/s memory bandwidth. The used GTS has a max. memory bandwidth of 48 GB/s (measured on system with bandwidth test included in the CUDA SDK), so we reach 87% of the maximum.

5

Conclusion

We have shown that a GPGPU approach can be viable not only on huge highperformance computing problems, but also on practical embedded applications handling much smaller data sets. The advantage of the GPU even over fast CPUs continues to grow with increasing data set size. Thus, with the trend towards higher resolutions, GPU use will become more widespread in embedded practical applications. Apart from our concrete application, we implemented very efficient sparse vector matrix multiplication for both non-transposed and transposed forms of a matrix, outperforming reference implementations for a GPU as well as a highly optimized SSE software version running on a state-of-the-art processor.

466

F. Stock and A. Koch

It is our hope that future GPUs will reduce the kernel invocation overhead, which really dominates execution time for smaller data sets, and also introduce atomic update operations for floating point numbers. The latter would allow new data and thread organization strategies to further reduce memory latencies.

Acknowledgements Thanks to our industrial partner for the fruitful cooperation.

References 1. Saad, Y.: Iterative Methods for Sparse Linear Systems, 2nd edn. Society for Industrial and Applied Mathematics, Philadelphia (2003) 2. ATI: AMD Stream Computing - Technical Overview. ATI (2008) 3. Khronos Group: OpenCL Specification 1.0 (June 2008) 4. NVIDIA Corp.: NVIDIA CUDA Compute Unified Device Architecture – Programming Guide (June 2007) 5. Kr¨ uger, J., Westermann, R.: Linear algebra operators for gpu implementation of numerical algorithms. In: SIGGRAPH 2003: ACM SIGGRAPH 2003 Papers, pp. 908–916. ACM, New York (2003) 6. Larsen, E.S., McAllister, D.: Fast matrix multiplies using graphics hardware. In: Supercomputing 2001: Proceedings of the 2001 ACM/IEEE Conference on Supercomputing (CDROM), p. 55. ACM, New York (2001) 7. Bell, N., Garland, M.: Efficient sparse matrix-vector multiplication on CUDA. NVIDIA Technical Report NVR-2008-004, NVIDIA Corporation (December 2008) 8. Sengupta, S., Harris, M., Zhang, Y., Owens, J.D.: Scan primitives for gpu computing. In: GH 2007: Proceedings of the 22nd ACM SIGGRAPH/EUROGRAPHICS Symposium on Graphics Hardware, Aire-la-Ville, pp. 97–106. Eurographics Association, Switzerland (2007) 9. Buatois, L., Caumon, G., Levy, B.: Concurrent number cruncher: a gpu implementation of a general sparse linear solver. Int. J. Parallel Emerg. Distrib. Syst. 24(3), 205–223 (2009) 10. Roux, F.X.: Acceleration of the outer conjugate gradient by reorthogonalization for a domain decomposition method for structural analysis problems. In: ICS 1989: Proceedings of the 3rd International Conference on Supercomputing, pp. 471–476. ACM, New York (1989) 11. Bolz, J., Farmer, I., Grinspun, E., Schr¨ ooder, P.: Sparse matrix solvers on the gpu: conjugate gradients and multigrid. In: SIGGRAPH 2003: ACM SIGGRAPH 2003 Papers, pp. 917–924. ACM, New York (2003) 12. Xilinx: Virtex 5 Family Overview. Xilinx (2008) 13. Williams, S., Shalf, J., Oliker, L., Kamil, S., Husbands, P., Yelick, K.: The potential of the cell processor for scientific computing. In: CF 2006: Proceedings of the 3rd Conference on Computing Frontiers, pp. 9–20. ACM Press, New York (2006)

Monte Carlo Simulations of Spin Glass Systems on the Cell Broadband Engine Francesco Belletti1 , Marco Guidetti1 , Andrea Maiorano2, Filippo Mantovani1 , Sebastiano Fabio Schifano3 , and Raffaele Tripiccione1 1

3

Dipartimento di Fisica, Universit` a di Ferrara and INFN Sezione di Ferrara, I-44100 Ferrara, Italy 2 Dipartimento di Fisica, Universit` a di Roma “La Sapienza”, I-00100 Roma, Italy Dipartimento di Matematica, Universit` a di Ferrara and INFN Sezione di Ferrara, I-44100 Ferrara, Italy

Abstract. We implement a Monte Carlo algorithm for spin glass systems and optimize for the Cell-BE processor, assessing the effectiveness of this architecture for state-of-the-art simulations. Recent developments in many-core processor architectures, like the IBM Cell BE seem to open new opportunities in this field, where computational requirements are so demanding that state-of-the-art simulations often use dedicated computing systems. We present our implementation, analyze results and compare performances with those of both commodity processors and dedicated systems.

1

Introduction

Several large-scale computational scientific problems still defy even state-of-theart computing resources. One such problem is the simulation of spin models. Spin models (among them, spin glasses) are relevant in many areas of condensedmatter physics. They describe systems characterized by phase transitions (such as the para-ferro transition in magnets) or by “frustrated” dynamics, appearing when the complex structure of the energy landscape of the system makes the approach to equilibrium very slow. These systems, extensively studied in the last two decades as paradigmatic examples of complexity, have been successfully applied to several scientific areas outside physics, such as quantitative biology, optimization, economics modeling and social studies [1,2]. A spin glass is a disordered magnetic system in which the interactions between the magnetic moments associated to the atoms of the system are mutually conflicting, due to some frozen-in structural disorder [3]. The macroscopic consequence is that it is difficult for the system to relax to equilibrium: glassy materials never truly reach equilibrium on laboratory-accessible times as relaxation times may be, for macroscopic samples, of the order of centuries. As a counterpart of this sluggish physical behavior, finding the state of the system of lowest energy is an NP-hard problem [4] and accurate numerical treatment is a true challenge. R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 467–476, 2010. c Springer-Verlag Berlin Heidelberg 2010 

468

F. Belletti et al.

Short-ranged spin glass models are defined on discrete, finite-connectivity regular grids (e.g., 3-D lattices, when modeling experiments on real magnetic samples) and are usually studied via Monte Carlo simulation techniques. The dynamical variables are the spins, sitting at the edges of the lattice and having a discrete and finite (usually small) set of possible values. State-of-the-art simulations may stretch for well over 1010 Monte Carlo updates of the full lattice, and have to be repeated on hundreds or thousands of different instantiations of the system (samples, see later for a detailed definition). In addition, each sample must be simulated more than once (replicas), as many properties of the model are encoded in the correlation between independent histories of the same system. For a 3-D lattice of linear size ∼ 80 (a typical value for a state-of-the-art investigation) these requirements translate into 1018...19 Monte Carlo-driven spin updates, a major computational challenge. It is trivial to implement sample and replica parallelism, since there is no information exchange between samples (or replicas) during the simulation. There is (see next section for details) a further very large amount of parallelism available in the algorithms associated to the simulation of each sample, that is not efficiently supported by commodity computer architectures but whose exploitation is nevertheless badly needed: indeed, simulating just one system of 803 sites for 1010 Monte Carlo steps implies % 5 × 1015 updates, that is, almost ten years on a commodity processor able to update one spin in % 50 ns (a typical figure). The computational challenge in this area is twofold: i) exploit the parallelism available in the simulation of each sample in order to reduce processing time to figures acceptable on human timescales, and ii) replicate (by the hundreds or thousands) whatever solution has been found to point i), to accumulate results on a large set of samples and replicas. In this paper, we call i) and ii) internal and external parallelism, respectively (we do not endorse in this paper the possibly misleading terminology used in spin glass jargon of synchronous and asynchronous parallelism, see ref. [5]). Correspondingly, we will use two performance metrics: system spin update time (SUT) and global spin update time (GUT). SUT is the average time needed by the Monte Carlo procedure to update one spin of one system. GUT is the average time needed to process one spin, taking into account that many systems may be concurrently handled by the simulation. SUT is a measure of how well we are able to exploit internal parallelism, while GUT measures the effects of compounding internal and external parallelism. In the last decade, application-driven machines, whose architectures are strongly focused and optimized for spin glass simulations, have provided a very effective answer to this challenge [6,7]. Recent projects are arrays of FPGAs [8,9,10]; each FPGA implements a very large number (∼ 1000) of concurrent spin-update engines. Performance on these systems is impressive: simulation time for each sample reduces from several years to just a few months. Application-driven systems imply a large development effort (in terms of costs and manpower) so any new opportunity is welcome. Recent developments in multi-core (and many-core) architectures promise to offer such an opportunity. Indeed, the availability of SIMD data-paths

Monte Carlo Simulations of Spin Glass Systems on the Cell BE

469

in each core helps efficiently exploit internal parallelism. On-chip memory systems (caches or otherwise), as well as efficient core-to-core communication mechanisms, reduce data-access latencies making it also possible to further expose parallelism among a large and increasing set of cores. In this paper, we address the problem of efficiently implementing Monte Carlo simulation for spin glasses on one specific many-core processor, the Cell-BE processor, assessing its performance and effectiveness for state-of-the-art simulations. The remainder of this paper is organized as follows: section 2 defines the Monte Carlo algorithm for spin glass simulations. Section 3 describes our implementation on the Cell BE. Section 4 presents our results and compares with production codes available for commodity x86 architectures, and with results on state-of-the-art dedicated architectures.

2

Monte Carlo Algorithms for Spin Glass Systems

In this paper we consider in details the Edwards-Anderson (EA) model, a widely studied theoretical model of a spin glass. Other models, equally popular in the field, share the same computational structure. The EA model lives on a D−dimensional (usually D = 3) square lattice of linear size L. Atomic magnetic moments are modeled by spins S, discrete variables that take just two values, S = ±1 (in other models, like the Potts model [11] spins are q-valued, q = 3 . . . 6). For each pair (i, j) of neighboring spins in the lattice the model defines an interaction term (coupling) Jij , needed to compute the energy. The latter is a functional of the configuration, i.e. of the set {S} of all spin-values:  E[{S}] = − Jij Si Sj , ij

where ij denotes the sum on all nearest-neighbor points. The Jij are randomly assigned, mimicking the structural disorder of the real system, and kept fixed during Monte Carlo simulation. A given assignment of the full set of Jij is a sample of the system. Physically relevant results are averages over a large number of independent samples. The probability distribution for the Jij reflects the properties of the system: in glassy systems one usually takes Jij = ±1 with 50% probability (binary EA model) or samples a Gaussian distribution with zero mean and unit variance (Gaussian EA model). As already remarked, for each sample we need to consider several replicas (i.e. several copies of the system sharing the same set of couplings and evolving independently during the Monte Carlo process). For each sample, physically relevant observables are the canonical (i.e., fixed temperature) averages of quantities that are themselves functionals  of the configuration – for instance magnetization is defined as M [{S}] = i Si . The canonical average (written as M ) at temperature T is 

M  = M [{S}] e−E[{S}]/T , (1) the sum stretching over all configurations; in words, the canonical average takes into account all configurations, weighted with a factor exponentially depending

470

F. Belletti et al. Require: set of S and J 1. loop {loop on Monte Carlo steps} 2. for all Si do {for each spin Si } 3. Si = (S i == 1) ? − 1 : 1 {flip value of Si } 4. ΔE = ij (Jij · Si · Sj ) − (Jij · Si · Sj ){compute change of energy} 5. if ΔE ≤ 0 then 6. S = S  {accept new value of S} 7. else 8. ρ = rnd() {compute a random number 0 ≤ ρ ≤ 1 with ρ ∈ Q} 9. if ρ < e−βΔE then 10. S = S‘ {accept new value of S} 11. end if 12. end if 13. end for 14. end loop

Algorithm 1. Metropolis Algorithm on their energies. Performing the full sum of eq. (1) is an obviously impossible task for typical lattice sizes, so an approximation is needed. Monte Carlo (MC) methods are one such option; a Monte Carlo process produces a sequence of configurations sampled with probability proportional to the exponential term in eq. (1): an unbiased sum on all configurations generated by the process converges to (1) after an infinite – in practice, a large enough – number of steps. Several Monte Carlo algorithms are known. The Metropolis procedure [12] – usually adopted in spin glasses – generates new configurations by tentatively changing spins, as described by algorithm 1. The visit order of the algorithm is not important, as long as all sites are visited an equal number of times on average (the process of visiting all sites, is called an MC sweep). All steps of the algorithm can be applied in parallel to any subset of spins that do not share a coupling term in the energy function (so we can correctly compute ΔE). We divide the lattice in a checkerboard scheme and apply the algorithm first to all black sites and then to all white ones, trivially identifying an available parallelism of degree up to L3 /2, linearly growing as the size of the system: in principle, we may schedule one full MC sweep for any lattice size in just two computational steps. This is still not possible today: what we have to do is to find ways to implement as large a subset of the available parallelism as possible.

3

Algorithm Implementation

In this section we discuss ways in which the available parallelism described above can be implemented on a multi-core architecture like the Cell-BE [13]. Our main priority is exposing internal parallelism, since external parallelism can be trivially farmed out; we are obviously allowed to mix strategies, if useful for performance. A first approach to internal parallelism uses SIMD instructions within each core, and processes several (white or black) spins in parallel. We consider later further level of internal parallelism, stretching across cores.

Monte Carlo Simulations of Spin Glass Systems on the Cell BE

471

Fig. 1. A data layout with external parallelism w and internal parallelism V allocates spins onto 128-bit SIMD vectors. Each vector has V slots, and each slot allocates w spins, one for each lattice.

External parallelism can be exploited on top of SIMD-based internal parallelism by mapping on the same machine variable the spins belonging to the same site of different samples. This technique is usually referred to as multi-spin coding [14,15]. Mixing internal and external parallelism is necessary, because the Cell-BE, like all commodity architectures, does not support efficiently the onebit-valued data types that represent spins and couplings. Data allocation and coding have to be carefully tuned to obtain an efficient implementation. We first consider data layout. We map V (V = 2, . . . , 64) binaryvalued spins of w = 128/V lattice samples on a 128-bit word as shown in Fig. 3. This data layout allows us to use SIMD instructions that update in parallel V spins for each of w independent lattices, compounding an internal parallelism of degree V and an external parallelism of degree w. The key idea is that all steps of algorithm 1 (except for random numbers, see later) for each spin can be coded as bit-valued logical operations, so many spins can be treated at the same time by SIMD logical instructions. We only need to represent each bit of the values of the energy (and other relevant variables) in a different 128-bit SIMD vector. It will turn out that a rather smallset of such vectors is really needed. The energy Ei of a spin at site i is Ei = − Si Jij Sj . This sum includes all sites j that are nearest-neighbors to site i; in 3-D, it takes all even integer values in the range [−6, 6]. We replace  S and J by the binary variables σ and σi xor λij xor σj . The energy of the spin λ (σ, λ ∈ {0, 1}) and compute Xi = at site i is Ei = 6 − 2Xi , Xi taking all integer values in [0, 6]. Flipping a spin changes the sign of its local energy, so the energy cost of a flip is simply ΔEi = −2Ei = 4Xi − 12, Xi ∈ {0, 1, 2, 3, 4, 5, 6} The Metropolis algorithm accepts the new state if one of the following conditions is true: i) Δ(Ei ) ≤ 0, ii) ρ ≤ e−βΔ(Ei ) (notice that condition i) implies condition

472

F. Belletti et al. Require: ρ pseudo-random number Require: ψ = int (−(1/4β) log ρ){encoded on two bits} Require: η = ( not Xi ){encoded on two bits} 1. c1 = ψ[0] and η[0] 2. c2 = (ψ[1] and η[1]) or ((ψ[1] or η[1]) and c1 ) 3. σi = σi xor (c2 or not Xi [2])

Algorithm 2. Bit-wise algorithm to update one spin 1. WHEEL[k] = WHEEL[k-24] + WHEEL[k-55] 2. ρ = WHEEL[k] ⊕ WHEEL[k-61]

Algorithm 3. Parisi-rapuano Algorithm. WHEEL is a circular array initialized with random 32-bit unsigned-integers, and ρ is the generated pseudo-random number ii) ). All Xi values in [0, 3] correspond to negative or null energy cost, so the spin flip is always accepted in these cases. In a three-bit representation of Xi , this corresponds to having the most significant bit of Xi set to 0. If the most significant bit of Xi is 1, we have to check for condition ii), which we rewrite as −(1/4β) log ρ + (7 − Xi ) ≥ 4 In a three-bit representation, 7 − Xi = not Xi . Since in this case condition i) is not verified, we only need two bits to represent relevant not Xi values in {1, 2, 3}. Any value of −(1/4β) log ρ ≥ 3 verifies the condition, independently of Xi . In addition, since − log(ρ/4β) is the only non-integer quantity involved, we only need its integer part. The inequality then reads: int (−(1/4β) log ρ) + ( not Xi ) ≥ 4 We call ψ = (int (−(1/4β) log ρ)) and η = (( not Xi )), both quantities being truncated to their two least significant bits, so the condition is verified if (ψ + η) generates a carry out of the second bit. Summing up, we compute the new spin value σi as a bit-wise expression defined by algorithm 2. We further optimize the evaluation of ψ replacing the computation of the logarithm by a look-up table. A more detailed description of this implementation is given in [16]. The advantages of this scheme are twofold: i) it exploits the SIMD capabilities of the architecture leveraging on internal and external parallelism and ii) it does not use conditional statements, badly impacting on performance. At each step of the algorithm we need V (pseudo-)random numbers, since the same random value can be shared among the w samples, but independent random numbers are mandatory to update different spins of the same system. We use the 32-bit Parisi-Rapuano generator [17], defined by algorithm 3, a popular choice for spin glass simulations. We use SIMD instructions to generate 4 random numbers in parallel. For values of V > 4, we iterate the code. So far, we have exploited SIMD parallelism within each core. We now consider multi-core parallelism. We split the lattice in C sub-lattices of contiguous planes

Monte Carlo Simulations of Spin Glass Systems on the Cell BE

473

(C is the number of cores). We map each sub-lattice (of L × L × L/C sites) onto a different core. We consider two cases: 1. each sub-lattice is small enough to fit the private memory of each core. 2. it does not fit fully within the private memory of its core. In this case, each core iteratively loads, updates and stores chunks of its sub-lattice till the whole sub-lattice is updated. The first case implies data exchange across the cores (and no data traffic to main memory), while in the second case data access patterns are between main memory and the cores. The algorithm describing our first approach is defined by a loop in which each core first updates all its white spins and then updates all the black ones (as required by the Metropolis algorithm). White and black spins are stored in data-structures called half-planes, each housing L2 /2 spins. Each core updates the half-planes of one color performing the steps described by algorithm 4 Note that the for all statement can be overlapped with the send and receive of the boundary half-planes. The term previous core and next core refer to the cores housing the two adjacent sub-lattices. In the second approach, data is allocated in main memory and split in blocks, small enough to fit inside core memories. Our implementation for this approach is a loop where each core first updates all its white half-planes in the range [0 ; (L/C) − 1], synchronizes with the other cores and then repeats these steps for the black half-planes. The update of each half-plane is performed using a double-buffer scheme and executing the steps of algorithm 5 for each of halfplane i ∈ [0 ; (L/C) − 1]. The approaches described above are adopted also for the Gaussian model. In this case the only internal-parallelism degree available is V = 4, because the couplings are represented by floating-point numbers , and the update computation can not be translated into a bit-wise expression like in the EA model.

4

Performance Results and Conclusions

For our tests we used an IBM QS22 system, with two tightly connected IBM PowerXCell 8i processors. In this setup, it is possible to use up to 16 cores, but each SPE can exchange data at full speed only with the SPEs and the main memory of the same processor. 1. 2. 3. 4. 5. 6.

update the boundaries half-plane (indexes (0) and ((L3 /C) − 1)). for all i ∈ [1..((L3 /C) − 2)] do update half-planes (i) end for send/receive half-plane (0) to the previous core. send/receive half-plane ((L3 /C) − 1) to the next core.

Algorithm 4. Private Memory Implementation

474

F. Belletti et al. 1. for all black,white do 2. for all half-plane (i) ∈ [0..(L/C) − 1] do 3. if i > 0 then 4. store half-plane (i − 1) to main memory 5. end if 6. if i < (L/C) − 1 then 7. load half-plane (i + 2) from main to local memory 8. end if 9. update half-plane (i) and proceed to half-plane (i) = (i + 1) 10. end for 11. synchronize all cores 12. end for

Algorithm 5. Global Memory Implementation We have made extensive performance tests of most allowed combinations of (L, V, C). We have found large variations in overall efficiency as we move in this parameter space; indeed, efficiency depends on a delicate balance of sustained memory bandwidth and processor performance (the latter affected by scheduling details and by the cost of generating random numbers). Our measured data is consistent with a simple theoretical model that will be presented elsewhere (see however [16] for some details). We summarize our main results for the binary and Gaussian model in Table 4, where values of the parameters relevant for tipical lattice sizes are given. For a given value of the lattice size, we list the values of the parameters that yield the best performance, measured as system update time (SUT) and global update time (GUT). This table can be used, when planning a simulation campaign, to identify the program version that provides the best performance (the best mix of external and internal parallelism) for a given size of the problem. We now compare our performance results with those of a previous implementation for an x86 architecture and of a state-of-the-art custom HPC systems optimized for spin glass applications. Let us start with the binary model. We c report here performance of a program running on a 2.4 GHz Intel Core2-Duo&. The program implements the multi-spin coding technique, and is configured for running two replicas of 128 samples, that is each core updates in parallel 128 spins (V = 1 and w = 128 in our previous notation). The program is not coded to use explicitely SSE instructions and registers. However, since the vector type provided by the gcc compiler is used, the compiler is able to exploit some SSE instructions. This program has a SUT of 92.16 ns/spin and a GUT of 0.72 ns/spin. Lattice size does not impact performance as long as it fits the cache memory. On the same processor, a routine partially exploiting internal parallelism, updates 64 spins of a single system of size L = 64 (energies are still computed in parallel using the multi-spin coding trick, but the Metropolis condition check is inherently sequential as different random numbers for every spin are needed in this case). In this case, SUT is 4.8 ns. We now compare our results with the performance of a recent dedicated machine, the Janus processor. Janus is the latest example of an application-driven

Monte Carlo Simulations of Spin Glass Systems on the Cell BE

475

machine strongly optimized for spin glass simulations [8]. Each Janus processor (called an SP) handles one lattice at a time, updating in parallel ∼ 1000 spins (in our notation, V = 1000, W = 1 for Janus) corresponding to a SUT of 0.016 ns/spin. Different copies of the system are handled by different SP on the machine. The smallest Janus system has 16 SPs delivering a GUT of 0.001 ns/spin. Summing up our comparison for the binary model, we can say that, before the introduction of the CBE architecture there was a factor 300X (SUT) or 75X (GUT) between commodity systems and application-driven machines. The Cell strongly reduces this gap to values in the range 10X . . . 60X but does not close it yet: a cluster of ∼ 20 − 60 Cells approaches the performance of one Janus core if we consider the global spin update time, but SUT performance still means that a CBE-based simulation may take several years. This comparison is particularly appropriate, since the cost of commercial Cell-based systems is not appreciably lower than one Janus processor. In any case many-cores are an extremely effective path to performance here and in the not-to-far future application-driven machines may become an obsolete approach to the problem. The scenario is somewhat different for the Gaussian model. In this case the couplings are floating-point numbers, and we cannot exploit multi-spin coding techniques. The code of the Metropolis algorithm available for x86 PC has 23 ns/spin GUT when sharing the same random number among 8 samples. The shortest SUT is % 65 ns when only one sample is simulated. On the other side, Table 1. System and global update time (SUT and GUT) for the binary (left) and Gaussian model (right). We list several allowed combinations of (L, V, C). Some values of L (like L = 80 for the binary) imply data mis-alignement, and a corresponding performance loss. For the Gaussian model, lattice sizes larger that L = 80, requiring changes in data organization to fit the CBE local-store, are not yet implemented.

L C 16 16 32 32 48 48 64 64 80 80 96 96 128 128

8 16 8 16 8 16 8 16 8 16 8 12 8 16

Binary model SUT GUT

V w Memory 8 8 16 16 8 8 32 32 8 8 16 16 64 64

16 local 16 local 8 local 8 local 16 local 16 local 4 local 4 local 16 local 16 local 8 global 8 global 2 global 2 local

(ns/spin) (ns/spin)

0.83 1.17 0.40 0.26 0.48 0.25 0.29 0.15 0.82 1.03 0.42 0.41 0.24 0.12

0.052 0.073 0.050 0.032 0.030 0.016 0.072 0.037 0.051 0.064 0.052 0.051 0.120 0.060

Gaussian model SUT GUT

V w Memory 4 4 4 4 4 4 4 4 4 4 -

1 1 1 1 1 1 1 1 1 1 -

local local local local global local global global global global -

(ns/spin) (ns/spin)

1.00 1.34 0.65 0.42 1.65 0.42 1.71 2.23 1.59 2.10 -

0.250 0.335 0.162 0.105 0.412 0.105 0.427 0.557 0.397 0.525 -

476

F. Belletti et al.

Janus, not well-suited for this case, has no published results for this model. Also for the Gaussian model our implementation is around 10X . . . 20X times faster. In conclusion, our implementation on the Cell-BE is roughly two order of magnitude faster than on an x86 PCs but still one order of magnitude far from the performance of a dedicated system. We might expect that forthcoming multicore architectures will be able to support large scale simulation campaigns. Also, graphics processors, with a much more fine-grained processing structure may offer an alternate approach to the problem: work is in progress to asses the efficiencies of both achitecures. Acknowledgements. We thank the J¨ulich Supercomputing Centre for access to QS22 systems, and members of the Janus collaboration for useful discussions.

References 1. M´ezard, M., Parisi, G., Virasoro, M.: Spin Glass Theory and Beyond. World Scientific, Singapore (1987) 2. Schulz, M.: Statistical Physics and Economics: Concepts, Tools, and Applications. Springer, Heidelberg (2003) 3. Binder, K., Young, A.P.: Spin Glasses: Experimental Facts, Theoretical Concepts and Open Questions. Rev. Mod. Phys. 58, 801–976 (1986) 4. Barahona, J.: On the computational complexity of Ising spin glass models. J. Phys. A: Math. Gen. 15, 3241–3253 (1982) 5. Newman, M.E.J., Barkema, G.T.: Monte Carlo Methods in Statistical Physics. Oxford University Press, USA (1999) 6. Condon, J.H., Ogielski, A.T.: Fast special purpose computer for Monte Carlo simulations in statistical physics. Rev. Sci. Instruments 56, 1691–1696 (1985) 7. Cruz, A., et al.: SUE: A Special Purpose Computer for Spin Glass Models. Comp. Phys. Comm. 133, 165–176 (2001) 8. Belletti, F., et al.: Simulating Spin Systems on Ianus, an FPGA-Based Computer. Comp. Phys. Comm. 178, 208–216 (2008) 9. Belletti, F., et al.: Ianus: an Adaptive FPGA Computer. Computing in Science and Engineering 8, 41–49 (2006) 10. Belletti, F., et al.: JANUS: an FPGA-based System for High Performance Scientific Computing. Computing in Science and Engineering 11, 48–58 (2009) 11. Wu, F.Y.: The Potts Model. Rev. Mod. Phys. 54, 235–268 (1982) 12. Metropolis, N., et al.: Equation of State Calculation by Fast Computing Machine. J. Chemical Physics 21, 1087–1092 (1953) 13. IBM Cell Broadband Engine Architecture, http://www-128.ibm.com/developerworks/power/cell/documents.html 14. Bhanot, G., Duke, D., Salvador, R.: Finite-size scaling and three dimensional Ising model. Phy. Rev. B 33, 7841–7844 (1986) 15. Michael, C.: Fast heat-bath algorithm for the Ising model. Phy. Rev. B 33, 7861– 7862 (1986) 16. F. Belletti Monte Carlo Simulations of Spin Glasses on Cell Broadband Engine Phd Thesis Univesit` a di Ferrara (2009) 17. Parisi, G., Rapuano, F.: Effects of the random number generator on computer simulations. Phys. Lett. B 157, 301–302 (1985)

Montgomery Multiplication on the Cell Joppe W. Bos and Marcelo E. Kaihara Laboratory for Cryptologic Algorithms, ´ Ecole Polytechnique F´ed´erale de Lausanne (EPFL), CH-1015 Lausanne, Switzerland {joppe.bos,marcelo.kaihara}@epfl.ch

Abstract. A technique to speed up Montgomery multiplication targeted at the Synergistic Processor Elements (SPE) of the Cell Broadband Engine is proposed. The technique consists of splitting a number into four consecutive parts. These parts are placed one by one in each of the four element positions of a vector, representing columns in a 4-SIMD organization. This representation enables arithmetic to be performed in a 4-SIMD fashion. An implementation of the Montgomery multiplication using this technique is up to 2.47 times faster compared to an unrolled implementation of Montgomery multiplication, which is part of the IBM multi-precision math library, for odd moduli of length 160 to 2048 bits. The presented technique can also be applied to speed up Montgomery multiplication on other SIMD-architectures. Keywords: Cell Broadband Engine, Cryptology, Computer Arithmetic, Montgomery Multiplication, Single Instruction Multiple Data (SIMD).

1

Introduction

Modular multiplication is one of the basic operations in almost all modern publickey cryptographic applications. For example, cryptographic operations in RSA [1], using practical security parameters, requires a sequence of modular multiplications using a composite modulus ranging from 1024 to 2048 bits. In elliptic curve cryptography (ECC) [2,3], the efficiency of elliptic curve arithmetic over large prime fields relies on the performance of modular multiplication. In ECC, the length of most commonly used (prime) moduli ranges from 160 to 512 bits. Among several approaches to speed up modular multiplication that have been proposed in the literature, a widely used choice is the Montgomery modular multiplication algorithm [4]. In the current work, we study Montgomery modular multiplication on the Cell Broadband Engine (Cell). The Cell is an heterogeneous processor and has been used as a cryptographic accelerator [5,6,7,8] as well as for cryptanalysis [9,10]. In this article, a technique to speed up Montgomery multiplication that exploits the capabilities of the SPE architecture is presented. This technique consists of splitting a number into four consecutive parts which are placed one by one in each of the four element positions of a vector. These parts can be seen as four columns in a 4-SIMD organization. This representation benefits from R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 477–485, 2010. c Springer-Verlag Berlin Heidelberg 2010 

478

J.W. Bos and M.E. Kaihara

the features of the Cell, e.g. it enables the use of the 4-SIMD multiply-and-add instruction and makes use of the large register file. The division by a power of 2, required in one iteration of Montgomery reduction, can be performed by a vector shift and an inexpensive circular change of the indices of the vectors that accumulate the partial products. Our experimental results show that an implementation of Montgomery multiplication, for moduli of sizes between 160 and 2048 bits, based on our newly proposed representation is up to 2.47 times faster compared to an unrolled implementation of Montgomery multiplication in the IBM’s Multi-Precision Math (MPM) Library [11]. The article is organized as follows. In Section 2, a brief explanation of the Cell broadband engine architecture is given. In Section 3, we describe the Montgomery multiplication algorithm. In Section 4, our new technique for Montgomery multiplication is presented. In Section 5 performance results of different implementations are given. Section 6 concludes this paper.

2

Cell Broadband Engine Architecture

The Cell architecture [12], developed by Sony, Toshiba and IBM, has as a main processing unit, a dual-threaded 64-bit Power Processing Element (PPE) and eight Synergistic Processing Elements (SPEs). The SPEs are the workhorses of the Cell and the main interest in this article. The SPE consist of a Synergistic Processor Unit (SPU) and a Memory Flow Controller (MFC). Every SPU has access to a register file of 128 entries, called vectors or quad-words of 128-bit length, and a 256 kilobyte Local Store (LS) with room for instructions and data. The main memory can be accessed through explicit direct memory access requests to the MFC. The SPUs have a 128-bit SIMD organization allowing sixteen 8-bit, eight 16-bit or four 32-bit integer computations in parallel. The SPUs are asymmetric processors that have two pipelines denoted as even and odd pipelines. This means that two instructions can be dispatched every clock cycle. Most of the arithmetic instructions are executed on the even pipeline and most of the memory instructions are executed on the odd pipeline. It is a challenge to fully utilize both pipelines always at the same time. The SPEs have no hardware branch-prediction. Instead, hints can be provided by the programmer (or the compiler) to the instruction fetch unit that specifies where a branch instruction will jump to. An additional advantage of the SPE architecture is the availability of a rich instruction set. With a single instruction, four 16-bit integer multiplication can be executed in parallel. An additional performance improvement may be achieved with the multiply-and-add instruction which performs a 16 × 16-bit unsigned multiplication and an addition of the 32-bit unsigned operand to the 32-bit multiplication result. This instruction has the same latency as a single 16 × 16bit multiplication and requires the 16-bit operands to be placed in the higher positions of the 32-bit sections (carries are not generated for this instruction). One of the first applications of the Cell processor was to serve as the heart of Sony’s PS3 video game console. The Cell contains eight SPEs, and in the

Montgomery Multiplication on the Cell

479

Input: M : r n−1 ≤ M < r n , 2  M 1: X, Y : 0 ≤ X, Y < M 2: Output: Z = X · Y · R−1 mod M 3: 4: 5: 6: 7: 8: 9: 10: 11: 12:

Z=0 for i = 0 to n − 1 do Z = Z + yi · X q = (−Z · M −1 ) mod r Z = (Z + q · M )/r end for if Z ≥ M then Z =Z−M end if

Algorithm 1. Radix-r Montgomery Multiplication[4] PS3 one of them is disabled. Another SPEs is reserved by Sony’s hypervisor (a software layer which is used to virtualize devices and other resources in order to provide a virtual machine environment to, e.g., Linux OS). In the end, six SPEs can be accessed when running Linux OS on the PS3.

3

Montgomery Modular Multiplication

The Montgomery modular multiplication method introduced in [4] consists of transforming each of the operands into a Montgomery representation and carry out the computation by replacing the conventional modular multiplications by Montgomery multiplications. This is suitable to speed up, for example, modular exponentiations which can be decomposed as a sequence of several modular multiplications. One of the advantages of this method is that the computational complexity is usually better compared to the classical method by a constant factor. Given an n-word odd modulus M , such that rn−1 ≤ M < rn , where r = 2w is theradix of the system, and w is the bit length of a word, and an integer n−1 /= X = i=0 xi ·2w·i , then the Montgomery residue of this integer is defined as X X · R mod M . The Montgomery radix R, is a constant such that gcd(R, M ) = 1 and R > M . For efficiency reasons, this is usually adjusted to R = rn . The / Y/ ) = X / · Y/ ·R−1 mod M . Montgomery product of two integers is defined as M (X, / = X·R mod M and Y/ = Y ·R mod M are Montgomery residues of X and Y , If X / = M (X, / Y/ ) = X ·Y ·R mod M is a Montgomery residue of X ·Y mod M . then Z Algorithm 1 describes the radix-r interleaved Montgomery algorithm. The conversion between the ordinary representation of an integer X to the / can be performed using the Montgomery algoMontgomery representation X / = M (X, R2 ), provided that the constant R2 mod M is rithm by computing X pre-computed. The conversion back from the Montgomery representation to the ordinary representation can be done by applying the Montgomery algorithm to / 1). the result and the number 1, i.e. Z = M (Z,

480

J.W. Bos and M.E. Kaihara

In cryptologic applications, where modular products are usually performed succeedingly, the final conditional subtraction, which is costly, is not needed until the end of a series of modular multiplications [13]. In order to avoid the last conditional subtraction (lines 7 to 9 of Algorithm 1), R is chosen such that 4M < R and inputs and output are represented as elements of Z/2M Z instead of Z/M Z, that is, operations are carried out in a redundant representation. It can be shown that throughout the series of modular multiplications, outputs from multiplications can be reused as inputs and these values remain bounded. This technique does not only speed-up modular multiplications but also prevents the success of timing attacks [14] as operations are data independent [13].

4

Montgomery Multiplication on the Cell

In this section, we present an implementation of the Montgomery multiplication algorithm that takes advantage of the features of the SPE; e.g. the SIMD architecture, the large register file and the rich instruction set. The multiplication algorithm implemented in the MPM library uses a radix r = 2128 to represent large numbers. Each of these 128-bit words is in turn represented using a vector of four consecutive 32-bit words in a SIMD fashion. One drawback of this representation is that operands whose sizes are slightly larger than a power of 2128 require an entire extra 128-bit word to be processed, waisting computational resources. Another drawback is that the addition of the resulting 32-bit product to a 32-bit value might produce a carry which is not detected. In contrast to the addition instruction, there is no carry generation instruction for the multiply-and-add operation and is not used in the Montgomery multiplication of the MPM library. The technique we present uses a radix r = 216 which enables a better division of large numbers into words that match the input sizes of the 4-SIMD multipliers of the Cell. This choice enables the use of the following property, and hence the multiply-and-add instruction: If a, b, c, d ∈ Z and 0 ≤ a, b, c, d < r, then a · b + c + d < r2 . Specifically, when r = 216 , this property enables the addition of a 16-bit word to the result of a 16 × 16-bit product (used for the multi-precision multiplication and accumulation) and an extra addition of 16-bit word (used for 16-carry propagation) so that the result is smaller than r2 = 232 and no overflow can occur. We will assume hereafter that the radix r = 216 . Given an odd 16n-bit modulus M , i.e. r(n−1) ≤ M < rn , a Montgomery > ? residue X, such that 0 ≤ X < 2M < r(n+1) , is represented using s = n+1 4 vectors of 128 bits. The extra 16-bit word is considered in the implementation because the intermediate accumulating result of Montgomery multiplication can be up to  2M . The Montgomery residue X is represented using a radix r system, n i.e. X = i=0 xi · ri . On the implementation level the 16-bit words xi are stored column-wise in the s 128-bit vectors Xj , where j ∈ [0, s − 1]. The four 32-bit parts of such a vector are denoted by Xj = {Xj [0], Xj [1], Xj [2], Xj [3]}, where Xj [0] and Xj [3] contain the least and most significant 32-bits of Xj respectively. Each of the (n + 1) 16-bit words xi of X is stored in the most significant 16 bits

Montgomery Multiplication on the Cell ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ Xs−1 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ . ⎪ ⎪ ⎨ .. X=

⎪ ⎪ Xj ⎪ ⎪ ⎪ ⎪ .. ⎪ ⎪ ⎪ . ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ X0 ⎪ ⎪ ⎪ ⎩

128-bit legth vector

@ =

AB

x3s−1 16-bit

16-bit

high

low

x2s−1

xs−1

481

C

B C@ A B C@ A .. . x3s+i

x2s+i

xs+i

xi

xs

x0

= .. . x3s

=

x2s

B

C@

A

the least significant 16-bit word of X

n i Fig. 1. The 16-bit words xi of a 16(n > n+1+? 1)-bit positive integer X = i=0 xi · r < 2M 128-bit vectors Xj on the SPE architecture are stored column-wise using s = 4

D E of Xi mod s [ si ]. A graphical representation of this arrangement is provided in Figure 1. We follow hereafter the same notation to represent the long integer numbers used. The Montgomery multiplication algorithm using this representation is described in Algorithm 2. The algorithm takes as inputs the modulus M of n words and the operands X and Y of (n + 1) words. The division by r, which is a shift by 16 bits of the accumulating partial product U , is implemented as a logical right shift by 32 bits of the vector that contains the least significant position of U and a change of the indices. That is, during each iteration, the indices of the vectors that contain the accumulating partial product U change circularly among the s registers without physical movement of data. In Algorithm 2, each 16-bit word of the inputs X, Y and M and the output Z is stored in the upper part (the most significant 16 bits) of each of the four 32-bit words in a 128-bit vector. The vector μ stores the replicated values of (−M )−1 mod r in the lower 16-bit positions of the words. The temporary vector K stores in the most significant 16-bit positions the replicated values of yi , i.e. each of the parsed coefficients of the multiplier Y corresponding to the i-th iteration of the main loop. The operation A ← muladd(B, c, D), which is a single instruction on the SPE, represents the operation of multiplying the vector B (where data are stored in the higher 16-bit positions of 32 bit words) by a vector with replicated 16-bit values of c across all higher positions of the 32-bit words. This product is added to D and the overall result is placed into A. The temporary vector V stores the replicated values of u0 in the least significant 16-bit words. This u0 refers n to the least significant 16-bit word of the updated value of U , i.e. U = j=0 uj · rj represented by s vectors of 128-bit Ui mod s , Ui+1 mod s , . . . , Ui+n mod s following the above explained notation (i refers to the index of the main loop). The vector Q is computed as an element-wise logical left shift by 16 bits of the 4-SIMD product of vectors V and μ. The propagation of the higher 16-bit carries of U(i+j) mod s described in lines 9 and 15 consist of extracting the higher 16-bit words of these vectors and placing

482

J.W. Bos and M.E. Kaihara Input: ⎧ ⎪ M represented by s 128-bit vectors: Ms−1 , . . . , M0 , such that ⎪ ⎪ n−1 ⎪ ≤ M < r n , 2  M, r = 216 ⎨ r X represented by s 128-bit vectors: Xs−1 , . . . , X0 , ⎪ ⎪ Y represented by s 128-bit vectors: Ys−1 , . . . , Y0 , such that 0 ≤ X, Y < 2M ⎪ ⎪ ⎩ μ : a 128-bit vector containing (−M )−1 mod r replicated in all 4 elements.  Z represented by s 128-bit vectors: Zs−1 , . . . , Z0 , such that Output: Z ≡ X · Y · r −(n+1) mod M, 0 ≤ Z < 2M 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21:

for j = 0 to s − 1 do Uj = 0 end for for i = 0 to n do K = {yi , yi , yi , yi } for j = 0 to s − 1 do U(i+j) mod s = muladd(Xj , K, U(i+j) mod s ) end for Perform 16-carry propagation on U(i+j) mod s for j = 0, . . . , s − 1 V = {u0 , u0 , u0 , u0 } Q =shiftleft(mul(V , μ), 16) /* Q = V · μ mod r */ for j = 0 to s − 1 do U(i+j) mod s = muladd(Mj , Q, U(i+j) mod s ) end for Perform 16-carry propagation on U(i+j) mod s for j = 0, . . . , s − 1 Ui mod s =vshiftright(Ui mod s , 32) /* Vector logical right shift by 32 bits*/ end for Perform carry propagation on Ui mod s for i = n + 1, . . . , 2n + 1 for j = 0 to s − 1 do Zj = U(n+j+1) mod s /* Place results in higher 16-bit positions*/ end for

Algorithm 2. Montgomery Multiplication Algorithm for the Cell Table 1. Performance results, expressed in nanoseconds, for the computation of one Montgomery multiplication for different bit-sizes on a single SPE on a PlayStation 3 Bit-size This work MPM Unrolled Ratio Ratio of moduli (ns) (ns) MPM (ns) 192 224 256 384 512 1024 2048

174 369 200 369 228 369 339 652 495 1,023 1,798 3,385 7,158 12,317

2.12 1.85 1.62 1.92 2.07 1.88 1.72

277 277 277 534 872 3,040 11,286

1.59 1.39 1.21 1.58 1.76 1.69 1.58

them into the lower 16-bit positions of temporary vectors. These vectors are then added to U(i+j+1) mod s correspondingly. The operation is carried out for the vectors with indices j ∈ [0, s − 2]. For j = s − 1, the temporary vector that contains

Montgomery Multiplication on the Cell

483

2.6

2.4

2.2

Times faster

2

1.8

1.6

1.4

1.2

1 256

512

768 1024 1280 Number of bits in the modulus

1536

1792

2048

Fig. 2. The speed-up of the new Montgomery multiplication compared to the unrolled Montgomery multiplication from MPM

the words is logically shifted 32 bits to the left and added to the vector Ui mod s . Similarly, the carry propagation of the higher words of U(i+j) mod s described in line 18 is performed with 16-bit word extraction and addition, but requires a sequential parsing over the (n + 1) 16-bit words. Note that U is represented with vectors whose values are placed in the lower 16-bit word positions.

5

Experimental Results

In this section, performance comparison of different software implementations of the Montgomery multiplication algorithm running on a single SPE of a PS3 using moduli of varying bit lengths is presented. The approach described in Section 3 is compared to the implementation in the Multi-Precision Math (MPM) Library [11]. The MPM library is developed by IBM and part of the software development kit [15] for the Cell. MPM consists of a set of routines that perform arithmetic on unsigned integers of a large number of bits. According to [11]: “All multiprecision numbers are expressed as an array of unsigned integer vectors (vector unsigned int) of user specified length (in quadwords). The numbers are assumed to be big endian ordered”. To enhance the performance and avoid expensive conditional branches, a code generator has been designed. This generator takes as input the bit-size

484

J.W. Bos and M.E. Kaihara

of the modulus and outputs the Montgomery multiplication code in the Cprogramming language that uses the SPU-intrinsics language extension. Such a generator has been made for both our approach as well as for the implementation of Montgomery multiplication in the MPM library. We refer to this faster version of MPM as unrolled MPM. The computational times, in nanoseconds, of a single Montgomery multiplication for cryptographically interesting bit sizes, are given in Table 1. Our Montgomery implementation is the subtraction-less variant (suitable for cryptographic applications) while the MPM version is the original Montgomery multiplication including the final subtraction. A subtraction-less variant of the MPM implementation is significantly slower since it requires the processing of an entire 128-bit vector for one extra bit needed to represent the operands in the interval of values [0, 2M ). Thus, it is not considered in our comparison. The performance results include the overhead of benchmarking, function calls, loading and storing the inputs and output back and forth between the register file and the local store. The stated ratios are the (unrolled) MPM results versus the new results. Figure 2 presents the overall speed-up ratios, compared to the unrolled implementation of MPM. Every 128 bits a performance peak can be observed in the figure, for moduli of 128i + 16 bits. This is because the (unrolled) MPM implementations work in radix r = 2128 and requires an entire 128-bit vector to be processed for these extra 16 bits. This peak is more considerable for smaller bit-lengths because of the significant relative extra work compared to the amount of extra bits. This effect becomes less noticeable for larger moduli. The drop in the performance ratio of our algorithm, that occurs every multiple of 64 bits in Figure 2, can be explained by the fact that moduli of these bit sizes require an extra vector to hold the number in redundant representation. Despite this fact our implementation outperforms the unrolled MPM (which is faster compared to generic MPM) because it only iterates over the 16-bit words that contain data. The only bit sizes where the unrolled MPM version is faster compared to our approach are in the range [112, 128]. This is due to a small constant overhead built in our approach and becomes negligible for larger bit-lengths. The maximal speed-up obtainable from our approach is 2.47 times compared to the unrolled Montgomery multiplication of the MPM library and occurs for moduli of size 528 bit.

6

Conclusions

We have presented a technique to speed up Montgomery multiplication on the SPEs of the Cell processor. The technique consist of representing the integers by grouping consecutive parts. These parts are placed one by one in each of the four element positions of a vector, representing columns in a 4-SIMD organization. In practice, this leads to a performance speed-up of up to 2.47 compared to an unrolled Montgomery multiplication implementation as in the MPM library for moduli of sizes 160 to 2048 bits which are of particular interest for publickey cryptography. Although presented for the Cell architecture, the proposed techniques can also be applied to other SIMD architectures.

Montgomery Multiplication on the Cell

485

References 1. Rivest, R.L., Shamir, A., Adleman, L.: A method for obtaining digital signatures and public-key cryptosystems. Communications of the ACM 21, 120–126 (1978) 2. Koblitz, N.: Elliptic curve cryptosystems. Mathematics of Computation 48, 203– 209 (1987) 3. Miller, V.S.: Use of elliptic curves in cryptography. In: Williams, H.C. (ed.) CRYPTO 1985. LNCS, vol. 218, pp. 417–426. Springer, Heidelberg (1986) 4. Montgomery, P.L.: Modular multiplication without trial division. Mathematics of Computation 44(170), 519–521 (1985) 5. Costigan, N., Scott, M.: Accelerating SSL using the vector processors in IBM’s Cell broadband engine for Sony’s playstation 3. Cryptology ePrint Archive, Report 2007/061 (2007), http://eprint.iacr.org/ 6. Bos, J.W., Casati, N., Osvik, D.A.: Multi-stream hashing on the PlayStation 3. In: PARA 2008 (2008) (to appear) 7. Bos, J.W., Osvik, D.A., Stefan, D.: Fast implementations of AES on various platforms. Cryptology ePrint Archive, Report 2009/501 (2009), http://eprint.iacr.org/ 8. Costigan, N., Schwabe, P.: Fast elliptic-curve cryptography on the Cell broadband engine. In: Preneel, B. (ed.) AFRICACRYPT 2009. LNCS, vol. 5580, pp. 368–385. Springer, Heidelberg (2009) 9. Bos, J.W., Kaihara, M.E., Montgomery, P.L.: Pollard rho on the PlayStation 3. In: SHARCS 2009, pp. 35–50 (2009) 10. Stevens, M., Sotirov, A., Appelbaum, J., Lenstra, A., Molnar, D., Osvik, D.A., de Weger, B.: Short chosen-prefix collisions for MD5 and the creation of a rogue CA certificate. In: Halevi, S. (ed.) Advances in Cryptology - CRYPTO 2009. LNCS, vol. 5677, pp. 55–69. Springer, Heidelberg (2009) 11. IBM: Multi-precision math library. Example Library API Reference, https://www.ibm.com/developerworks/power/cell/documents.html 12. Hofstee, H.P.: Power efficient processor architecture and the Cell processor. In: HPCA 2005. IEEE Computer Society, Los Alamitos (2005) 13. Walter, C.D.: Montgomery exponentiation needs no final subtractions. Electronics Letters 35(21), 1831–1832 (1999) 14. Kocher, P.C.: Timing attacks on implementations of Diffie-Hellman, RSA, DSS, and other systems. In: Koblitz, N. (ed.) CRYPTO 1996. LNCS, vol. 1109, pp. 104–113. Springer, Heidelberg (1996) 15. IBM: Software Development Kit (SDK) 3.1 (2007), http://www.ibm.com/developerworks/power/cell/documents.html

An Exploration of CUDA and CBEA for Einstein@Home Jens Breitbart1 and Gaurav Khanna2 1

2

Research Group Programming Languages / Methodologies Universit¨ at Kassel Kassel, Germany [email protected] Physics Department, University of Massachusetts at Dartmouth North Dartmouth, MA, USA [email protected]

Abstract. We present a detailed approach for making use of two new computer hardware architectures–CBEA and CUDA–for accelerating a scientific data-analysis application (Einstein@Home). Our results suggest that both the architectures suit the application quite well and the achievable performance in the same software developmental time-frame is nearly identical.

1

Introduction

The performance of computing technologies like graphics cards or gaming consoles is increasing at a rapid rate, thus making general-purpose computing on such devices a tantalizing possibility. Both Compute Unified Device Architecture (CUDA) and Cell Broadband Engine Architecture (CBEA) are new hardware architectures designed to provide high performance and scaling for multiple hardware generations. CUDA is NVIDIA’s general-purpose software development system for graphics processing units (GPUs) and offers the programmability of their GPUs in the ubiquitous C programming language. The Cell Broadband Engine (CBE), which is the first incarnation of the CBEA, was designed by a collaboration between Sony, Toshiba and IBM (so-called STI). The CBE was originally intended to be used in gaming consoles (namely Sony’s Playstation 3), but the CBEA itself was not solely designed for this purpose and has been used in areas such as high-performance computing as well. In this article, we will compare the current state of both CUDA and the CBEA by modifying the Einstein@Home client application, to enable it to take advantage of these different hardware architectures. Einstein@Home is a distributed computing project that uses the computing power volunteered by end users running its client application. The computation performed by this client application can be executed in a data-parallel fashion, which suits both CUDA and the CBE well. We took approximately the same time for developing the client on both architectures and achieved similar performance, but faced very R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 486–495, 2010. c Springer-Verlag Berlin Heidelberg 2010 

An Exploration of CUDA and CBEA for Einstein@Home

487

different challenges. Developing the CUDA client required us to redesign the existing data structure, which we could use for the CBE client as well. However, on the CBEA we could not use all hardware features due to the low amount of on-chip memory and thus likely lost some performance. This article is organized as follows. First, Sect. 2 gives an overview of the Einstein@Home client application. The next two sections, describe the CBEA architecture (Sect. 4) and our newly developed CBEA application (Sect. 5). The following two sections give an overview of CUDA (Sect. 6) and our experiences with that architecture (Sect. 7). Finally, in Sect. 8 we compare both implementations and outline the benefits and drawbacks of CUDA and the CBE. Section 9 describes related work, whereas Sect. 10 summarizes this work.

2

Einstein@Home

Einstein@Home is a BOINC [1] based, public distributed computing project that offloads the data-analysis associated to these observatories to volunteers worldwide. The goal of Einstein@Home is to search for gravitational waves emitted by neutron stars (pulsars), by running a brute force search algorithm for different waveforms in a large data-set. We consider the Einstein@Home client a meaningful test application for our comparison of CUDA and the CBEA since it is very compute intensive, and its parallelization is quite straightforward. The computation of the application can be roughly divided into two parts – the so-called F-Statistics computation, and a Hough-transformation. We will concentrate on the F-Statistics computation in this work, because the Hough code will be replaced by an alternative, extremely efficient algorithm soon. We provide below a brief overview of F-Statistics and its data dependencies; a detailed discussion can be found in [2]. Listing 1.1 provides an overview of the F-Statistics code. The code uses for each whenever the loop can be carried out in parallel with no or minimal changes to the loop body. + is used for all commutative calculations. The parameters of a function denote all the values a function depends on, however we simply use “...” as a reference to all parameters passed to the calling function. The F-Statistics code consist of three nested loops, each looping through a different part of the input data set, namely: a frequency band, all used detectors and an SFT of each signal. Listing 1.1. F-Statistics pseudo code

T [ ] ComputeFStatFreqBand ( . . . ) { T result [ ] ; int i = 0; f o r ea ch ( f r e q u e n c y i n f r e q u e n c y b a n d ( . . . ) ) r e s u l t [ i ++]= ComputeFStat ( . . . , f r e q u e n c y ) ; return r e s u l t ; }

488

J. Breitbart and G. Khanna

T ComputeFStat ( . . . ) { T result ; f o r ea ch ( d e t e c t o r i n d e t e c t o r s ) r e s u l t += ComputeFaFb ( . . . , d e t e c t o r ) ; return r e s u l t ; } T ComputeFaFb ( . . . ) { T result ; f o r ea ch (SFT i n SFTs ( f r e q u e n c y ) ) r e s u l t += s o m e c a l c u l a t i o n s ( . . . , SFT ) ; return normalized ( r e s u l t ) ; } In this article, we will indicate the current state of the calculation, as the presently computed upon SFT, detector and frequency. More formally speaking – the current state is the tuple of (f requency, detector, SF T ) using the variable names used in the listing 1.1.

3

Test Cases

The various measurements presented in this article were done with two different kinds of data-sets that not only differ in the overall runtime, but also in their memory requirements and performance characteristics. The small test case is based on a data-set used to check if the Einstein@Home client application is producing correct results. It uses a small number of frequencies in contrast to a full Einstein@Home work unit. In the small data set case, the F-Statistics takes nearly 90% of the overall runtime, while in the full work unit case its share is nearly 50%. A full work unit consists of multiple, so-called sky points. The calculation done for one sky point consists of both the F-Statistics and the Hough transformation and the runtime required to compute one sky point is nearly identical to the one required for the other sky points. For this work, we measure the time required to calculate one sky point for the full work unit.

4

Cell Broadband Engine

The CBE has a general purpose CPU core, called the PPE, which can run 2 software threads simultaneously, and 8 specialized SIMD compute engines, called SPEs available for numerical computation. All these compute elements are connected to each other through a high-speed interconnect bus (EIB). We will not attempt to go into much detail concerning the CBE’s design here, rather we will simply point out one feature that addresses the issue of the memory wall. The memory wall refers to the large (and increasing) gap between processor and memory performance, causing the slower memory speeds to become a significant bottleneck. The CBE allows the programmer to overlap

An Exploration of CUDA and CBEA for Einstein@Home

489

memory access and the actual computation (“double buffering”), in order to hide the time it takes to access memory, thus providing a workaround for the memory wall issue. In addition, the parallel programming model on the CBE allows for the use of the SPEs for performing different tasks in a workflow (“task parallel” model) or performing the same task on different data (“data parallel” model). We use the data parallel model in our implementations. One challenge introduced by this design, is that the programmer typically has to explicitly manage the memory transfer between the PPE and the SPEs. The PPE and SPEs are equipped with a DMA engine – a mechanism that enables data transfer to and from, main memory and each other. The PPE can access main memory directly, but the SPEs can only directly access their own, rather limited (256KB) local store. This poses a challenge for some applications, including the Einstein@Home client application. However, compilers (e. g. IBM XLC/C++) are now available that enable a software caching mechanism that allow for the use of the SPE local store as a conventional cache, thus negating the need of transferring data manually to and from main memory. Another important mechanism that allows communication between the the different elements (PPE, SPEs) of the CBE is the use of mailboxes. These are special purpose registers that can be used for uni-directional communication. They are typically used for synchronizing the computation across the SPEs and the PPE, and that is primarily how we made use of these registers as well. Details on our specific use of these various aspects of the CBEA for the Einstein@Home client application appear in the next section of this article.

5

Implementation on the Cell Broadband Engine

The F-Statistics calculation consists of multiple nested loops that can be carried out in parallel, as seen in Listing 1.1. The outer loop iterations are independent from one another and each iteration calculates only one result, which is written to a memory location that is different for every loop iteration. The results written in an outer loop iteration are a reduction of all inner loop iterations. We parallelized the F-Statistics calculation by splitting the outer loop into equal-sized parts that can be executed in parallel. We chose this approach, since it does not require any communication and still provides almost perfect work distribution, since the number of inner loop iterations varies only be a small margin. Each such part is assigned to an SPE, and each SPE calculates the results for the assigned part independently. The PPE is only used to start the SPEs and waits for them to complete the F-Statistics calculation before continuing with the Einstein@Home application. The code executed by the SPEs is identical to the original code, except for the modification described below. We developed two F-Statistics implementations for the CBE. In our first implementation, we manually manage transfers to and from the local store of the SPEs, whereas our second implementation relies on a software cache, which must be provided by a compiler. In this section, we first describe the two implementations and

490

J. Breitbart and G. Khanna

discuss the benefits of each later. We refer to the first implementation as DMAFstat, whereas the second implementation is called EA-Fstat. In DMA-Fstat the PPE creates multiple threads, each of which is used to control a single SPE. After the threads are created, the PPE inputs the data structure addresses used by F-Statistics into the mailboxes of each SPE. This communication is also used to notify the SPEs to begin work. After the SPEs have received the addresses, they use DMA transfers to fetch all data required for the complete computation. We cannot use double buffering for all data structures because the data that is needed for the calculation is computed on-the-fly for some of these. We could implement double buffering for some data structures, but we did not do so. It turns out that DMA-Fstat cannot be used for a full work unit, because the data needed cannot not fit into the SPE local store. Therefore, we did not optimize DMA-Fstat any further. Since we did not use double buffering, that diminishes the possible performance gain we could achieve with this implementation. After the data is processed, the SPEs write their results back to main memory by using DMA transfers and place a “work finished” message in the mailbox. The PPE waits until all SPEs have placed this message in their mailbox, before the Einstein@Home client is executed any further. DMAFstat produces correct results for the small test case. We developed EA-Fstat to no longer be limited by the amount of data that can be processed. EA-Fstat relies on a SPE software cache implementation provided by e.g. IBMs XLC/C++ compiler, which frees us from manually transferring data to the local store. We only needed to guarantee that in the SPE code all pointers, pointing to main memory are qualified with the __ea qualifier. Since the data structures used by the Einstein@Home client are deeply pointer based, this modification took some time, but by itself was not very challenging. The initial communication is done identically as in DMA-Fstat, meaning that the addresses of the data structures in main memory are sent to the SPE mailboxes. These addresses are assigned to __ea qualified pointers and then used as if they point to locations in the SPE’s local store. The synchronization of the SPEs and the PPE is done identically to that of DMA-Fstat. However before the SPE sends out the “work finished” message in the mailbox, it writes back all data stored in the software cache. The cache write back is done by manually calling a function. The benefit of EA-Fstat compared to DMA-Fstat is that the developer no longer has to worry about the size of the local store and can automatically benefit from a possible larger local store in future hardware generations. Furthermore, relying on the software cache reduces the complexity of the SPE program code: DMA-Fstat requires 122 lines of code (not counting comments) consisting of memory allocation and pointer arithmetic, whereas the EA-Fstat implementation only consists of 9 library function calls, that read the data structure addresses out of the mailboxes. All performance comparisons of our CBE clients were done by running the codes on a Sony Playstation 3, that allows the use of a maximum of 6 SPEs running at a clock frequency of 3.2 GHz. The best performance in the small

An Exploration of CUDA and CBEA for Einstein@Home

491

Table 1. Time per sky point for the CBE client for a full work unit case Processing elements SPEs 1 2 3 4 5 6 22 min 20 min 16 min 14.5 min 13.75 min 13.5 min 13 min PPE

test case is achieved by using 3 SPEs, since the amount of work in this test case is rather small. We gain a factor of 3.6 in performance over the PPEonly version. To finish the small test case, DMA-Fstat requires 2:30 minutes, whereas EA-Fstat needs 2:34 minutes. The comparison of both DMA-Fstat and EA-Fstat shows the performance hit of using the software cache mechanism is only about 2.5%. The low performance loss is partly due to the fact that we did not use double buffering for DMA-Fstat, which would have likely increased its performance, and that all the data required fits into the local store of an SPE. We show the performance of our EA-Fstat solution in Table 1. When running a full work unit the EA-Fstat client cannot outperform the PPE version as well as it did in the small data-set test case, since 50% of the runtime is used up by the hough transformation. When running the client with 6 SPEs, the client finishes in about 59% of the runtime of the PPE-only client – we gain a factor of 1.7 in overall application performance, by making use of the CBE architecture. The performance hardly seems to be limited by the size of the used software cache – halving the cache size to 64KB reduces the performance by 2%. When considering the F-Statistics computation alone, the performance improved by a factor of about 5.5 upon using 6 SPEs – F-Statistics requires less than two minutes of the overall runtime. The best overall performance of the Einstein@Home client on the CBE platform is probably achieved by running multiple clients on one system, so the PPE’s ability of running 2 software threads can be used and all SPEs are kept busy, even when one client is currently executing the Hough transformation. Our experimentation suggests that one can gain an additional 30% in overall application performance in this manner. Recall, that the Hough code is soon due to be replaced by an extremely efficient algorithm – when that happens, we expect to gain over a factor of 5 in the overall application performance.

6

CUDA Architecture

CUDA is a general-purpose programming system currently only available for NVIDIA GPUs and was first publicly released in late 2007. Through CUDA, the GPU (called device) is exposed to the CPU (called host ) as a co-processor with its own memory. The device executes a function (called kernel ) in the SPMD model, which means that a user-configured number of threads runs the same program on different data. From the host viewpoint, a kernel call is identical to an asynchronous function call. Threads executing a kernel must be organized within so called thread blocks, that may consist of up to 512 threads; multiple thread

492

J. Breitbart and G. Khanna

blocks are organized into a grid, that may consist of up to 217 thread blocks. Thread blocks are important for algorithm design, since only threads within a thread block can be synchronized. NVIDIA suggests having at least 64 threads in one thread block and up to multiple thousands of thread blocks to achieve high performance. Threads within thread blocks can be addressed with one-, two- or three-dimensional indexes; thread blocks within a grid can addressed with one- or two-dimensional indexes. We call the thread index threadIdx and the dimensions of the thread block x, y and z (therefore threadIdx.x is the first dimension of the thread index). We refer to the thread block index as blockIdx and use the same names for the dimensions. The high number of threads is used by the thread scheduler to hide memory access latency (see memory wall in Sect. 4) for accesses to the so-called, global memory. Global memory is located on the device and can be accessed by both host and device. In contrast to host memory, global memory is not cached so accesses to this memory cost an order-of-magnitude more than most other operations at the device. For example, global memory access can cost up to 600 clock cycles, whereas an addition or the synchronization of a thread block is performed in about 4 clock cycles. Another way to hide global memory access latency is by using so-called shared memory as a cache. Shared memory is fast on-chip memory that is shared by all threads of a thread block. However, using global memory cannot be avoided, because it is the only kind of memory, that can be accessed by both the host and the device. Data stored in main memory must be copied to global memory, if it is needed by the device. Results of a kernel that need to be used by the CPU must be stored in global memory and the CPU must copy it to main memory to use them. All transfers done to or from global memory are DMA transfers and have a high cost of initialization and a rather low cost for the actual data transfer itself.

7

Implementation Using CUDA

The development of the CUDA based F-Statistics was an evolutionary process. We developed three different versions, each solving the problems that emerged in the previous version. Version 1 executes the innermost loop on the device by using one thread per loop iteration i. e. we calculate the states (f requency, detector, threadIdx.x) with one kernel call. We can parallelize the loop this way because, there are no dependencies between the loop iterations, except a reduction for all the results of the loop iterations. The reduction is done by storing all results in shared memory and using one thread to sum up the results and store it back into global memory. Our implementation works, because the number of loop iterations of the inner loop is always smaller than the maximum number of threads allowed within a thread block. Now, we could easily implement a parallel reduction, e. g. NVIDIA provides a parallel reduction example [3] that could be used. However, these calculations are not the performance bottleneck for version 1.

An Exploration of CUDA and CBEA for Einstein@Home

493

Version 1 requires about 70 times the runtime of the original Einstein@Home client, whereas 95% of this time is used for memory management. This results from the fact that the data structures used by Einstein@Home consist of a high amount of small memory blocks and version 1 copies all these memory blocks to global memory one after another. Since all these memory transfers have a rather high cost of initialization, we cannot achieve a high performance with this solution. Our second implementation continues to calculate the innermost loop of the F-Statistics calculation in parallel, but uses a new data structure, which solves the performance problems when copying the data. Our new data structure is an aggregation of a small number of arrays. We group the arrays in two types. One type is called data array – these store the data from the original Einstein@Home data structures. The Einstein@Home data is copied into the data arrays one after another. The second array type is called offset arrays and are used to identify the original memory blocks inside the data arrays, by storing the starting point of the previously independent memory block inside the data array. By using this data structure, the performance drastically improved to about the same level as that of the CPU based application. Since version 2 still only calculates (f requency, detector, threadIdx.x) with one kernel call and thereby only uses one thread block, whereas the device is designed to run multiple thread blocks at once, most of the processing power of the device is unused. Version 3 uses multiple thread blocks to calculate all three loops of F-Statistics, in parallel. A single loop iteration of the outer loop is calculated by one thread block, whereas each iteration of the inner loops is calculated by one thread. More formally speaking, version 3 calculates (blockIdx.x, threadIdx.y, threadIdx.x) with one kernel call. We chose this approach, because the number of loop iterations of the inner loops is always less than the maximum number of threads allowed inside one thread block. This approach allows us to easily implement the reduction that is executed on all results of the innermost loops, since we can synchronize all threads within the thread block. We continue to use the reduction introduced in version 1, in this version. The number of threads within one thread block is identical for all thread blocks of a kernel call, however the number of loop iterations done by the inner loops depends on the index of the outer loop. Therefore, we have to identify the maximum number of the iterations of the inner loops at the host and use this maximum number for the kernel call. This approach results in idle threads for all thread blocks that have more threads than loop iterations that must be calculated. Our evaluation shows that there are only a small fraction of idle threads – typically, 6 threads per thread block, with the maximum being 10. This is our final implementation of the Einstein@Home CUDA application. The development of all three versions was done on a device, that did not support double-precision floating point operations, which are required by Einstein@Home to produce correct results. This client therefore, does not use double-precision and the results are of no scientific value.

494

J. Breitbart and G. Khanna Table 2. Time measurements of the CUDA based client test case CPU GPU small 2:19 min 1:22 min full (per sky point) 11:00 min 7:00 min

Upon running our implementation on a test system with a GeForce GTX 280 and two AMD Opteron 270 (2 GHz) the CUDA version performs about 1.7 times as fast as the CPU version for the small test case. When run on a full work unit, the CUDA client performs about 1.6 times as fast as the CPU version. Note that both the CPU and the GPU version only use one core when running the Einstein@Home client. Considering only F-Statistics, the performance was improved by more than a factor of about 3.5 – the F-Statistics calculation only takes about 1.5 minutes.

8

Comparison of CUDA and the CBE

One of the most important factors when developing for the CBEA is the size of local store, especially when the developer manually manages the data in the local store. The data required by our application does not fit in the local store. An easy way to overcome this is to use a software cache – however, by using this technique, the developer loses the chance to explicitly utilize double buffering, which is one of the most important benefits of the CBEA. Using CUDA required us to change the underlying data structure of our application. In our case the data structure redesign was rather easy, however in other cases this may be more problematic. The main benefit of using CUDA is the hardware, that increases performance at a much higher rate than multi-core CPUs or the CBEA. Furthermore, software written for CUDA can easily scale with new hardware. However, writing software that achieves close-to-the-maximum performance with CUDA is very different when compared to most other programming systems, since thread synchronization or calculations are typically not the time consuming parts, but memory accesses to global memory are. We did not optimize any of our implementations to the maximum possible performance, instead invested a similar amount of development time into both the CBE and the CUDA client. Therefore, our performance measurements should not be considered as what is possible with the hardware, but rather what performance can be achieved within a reasonable development time-frame.

9

Related Work

Scherl et al. [4] compare CUDA and the CBEA for the so called FDK method for which CUDA seems to be the better option. Christen et al. [5] explore the usage of CUDA and the CBEA for stencil-based computation, which however does not give a clear winner. In contrast, our work does not strive for the maximum possible performance and relies on very low cost hardware.

An Exploration of CUDA and CBEA for Einstein@Home

10

495

Conclusion

The main outcome of our work presented in this article, is that new upcoming architectures such as CBEA and CUDA have strong potential for significant performance gains in scientific computing. In our work, we focused on a specific, data-analysis application from the gravitational physics community, called Einstein@Home. Using these architectures, we successfully accelerated by several fold, one of the two computationally intensive routines of the Einstein@Home client application. Our final CBEA and CUDA implementations yield comparable performance and both architectures appear to be a good match for the Einstein@Home application.

Acknowledgment The authors would like to thank NVIDIA and Sony for providing the hardware that was used for development, testing and benchmarking our codes. Gaurav Khanna would also like to acknowledge support from the National Science Foundation (grant number: PHY-0831631).

References 1. Anderson, D.P.: Boinc: A system for public-resource computing and storage. In: GRID 2004: Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing, Washington, DC, USA, pp. 4–10. IEEE Computer Society, Los Alamitos (2004) 2. Breitbart, J.: Case studies on gpu usage and data structure design. Master’s thesis, University of Kassel (2008) 3. NVIDIA Corporation: CUDA parallel reduction (2007), http://developer.download.nvidia.com/compute/cuda/1 1/Website/ Data-Parallel Algorithms.html#reduction 4. Scherl, H., Keck, B., Kowarschik, M., Hornegger, J.: Fast GPU-Based CT Reconstruction using the Common Unified Device Architecture (CUDA). In: Frey, E.C. (ed.) Nuclear Science Symposium, Medical Imaging Conference 2007, NSS 2007. Nuclear Science Symposium Conference Record, vol. 6, pp. 4464–4466. IEEE, Los Alamitos (2007) 5. Christen, M., Schenk, O., Messmer, P., Neufeld, E., Burkhart, H.: Accelerating Stencil-Based Computations by Increased Temporal Locality on Modern Multi- and Many-Core Architectures (2008)

Introducing the Semi-stencil Algorithm Ra´ ul de la Cruz, Mauricio Araya-Polo, and Jos´e Mar´ıa Cela Barcelona Supercomputing Center, 29 Jordi Girona, 08034 Barcelona, Spain {raul.delacruz,mauricio.araya}@bsc.es

Abstract. Finite Difference (FD) is a widely used method to solve Partial Differential Equations (PDE). PDEs are the core of many simulations in different scientific fields, e.g. geophysics, astrophysics, etc. The typical FD solver performs stencil computations for the entire 3D domain, thus solving the differential operator. This computation consists on accumulating the contribution of the neighbor points along the cartesian axis. It is performance-bound by two main problems: the memory access pattern and the inefficient re-utilization of the data. We propose a novel algorithm, named ”semi-stencil”, that tackle those two problems. Our first target architecture for testing is Cell/B.E., where the implementation reaches 12.4 GFlops (49% peak performance) per SPE, while the classical stencil computation only reaches 34%. Further, we successfully apply this code optimization to an industrial-strength application (Reverse-Time Migration). These results show that semi-stencil is useful stencil computation optimization. Keywords: Stencil computation, Reverse-Time Migration, High Performance Computing, Cell/B.E., heterogeneous multi-core.

1

Introduction

Astrophysics [1], Geophysics [2], Quantum Chemistry [3] and Oceanography [4] are examples of scientific fields where large computer simulations are frequently deployed. These simulations have something in common, the mathematical models are represented by Partial Differential Equations (PDE), which are mainly solved by the Finite Difference (FD) method. Large simulations may consume days of supercomputer time, if they are PDE+FD based, most of this execution time is spent in stencil computation. For instance, Reverse-Time Migration (RTM) is a seismic imaging technique in geophysics, of the RTM kernel execution time up to 80% [5] is spent in the stencil computation. In this paragraph, we first review the stencil computation characteristics, then we identify its main problems. Basically, the stencil central point accumulates the contribution of neighbor points in every axis of the cartesian system. This operation is repeated for every point in the computational domain, thus solving the spatial differential operator, which is the most computational expensive segment of a PDE+FD solver. We identify two main problems: R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 496–506, 2010. c Springer-Verlag Berlin Heidelberg 2010 

Introducing the Semi-stencil Algorithm

497

– first, the non-contiguous memory access pattern. In order to compute the central point of the stencil, a set of neighbors have to be accessed, some of these neighbor points are far in the memory hierarchy, thus paying many cycles in latencies. Furthermore, depending on the stencil order (associated to the number of neighbors that contribute in the spatial differential operator) this problem becomes even more important, due to that the required data points are even more expensive to access. – Second, the low computation/access and re-utilization ratios. After gather the set of data points just one central point is computed, and only some of those accessed data points will be useful for the computation of the next central point. We introduce the semi-stencil algorithm, which improves the stencil computation tackling the above mentioned problems. We remark that this algorithm changes the structure of the stencil computation, but anyway it can be generally applied to most of the stencil based problems. The semi-stencil algorithm computes half the contributions required by a central point, but for two central points at the same time, with one a stencil computation is completed, and the other half central point pre-computes the stencil computation of another point. This algorithm reduces the number of neighboring points loaded per computation, this is achieved by just accessing the points required to compute half the stencil. This helps with the first problem of the stencil computation. At every step of the algorithm half of the stencil for the next step is pre-computed. The number of floating point operations remain the same, but because the load number is reduced the computation-access ratio is higher. So, tackling the second problem of the stencil computation. Currently, the GHz race for higher frequencies has slowed down due to technological issues. Therefore, the main source of performance comes from the exploitation of multi-core architectures. Among such architectures, the Cell/B.E. (our target achitecture) exhibit appealing features as energy efficiency and remarkable computing power, which is demanded by the applications which rely on stencil computations. The remainder of this paper is organized as follows: Section 2 introduces the classical stencil algorithm and its problems. In Section 3 we review the principal techniques that help to cope with the stencil problem. Section 4 introduces the novel semi-stencil algorithm, its internals, features and considerations. In Section 5, we evaluate the performance. Finally, Section 6 presents our conclusions.

2

The Stencil Problem

The stencil computes the spatial differential operator, which is required to solve a PDE+FD. A multidimensional grid (often a huge 3D data structure) is traversed, where the elements are updated with weighted contributions. Figure 1.b) depicts classical generic stencil structure. The parameter represents the number of neighbors to be used at each direction of the cartesian axis.

498

R. de la Cruz, M. Araya-Polo, and J.M. Cela

Two major problems can be inferred from this computation, the first one is the sparse memory access pattern [6,7]. Data is stored in Z-major form, therefore accesses across the other two axis (X and Y ) may be significantly more expensive (see Figure 1(a)) latency-wise. The parameter has a direct impact on this problem, the larger the value the more neighbors of each axis have to be t loaded to compute Xi,j,k . The points of the Z axis are stored sequentially in memory, and the cost to fetch them is low.

Fig. 1. (a) The memory access pattern for a 7-point stencil. The sparsity of the required data to compute the stencil is higher in the last axis. (b) The generic stencil structure.

The second problem has two faces: the low floating-point instruction to memory access ratio, and the poor reuse of the accessed data [8]. We state a simple metric to expose these two faces: FP/Mem. This ratio is the result of dividing the number of floating-points instructions (Multiply-Adds) per loads and stores during one X t computation step. M ultiplyAdd Instructions F loatingP oint Instructions = M emory Accesses X t−1 Loads + X t Stores 2 ∗ dim ∗ + 1 2 ∗ dim ∗ + 1 = = (2 ∗ (dim − 1) ∗ + 1) + (1) 2 ∗ dim ∗ − 2 ∗ + 2

F P/M emClassical =

(1)

Equation 1 states this metric for the classical stencil. dim is the number of dimensions of our PDE, where for each dimension 2 ∗ Multiply-Add instructions are required. Also, one extra Multiply-Add instruction must be considered for

Introducing the Semi-stencil Algorithm

499

t−1 self-contribution (Xi,j,k ). The number of loads needed to compute the stencil change depending on the axis. Those dimensions that are not stride direction (X and Y ) require 2 ∗ loads each step iteration. On the other hand, the stride direction requires only 1 load because the remaining loads can be reused from t ) is needed. previous iterations. Finally, 1 store for saving the result (Xi,j,k FP/Mem ratio depends on variables dim and , taking into account that is the only variable that may grow, the ratio tends to 1.5, which is very poor. This result along with previous research [9,10,8] shows that stencil computation is usually memory-bound. In other words, it is not possible to feed the computation with enough data to keep a high arithmetic throughput, thus the performance of this kind of algorithm is limited. These concerns force us to pay special attention on how data is accessed. It is crucial to improve the memory access pattern (reducing the overall amount of data transfers), and to exploit the memory hierarchy as much as possible (reducing the overall transfer latency). Next section review the main approaches to the problem found in the literature.

3

State of the Art

Most of the contributions in the stencil computations field can be divided into three dissimilar groups: space blocking, time blocking and pipeline optimizations. The first two are related with blocking strategies widely used in cache architectures, while the last one is used to improve the performance at the pipeline level.

Space Blocking Space blocking algorithms try to minimize the access pattern problem of stencil computations. Space blocking is specially useful when the data structure does not fit in the memory hierarchy. The most representative algorithms of this kind are: tiling or blocking [9] and circular queue [11].

Time Blocking Time blocking algorithms perform loop unrolling over the time steps to exploit as much as possible a data already read, therefore increasing the data reuse. Time blocking may be used where there is no other computation between stencil sweeps such as: boundary condition, communication, I/O, where a time dependency exists. Optimizations of the time blocking can be divided in explicitly and implicitly algorithms: time-skewing [10] and cache-oblivious [8].

Pipeline Optimizations Low level optimizations include well-known techniques suitable to loops, like unrolling, fission, fusion, prefetch and software pipelining [12,13]. All those techniques have been successfully used for a long time in many other computational fields to improve the processor throughput and decrease the CPI (Cycle per Instruction).

500

4

R. de la Cruz, M. Araya-Polo, and J.M. Cela

The Semi-stencil Algorithm

The semi-stencil algorithm changes in noticeable way the structure of the stencil computation as well as the memory access pattern of the stencil computation. This new computation structure (depicted in Figure 2) consist in two phases: forward and backward, which are described in detail in Section 4.1. Besides, head and tail computations are described in last part of the chapter. The semistencil tackles the problems described in Section 2 in the following ways: – It improves data locality since less data is required on each axis per iteration. This may have an important benefit for hierarchy cache architectures, where the non-contiguous axes of the 3D domain are more expensive latency-wise. – The second effect of this new memory access pattern is the reduction of the total amount of loads per inner loop iteration, but keeping the same number of floating-point operations. Thereby it increases the data re-utilization and the F P/M em ratio. We calculate this ratio for the semi-stencil as follows: F P/M emSemi =

2 ∗ dim ∗ + 1 M ultiplyAdd Instructions (2) = X t−1 Loads + X t Stores dim ∗ − + dim + 2

In Equation 2, and regarding the data reuse in the stride dimension, the number of loads for X t−1 have decreased substantially, almost by a factor of 2. Due to the reduced number of loads less cycles are needed to compute internal loop, also it has the benefit of using fewer registers per iteration. This gives a chance to perform more aggressive low level optimizations, and avoid instruction pipeline stalls. As shown in Figure 2.Left, the semi-stencil algorithm updates two points per step by reusing X t−1 loads, but at the same time doubles the number of stores needed per step of X t . Depending on the architecture, this could be a problem and a source of performance loss. For instance, hierarchy cache architectures with write allocate policy. Nowadays, some hierarchy cache architectures implement cache-bypass techniques as a workaround for this problem. On the other hand, the scratchpad memory based (SPM) architectures, like Cell/B.E., are immune to this problem. Therefore, semi-stencil mapping on the Cell/B.E. architecture should not be affected by this issue. It is worth pointing out that the semi-stencil algorithm can be applied to any axis of a 3D stencil computation. This means that it can be combined with the classical stencil algorithm. Moreover, this novel algorithm is independent of any other optimization technique, like blocking or pipeline optimizations. In fact, we can stack semi-stencil algorithm over any other known technique. In the following sections we will elaborate on the semi-stencil two phases. 4.1

Forward and Backward Update

Forward update is the first contribution that a point receives at time-step t. In this phase, when step i of the sweep direction axis is being computed, the

Introducing the Semi-stencil Algorithm

501

Fig. 2. Left: Detail of the two phases for the semi-stencil algorithm at step i. a) Forward t t−1 point and b) Backward update on Xit point. Notice that the Xi+1 to update on Xi+ t−1 Xi+ −1 loads can be reused for both phases. Right: Execution example of semi-stencil algorithm in a 1D problem, where = 4. F stands for a forward update and B stands for a backward update. t t point Xi+ is updated (producing Xi+ ) with the t − 1 rear contributions (Figure 2.Left.a and Equation 3 depicts this operation). In the following equations, prime character denotes the point that has been partially computed. t−1 t−1 t−1 t Xi+ = C1 ∗ Xi+−1 + C2 ∗ Xi+−2 + · · · + C−1 ∗ Xi+1 + C ∗ Xit−1 t−1 t−1 + C2 ∗ Xi+2 Xit = Xit + C0 ∗ Xit−1 + C1 ∗ Xi+1

+

···

t−1 t−1 + C−1 ∗ Xi+−1 + C ∗ Xi+

(3)

(4)

In the backward update, the X  ti point, computed in a previous forward step, t−1 t−1 to Xi+ ) is completed with the front contributions of the axis (points Xi+1 (Figure 2.Left.b and Equation 4). Remark, only 1 load is required since almost all data points of time-step t − 1 were loaded in the forward update. Therefore, t−1 point, 1 store for Xit value and finally + 1 this phase needs 1 load for Xi+ Multiply-Add instructions ( neighbors + self-contribution). 4.2

Head, Body and Tail Computations

The FD methods require interior points (inside the solution domain) and ghost points (outside the solution domain). To obtain the correct results on border interior points, the algorithm must be split into three different parts: head, body

502

R. de la Cruz, M. Araya-Polo, and J.M. Cela

and tail. The head segment updates the first interior points with the rear contributions (forward phase). In the body segment, the interior points are updated with neighbor interior elements (forward and backward phases). Finally, in the tail segment, the last interior points of the axis are updated with the front contributions (backward phase). Figure 2.Right shows an example execution of the algorithm with the three segments.

5

Performance Evaluation and Analysis

We evaluate the performance of the semi-stencil algorithm in two steps: as single SPE implementation, and then integrating the implementation into a numerical kernel of a real-life scientific application. The purpose of the first implementation is to show the local behavior of the algorithm, the second implementation purpose is twofold: measure the expected positive impact of the algorithm on a real application, and expose the impact of the architecture on the algorithm performance. After the evaluation, we analyze the results taking the formulas from Sections 2 and 4 into account. As we have seen in Section 3, it exists other techniques to deal with stencil computations. Time blocking methods, due to their inherent time dependency, may pose integration problems (boundary conditions, communication, I/O) with scientific numerical kernels. Also, space blocking is not useful on SPM architecture like Cell/B.E., this due to the constant access cost to memory. Before to present the results, we briefly review the Cell/B.E. architecture. Each Cell/B.E. processor on a QS22 blade contains a general-purpose 64-bit PowerPC-type PPE (3.2 GHz) with 8 GB of RAM. The SPEs have a 128-bit wide SIMD instruction set, which allows to process simultaneously 4 single-precision floating-point operands. Also, the SPEs have software-based SPM called Local Stores (LS) of 256 KB. The 8 SPEs are connected with the PPE by the Element Interconnect Bus of 200 GB/s of peak bandwidth, but the bandwidth to main memory is only 25.6 GB/s.

Single-SPE Implementation In the first porting of our semi-stencil algorithm to Cell/B.E. we focus only on the SPE’s code, the PPE’s code and the techniques required to make the main memory transfers efficient are taking into account in our second semi-stencil implementation. Our implementation is fully handcoded SIMDized (as well as the classical stencil code). The classical stencil implementation takes advantage of optimization techniques such as: software prefetching, software pipelining and loop unrolling. The following results are expressed in terms of elapsed time and floating-point operations. Notice that the results were computed in single precision arithmetic, and the stencil size ( ) is 4. Our purpose with the results is to compare both approaches implementation. Also, the experiments look for situate the proposed algorithm with respect the peak performance of the Cell/B.E. architecture.

Introducing the Semi-stencil Algorithm

503

Table 1. Performance results for the single SPE implementations. These results were obtained with IBM XL C/C++ for Multi-core Acceleration for Linux, V9.0 and GCC 4.1.1. No auto-vectorization flags were used, all codes were vectorized by hand. Stencil size = 4. Compiler (Optimization) XLC -O3 XLC -O5 GCC -O3

Classical Stencil Time Performance [ms] [GFlops] 6.84 4.32 3.44 8.61 7.83 3.78

Semi-stencil Time Performance [ms] [GFlops] 3.35 8.83 2.38 12.44 4.57 6.47

The peak performance achieved by the semi-stencil is 12.44 GFlops (Table 1) which corresponds to 49% of the SPE peak performance. Under the same experimental setup the classical stencil reaches 8.61 GFlops (34% of the SPE peak performance). This means that the semi-stencil algorithm is 44% faster than the classical stencil. The projected aggregated performance of this algorithm is 99.52 GFlops for one Cell/B.E., which is to the best of our knowledge [11], the fastest stencil computation on this architecture. Table 2. Pipeline statistics for the single SPE implementations. Obtained with the IBM Full-System Simulator. Classical Stencil Semi-stencil Semi-stencil gain Total cycle count [cycles] 11460147 7592643 33.7% CPI [cycles/instructions] 0.69 0.75 -9% Load/Store instr. [instr.] 4236607 2618079 38.2% Floating point instr. 5800000 4652400 19.8% Ratio FP/Mem 1.36 1.78 24.4%

In Table 2 every metric is in semi-stencil favor, except the CPI measure, which is 9% better for the classical stencil algorithm. In any case, the most important metric and the one that summarize the performance is the FP/Mem ratio. This ratio is 24% better for the semi-stencil than the classical stencil computation.

Real-life Scientific Application Implementation We integrate the semi-stencil implementation with a Reverse-Time Migration (RTM) implementation [5]. RTM is the tool of choice when complex subsurface areas are intended to be clearly defined. RTM has proven to be very useful for the subsaly oil discoveries of the Gulf of Mexico. The core of the RTM is an acoustic wave propagation (PDE+FD) solver, which suits as test case for our proposed algorithm. The RTM+semi-stencil is 16% faster than RTM+classical, which partially (44% in the ideal case, see Table 1) keeps the performance distance presented

504

R. de la Cruz, M. Araya-Polo, and J.M. Cela

Table 3. Performance results for the RTM implementations. These results were obtained with GCC 4.1.1, using only one Cell/B.E. of a QS22 blade. Each experiment carried 500 steps for a 512x512x512 data set. Total time experiments cover both computation and communication times. Stencil size = 4. Total Only Computation Only Communication Unbalance Time Performance Time Performance Time % [s] [GFlops] [s] [GFlops] [s] Algorithm RTM+classical 74.24 39.15 71.92 41.06 70.33 2.3 RTM+semi-stencil 63.33 46.62 48.13 61.35 64.62 25.6

in the previous tests. This is because RTM and the data transfers introduce extra complexity to the implementation, RTM has several segments that demand computational resources, e.g. boundary conditions, I/O, etc. As can be seen in Table 3, the unbalance between communication (data transfers from/to main memory to/from LS) thwart the performance. This is especially hard with the RTM+semi-stencil, this implementation lost performance up to 25.6%, even after using multi-buffering technique. If we add this 25.6% to the already gained 16%, we recover the 44% of advantage of the semi-stencil over the classical stencil algorithm.

Analysis In this section, given that the experimental results show a clear lead for the semi-stencil algorithm, we confront these results with the theoretical estimation based on the FP/Mem ratio formulas of Section 2 and 4.

Fig. 3. Projected FP/Mem ratio for different stencil size. Notice that the stencils size in the central segment are the most commonly used.

From Figure 3, the theoretical FP/Mem ratio for the classical stencil algorithm is 1.39, while in Table 2 is 1.36. In the semi-stencil case, the theoretical FP/Mem

Introducing the Semi-stencil Algorithm

505

ratio is 1.92 and in Table 2 is 1.78. These figures show that our model of FP/Mem is robust and reliable. Also, both implementation have room for improvements, where for the classical stencil it may only represents a 2.1%. Implementation of the semi-stencil may improve up to 7.3%, but due to its logic complexity this is hard to achieve.

6

Conclusions and Future Work

In this paper we have presented a new generic strategy for finite difference stencil computations. This new algorithm approach, called semi-stencil, is especially well suited for scientific applications on heterogeneous architectures, like Cell/B.E., where a low latency SPM exists. Semi-stencil has shown two benefits compared to the classical stencil algorithm: – It reduces the minimum data working set, reducing the required space in the SPM for each processing unit. This effect becomes more critical for high order stencils. – It increases data locality, by reducing the number of loads but increasing the number of stores. In a heterogeneous system, these additional stores are executed in the processing unit’s SPM, removing the negative impact on the global performance. For a PDE+FD scheme, the best implementations of the classical stencil computation typically are able to reach to 30% of the processor peak performance. Under the same conditions, the semi-stencil algorithm achieve up to 49% of the peak performance. Also, this improvement revamps the performance of already developed code, as our RTM application. Future work will focus on research of this novel algorithm on cache-based architectures, where the write policy for the cache hierarchy may have a negative impact on the performance due to the increased number of stores.

References 1. Brandenburg, A.: Computational aspects of astrophysical MHD and turbulence, vol. 9. CRC, Boca Raton (April 2003) 2. Operto, S., Virieux, J., Amestoy, P., Giraud, L., L’Excellent, J.Y.: 3D frequencydomain finite-difference modeling of acoustic wave propagation using a massively parallel direct solver: a feasibility study. In: SEG Technical Program Expanded Abstracts, pp. 2265–2269 (2006) 3. Alonso, J.L., Andrade, X., Echenique, P., Falceto, F., Prada-Gracia, D., Rubio, A.: Efficient formalism for large-scale ab initio molecular dynamics based on timedependent density functional theory. Physical Review Letters 101 (August 2008) 4. Groot-Hedlin, C.D.: A finite difference solution to the helmholtz equation in a radially symmetric waveguide: Application to near-source scattering in ocean acoustics. Journal of Computational Acoustics 16, 447–464 (2008)

506

R. de la Cruz, M. Araya-Polo, and J.M. Cela

5. Araya-Polo, M., Rubio, F., Hanzich, M., de la Cruz, R., Cela, J.M., Scarpazza, D.P.: 3D seismic imaging through reverse-time migration on homogeneous and heterogeneous multi-core processors. Scientific Programming: Special Issue on the Cell Processor 16 (December 2008) 6. Kamil, S., Husbands, P., Oliker, L., Shalf, J., Yelick, K.: Impact of modern memory subsystems on cache optimizations for stencil computations. In: MSP 2005: Proceedings of the 2005 Workshop on Memory System Performance, pp. 36–43. ACM Press, New York (2005) 7. Kamil, S., Datta, K., Williams, S., Oliker, L., Shalf, J., Yelick, K.: Implicit and explicit optimizations for stencil computations. In: MSPC 2006: Proceedings of the 2006 Workshop on Memory System Performance and Correctness, pp. 51–60. ACM, New York (2006) 8. Frigo, M., Strumpen, V.: Cache oblivious stencil computations. In: 19th ACM International Conference on Supercomputing, pp. 361–366 (June 2005) 9. Rivera, G., Tseng, C.W.: Tiling optimizations for 3D scientific computations. In: Proc. ACM/IEEE Supercomputing Conference (SC 2000), p. 32 (November 2000) 10. Wonnacott, D.: Time skewing for parallel computers. In: Carter, L., Ferrante, J. (eds.) LCPC 1999. LNCS, vol. 1863, pp. 477–480. Springer, Heidelberg (2000) 11. Datta, K., Murphy, M., Volkov, V., Williams, S., Carter, J., Oliker, L., Patterson, D., Shalf, J., Yelick, K.: Stencil computation optimization and auto-tuning on stateof-the-art multicore architectures. In: SC 2008: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing, Piscataway, NJ, USA, pp. 1–12. IEEE Press, Los Alamitos (2008) 12. Allan, V.H., Jones, R.B., Lee, R.M., Allan, S.J.: Software pipelining. ACM Comput. Surv. 27(3), 367–432 (1995) 13. Callahan, D., Kennedy, K., Porterfield, A.: Software prefetching. In: ASPLOSIV: Proceedings of the Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 40–52. ACM, New York (1991)

Astronomical Period Searching on the Cell Broadband Engine Maciej Cytowski1 , Maciej Remiszewski2 , and Igor Soszy´ nski3 1

Interdisciplinary Centre for Mathematical and Computational Modelling, University of Warsaw 2 IBM Deep Computing, Central and Eastern Europe 3 Department of Observational Astrophysics, Faculty of Physics, University of Warsaw

Abstract. Period searching is widely used in astronomy for analyzing observatory data. Computer programs used for these tasks can be opitmized to work on novel computer architectures, such as Cell Broadband Engine. In this article we present performance results of one of these programs on the Cell processor. Important programming steps and performance comparisons are shown and discussed. Keywords: Period searching, Cell architecture, astrophysics.

1

Introduction

The Cell Broadband Engine is a novel and pioneering processor architecture. It allows for a very interesting study of inside chip parallelization over short vector SIMD cores. There are many codes which performance can benefit from Cell BE specific architecture. In this paper we describe the Cell BE implementation of an astronomical period-searching program used in operational computations in the Optical Gravitational Lensing Experiment [1]. The scientific results obtained with the use of Cell-accelerated computer program where presented in [2], [3] and [4] . The original code was ported using the Cell SDK and its SPE library in a relatively short time. We achieved an overall speedup of 9.68 over current workstations.In the course of our work we encountered many important aspects of Cell programming and learned how to identify codes whose performance could benefit from this architecture. We are currently working on porting two other astrophysical applications to Cell. Presented work can serve as a fundament for creating a general framework for period searching on the Cell processor. The complexity of such tasks is often induced by the vast amount of data to be processed. Parallelization can therefore easily be achieved by partitioning the data across available compute cores. Astronomical period searching is a good example here. Computations could be realized independently on a number of cores independently for each individual star under consideration. R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 507–516, 2010. c Springer-Verlag Berlin Heidelberg 2010 

508

M. Cytowski, M. Remiszewski, and I. Soszy´ nski

Moreover the possibility of vector processing could easily be exploited for these computations. Time comparisons presented here show that vector implementations on short vector SIMD cores in current processor architectures are approximately 2.4 times faster than their scalar counterparts.

2

Optical Gravitational Lensing Experiment

The Optical Gravitational Lensing Experiment (OGLE) project is a continuos effort devoted mainly to the search for dark matter using microlensing phenomena as originally proposed by Paczy´ nski [5] [6]. As described in great detail in [1] long term observations of the Magellanic Clouds (MC) and the Galactic Bulge (GB) are conducted from the observatory in Las Campanas, Chile. Photometric data is collected via a CCD camera over extensive periods to allow for both detection and collection of statistically significant samples of microlensing events. The result is a vast amount of experimental data, reaching 4TB per year with OGLE-III. An annual total of 10 Billion photometric measurements record light curves for 200 Million individual stars from the Large and Small Magellanic Clouds (LMC and SMC respectively) as well as GB. In further data processing each light curve is analysed for distinctive patterns which can indicate a microlensing event taking place. Additional improvement to the search method used by OGLE came from incorporating the Transit Method, as proposed in [7]. Since 1992, throughout the three phases of the OGLE project (OGLE, OGLE-II and OGLE-III) 13 planets outside our solar system have been discovered, six of which were found using the microlensing and seven the transit method. While the search for planets and other forms of dark matter is the leading target of the OGLE project, the data collected in the process is very well suited for identification of different kinds of variable stars. The light curve analysis for periodicity is performed by the Fnpeaks code written by W.Hebisch, Z.Kolaczkowski and G.Kopacki and is a process performed for each star individually. The work described in this paper relates to this very procedure for the OGLE-III experimental data. Because of the huge amount of processing to be performed, the OGLE team requested the support of ICM in conducting the calculations at their facility. At the same time, due to an ongoing collaboration between ICM and IBM, it was decided to port the Fnpeaks code to the Cell Broadband Engine Architecture and use Cell BE based machines in the data processing as well. The scientific results were presented in separate publications in [2], [3] and [4]. The purpose of this paper is to describe how the Fnpeaks code has been ported and tuned on the Cell BE and how processing performance on this architecture compares to x86. It is worth noticing that while a large amount of data has already been processed, more input is still to follow, enabling the Cell Broadband Engine Architecture to continue to accelerate processing of experimental data for OGLE.

Astronomical Period Searching on the Cell Broadband Engine

3

509

Porting Process

The parallelism on the Cell BE processor can be exploited at multiple levels. Each Cell BE chip has eight SPEs with two-way instruction-level parallelism. Moreover each SPE supports single instruction multiple data (SIMD) operations [10]. The Power Processing Unit of Cell BE processor is usually used as a management thread. It can read and preprocess data, create SPE contexts and then run an appropriate SPE program. The SPE program can copy data from main memory to the Local Store through the Element Interconnect Bus. After computation it can store results back into Main Memory. Porting of an arbitrary code to the SPE may be challenging since the local memory space (Local Store) is only 256 KB. This rather small space was designed to accommodate both instructions and data. The Local Store can be regarded as a software-controlled cache that is filled and emptied by DMA transfers. The programmer has to decide how to partition the data and how to organize those transfers during execution. We have ported the Fnpeaks period-searching code (Z.Kolaczkowski, 2003) used in the OGLE project [9] to the Cell BE architecture. Fnpeaks implements the Discrete Fourier Transform algorithm. It is executed in an iterative process. After finding the highest peak in the power spectrum, a third order Fourier series is fitted to the folded light curve (Harmonphfit program) and the corresponding function is subtracted from the data. Afterwards the Fnpeaks program is once again executed on the modified data set. The entire process is repeated twice. The single threaded Fnpeaks program was already vectorized and achieved good performance when using SSE instructions on the AMD Opteron processor. Hence the porting process was first focused on compiling the code to work on a single SPE and using the appropriate vector instructions of the SPE API. To achieve better performance we have also analyzed performance with available Cell tools like Oprofile, Visual Performance Analyzer and spu timing. The parallel scheme based on pthreads library was designed and implemented subsequently. The PPE works as a management and working thread as we make use of its 2-way multithreaded core architecture. This chapter describes in detail all consecutive steps. The Fnpeaks code is not memory consuming since the observation data for one star (single computational element) consisted of a maximum of 6000 single precision floating point numbers. Thanks to this property we didn’t need to empty or refill the Local Store with incoming partitioned data sets. One SPE thread (context) performs a DMA load command at the begining and stores results back into main memory at the end. 3.1

SIMD Optimization

The Fnpeaks program was originally written in a vector fashion using Streaming SIMD Extensions (SSE). It achieved good performance when compared to a simple scalar version. This could automatically be adopted to fit the Cell BE vector processing capabilities. The notation used for implementing the SSE version

510

M. Cytowski, M. Remiszewski, and I. Soszy´ nski

had to be simply rewritten into the Cell BE SDK language. The main vector instructions used in the computational SPU kernel were: spu mul, spu madd and spu msub. All of these instructions were operating on float vectors of size 4. The computational SPU kernel was optimized with the use of spu timing tool to achieve best possible performance with the use of dual-pipe vector processing. Below we present an excerpt of the computational SPU kernel of the application.

void do_row( .. ) { .. for(i=0;i  do ˆ = argmax |p| 3 k k

4 5 6 7 8 9 10 11 12 13 14 15

ˆ Ii = Ii−1 ∪ k if i > 1 then w = Solve: Li−1 w = GIi−1 ,kˆ ( ' Li−1 √ 0 where Li = wT 1 − wT w end if aIi = Solve: Li (Li )T aIi = p0Ii β = GIi aIi p = p0 − β δi = aTIi βIi i = i−1 − δi + δi−1 i= i+1 end while

Due to orthogonalization, the matrix (DTI DI ) is symmetric positive definite, which allows the Cholesky decomposition. In each iteration the triangular matrix L is enlarged by another row. The non-zero element coefficient vector aIi is computed in line 9 by means of a forward- and backward substitution. In line 11 we update the projection p = DT r = p0 − GI (DI )+ x .

(3)

When an error-constrained sparse approximation problem is to be solved, the residual is required to check the termination criterion. The 2 norm of the residual i is computed in line 13. Image denoising: Image denoising tries to remove noise from a given image X and recover the original noise-free image X0 that is corrupted by additive, zeromean, white and homogeneous Gaussian noise g with standard deviation σ [12] X = X0 + g .

(4)

Sparse representations can be used for image denoising, if a dictionary containing noise-free image components is given. The image X is decomposed into small overlapping patches x. After having computed the noise-free sparse representation of each patch by means of Batch-OMP, the denoised image is given by a linear combination of the noisy image and an average of denoised patches. A suitable dictionary, that we use throughout our paper, is a frame derived from cosine functions. An alternative would be dictionary training with the KSVD algorithm [13].

560

3

D. Bartuschat, M. St¨ urmer, and H. K¨ ostler

Implementation on the Cell Processor

The Cell Broadband Engine consists of a general-purpose PowerPC Processor Element (PPE) for control-intensive tasks and several accelerator cores called Synergistic Processor Elements (SPEs), short-vector RISC processors specialized for data-rich and compute-intensive SIMD applications. The SPEs cannot directly work on main memory, but contain a small, private memory instead of a cache. This so-called Local Storage (LS) holds both instructions and data, which must be copied from and to main memory by means of asynchronous DMA transfers. This design explicitly parallelizes computation and transfer of data and instructions [14]. In order to achieve maximum performance, the programmer has to exploit this kind of parallelism, together with data-level parallelism through SIMD vector processing and thread-level parallelism [15]. For denoising, we decompose the image into fully overlapping patches of 8 × 8 pixels. Due to their superior performance, all work is done by the SPEs, which must extract and copy the patches from main memory to their LS, sparse-code, reconstruct, and finally combine them to the denoised image. Depending on the local image content, the effort for sparse-coding a patch varies. To address this possible imbalance, patches are distributed dynamically to the SPEs with a granularity of four whole lines of patches, requiring data from 11 subsequent image rows. As only an even smaller window of the image can be held in the LS at a time, the row needs to be traversed in several steps. On the other hand, this allows for optimal overlapping of data transfer and computation by means of multibuffering. Separate target buffers are provided for each SPE, because otherwise additional synchronization of the SPEs’ write accesses is required. The most time consuming part of the denoising is the Batch-OMP algorithm. Its data structures and kernels must be tuned to meet the requirements of the SPE as good as possible. Using SIMD is especially important, as this is the default case on an SPE and scalar operations are usually much more expensive. Further aspects are the in-order execution and the relatively high branchmiss penalty that both make loops with short bodies and few iterations very inefficient. Hence, the size of the atoms and of the dictionary were restricted to multiples of eight and fixed at compile time, as well as the maximum number of atoms. The static data structures and known loop ranges generally simplify SIMD vectorization, address calculations, and enable better unrolling of loops. Typically, for a sufficient representation only a small but varying number of atoms needs to be chosen. As they possess different complexity, the importance of the several kernels can change in this range a lot, and it is advisable to put at least some effort into optimizing all parts. The following types of kernels are required: Operations on dense vectors: These are computation of the dot product (line 1 and 12), subtraction of a vector from another (line 11), and determining the index of the element with the maximal absolute value (line 3). For all of them, loop unrolling and SIMD vectorization is applied, which is straight-forward for the subtraction. As the SPEs’ instruction set architecture

Orthogonal Matching Pursuit Algorithm for Image Denoising on the Cell BE

561

offers solely vertical SIMD operations for floating point values, only the four dot products of every fourth element can be computed efficiently. The final sum of the four partial results requires multiple instructions that can reorder SIMD vectors. A similar approach is used for the index search, where again four maximal value and the corresponding index are determined first. Only vector lengths a multiple of four, i.e. full SIMD vectors, need to be considered, as the vector lengths are either restricted to a multiple of four or padded by zeros. Gather operations: Creating vectors by gathering elements according to an index list is required in several steps of the algorithm (line 6, 9, and 12). As the SPEs only support natively aligned SIMD load and store operations, it is advisable to combine four elements of a target vector in a SIMD register before storing them to the LS. A trick is required to handle vectors with a size not a multiple of four: All source vectors are padded with an additional zero, and the index arrays are initialized with that offset to their maximal length beforehand. When rounding the number of indices up to a multiple of four in the gather kernels, no special handling of other sizes is required and target vectors are automatically zeropadded to whole SIMD vectors. To extract values from the symmetric Gram matrix, an additional row of all zeros is added, and addressing is done rowinstead of column-wise. Matrix-vector-multiplications: Multiplication of a dense matrix with a dense vector during initialization (line 1) and with a sparse vector for the projection of the Gram matrix onto the residual (line 10) need to be performed. In addition to loop unrolling and SIMD vectorization, register blocking is required to reduce the number of load and store operations. The purely dense multiplication is comparable to the SPE function of IBM’s Cell-enhanced BLAS library, but has fewer restrictions on the matrix and vector size. For multiplication with the sparse vector, one to four lines of the matrix are scaled and added at once. Cholesky substitution: The lower triangular matrix and its upper triangular transpose both are stored in column-major order, as this is required to enable some SIMD processing in the forward and backward substitutions (line 5 and 9). The reciprocal of the diagonal is stored separately, for each element a whole SIMD vector is used with the i − th coefficient in the (i mod 4) − th slot and all other slots being zero. Although this makes it more complex to append a row or colum to the triangular matrices (line 7), the possibility to use SIMD operations throughout the whole substitutions more than compensates. A corresponding formulation of the forward substitution Ly = b can be seen in algorithm 2. Note that for the given storage scheme all operations, especially in lines 3 and 5, can be done in SIMD – the ranges of the inner loop at 4 need be rounded to multiples of four in this case. The backward substitution is done analogously. To speed up the computation further, separate kernels for a size of 1, 2 to 4, 5 to 8, 9 to 12, 13 to 16, and 17 and 20 have been created, which allows to split the outer and fully unroll the inner loop and thus hold the whole temporary vector t in registers. Otherwise

562

D. Bartuschat, M. St¨ urmer, and H. K¨ ostler

Algorithm 2. Forward substituion y = L−1 b 1 Init: y = 0, t = b 2 for j = 1 to n do 3 yi = yi + ti · L1j,j 4 for i = j + 1 to n do 5 ti = ti − yi · Lj,i 6 end for 7 end for

this loop would require to load and store all operands, as no registers cannot be addressed according to the loop variable.

4

Results

Batch-OMP kernel: We use the IBM systemsim Cell-Simulator to investigate the performance of the Batch-OMP kernel itself. Like for all other experiments presented, a dictionary containing K = 128 atoms of 64 elements each is used. The time required for denoising a single patch is shown in Fig. 1. Initialization (constant run time) and reconstruction of the patch (run time depending on the number of atoms chosen) are executed only once. The other kernels need to be executed whenever a new atom is chosen. As they possess different complexity, their contribution is analyzed in more detail in Fig. 2.

Fig. 1. Time required for sparse coding and reconstruction of a single 64-element patch with a dictionary of 128 atoms for a fixed number of atoms to be chosen

About 94 % of the initialization is spent with multiplying the dictionary matrix D with the current patch vector x. If only a small number of atoms is chosen, it dominates the overall runtime. Index search (line 3 of algorithm 1) and vector subtraction (line 11) only dependent on the dictionary size. They play some role as long as the triangular

Orthogonal Matching Pursuit Algorithm for Image Denoising on the Cell BE

563

matrix L is very small, but become less and less important if more than four atoms are chosen. One would expect the substitution kernels to become dominant soon. While all kernels not mentioned yet have linear complexity, these are the only ones with complexity O(i2 ). When the whole Batch-OMP is considered, the effort should even increase cubically with the number of atoms chosen. For the set sizes evaluated, however, this cannot be observed. This is related to the unrolling of the inner loop of the substitution kernel of algorithm 2 starting at line 4. As the temporary vector t corresponds to very few SIMD vectors, all succeeding computations fall into the latency of the first update, which is on the critical path. The substitution kernels therefore exhibit linear increase of runtime for small triangular matrices, and quadratic only for larger ones.

Fig. 2. Contribution of the various kernel types to the overall runtime

In general, the multiplications of a matrix with dense (during initialization) or sparse vectors (projection of the Gram matrix onto the residual and reconstruction of the denoised patch) together with the substitutions need at least 3/4 of the time. While the former generally exhibit a high computational density, the substitutions also suffer from latencies as long as only very few atoms have been chosen. Denoising performance: We run our Cell implementation to denoise an image R 3 (PS3), an IBM QS22 blade and a current of size 512 × 512 on a Playstation multicore CPU. Each of the 255025 overlapping patches corresponds to 8 × 8 pixels. As the rounding mode of the SPEs yield results slightly different from the CPU’s, for each patch a sparse representation with 16 atoms is computed to ensure that the same amount of work is done. The measured runtime excluding reading the noisy image and writing the denoised image are shown for the blade and for the PS3 in Fig. 3 together with the parallel efficiency for different numbers of used SPEs on the blade. Here the

564

D. Bartuschat, M. St¨ urmer, and H. K¨ ostler

parallel efficiency is considered with respect to the runtime on one SPE. The results from the PS3 resemble the results from the blade up to the number of six available SPEs scaled by the slightly lower CPU frequency of 3.188 instead of 3.2 GHz. Based on the analysis of the Batch-OMP kernel, the overhead can be estimated. In the serial case, the Batch-OMP kernel is executed 99.1 % of the time, and even when using 16 SPEs for 96.6 % on each of them. Orchestrating the DMA transfers and combining the denoised images takes only a small amount of time. The parallel efficiency fluctuates mainly due to the granularity of four lines that are assigned to an SPE at a time. Less apparent, a continuous decrease of efficiency is observed. The main reason here is the increasing work to combine the partial results each SPE creates separately. Both effects could partly be countervailed, but the effort does not seem worthwhile with 92.7 % parallel efficiency in the worst case of 15 SPEs.

Fig. 3. Runtimes and parallel efficiency of image denoising on Cell Broadband Engine

In Tab. 1 the runtimes of the algorithm on the blade from Fig. 3 are compared to runtimes of a (not hand-optimized) parallel OpenMP implementation TM R Core i7 940 for multicore CPUs. The runtimes were measured on an Intel with 2.93GHz (Nehalem). It can be seen that the per-core performance of the CBEA implementation is about 5 times higher. Denoising results: A comparison of a noisy image and the denoised image resulting from the algorithm described in this paper, is presented in Fig. 4. Fig. 4(a) Table 1. Comparison of runtimes of the image denoising algorithm on a current multicore CPU and Cell for different numbers of cores or SPEs Cores / SPEs 1 2 4 Runtime [s] on QS22 3.51 1.76 0.89 Runtime [s] on Nehalem 17.9 9.52 4.79

Orthogonal Matching Pursuit Algorithm for Image Denoising on the Cell BE

565

depicts the original image and in Fig. 4(b) the same image corrupted by Gaussian noise with σ = 20 is shown. As can be seen from Fig. 4(c), the noise has been reduced significantly, while the underlying structure and edges have been preserved. Thus, the difference image in Fig. 4(d) contains much noise, but only little structure. The denoising of the shown image with 16 SPEs on a blade takes 59 ms and on average 4 atoms per patch are used.

(a) original image

(b) noisy image (Inoise )

(c) denoised image (Iden )

(d) difference image (Inoise − Iden )

Fig. 4. Denoising results for Lena image of size 512 × 512 pixels

5

Future Work

Currently we work on a comparison of the CBEA and a modern Graphic Processing Units with respect to the performance of the denoising algorithm. In addition to that also new architectures like the Intel Larrabee processor are worthwhile to consider.

566

D. Bartuschat, M. St¨ urmer, and H. K¨ ostler

References 1. Starck, J., Elad, M., Donoho, D.: Image decomposition via the combination of sparse representations and a variational approach. IEEE Transactions on Image Processing 14(10), 1570–1582 (2005) 2. Tropp, J.: Topics in Sparse Approximation. PhD thesis, The University of Texas at Austin (2004) 3. Aharon, M., Elad, M., Bruckstein, A.: On the uniqueness of overcomplete dictionaries, and a practical way to retrieve them. Linear Algebra and Its Applications 416(1), 48–67 (2006) 4. Borsdorf, A., Raupach, R., Hornegger, J.: Wavelet based Noise Reduction by Identification of Correlation. In: Franke, K., M¨ uller, K.-R., Nickolay, B., Sch¨ afer, R. (eds.) DAGM 2006. LNCS, vol. 4174, pp. 21–30. Springer, Heidelberg (2006) 5. Borsdorf, A., Raupach, R., Hornegger, J.: Separate CT-Reconstruction for 3D Wavelet Based Noise Reduction Using Correlation Analysis. In: Yu, B. (ed.) IEEE NSS/MIC Conference Record., pp. 2633–2638 (2007) 6. Mayer, M., Borsdorf, A., K¨ ostler, H., Hornegger, J., R¨ ude, U.: Nonlinear Diffusion vs. Wavelet Based Noise Reduction in CT Using Correlation Analysis. In: Lensch, H., Rosenhahn, B., Seidel, H.P., Slusallek, P., Weickert, J. (eds.) Vision, Modeling, and Visualization 2007, pp. 223–232 (2007) 7. Bartuschat, D., Borsdorf, A., K¨ ostler, H., Rubinstein, R., St¨ urmer, M.: A parallel K-SVD implementation for CT image denoising. Technical report, Department of Computer Science 10 (System Simulation), Friedrich-Alexander-University of Erlangen-Nuremberg, Germany (2009) 8. K¨ ostler, H.: A Multigrid Framework for Variational Approaches in Medical Image Processing and Computer Vision. Verlag Dr. Hut, M¨ unchen (2008) 9. Davis, G., Mallat, S., Avellaneda, M.: Adaptive greedy approximations. Constructive Approximation 13(1), 57–98 (1997) 10. Rubinstein, R., Zibulevsky, M., Elad, M.: Efficient Implementation of the K-SVD Algorithm and the Batch-OMP Method 11. Donoho, D.L., Elad, M.: Optimally sparse representations in general (nonorthogonal) dictionaries via l1 minimization. Proc. Nat. Acad. Sci. 100, 2197–2202 (2002) 12. Aubert, G., Kornprobst, P.: Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations. Applied Mathematical Sciences, 2nd edn., vol. 147. Springer, Heidelberg (2006) 13. Elad, M., Aharon, M.: Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process 15(12), 3736–3745 (2006) 14. IBM Corporation Rochester MN, USA: Programming Tutorial, Software Development Kit for Multicore Acceleration, Version 3.0 (2007) 15. Gschwind, M.: The Cell Broadband Engine: Exploiting Multiple Levels of Parallelism in a Chip Multiprocessor. International Journal of Parallel Programming 35(3), 233–262 (2007)

A Blocking Strategy on Multicore Architectures for Dynamically Adaptive PDE Solvers Wolfgang Eckhardt and Tobias Weinzierl Technische Universit¨ at M¨ unchen, 85748 Garching, Germany {eckhardw,weinzier}@in.tum.de http://www5.in.tum.de

Abstract. This paper analyses a PDE solver working on adaptive Cartesian grids. While a rigorous element-wise formulation of this solver offers great flexibility concerning dynamic adaptivity, and while it comes along with very low memory requirements, the realisation’s speed can not cope with codes working on patches of regular grids—in particular, if the latter deploy patches to several cores. Instead of composing a grid of regular patches, we suggest to identify regular patches throughout the recursive, element-wise grid traversal. Our code then unrolls the recursion for these regular grid blocks automatically, and it deploys their computations to several cores. It hence benefits from multicores on regular subdomains, but preserves its simple, element-wise character and its ability to handle arbitrary dynamic refinement and domain topology changes.

1

Introduction

Besides experiments and theoretical models, computational science and engineering is a driving force for new scientific insights, and, thus, there is an everlasting need for more detailed and accurate numerical results. While scientific codes and their results have been benefiting from increasing clock speeds for decades, hardware evolution nowadays comprises an increase in the number of cores. Software has to exploit multicores [9]. Tree data structures and element-wise traversals are of great value for the management of adaptive Cartesian grids for partial differential equations (PDEs). The combination facilitates dynamic h-adaptivity together with geometric multigrid algorithms, it yields storage schemes with low memory requirements, and the tree topology incorporates a domain decomposition [2,5,10], i.e. our PDE solvers based on this combination all run on distributed memory machines. In this paper, we concentrate on the shared memory parallelisation for multicore systems where running multiple domain decomposition codes on one multicore machine is not an option—here, a single element-wise traversal has to exploit the cores. On a distributed memory cluster with multicore nodes, the techniques then are one building block of a heterogeneous parallelisation strategy. Dynamic adaptivity—neither structure, location nor moment of refinement are known a priori—is a key ingredient of sophisticated PDE solvers. For such R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 567–575, 2010. c Springer-Verlag Berlin Heidelberg 2010 

568

W. Eckhardt and T. Weinzierl

codes, it is convenient to elaborate an algorithm on the smallest work unit available. The solver then arises from the combination of individual work units and the actual grid. An element-wise grid traversal mirrors this fact. A rigorous sequential, element-wise formulation often introduces runtime penalties, as the number of computations per element is typically small. With a small set of operations, a code neither exploits the vast of registers available (due to VLIW or SSE), nor can it use multicore architectures economically. Many PDE solvers hence do not support arbitrary dynamic grids, but restrict themselves to adaptive grids consisting of regular refined grid patches ([1,3], e.g.). On these patches, they switch from an element-wise formulation to a holistic point of view, optimise the calculations ([1,4,7], e.g.), and outperform elementwise approaches. This paper combines advantages of both worlds, as it allows to have arbitrary adaptive and changing grids but nevertheless benefits from regularly refined patches. A synthesised attribute [6] acts as marker on the tree representing the adaptive Cartesian grid. This marker is updated on-the-fly and identifies invariant, i.e. unchanging and regular refined subdomains. Throughout the traversal, the algorithm analyses the marker and either processes the grid element-wise or switches to a block-wise processing: The latter holds complete regular subgrids in one array. The corresponding computations fit to a shared-memory paradigm and, thus, fit to a multicore parallelisation. The results exhibit a high level of concurrency albeit the original algorithm is a pure element-wise formulation and although the grid supports arbitrarily dynamic refinement. Since the tree traversal’s formulation is recursive, such a blocking of computations equals a recursion unrolling [8]: Our approach tessellates a domain with an arbitrary refined grid, but then identifies and optimises regular subregions throughout the computation. The remainder of the paper is organised as follows: First, we introduce our dynamic grids and the corresponding element-wise traversal. These grids require a very small amount of memory to store them, they support dynamic h-refinement, and they resolve all grid resolutions simultaneously, i.e. they yield a sequence of regular Cartesian grids. The latter is an important ingredient for geometric multigrid algorithms. Second, we introduce our recursion unrolling and blocking approach, and we give the tree grammar identifying regularly refined subgrids. The numerical results, third, illustrate the runtime improvement due to this blocking, i.e. they illustrate the advantages for multicore architectures. Although the experiments concentrate on a matrix-free solver of the Poisson equation, the insights hold for many PDEs, and a short conclusion picks up this fact and closes the discussion.

2

k-Spacetrees

Let d denote the spatial dimension of the PDE. The adaptive Cartesian grids in this paper result from k-spacetrees: First, we embed the computational domain into a hypercube. Second, we cut this hypercube into k parts, k ≥ 2, along each coordinate axis, and, thus, end up with k d small hypercubes. Third,

Blocking Strategy on Multicores for Dynamically Adaptive Solvers

569

Fig. 1. k-spacetree construction process for d = 2 and k = 3 for a square (left). The square’s boundary is resolved with a smaller resolution than the square’s area. Cut through adaptive spacetree for a sphere (right).

we individually decide for each small hypercube whether to apply the refinement scheme recursively. The construction scheme equals, on the one hand, an adaptive Cartesian grid (Figure 1): Each tree level yields one regular Cartesian grid, while the individual regular grids are embedded into each other. The tree’s leaves (unrefined cells) in turn give an adaptive Cartesian grid. On the other hand, the construction imposes a tree topology on the grid’s elements. As our adaptive grids and k-spacetrees are equivalent, we hold the k-spacetree instead of the grid itself. This tree equals a quadtree for k = 2, d = 2. For k = 2, d = 3, we end up with an octree. The number of recursive steps depends on the boundary approximation, on the computational domain’s shape, and on the numerical accuracy to be obtained. Having a tree structure holding all the levels of the grid construction process is of great value for geometric multigrid algorithms [10]. Furthermore, dynamic h-adaptivity and dynamic coarsening fit perfectly to this approach, since they just add elements to or remove elements from the k-spacetree. Besides the flexibility, k-spacetrees also facilitate an efficient encoding: Throughout a depth-first traversal of the tree, one bit per cell is sufficient to decide whether the cell is refined or not, i.e. if the spacetree is encoded along a depth-first traversal, a bit stream of length n is sufficient; n being the number of spacetree elements. A fixed number of additional bits per vertex or cell, respectively, then allows to encode the geometry due to a marker-and-cell approach, i.e. we denote for each cell whether it is inside the computational domain or not, and it allows to encode the boundary conditions. We end up with a storage scheme with very modest memory requirements [2,5], although it supports dynamic, arbitrary adaptivity. Hence, it is an obvious idea to use a depth-first traversal to run through the spacetree. Such a traversal equals an element-wise traversal of the corresponding adaptive Cartesian grid levels where each element has 2d adjacent elements. The vertices have either 2d adjacent cells if they are part of a regular Cartesian subgrid, or less than 2d cells. The latter vertices are hanging vertices, they border

570

W. Eckhardt and T. Weinzierl

a regular Cartesian grid of one level, and they require a special mathematical treatment. A PDE solver finally is an element-wise assembly or an element-wise matrix-vector evaluation if we are implementing a matrix-free method. It is typically realised recursively.

3

Blocking

A recursive, element-wise traversal of the spatial discretisation given by a k-spacetree is flexible with respect to changing grids, and it comes along with low memory requirements. Yet, the work per element typically comprises only a small number of floating point operations. This reveals two shortcomings of a strict element-wise approach: It is first of all difficult to exploit huge numbers of registers and instruction level parallelism (VLIW or SSE) with few instructions—with only a small number of optimisations at hand, one has neither a broad amount of opportunities to tune the data layout and access ordering manually, nor can the compiler optimise the instruction scheduling [4]. Furthermore, the traversal comprises integer arithmetics. In the worst case, these integer arithmetics dominate the execution time, i.e. the traversal’s integer evaluations slow down the floating point computations. Solvers working on regular Cartesian grids do not have these very problems: Integer arithmetics for regular data structures are uncomplicated, and the regularity of the data structures facilitates permutations such as loop unrolling, loop merging, and blocking [4,7]. They make the code utilize advanced hardware features. Regular grids also fit to a shared memory parallelisation, where the set of computations is split among several cores sharing memory or caches. They fit to multicores. Our tree algorithm is very flexible and allows the treatment of adaptive grids. Whenever it resolves regular grids, the recursive formulation however can not cope with codes optimised for and benefiting from regular data structures. In particular, it is difficult to port it to a multicore architecture. As we do not want to restrict any flexibility, we make our adaptive algorithm identify regular subgrids throughout the traversal. For each regular subgrid, we replace the recursive element-wise formulation with tailored patch-based algorithmics. 3.1

Patch Identification

The traversal splits up the adaptive refinement into a two-step process: Whenever the algorithm wants to refine a spacetree leaf, it sets a refinement flag for the leaf. We then add the k d additional spacetree nodes throughout the subsequent traversal. Let p : E #→ N+ 0 ∪ {⊥} assign each cell of the k-spacetree a number. This number is a pattern identifier. E denotes the nodes of the k-spacetree. For all leaves e ∈ E ⎧ no refinement flag is set, no adjacent vertex is hanging, ⎨ 0 (1) p(e) = and all adjacent vertices are inside the domain, ⎩ ⊥ else

Blocking Strategy on Multicores for Dynamically Adaptive Solvers

571

holds. Let ( denote the k-spacetree’s topology, i.e. for a, b ∈ E with a ( b, a is a child of b. With (1), the traversal assigns each refined node e a pattern number  i+1 ∃i ≥ 0 : ∀c ( e : p(c) = i p(e) = (2) ⊥ else throughout the steps-up of the traversal, i.e. whenever it backtracks the depthfirst traversal and unrolls the call stack, p is set. This p is an analysed tree attribute [6]. The coarsening is treated analogously and resets pattern identifiers. Due to the attribute, we need one additional integer per spacetree node. Each refined node and its descendants compose a (sub-)k-spacetree. Each time the algorithm encounters a pattern number greater or equal one, we know that this subtree corresponds to a cascade of regular grids due to the universal quantifier in (2). We furthermore know that it will not change throughout the traversal and comprises exclusively non-hanging inner nodes. p(e) gives us the height of this subtree. The corresponding levels of the subtree yield regular Cartesian grids, and the identifier’s value describes the size of these grids. For the vertices within this gird, we can omit any case distinctions handling hanging nodes, boundary nodes, and so forth. 3.2

Recursion Unrolling

The operations executed by the recursive traversal per element split up into three phases: First, the traversal loads the data belonging to a spacetree node (vertex and cell data). Second, it triggers the element-wise operator evaluations or the assembly. Afterwards, it steps down recursively. Third, it writes back the node’s data. Our multicore strategy merges these three steps for regular subtrees. When the algorithm encounters an element e with p(e) = i ≥ 1, the algorithm allocates memory for i + 1 regular Cartesian grids with 1, k d , k 2d , . . . cells each. Afterwards, it reproduces the three processing phases: First, it invokes the load operations for all the elements. The data is stored in the regular Cartesian grids. Second, it applies the operator on all the regular Cartesian grids. Third, it invokes the store operations. The modified traversal transforms the recursive formulation into a sequential algorithm. It is a recursion unrolling where the three different processing steps are merged. While the traversal still works on an arbitrary adaptive grid, and while the grid may change between two traversals, the algorithm identifies regular patches on-the-fly and processes them with the modified scheme. For the remaining (irregular) grid parts, it preserves its recursive nature. 3.3

Blocking and Multicore

The recursion unrolling first eliminates several loads and stores for elements sharing vertices within a patch, e.g., and, thus, it improves the performance. Second, having regular Cartesian grids at hand allows to realise sophisticated solvers processing whole blocks of unknowns or discretisations or shape functions, respectively. In this paper, we concentrate on the performance aspect and neglect

572

W. Eckhardt and T. Weinzierl

solver modifications. However, all solvers, third, benefit from the application of source code optimisations such as loop unrolling and loop merging. Finally, the blocks provide big, homogeneous data structures. Multicores benefit from the latter. Our concurrency strategy is twofold: The original, recursive, element-wise space-tree traversal runs within one single thread. This thread also computes the pattern identifiers. As soon as the single thread encounters a refined element with a positive identifier, it switches to the block algorithm. This block algorithm in turn distributes its computing phase among several threads. As splitting up uniform work packages on a regular grid is straightforward, the thread work assignment and scheduling is straightforward, too. Grids with many huge patches of regular Cartesian grids benefit from this multicore strategy. They profit in particular, if the regular subgrids remain unchanged throughout the simulation. Nevertheless, the flexibility of the overall algorithm is preserved, i.e. it still handles arbitrary refined and changing grids.

4

Numerical Results

We tested the blocking and multicore strategy on AMD Opteron and Intel Itanium2 processors at the Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities. The latter were part of SGI’s Altix 4700 platform with up to four dual cores per blade, while the Opteron cluster provided up to four processors per node. A matrix-free solver for a simple Poisson equation discretised with linear finite elements on the unit square/cube (d ∈ {2, 3}) acted as test PDE solved by a Jacobi scheme. We started with a regular grid and, afterwards, applied an error estimator refining the grid where the second derivative became big. The figures reveal a significant speedup for regular grids (Table 1). With smaller mesh widths, the k-spacetree’s depth grows (second column), and the Table 1. Speedup on a regular grid with mesh width h. The upper part of the table gives results for d = 2, the lower part for d = 3.

Depth Patches h 6.6 · 10−3 5 1 × 4, 40 × 3, 184 × 2,. . . 3.3 · 10−3 6 1 × 5, 40 × 4 184 × 3, 616 × 2,. . . 6.6 · 10−4 7 1 × 6, 40 × . . . 8 ... 3.3 · 10−4 3.3 · 10−2 4 1 × 3, 316 × 2 5 1 × 4, 316 × 3 6.6 · 10−3 6364 × 2, . . .

Opteron Itanium2 Vertices 2 Cores 4 Cores 2 Cores 4 Cores 8 Cores 5.86 · 104 1.18 1.18 1.23 1.14 1.14 5.30 · 105

1.42

1.52

1.41

1.47

1.57

4.78 · 106 4.30 · 107 5.12 · 105 1.42 · 107

1.68 1.83 1.07 1.36

2.28 2.99 1.06 1.47

1.66 1.84 1.09 1.41

2.29 2.93 1.10 1.62

2.64 4.43 1.06 1.31

Blocking Strategy on Multicores for Dynamically Adaptive Solvers

573

Table 2. Speedup for d = 2 on the Opteron. The error estimator updates the grid several times (left). Adaptive grid after three iterations (right).

Patches it #vertices 1 5.414 · 106 1 × 7, 40 × 6, 184 × 5, 616 × 4, 1912 × 3 2 1.971 · 107 447 × 4, 15519 × 3 3 2.490 · 107 176 × 4, 7122 × 3 4 1.318 · 108 2 × 5, 443 × 4, 6582 × 3 5 3.208 · 108 1 × 5, 363 × 4, 19005 × 3 6 3.251 · 108 275 × 4, 18893 × 3 ... ...

Cores 2 4 1.71 2.28

1.02 1.00 1.00 1.02 1.02 1.02 1.18 1.16 1.23 1.23 ...

number of regular subtrees without boundary vertices increases (third column with the second digit giving the recursion unrolling depth p(e))—the path numbers follow a simple analytical formula. If cells adjacent to the boundary were not excluded from the recursion unrolling, the whole grid would fit into one patch. Multicore systems benefit from big patches, as they facilitate a high concurrency level and big grain sizes with low synchronisation and balancing overhead. Small patches can not benefit from high core numbers according to Amdahl’s law, i.e. the parallel efficiency declines with an increasing number of cores, and it increases with an increasing tree depth. Furthermore, the main memory restricts the maximum recursion unrolling. The parallel speedup hence suffers from a big dimension d: the patch size of a fixed recursion unrolling level grows exponentially, and the ratio of big to smaller patches drops, i.e. few relatively small patches already occupy the complete memory. Nevertheless, the results reflect that for d = 3 a pure recursion unrolling is not sufficient to obtain a better performance—here, the unrolled implementation has to be tuned, too. In Table 2, we study a dynamic changing grid. Here, the runtime profits from the recursion unrolling for the initial regular grid. Afterwards, the speedup breaks down due to the adaptive refinement. If the grid becomes quite regular again after a couple of iterations—the figures track the grid construction phase, while an almost stationary grid again yields figures similar to those in Table 1— the speedup raises again. While the parallelisation can not exploit several cores throughout the grid refinement phase, it is robust for all experiments, i.e. it never introduces a runtime penalty.

5

Conclusion

This paper starts from the observation that regular data structures in particular profit from multicores. It suggests a paradigm shift for adaptive mesh refinement:

574

W. Eckhardt and T. Weinzierl

Instead of composing a grid of small regular grids, it identifies regular subgrids on-the-fly, and switches to an optimised, multithreaded traversal code for these subgrids. Hence, it can tackle grids changing their structure and topology permanently. As our approach restricts to an algorithmic optimisation of the tree traversal, matrix-free PDE solvers such as the solver studied here benefit immediately from the traversal tuning. If the system matrices are set up explicitly, this approach speeds up solely the assembly (and, perhaps, some pre- and postprocessing)—a phase that typically does not dominate the overall solver runtime. However, the assembly workload becomes the more critical the more often the mesh changes, or, in turn, our optimisation enables the code to remesh more frequently. The effect of the blocking on the memory overhead of an explicit matrix setup— matrices corresponding to regular Cartesian grids can be stored more efficiently than flexible sparse matrices—is another issue yet to be studied. The numerical results restrict themselves to the Poisson equation. Nevertheless, the approach works for every PDE solver realised on k-spacetrees, and every solver holding big regular refined stationary grid patches benefits from the recursion unrolling. The primary application scope of the code here is computational fluid dynamics and fluid-interaction problems [2], where the computational domain typically consists of huge areas covered by regular grids. On the permanently changing boundary areas, a fine, adaptive grid resolution is of great value. The paper shows that a rigorous recursive, element-wise formulation for such a problem still benefits from multicore architectures—especially, if we adopt the refinement criteria and the maximum unrolling depth to the actual multicore architecture. Nevertheless, it is still a fact that a problem’s structure has to fit to a multicore architecture, i.e. if the grid consists exclusively of irregular subdomains, the approach here does not result in a multicore speedup.

Acknowledgments Thanks are due to Hans-Joachim Bungartz and Miriam Mehl for supervising the underlying diploma and Ph.D. thesis, as well as the valuable contributions and encouragement.

References 1. Bergen, B., Wellein, G., H¨ ulsemann, F., R¨ ude, U.: Hierarchical hybrid grids: achieving TERAFLOP performance on large scale finite element simulations. International Journal of Parallel, Emergent and Distributed Systems 4(22), 311–329 (2007) 2. Brenk, M., Bungartz, H.-J., Mehl, M., Muntean, I.L., Neckel, T., Weinzierl, T.: Numerical Simulation of Particle Transport in a Drift Ratchet. SIAM Journal of Scientific Computing 30(6), 2777–2798 (2008) 3. de St. Germain, J.D., McCorquodale, J., Parker, S.G., Johnson, C.R.: Uintah: a massively parallel problem solving environment. In: The Ninth International Symposium on High-Performance Distributed Computing, pp. 33–41 (2000)

Blocking Strategy on Multicores for Dynamically Adaptive Solvers

575

4. Gropp, W.D., Kaushik, D.K., Keyes, D.E., Smith, B.F.: High-performacne parallel implicit CFD. Parallel Computing 27(4), 337–362 (2001) 5. G¨ unther, F., Mehl, M., P¨ ogl, M., Zenger, C.: A cache-aware algorithm for PDEs on hierarchical data structures based on space-filling curves. SIAM Journal on Scientific Computing 28(5), 1634–1650 (2006) 6. Knuth, D.E.: The genesis of attribute grammars. In: WAGA: Proceedings of the International Conference on Attribute Grammars and Their Applications, pp. 1–12 (1998) 7. Kowarschik, M., Weiß, C.: An Overview of Cache Optimization Techniques and Cache-Aware Numerical Algorithms. In: Algorithms for Memory Hierarchies 2002, pp. 213–232 (2003) 8. Rugina, R., Rinard, M.C.: Recursion Unrolling for Divide and Conquer Programs. In: Midkiff, S.P., Moreira, J.E., Gupta, M., Chatterjee, S., Ferrante, J., Prins, J.F., Pugh, B., Tseng, C.-W. (eds.) LCPC 2000. LNCS, vol. 2017, pp. 34–48. Springer, Heidelberg (2001) 9. Sutter, H.: The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software. Dr. Dobb’s Journal 3(30), 202–210 (2005) 10. Weinzierl, T.: A Framework for Parallel PDE Solvers on Multiscale Adaptive Cartesian Grids. Verlag Dr. Hut (2009)

Affinity-On-Next-Touch: An Extension to the Linux Kernel for NUMA Architectures Stefan Lankes, Boris Bierbaum, and Thomas Bemmerl Chair for Operating Systems RWTH Aachen University 52056 Aachen, Germany {lankes,bierbaum,bemmerl}@lfbs.rwth-aachen.de

Abstract. For many years now, NUMA architectures are being used in the design of large shared memory computers and they are gaining importance even for smaller-scale systems. On a NUMA machine, the distribution of data has a significant impact on the performance and scalability of data-intensive programs, because of the difference in access speed between local and remote parts of the memory system. Unfortunately, memory access patterns are often very complex and difficult to predict. Affinity-on-next-touch may be a useful page placement strategy to distribute the data in a suitable manner, but support for it is missing from the current Linux kernel. In this paper, we present an extension to the Linux kernel which implements this strategy and compare it with alternative approaches.

1

Introduction

Today, shared memory multiprocessor computers increasingly often employ a NUMA (Non-Unified Memory Access) design, even ones with only a single mainboard. In such systems, the memory is directly attached to the processors via integrated memory controllers, each processor/memory pair forming a NUMA node. Node-local memory access is faster than access to remote memory, which is reached via some system interconnect, leading to a non-uniform memory access performance characteristic in the system as a whole. Also, the performance may be constrained by congested data paths to and from NUMA nodes, if too many remote accesses occur. Hence, on NUMA systems, the distribution of data typically has a significant impact on the performance of applications which process large amounts of data. To parallelize data-intensive algorithms, the shared memory programming model OpenMP [1] is often preferred over alternatives like MPI or PGAS because it allows an incremental approach to parallelism and obviates the need to specify the distribution of data over the NUMA nodes. However, the flat memory model of OpenMP needs to be properly mapped onto the hierarchical memory system of a NUMA architecture to avoid the performance issues mentioned above. A feasible approach may be to analyze the application’s data access patterns and try to minimize the number of remote memory accesses by a proper data R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 576–585, 2010. c Springer-Verlag Berlin Heidelberg 2010 

Affinity-On-Next-Touch

577

distribution [2,3]. But this may be a tediuos task if the memory access patterns are complex and dynamically changing during runtime. In such situations, an adaptive data distribution mechanism, supported by the operating system or runtime system, may be more beneficial, if the overhead to determine the actual access pattern and migrate the data can be kept low. Affinity-on-next-touch is a promising adaptive data distribution strategy. The basic idea is this: Via some runtime mechanism, a user-level process activates affinity-on-next-touch for a certain region of its virtual memory space. Afterwards, each page in this region will be migrated to that node which next tries to access it. Noordergraaf and van der Pas [4] have proposed to extend the OpenMP standard to support this strategy. That proposal was first implemented in Compaq’s OpenMP compiler by Bircsak et al. [5]. Since version 9, the affinity-onnext-touch mechanism is available in the Solaris operating system and can be triggered via the madvise system call. L¨ of and Homgren [6] und Terboven et al. [7] have described their encouraging experiences with this implementation. Linux, one of the most important operating systems in HPC, does not yet support affinity-on-next-touch. Terboven et al. [7] have presented a user-level implementation of this strategy for Linux. Unfortunately, its overhead is very high, as is shown in Sect. 2. In Sect. 3, we present a new, kernel-based solution, and in Sect. 4, we evaluate its performance.

2

Analysis of the User-Level Implementation

To realize affinity-on-next-touch in user space, Terboven et al. [7] protect a specific memory area from read and write accesses and install a signal handler to catch access violations. If a thread accesses a page in the protected memory area, the signal handler migrates the page to the node which handled the access violation. An access violation raises a synchronous signal which is handled on the node of the interrupted thread. Therefore, the page migrates to the node of the thread that tries to access the page. Afterwards, the signal handler clears the page protection and the interrupted thread is resumed. To evaluate the overhead of this user-level solution, we use a small OpenMP program, in which a parallel for-loop increments each element of an array. We measure the runtime of this loop during consecutive iterations. We expect that during the first iteration the pages are migrated and therefore, the difference between the time for the first iteration and the subsequent iterations shows the overhead of the page migration mechanism. We ran this benchmark on devon, a dual-socket, quad-core Opteron 2376 (2.3 GHz) system with 512 KB L2 cache per core, 6 MB shared L3 cache per processor and 32 GB of main memory running Fedora 10, which is based on kernel version 2.6.27. The benchmark was compiled with gcc 4.3.2 and uses an array size of 512 MB and 8 threads in the parallel region. Each thread is bound to one core of the NUMA system. We compare the results with those using Linux’s default first touch page placement strategy, which puts each page next

578

S. Lankes, B. Bierbaum, and T. Bemmerl

to the processor first accessing it. Because the benchmark initializes the array in a sequential loop, it is entirely placed on one node in this case. As the 3rd column of Tab. 1 shows, the user-level solution suffers from a high migration overhead. After the pages have been migrated, accesses to the array are significantly faster than with first touch, because remote accesses are completely avoided. The time for the first iteration shows the overhead which is too high to make this implementation really beneficial. Table 1. Measured time per loop iteration over the array (8 threads) Iteration First touch Next Touch Next Touch (User-Level) (Kernel-Level) 1 127.23 ms 5179.09 ms 418.75 ms 2 128.40 ms 66.82 ms 66.45 ms 3 127.99 ms 66.96 ms 66.72 ms 4 128.80 ms 66.74 ms 66.83 ms 5 128.45 ms 66.64 ms 66.94 ms 6 128.41 ms 66.60 ms 66.74 ms

This overhead may be caused by switching the thread context to the signal handler or by running the signal handler itself. Coarsely speaking, the signal handler consists of three system calls: getcpu is used to get the number of the thread’s NUMA node, move_pages to migrate the page and mprotect to change the access permissions. To understand what causes the overhead, we measured the system call time per thread and the total time for the first loop iteration with an increasing thread count. The difference between the loop runtime and the sum of the system call times is the time needed to create the parallel forloop, to increment the elements, to handle the access violation and to switch the thread context to the the signal handler. Fig. 1 shows that the loop runtime keeps almost constant while the thread count increases, a sign of quite bad scalability, because the more threads there are, the less each single thread needs to do. As long as the system’s data paths are not congested by the migration traffic, the loop should run faster with more threads. The main reason for the observed behavior is the increasing time for mprotect and the bad scalability of move_pages. The time for getcpu ist negligible. Besides this, the difference between the loop runtime and the time needed for the system calls is pretty large and increases when starting more threads. That time is primarily spent for access violation handling and context switching. Creating the parallel region and incrementing the array elements does not take very long in comparison as can be seen in Tab. 1, because this is done in the subsequent loop iterations as well. Without significant changes to the Linux kernel it is not possible to reduce the overhead of handling the access violation. For this work, we concentrated on reducing the time spent in mprotect and move_pages, because these system calls primarily cause the bad scalability of the user-level solution.

Affinity-On-Next-Touch 8000

579

getcpu mprotect move_pages difference to loop runtime

7000

time [ms]

6000 5000 4000 3000 2000 1000 0 1 Thread

2 Threads

4 Threads

8 Threads

Fig. 1. Time per thread for 1st loop iteration (Next Touch, User-Level)

To explain the behavior of mprotect, we need to take a closer look at the memory management of the Linux kernel [8]. A process’s virtual memory is sliced into Virtual Memory Areas (VMA), each of which covers a contiguous address range made up of pages with identical properties, e.g. access permissions. VMAs are represented by a record of type vm_area_struct and stored in a single list per process. In our case, the user-level solution creates a VMA with read and write permissions cleared and sets the same permissions in the page table entries. On access to one page in this VMA, the signal handler migrates the page to the current node and allows read and write access to it, splitting the VMA up into two new ones for the pages left and right of the affected one (if any) and one for the page itself. To reduce the number of VMAs, the Linux kernel tries to merge the new memory areas with their predecessors and successors to create one new area with a contiguous address range and uniform access permissions. During all of this, the VMA list needs to be updated, generating a lot of write accesses to the list. To avoid race conditions, the list is protected by a lock, which essentially serializes our code and slows down memory access. Consequently, we developed a new kernel extension to realize affinity-on-next-touch, which does not need write access to the list of virtual memory areas. Compared to the user-level implementation, this approach promises better performance. In addition to mprotect, move_pages does not scale well, too. To explain this phenomenon, we need to look to at how the Linux kernel handles demand paging. For each page frame, there exists a record page that contains information about the pages mapped onto this page frame. Every such record is added to an LRU page cache for fast lookup. This cache is made up of two lists: The first one is called active list and contains recently referenced pages, while the second list contains a set of inactive pages. If Linux needs to swap a page out from main memory, the kernel uses one from the inactive list. To accelerate

580

S. Lankes, B. Bierbaum, and T. Bemmerl

list operations and avoid locking, each list operation is delayed and temporarily buffered. Each core has got its own such buffer and exclusive access to it. To migrate a page, move_pages needs to remove the page from the page cache. Afterwards, the system call allocates a new page frame on a specific node, copies the content of the page to the new page frame and puts a new page record into the page cache. When doing so, it assumes that the LRU lists are upto-date. Therefore, move_pages executes the buffered list operations and sends a messages to all other cores to make them drain their buffers as well. This interrupts the computation on all cores and decreases the scalability. Disabling the demand paging algorithm does not solve this problem because the page cache is also used in other parts of the kernel. Therefore, we developed a kernel extension, which minimizes the need to drain the buffers on all cores.

3

An Extension to the Linux Kernel

Like in Solaris, our kernel extension can be triggered via the system call madvise with the new parameter MADV_ACCESS_LWP. Similar to the user-level solution, madvise protects the memory area against read and write accesses, but it changes the permissions only in the page table entries and does not update the corresponding vm_area_struct. Therefore, madvise does not need write access to the list of memory areas. Now, the permissions stored in vm_area_struct differ from the permissions set on a per-page basis. An attempt to access a page inside of this VMA triggers an interrupt, which is handled by the Linux kernel. If it detects that the page, which the thread tried to access, uses affinity-on-next-touch, the Linux kernel reads the original permissions from vm_area_struct, restores them in the page tables and migrates the page to the current node. To store the information that a page is using affinity-on-next-touch, madvise sets one bit inside of the page record, which can be found without traversing a linear list or a similar data structure. Depending on the memory model of the Linux kernel, only some simple shift operations are necessary. The page fault handler clears the affinity-on-next-touch bit with the atomic operation test_and_set and checks if the bit was set. If it was, the page fault handler restores the original permissions and migrates the page to the current node. To accelerate the page migration process, we make draining the list operation buffers on all cores less often necessary. Bevor returning from madvise, the kernel drains the buffer on the current core. If another core has buffered list operations for a page which is to be migrated, the migration fails. If this happens, the buffers on all cores are drained and the page migration is restarted. The probability that such a second attempt on migration is necessary is quite small. Our benchmark results, shown in the 4th column of Tab. 1, prove our assumptions and show that the kernel-level solution is more than twelve-fold faster than the user-level solution. By using the kernel-level solution, the average time for migrating one page is to 2.8 μs, while the user-level solutions needs 39.4 μs.

Affinity-On-Next-Touch

4

581

Performance Evaluation

4.1

The stream Benchmark

To evaluate the achievable memory bandwidth of our kernel solution, we ran the OpenMP version of McCalpin’s popular stream benchmark1 [9], modified to trigger page migration via next-touch, on the devon system described in Sect. 2. Fig. 2 shows the measured results for different page placement strategies. If the arrays are initialized by the master thread and first touch is used, all data are located on the master’s node. Access to this node’s memory becomes a bottleneck, constraining the achievable bandwidth (a). If each thread initializes the chunks of the arrays which it uses later during the computation, the benchmark achieves peak bandwidth (b). (c) - (f) show the memory bandwidth when using next-touch. The first iteration over the array includes the cost of page migration, if necessary, and of restoring the access permissions. Therefore, the memory bandwidth is much lower for (c) and (e). Further iterations achieve the peak bandwidth because the pages are ideally located then.

14

1 thread 2 threads 4 threads 8 threads

12

memory bandwith [GB/s]

10

8

6

4

2

0 first touch by master (a)

first touch next touch next touch next touch next touch by all (user-level, (user-level, (kernel, (kernel, (b) first iter.) best iter.) first iter.) best iter.) (c) (d) (e) (f)

Fig. 2. Memory bandwidth measured with a modified stream benchmark

A closer look at the results of the first iteration shows that the kernel-level solution reaches its peak bandwidth of 2.19 GB/s by using just one thread, because the pages are already located on the correct (the only) node. Therefore, no page 1

http://www.cs.virginia.edu/stream/

582

S. Lankes, B. Bierbaum, and T. Bemmerl

migration is necessary and the kernel just needs to restore the original access permissions of the pages. When using more threads, the kernel-level approach reaches a memory bandwidth up to 1.59 GB/s. This is more than seventeen-fold faster than the peak memory bandwidth of the user-level solution (91 MB/s). 4.2

The Jacobi Solver

We studied the overhead of adaptive data distribution by running a simple Jacobi solver, parallelized with OpenMP, on devon. The euclidian norm is used to detect that the solver found a suitable solution vector, which happens after 492 iterations using a 10000x10000 matrix. Fig. 3 shows the time to solution for different page placement strategies. In (a) - (c) the data is initialized by the master thread. Like for stream, this constrains the scalability when using first touch (a), affinity-on-next-touch provides a significant improvement here. The difference between the user-level and kernel-level solution is explicable by the additional time which the user-level solution needs to update the list of virtual memory areas. For the ideal solution (d) the memory access pattern of the solver has been analyzed and the matrix has been ideally distributed during the intialization phase. The difference between the ideal solution and the kernel-level implementation of affinity-on-next-touch is quite small, because the time for doing the calculations dominates the additional overhead of page migration. 4.3

An Adaptive PDE Solver

The stream benchmark and the Jacobi solver have static memory access patterns and are easy to optimize for a NUMA system. Programs like these are not the 160

first touch by master (a) next touch (user-level) (b) next touch (kernel-level) (c) ideal data distribution (d)

140 120

time [s]

100 80 60 40 20 0 1

2

4 number of threads

Fig. 3. Time to solution with the Jacobi solver

8

Affinity-On-Next-Touch

583

typical domain for an adaptive page placement strategy like affinity-on-nexttouch, but the large number of applications with a dynamic memory access pattern. Examples of this class of applications are PDE solvers using adaptive mesh refinements (AMR). In this case the work and data need to be dynamically repartitioned at runtime to get good parallel performance. Norden et al. [10] have presented a parallel structured adaptive mesh refinements (SAMR) PDE solver which has been parallelized with OpenMP. A specific aspect of this particular program is that is already optimized for NUMA architectures and dynamically triggers the affinity-on-next-touch mechanism. In addition to devon, we also ran the SAMR PDE solver on a quad-socket, dual-core Opteron 875 (2.2 GHz) system with 1 MB L2 cache per core and 16 GB of RAM running CentOS 5.2 (kernel version 2.6.18). The code was compiled with the Intel Fortran compiler 11.0. Tab. 2 shows the time to solution with various experimental setups. Data redistribution happens more often with a higher number of NUMA nodes. Table 2. Time to solution with the SAMR PDE solver (20000 iterations, 8 threads) First Touch Next Touch Next Touch by all (user-level) (kernel-level) 4-socket Opteron 875 sytem 2-socket Opteron 2376 sytem

5

5318.646s

4514.408s

4489.375s

3840.465s

3904.923s

3777.936s

Related Work

Currently, the Linux scheduler does not take NUMA locality into account when migrating tasks between NUMA nodes. In [11] L. T. Shermerhorn presents an extension to the Linux kernel which permits a task to pull pages close to itself after having been migrated to a new node. To realize this, he uses a mechanism similar to affinity-on-next-touch called migration on fault. His kernel extension unmaps the eligible pages, making the virtual-to-physical address translation fail on access by a thread. After having entered the fault path, the kernel migrates the page to the current node and remaps it. The ideas behind affinity-on-nexttouch and migration on fault are nearly the same. Shermerhorn’s extension uses the fault path to migrate the pages, while the kernel extension proposed in this paper uses an access violation to detect the necessity for a page migration. The performance differences between both solutions are negligible on small NUMA systems. On larger systems, we have to evaluate the scalability of both extensions. This needs to be examined further. Shortly after this paper had been accepted for publication, B. Goglin and N. Furmento published their work about an extension to the Linux kernel which implements affinity-on-next-touch, too [12]. A quick analysis of their patch showed our optimization of move_pages reaching a higher bandwidth. In the future, we need to further compare both kernel extensions.

584

6

S. Lankes, B. Bierbaum, and T. Bemmerl

Outlook and Conclusions

Affinity-on-next-touch is a promising page placement strategy for NUMA machines. However, this paper shows that without changes to the kernel it is not possible to implement it on Linux systems with satisfying performance. Additionally, the bad scalability of mprotect is a problem for any distributed shared memory system using this system call to intercept memory accesses. Our kernel extension provides significantly better performance than the user level implementation, as shown by benchmark results. The mprotect bottleneck was removed and page migration accelerated. In the near future, we want to evaluate the behavior of our kernel extension on systems with a higher number of cores and compare the results with alternative extensions [11,12]. We’d like to thank the authors of [7,10] for their support.

References 1. Dagum, L., Menon, R.: OpenMP: An Industry-Standard API for SharedMemory Programming. IEEE Computational Science & Engineering 5(1), 46–55 (1998) 2. Dormanns, M., Lankes, S., Bemmerl, T., Bolz, G., Pfeiffle, E.: Parallelization of an Airline Flight-Scheduling Module on a SCI-Coupled NUMA Shared Memory Cluster. In: High Performance Computing Systems and Applications (HPCS), Kingston, Canada (June 1999) 3. Dormanns, M.: Shared-Memory Parallelization of the GROMOS 1996 Molecular Dynamics Code. In: Hellwagner, H., Reinefeld, A. (eds.). SCI: Scalable Coherent Interface. Springer, Heidelberg (1999) 4. Noordergraaf, L., van der Pas, R.: Performance Experiences on Sun’s WildFire Prototype. In: Proceedings of the 1999 ACM/IEEE Conference on Supercomputing, Portland, Oregon, USA (November 1999) 5. Bircsak, J., Craig, P., Crowell, R., Cvetanovic, Z., Harris, J., Nelson, C.A., Offner, C.D.: Extending OpenMP for NUMA machines. In: Proceedings of the 2000 ACM/IEEE Conference on Supercomputing, Dallas, Texas, USA (November 2000) 6. L¨ of, H., Holmgren, S.: Affinity-on-next-touch: Increasing the Performance of an Industrial PDE Solver on a cc-NUMA System. In: Proceedings of the 19th Annual International Conference on Supercomputing, Cambridge, Massachusetts, USA, pp. 387–392 (June 2005) 7. Terboven, C., an Mey, D., Schmidl, D., Jin, H., Reichstein, T.: Data and Thread Affinity in OpenMP Programs. In: Proceedings of the 2008 Workshop on Memory Access on future Processors: A solved problem? ACM International Conference on Computing Frontiers, Ischia, Italy, pp. 377–384 (May 2008) 8. Love, R.: Linux Kernel Development, 2nd edn. Novell Press (2005) 9. McCalpin, J.D.: Memory Bandwidth and Machine Balance in Current High Performance Computers. In: IEEE Computer Society Technical Committee on Computer Architecture (TCCA) Newsletter, pp. 19–25 (December 1995)

Affinity-On-Next-Touch

585

10. Nord´en, M., L¨ of, H., Rantakokko, J., Holmgren, S.: Geographical Locality and Dynamic Data Migration for OpenMP Implementations of Adaptive PDE Solvers. In: Proceedings of the 2nd International Workshop on OpenMP (IWOMP), Reims, France, pp. 382–393 (June 2006) 11. Schermerhorn, L.T.: Automatic page migration for linux (a matter of hygiene). In: linux.conf.au 2007 (2007) 12. Goglin, B., Furmento, N.: Enabling High-Performance Memory Migration for Multithreaded Applications on Linux. In: Proceedings of the 23rd IEEE International Parallel and Distributed Processing Symposium (IPDPS 2009), Workshop on Multithreaded Architectures and Applications (MTAAP 2009), Rome, Italy (2009)

Multi–CMP Module System Based on a Look-Ahead Configured Global Network Eryk Laskowski1, L  ukasz Ma´sko1, and Marek Tudruj1,2 1

2

Institute of Computer Science PAS, 01-237 Warsaw, Ordona 21, Poland Polish-Japanese Institute of Information Technology, ul. Koszykowa 86, 02-008 Warsaw, Poland {laskowsk,masko,tudruj}@ipipan.waw.pl

Abstract. The impressive progress of Systems on Chip (SoC) design enables a revival of efficient massively parallel systems based on many Chip Multiprocessor (CMP) modules interconnected by global networks. The paper presents methods for the optimized program execution control for such modular CMP systems. At the CMP module level, communication through shared memory is applied, improved by a novel efficient group communication mechanism (reads on the fly). The inter–module global communication is implemented as a message passing between module memories placed in a shared address space. A two–phase structuring algorithm is described for programs represented as macro data-flow graphs. In the first phase, program tasks inside the CMP modules are scheduled, using an algorithm based on the notion of moldable tasks. In the next phase, the moldable task graph is structured for optimized communication execution in the global interconnection network according to the look-ahead link connection setting paradigm. Simulation experiments evaluate the efficiency and other properties of the proposed architectural solutions.

1

Introduction

Systems on Chip (SoC) and Network on Chip (NoC) technologies [6] which are different forms of Chip Multiprocessors (CMP), are based on many processor cores implemented together on single chips. In recent years, they have given strong arguments for fulfillment of designer’s and researcher’s dream of massive parallelism. Although the most mature CMP systems such as Cell BE and Niagara are still based on a relatively small number of cores which is 8, new ones, much more developed appear, such as 128–core Nvidia GeForce 8800GTX [2] and 188–core Cisco Silicon Packet Processor [1]. Challenges of computing power maximization and technology problems of heat dissipation and communication in multicore systems currently stimulate research in the domain of efficient architectural paradigms and new parallel programming styles. A processor centric design style, commonly used at an early stage of CMP technique, has been now transformed into the interconnect–centric design. System design experience accumulated so far, demonstrates that cluster– based systems, supported by adequate intra–cluster communication solutions, R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 586–595, 2010. c Springer-Verlag Berlin Heidelberg 2010 

CMP Module System Based on a Look-Ahead Configured Global Network

587

can strongly increase execution efficiency of parallel programs and scalability. Highly modular properties of CMPs direct structural research towards hierarchical network structures. Instead of a large connection switching network connecting all processor cores, local core sub-networks can be arranged to be next connected by a central global network. Following this new idea, efficient massively parallel systems can be built today. An early implementation of such concept is the RoadRunner system [3]. This paper concerns optimization of global communication in the system with multiple CMP modules interconnected by a global network. We consider a global network based on a circuit–switching and message passing of cacheable blocks of data. We assume that both intra– and inter–CMP module communication is located at the application program level. We assume redundant communication resources in the global communication network (e.g. multiple crossbar switches or multiple communication links), so that inter CMP module connections can be look-ahead configured to provide time transparency of connection setting [8]. This factor is important for communication at application program level with very small control overheads. The paper presents algorithms for global communication structuring based on heuristic principles, which reduces application program execution time. The algorithms are verified experimentally using a simulator, which executes application program graphs with structured global communication and evaluate their parallel efficiency.

2

The System Architecture

In the paper, we consider parallel multi–CMP systems, whose general structure is presented in Fig. 1. Basic system elements are CMP modules interconnected by a global network. A single CMP module, Fig. 2, consists of a number of processors and shared cache memory modules connected via a local network. A CMP module is implemented as a single integrated circuit. The module contains a number of processor cores (each with its local L1 data cache) and a number of shared L2 data cache banks interconnected through a local network. Additional instruction caches are provided, but they are out of the interest of this paper, so all discussed caches are data caches. Dynamic core clusters can be created around L2 banks using local data exchange networks (L2 buses). All L2 cache

Fig. 1. General system structure

588

E. Laskowski, L  . Ma´sko, and M. Tudruj

Fig. 2. The structure of a CMP module

banks of the CMP module are connected to the local fragment of distributed memory shared by all CMP modules in the system. Tasks in programs are built according a cache–controlled macro data-flow paradigm, so, all data have to be pre-fetched to processor L1 data cache before a task begins and L1 cache reloading is disabled. Operations on data include: data pre-fetch from memory (L2 cache) to a core L1, write from L1 to memory or L2, read on the fly (similar to cache injection) from a L2 bank bus to many L1s in parallel, core switching between L2 buses (clusters). Current task results are sent to the L2 cache memory module only after task completes. This program execution paradigm completely prevents L1 data cache thrashing. New features of the data cache organization consist also in multi–ported structure of L1 data caches (multiple banks are used, which can be connected to many L2 buses). It enables parallel loading of arguments of subsequent numerical operations and many communications (or reads) on the fly performed at a time for a processor (core). The system provides a new mechanism to provide L1 and L2 data caches synchronization in the context of processor switching and reads on the fly. If a processor is switched from one L2 module to another and is to write some of its data from L1 to this new L2 module (for instance to enable other processors to read these data through reads on the fly or simply to do a cache block flush), a respective new line in a target L2 data cache must be provided. This operation is not performed in a standard manner (by transmitting proper data from system memory), but instead, just before the L2 write operation, a dummy empty line in L2 is generated together with a mask, which controls in terms of L1 line blocks validity, which data will be written by the considered data transfer. This line is then filled with new data transferred via a L1–L2 bus. When desired, the operating memory will be updated only with the new validated parts of the L2 lines.

CMP Module System Based on a Look-Ahead Configured Global Network

589

a)

b) Fig. 3. The system based on multiple: a) crossbar switches, b) module communication links

Similar actions are performed in L1 caches, when data are read to L1 to new addresses to comply with the assumed single-assignment principle, which eliminates consistency problems for multiple copies of modified shared data (data read for subsequent modification are stored under new addresses). This imposes, that a new dummy lines in L1 data caches are provided with similar control fields. Only on L1 to L2 flushing or reads on the fly on the L2 buses corresponding L2 dummy lines will be generated (a lazy synchronization is used). More details on the proposed architecture of the CMP module and system execution control but with a single level data caches can be found in [7]. The global interconnection network allows every processor to read or write data to any shared memory module present in the system. This network provides standard data exchange between CMP modules, but at the cost of higher data access latency. The global network can be implemented as an on-board network placed on a backplane with sockets for CMP modules. Due to technology constraints, the latency of global interconnection network will be at least tenfold (or even several dozen times) bigger than the latency of local networks. Thus, proper optimization of global communication can play significant role in overall parallel execution efficiency. In the paper, we consider a global network based on a circuit–switching and message passing of cacheable blocks of data. In such a network, before a message can be sent, the connection between the sender and the receiver has to be created. We assume that both intra– and inter–CMP module communication is located at

590

E. Laskowski, L  . Ma´sko, and M. Tudruj

the application program level. It means that data transfers are not processed by the operating system nor any communication library, so the transfers have very small start-up time and control overheads. In the simplest case, we can assume dynamic on request connection reconfiguration, where each communication in an application program generates a connection request directed to the global network controller. However, we can provide some redundant communication resources in the global network (see Fig. 3), to be able to apply the look-ahead dynamic connection reconfiguration paradigm [8]. It is based on anticipated connection setting in the redundant network resources to eliminate connection creation time overhead. The redundant resources considered in this paper are multiple global link connection switches (crossbar switches, multistage connection networks) and CMP module communication link sets. With the look-ahead dynamic reconfigurability, an application program is divided into communication–disjoint sections. Sections are executed using connections, which were created in advance in some connection switching devices. The connections do not change during execution of the current section. Special inter–module synchronization hardware has to be included in the system to enable execution of many program sections in parallel with the link connection reconfiguration. The application program has to be divided into sections at compile time, as the result of an analysis of the program communication.

3

The Algorithm

In the paper we propose a program structuring algorithm covering global communication in the multiple CMP system. The communication structuring assumes the look-ahead dynamic inter–module connection reconfiguration. The proposed communication structuring method is best suited for circuit–switched interconnection networks with application program level communication, where link creation overhead can not be neglected. Such interconnects include crossbars switches, multistage networks and multibus structures. A two–phase approach is used, which comprizes on one side program graph scheduling onto CMP modules and CMP external communication links, and on the other side program graph partitioning into sections based on the look-ahead created connections between CMP modules. The outcome of the scheduling phase of the algorithm is the scheduled program graph, assigned to computational and communication resources in CMP modules. Global communication is scheduled to the communication link(s) of the CMP modules, but the partitioning into sections and mapping of global communication to redundant communication resources is not defined. The scheduling phase applies the moldable task concept [5]. Moldable tasks (MT) are parallel tasks, for which the number of assigned processors can be variable, but is determined before task execution and then doesn’t change. The scheduling phase consists of three steps:

CMP Module System Based on a Look-Ahead Configured Global Network

591

Fig. 4. The graph partitioning heuristics

1) building a MT data flow graph, based on the program macro data-flow graph, 2) defining the best internal structure of each MT node for each number of processors (schedule of component nodes to logical processors inside CMP modules of different sizes), 3) defining an assignment of resources to MTs (allotment) and scheduling the MT graph in the architecture with simplified inter-CMP connections (fully connected network). In the partitioning phase, scheduled program graph is divided into sections for the look-ahead execution of global communication in the assumed environment. Program graph partitioning into sections is defined using the Communication Activation Graph (CAG). The CAG is extracted from the scheduled program graph and constitutes the basic graph structure for partitioning heuristics. The program graph partitioning heuristics used to find program sections is described in Fig. 4. Besides the partition of the graph, the heuristics assigns a communication resource (e.g. crossbar switch) to each section. The algorithm starts with an initial partition, which consists of sections built of a single communication and assigned to the same crossbar switch. In each step, a vertex of CAG is selected and then the algorithm tries to include this vertex to a union of existing sections determined by edges of the current vertex. The heuristics tries to find such a union of sections, which doesn’t break rules of graph partitioning. The union, which gives the shortest program execution time is selected. See [4] for detailed description of the partitioning heuristics. The program execution time is obtained by simulated execution of the partitioned graph in the presented system. The functioning of the global network, Network Interface Controller and the Global Network Controller is modeled as subgraphs executed on virtual additional processors added to the initial application graph. Weights in the graph nodes correspond to latencies of respective control actions, such as crossbar switch reconfiguration, bus latency, connection setting and activation of program sections. These weights are expressed as control parameters of the simulation.

4

Experimental Results

The presented program optimization algorithms have been verified experimentally and used for evaluation of execution efficiency of programs for different program execution control strategies and system parameters. The results presented in the paper are obtained for four families of synthetic sample program

592

E. Laskowski, L  . Ma´sko, and M. Tudruj

graphs (G-A2, G-A5, G-B2, G-B5), which were randomly generated for the same set of parameters. These graphs have similar granularity to fine–grain numeric applications programs (e.g. Strassen matrix multiplication with two recursion levels). The structure of experimental graph is shown in Fig. 5. The assumed parameters for the program graph generator are the following: the number of levels – 10, the number of subgraphs – 8, the width of a level in a subgraph – 4, the total number of nodes in a graph – 320, the node weight 100–200, the communication edge weight – 20–30 (G-A2, G-B2) or 50–75 (G-A5, G-B5), the node input degree – 1–2 (G-A2, G-A5) or 3–4 (G-B2, G-B5). Up to 100 additional inter–subgraph data transfers have been added to each graph. The initial program graphs were first processed by the task scheduling algorithm based on moldable task approach for execution in the assumed modular system with 32 processors organized to contain 1, 2, 4, 8 CMP modules or 32 single–processor nodes interconnected by a global network. The programs were executed using the look-ahead paradigm in the following system architectural models: system with redundant connection switches (R2 – 2 crossbar switches), system with partitioned module communication link sets (L2 – 2 links in each CMP module/node), and system with single connection switch with on-request reconfiguration (as a reference system). A hardware synchronization was used in the system with multiple crossbar switches. No synchronization was required for model L2. System parameters that were used during experiments: tR – reconfiguration time of a single connection, range 1–400, tV – section activation time overhead, range 2–256. The latency of the global network is assumed to be 2, 8, 16 times bigger than that of the intra–module bus (X2, X8, X16 global network speed coefficient, respectively). Program execution speedup (against sequential execution) for different numbers of CMP modules and different global network latency, for L2 architecture is shown in Fig. 6a – Fig. 6b. The speedup degrades with the increasing number of modules or the latency of the global network. This is due to the increasing volume of global transfers, which are substantially slower than intra–module communication. Thus, in most cases, for optimal program execution the number of CMP modules used for tested graphs should be set as low as possible (the lower limit depends on the efficiency of intra–module communication subsystem). An exception is the G-B5 graph with the biggest volume of global communications,

Fig. 5. Experimental graph structure

CMP Module System Based on a Look-Ahead Configured Global Network

a)

593

b)

Fig. 6. Speedup for L2 architecture as a function of the number of CMP modules: a) G-B5 graph, b) G-B2 graph

for which the execution on 8 CMP modules with slow global network gives better speedup than on 2 or 4 modules (Fig. 6a). It is since a denser global communication in the G-B5 graph is distributed among a bigger number of CMP module external links, thus reducing the possible communication contention on 8 modules links comparing the case of the system with 2 or 4 modules. The number of processor cores in a CMP module is bounded by the current level of the VLSI technology and the kind of the assumed CMP module internal data network (a shared memory bus in the assumed environment). The number of modules used for execution of a parallel program will be governed by a tradeoff between the parallel speedup and the limitations of the CMP module technology. The evaluation of different architectures of communication and reconfiguration control subsystem is shown in Fig. 7 – Fig. 8. The system based on multiple crossbar switches (R2) is able to reduce the reconfiguration time overhead better than L2 architecture. It is confirmed in Fig. 8a by the best reduction

a)

b)

Fig. 7. Average reduction of reconfiguration overhead for L2 architecture as a function of the number of CMP modules: a) tR maximal, tV minimal, b) both tR and tV maximal

594

a)

E. Laskowski, L  . Ma´sko, and M. Tudruj

b)

Fig. 8. Average reduction of reconfiguration overhead for R2 architecture as a function of the number of CMP modules: a) tR maximal, tV minimal, b) both tR and tV maximal

of reconfiguration overhead for X2 global network latency, when the amount of the global communication is the largest. On the other side, the worst speedup of R2 architecture for big value of section activation time overhead (tV parameter) indicates that a system with multiple connection switches employs much more control communication (e.g. barrier synchronization is applied at the end of program section before switching processor links to an other crossbar) than L2 architecture. Thus, an efficient synchronization and section activation subsystem is essential for good performance of the R2 architectural model. The overall assessment of profits coming from the look-ahead reconfiguration in global interconnection networks is shown in Fig. 9. The use of the look-ahead reconfiguration provides a 50% increase of program execution speedup, comparing execution of the same program using an on-request reconfigurable global network. This big increase is obtained when reconfiguration efficiency is low, that is when classical approach does not give good results.

a)

b)

Fig. 9. Speedup increase against on-request reconfiguration as a function of the number of CMP modules: a) L2, both tR and tV maximal, b) R2, tR maximal, tV minimal

CMP Module System Based on a Look-Ahead Configured Global Network

5

595

Conclusions

In the paper, we have discussed the look-ahead dynamic connection reconfiguration in parallel multi–processor system implemented as many CMP modules interconnected by global network. The presented communication and reconfiguration control is supported by program structuring algorithm, based on program graph partitioning. Dynamic connection reconfiguration enables optimization of communication infrastructure according to application program requirements. Experimental results have shown that the systems based on multiple communication resources (link sets or crossbars switches) and optimized reconfiguration control through program graph partitioning into sections give better speedups than classic on-request reconfiguration control, especially when the reconfiguration subsystem is not efficient enough or not adjusted to application communication requirements. To summarize, the lower is the reconfiguration efficiency of the system, the better results are obtained from the application of the look-ahead dynamic connection reconfiguration. Thus, the look-ahead reconfiguration is especially useful for execution of fine-grain parallel programs, which have time–critical reconfiguration requirements.

References 1. Asanovic, K., et al.: A View of the Parallel Computing Landscape. Communications of the ACM 52(10), 56–87 2. Che, S., et al.: A Performance Study of General Purpose Applications on Graphics Processors. In: 1st Workshop on General Purpose Processing on Graphics Processing Units, Boston (October 2007) 3. Koch, K.: Roadrunner System Overview, Los Alamos National Laboratory, http://www.lanl.gov/orgs/hpc/roadrunner/rrinfo/RR%20webPDFs/ RRSystemOversm.pdf 4. Laskowski, E., Tudruj, M.: Efficient Parallel Embedded Computing Through LookAhead Configured Dynamic Inter–Processor Connections. In: 5th Int. Symp. on Parallel Computing in Electrical Engineering, PARELEC 2006, Bialystok, Poland, September 2006, pp. 115–122. IEEE CS, Los Alamitos (2006) 5. Ma´sko, L  ., Dutot, P.–F., Mouni´e, G., Trystram, D., Tudruj, M.: Scheduling Moldable Tasks for Dynamic SMP Clusters in SoC Technology. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Wa´sniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, pp. 879–887. Springer, Heidelberg (2006) 6. Owens, J.D., et al.: Research Challenges for On–Chip Interconnection Networks. In: IEEE MICRO, pp. 96–108 (September-October 2007) 7. Tudruj, M., Ma´sko, L  .: Towards Massively Parallel Computations Based on Dynamic SMP Clusters wih Communication on the Fly. In: 4th Int. Symp. on Parallel and Distrib. Computing, ISPDC 2005, Lille, France, July 2005, pp. 155–162. IEEE CS, Los Alamitos (2005) 8. Tudruj, M.: Look-Ahead Dynamic Reconfiguration of Link Connections in Multi– Processor Architectures. In: Parallel Computing 1995, Gent, pp. 539–546 (September 1995)

Empirical Analysis of Parallelism Overheads on CMPs Ami Marowka Department of Computer Science, Bar-Ilan University, Israel [email protected]

Abstract. OpenMP and Intel Threading Building Blocks (TBB) are two parallel programming paradigms for multicore processors. They have a lot in common but were designed in mind for different parallel execution models. Comparing the performance gain of these two paradigms depends to a great extent on the parallelization overheads of their parallel mechanisms. Parallel overheads are inevitable and therefore understanding their potential costs can help developers to design more scalable applications. This paper presents a comparative study of OpenMP and TBB parallelization overheads. The study was conducted on a dual-core machine with two different compilers, Intel compiler and Microsoft Visual Studio C++ 2008, and shows that Intel compiler outperforms Microsoft compiler. Nevertheless, the relative performance of TBB versus OpenMP is mainly depends on the implementation of the parallel constructs of a specific compiler. Keywords: OpenMP, TBB, Benchmarks, Multicore, Parallel Overhead.

1

Introduction

Multi-core processing has become ubiquitous, from laptops to supercomputers, and everywhere in between. The appearance of Multicore processors brings high performance computing to the desktop and opens the door of mainstream computing for parallel computing. Unfortunately, writing parallel code is more complex than writing serial code. Parallel programming is cumbersome, error prone, and difficult [1,2,3,4,5,6,7,8]. OpenMP [9,10] and Intel Threading Building Blocks (TBB) [11,12] are two parallel programming paradigms for multicore processors. OpenMP and TBB have a lot in common but were designed for different parallel execution models. Both are shared-memory data-parallel programming models and are based on multithreading programming to maximize utilization of multicore processors. However, OpenMP does not free the programmer from most of the tedious issues of parallel programming. The programmer has much to understand, including: the relationship between the logical threads and the underlying physical processors and cores; how threads communicate and synchronize; how to R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 596–605, 2010. c Springer-Verlag Berlin Heidelberg 2010 

Empirical Analysis of Parallelism Overheads on CMPs

597

measure performance in a parallel environment; and the sources of load unbalancing. The programmer must check for dependencies, deadlocks, conflicts, race conditions, and other issues related to parallel programming. On the other hand, the Intel Threading Building Blocks (TBB) is a new high-level library that hides some the issues mentioned above from the programmer and automated the data decomposition and tasks scheduling in an efficient manner. As multi-core architectures continue to evolve, however, they will require developers to refine their threading techniques as a core aspect of their solutions rather than as merely a desirable feature. Overheads associated with operations like thread creation, synchronization and locking, threading granularity, scheduling, and process management will become more pronounced as time goes on, and the necessity of planning for parallel scalability will become more and more important. The contribution of this paper is twofold. First, we present a TBB microbenchmarks suite called TBBench that was developed for benchmarking TBB parallel construct overheads. The design of the TBB micro-benchmark follows the design methodology of the OpenMP EPCC micro-benchmarks [13,14] for enabling accurate comparison between the measured overheads of the two programming models. Second, we use the TBB and OpenMP micro-benchmarks to study the parallelization overheads on a dual-core machine with two different compiler, Intel compiler and Microsoft Visual Studio C++ 2008. We report in details the various results of this study. The rest of this paper is organized as follows. In Section 2, the methodology design of TBBench and its contents are presented. Section 3 presents an in-depth analysis of the running results of OpenMP and TBB benchmarks, and Section 4 concludes the paper.

2

TBBench: A TBB Micro-Benchmarks Suite

TBBench is a suite of core benchmarks for measuring the overheads associated with the execution of TBB parallel constructs for synchronization, scheduling, and work-sharing. The design of TBBench follows the design concepts of the OpenMP EPCC micro-benchmarks suite. In this way, a more accurate comparison between the parallel constructs of the two programming models can be achieved. The approach used by EPCC Micro-benchmarks to measure the overhead associated with a parallel construct is to compare the running time of a regionof-code running in parallel on P cores (Tp ) to the running time of the same region-of-code running sequentially (Ts ). The calculated overhead is given by taking the difference Tp -Ts /p. For example, by measuring the overhead associated with the OpenMP parallel for directive, EPCC Micro-benchmarks measures the time to run the following region-of-code:

598

A. Marowka

f o r ( j =0; j