Encyclopedia of applied and computational mathematics Volume 1. A-K 9783540705284, 9783540705291, 9783540705307, 3540705287

Citation preview

Björn Engquist Editor

Encyclopedia of Applied and Computational Mathematics 1 3Reference

Encyclopedia of Applied and Computational Mathematics

¨ Engquist Bjorn Editor

Encyclopedia of Applied and Computational Mathematics Volume 2 L–Z

With 361 Figures and 33 Tables

Editor Bj¨orn Engquist University of Texas at Austin Austin, TX, USA

ISBN 978-3-540-70528-4 ISBN 978-3-540-70529-1 ISBN 978-3-540-70530-7 (print and electronic bundle) DOI 10.1007/ 978-3-540-70529-1

(eBook)

Library of Congress Control Number: 2015953230 Springer Heidelberg New York Dordrecht London © Springer-Verlag Berlin Heidelberg 2015 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. Printed on acid-free paper Springer-Verlag GmbH Berlin Heidelberg is part of Springer Science+Business Media (www.springer.com)

Preface

The scientific field of applied and computational mathematics has evolved and expanded at a very rapid rate during the last few decades. Many subfields have matured, and it is therefore natural to consider publishing an encyclopedia for the field. Traditional encyclopedias are, however, becoming part of history. For fast, simple, and up-to-date facts they cannot compete with web search engines and web-based versions of the Wikipedia type. For much more extensive and complete treatments of topics, traditional monographs and review articles in specialized journals are common. There is an advantage with web-based articles as in Wikipedia, which constantly evolves and adapts to changes. There is also an advantage with articles that do not change, have known authors, and can be referred to in other publications. With the Encyclopedia for Applied and Computational Mathematics (EACM), we are aiming at achieving the best of these two models. The goal with EACM is a publication with broad coverage by many articles, which are quality controlled through a traditional peer review process. The articles can be formally cited, and the authors can take credit for their contributions. This publication will be frozen in its current form, obviously on paper as well as on the web. In parallel there will be an electronic version, where the authors can make changes to their articles and where new articles will be added. After a couple of years, this dynamic version will result in a publication of a new edition of EACM, which can be referenced while the dynamic electronic version continues to evolve. The length of the articles is also chosen to fill the gap between the common shorter web versions and more specialized longer publications. They are here typically between 5 and 10 pages. A few are introductory overviews and a bit longer than the average article. An encyclopedia will never be complete, and the decision to define the first edition at this time is based on a compromise between the desire of covering the field well and a timely published version. This first edition has 312 articles with the overall number of 1,575 pages in 2 volumes. There are few contributions with animations, which will appear in the electronic version. This will be expanded in the future. The above discussion was about the “E” in EACM. Now we turn to rest of the acronym, “ACM.” Modern applied and computational mathematics is more applied, more computational, and more mathematical than ever. The exponential growth of computational power has allowed for much more complex mathematical models, and these new models are typically more realistic for applications. It is natural to include both applied and computational mathematics in the encyclopedia even though these two fields are philosophically different. In computational mathematics, algorithms are developed and analyzed and may in principle be independent from applications. The two fields are, however, now very tightly coupled in practice.

v

vi

Almost all applied mathematics has some computational components, and most computational mathematics is directly developed for applications. Computation is now mentioned as the third pillar of science together with the classical theory and experiments. Scientific progress of today is often based on critical computational components, which can be seen in the growing contribution from computations in recent Nobel Prizes. The importance of mathematical modeling and scientific computing in engineering is even more obvious. The expression “Computational Science and Engineering” has emerged to describe a scientific field where computations have merged with applications. Classical fields of applied mathematics, for example, asymptotic analysis and homogenization, are today not so often used for achieving quantitative results. They are, however, very important in mathematical modeling and in deriving and understanding a variety of numerical techniques for multiscale simulations. This is explained in different settings throughout EACM. We mentioned above that modern applied and computational mathematics are more mathematical than ever. In the early days, this coupling was natural. Many algorithms that are used today and also discussed in this encyclopedia have the names of Newton and Gauss. However, during the century before the modern computer, mathematics in its pure form evolved rapidly and became more disconnected from applications. The computational tools of pen and paper, the slide rule, and simple mechanical devices stayed roughly the same. We got a clear division between pure and applied mathematics. This has changed with the emergence of the modern computer. Models based on much more sophisticated mathematics are now bases for the quantitative computations and thus practical applications. There are many examples of this tighter coupling between applied and computational mathematics on the one hand and what we regard as pure mathematics on the other. Harmonic analysis is a typical example. It had its origin in applied and computational mathematics with the work of Fourier on heat conduction. However, only very special cases can be studied in a quantitative way by hand. In the years after this beginning, there was substantial progress in pure directions of harmonic analysis. The emergence of powerful computers and the fast Fourier transform (FFT) algorithm drastically changed the scene. This resulted, for example, in wavelets, the inverse Radon transform, a variety of spectral techniques for PDEs, computational information, and sampling theory and compressed sensing. The reader will see illustrative examples in EACM. Partial differential equations have also had a recent development where ideas have bounced back and forth between applications, computations, and fundamental theory. The field of applied and computational mathematics is of course not well defined. We will use the term in a broad sense, but we have not included areas that have their own identity and where applied and computational mathematics is not what you think of even if in a strict sense applied and computational mathematics would be correct. Statistics is the most prominent example. This was an editorial decision. Through the process of producing the EACM, there has been a form of self-selection. When section editors and authors were asked to cover an area or a topic closer to the core of applied and computational mathematics, the success rate was very high. Examples are numerical analysis and inverse problems. Many applied areas are also well covered, ranging from general topics in fluid and solid mechanics to computational aspects of chemistry and the mathematics of atmosphere and ocean science.

Preface

Preface

vii

In fields further from the core, the response was less complete. Examples of the latter are the mathematical aspects of computer science and physics, where the researchers generally do not think of themselves as doing applied and computational mathematics even when they are. The current encyclopedia naturally does not cover all topics that should ideally have their own articles. We hope to fill these holes in the evolving web-based version and then in the future editions. Finally, I would like to thank all section editors and authors for their outstanding contributions and their patience. I also hope that you will continue to improve EACM in its dynamic form and in future editions. Joachim Heinze and Martin Peters at Springer initiated the process when they came with the idea of an encyclopedia. Martin Peters’ highly professional supervision of the development and publication process has absolutely been critical. I am also very grateful for the excellent support from Ruth Allewelt and Tina Shelton at Springer. Austin, USA September 2015

Bj¨orn Engquist

About the Editor

Bj¨orn Engquist received his Ph.D. in Numerical Analysis from Uppsala University in the year 1975. He has been Professor of Mathematics at UCLA, Uppsala University, and the Royal Institute of Technology, Stockholm. He was Michael Henry Strater University Professor of Mathematics and Applied and Computational Mathematics at Princeton University and now holds the Computational and Applied Mathematics Chair I at the University of Texas at Austin. He was Director of the Research Institute for Industrial Applications of Scientific Computing and of the Centre for Parallel Computers at the Royal Institute of Technology, Stockholm. At Princeton University, he was Director of the Program in Applied and Computational Mathematics and the Princeton Institute for Computational Science, and he is now the Director of the ICES Center for Numerical Analysis in Austin. Engquist is a member of the American Association for the Advancement of Science, the Royal Swedish Academy of Sciences, the Royal Swedish Academy of Engineering Sciences, and the Norwegian Academy of Science and Letters. He was a Guggenheim fellow and received the first SIAM Prize in Scientific Computing 1982, the Celsius Medal 1992, the Henrici Prize 2011, the George David Birkhoff Prize in Applied Mathematics 2012, and the ICIAM Pioneer Prize 2015. He was an ICM speaker in the years 1982 and 1998. His research field is development, analysis, and application of numerical methods for differential equations. A particular focus has been multiscale problems and applications to fluid mechanics and wave propagation. He has had 40 Ph.D. students.

ix

Section Editors

Mark Alber Department of Applied and Computational Mathematics and Statistics University of Notre Dame Notre Dame, IN, USA

Ernst Hairer Section de Math´ematiques Universit´e de Gen`eve Gen`eve, Switzerland

Johan H˚astad Royal Insitute of Technology Stockholm, Sweden

xi

xii

Section Editors

Arieh Iserles Department of Applied Mathematics and Theoretical Physics Centre for Mathematical Sciences University of Cambridge Cambridge, UK

Hans Petter Langtangen Simula Research Laboratory Center for Biomedical Computing Fornebu, Norway Department of Informatics University of Oslo, Oslo, Norway

Claude Le Bris Ecole des Ponts – INRIA Paris, France

Christian Lubich Mathematisches Institut University of T¨ubingen T¨ubingen, Germany

Section Editors

xiii

Andrew J. Majda Department of Mathematics and Climate Atmosphere, Ocean Science (CAOS) Courant Institute of Mathematical Sciences New York University New York, NY, USA

Joyce R. McLaughlin Department of Mathematical Sciences Rensselaer Polytechnic Institute Troy, NY, USA

Risto Nieminen School of Science Aalto University Espoo, Finland

J. Tinsley Oden Institute for Computational Engineering and Science The University of Texas at Austin Austin, TX, USA

xiv

Section Editors

Aslak Tveito Simula Research Laboratory Center for Biomedical Computing Fornebu, Norway Department of Informatics University of Oslo, Oslo, Norway

Contributors

´ Assyr Abdulle Mathematics Section, Ecole Polytechnique F´ed´erale de Lausanne (EPFL), Lausanne, Switzerland Andrew Adamatzky Unconventional Computing Centre, University of the West of England, Bristol, UK Todd Arbogast Institute for Computational Engineering and Sciences, University of Texas, Austin, TX, USA Douglas N. Arnold School of Mathematics, University of Minnesota, Minneapolis, MN, USA Simon R. Arridge Department of Computer Science, Center for Medical Image Computing, University College London, London, UK Uri Ascher Department of Computer Science, University of British Columbia, Vancouver, BC, Canada Kendall E. Atkinson Department of Mathematics and Department of Computer Science, University of Iowa, Iowa City, IA, USA Paul J. Atzberger Department of Mathematics, University of California Santa Barbara (UCSB), Santa Barbara, CA, USA Florian Augustin Technische Universit¨at M¨unchen, Fakult¨at Mathematik, Munich, Germany Winfried Auzinger Institute for Analysis und Scientific Computing, Technische Universit¨at Wien, Wien, Austria Owe Axelsson Division of Scientific Computing, Department of Information Technology, Uppsala University, Uppsala, Sweden Institute of Genomics, ASCR, Ostrava, Czech Republic Ruth E. Baker Centre for Mathematical Biology, Mathematical Institute, University of Oxford, Oxford, UK Guillaume Bal Department of Applied Physics and Applied Mathematics, Columbia University, New York, NY, USA Vijay Balasubramanian Department of Physics and Astronomy, Department of Neuroscience, University of Pennsylvania, Philadelphia, PA, USA

xv

xvi

Contributors

Roberto Barrio Departamento de Matem´atica Aplicada and IUMA, University of Zaragoza, Zaragoza, Spain Timothy Barth NASA Ames Research Center, Moffett Field, CA, USA Catherine A.A. Beauchemin Department of Physics, Ryerson University, Toronto, ON, Canada Margaret Beck Department of Mathematics, Heriot-Watt University, Edinburgh, UK Mikhail I. Belishev PDMI, Saint-Petersburg, Russia Alfredo Bellen Department of Mathematics and Geosciences, University of Trieste, Trieste, Italy Ted Belytschko Department of Mechanical Engineering, Northwestern University, Evanston, IL, USA Rafael D. Benguria Departamento de F´ısica, Pontificia Universidad Cat´olica de Chile, Santiago de Chile, Chile Fredrik Bengzon Department of Mathematics and Mathematical Statistics, Ume˚a University, Ume˚a, Sweden Jean–Paul Berrut D´epartement de Math´ematiques, Universit´e de Fribourg, Fribourg/P´erolles, Switzerland ˚ Bj¨orck Department of Mathematics, Link¨oping University, Link¨oping, Sweden Ake Petter E. Bjørstad Department of Informatics, University of Bergen, Bergen, Norway Sergio Blanes Instituto de Matem´atica Multidisciplinar, Universitat Polit`ecnica de Val`encia, Val`encia, Spain Pavel Bochev Computational Albuquerque, NM, USA

Mathematics,

Sandia

National

Laboratories,

Liliana Borcea Department of Mathematics, University of Michigan, Ann Arbor, MI, USA Brett Borden Physics Department, Naval Postgraduate School, Monterey, CA, USA John P. Boyd Department of Atmospheric, Oceanic and Space Science, University of Michigan, Ann Arbor, MI, USA Michal Branicki School of Mathematics, The University of Edinburgh, Edinburgh, UK Claude Brezinski Laboratoire Paul Painlev´e, UMR CNRS 8524, UFR de Math´ematiques Pures et Appliqu´ees, Universit´e des Sciences et Technologies de Lille, Villeneuve d’Ascq, France Hermann Brunner Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Hong Kong SAR, China Department of Mathematics and Statistics, Memorial University of Newfoundland, St. John’s, NL, Canada

Contributors

xvii

Martin Buhmann Mathematisches Institut, Justus-Liebig-Universit¨at, Giessen, Germany Martin Burger Institute for Computational and Applied Mathematics, Westf¨alische Wilhelms-Universit¨at (WWU) M¨unster, M¨unster, Germany John C. Butcher Department of Mathematics, University of Auckland, Auckland, New Zealand Michel Caffarel Laboratoire de Chimie et Physique Quantiques, IRSAMC, Universit´e de Toulouse, Toulouse, France Russel Caflisch UCLA – Department of Mathematics, Institute for Pure and Applied Mathematics, Los Angeles, CA, USA Xing Cai Simula Research Laboratory, Center for Biomedical Computing, Fornebu, Norway University of Oslo, Oslo, Norway Fioralba Cakoni Department of Mathematics, Rutgers University, New Brunswick, NJ, USA Daniela Calvetti Department of Mathematics, Applied Mathematics, and Statistics, Case Western Reserve University, Cleveland, OH, USA Mari Paz Calvo Departamento de Matem´atica Aplicada, Universidad de Valladolid, Valladolid, Spain Eric Canc`es Ecole des Ponts ParisTech – INRIA, Universit´e Paris Est, CERMICS, Projet MICMAC, Marne-la-Vall`ee, Paris, France Fernando Casas Departament de Matem`atiques and IMAC, Universitat Jaume I, Castell´on, Spain Jeff R. Cash Department of Mathematics, Imperial College, London, England Carlos Castillo-Chavez Mathematical and Computational Modeling Sciences Center, School of Human Evolution and Social Change, School of Sustainability, Arizona State University, Tempe, AZ, USA Santa Fe Institute, Santa Fe, NM, USA Isabelle Catto CEREMADE UMR 7534, CNRS and Universit´e Paris-Dauphine, Paris, France ˇ ık Los Alamos National Laboratory, Los Alamos, NM, USA Ondˇrej Cert´ Raymond Chan Department of Mathematics, The Chinese University of Hong Kong, Shatin, Hong Kong Philippe Chartier INRIA-ENS Cachan, Rennes, France Gui-Qiang G. Chen Mathematical Institute, University of Oxford, Oxford, UK Jiun-Shyan Chen Department of Structural Engineering, University of California, San Diego, CA, USA Margaret Cheney Department of Mathematics, Colorado State University, Fort Collins, CO, USA

xviii

Christophe Chipot Laboratoire International Associ´e CNRS, UMR 7565, Universit´e de Lorraine, Vandœuvre-l`es-Nancy, France Theoretical and Computational Biophysics Group, Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana-Champaign, UrbanaChampaign, IL, USA Emiliano Cristiani Istituto per le Applicazioni del Calcolo “Mauro Picone”, Consiglio Nazionale delle Ricerche, Rome, RM, Italy Daan Crommelin Scientific Computing Group, Centrum Wiskunde and Informatica (CWI), Amsterdam, The Netherlands Korteweg-de Vries Institute for Mathematics, University of Amsterdam, Amsterdam, The Netherlands Felipe Cucker Department of Mathematics, City University of Hong Kong, Kowloon Tong, Hong Kong Constantine M. Dafermos Division of Applied Mathematics, Brown University, Providence, RI, USA Eric Darve Mechanical Engineering Department, Institute for Computational and Mathematical Engineering, Stanford University, Stanford, CA, USA Clint N. Dawson Institute for Computational Engineering and Sciences, University of Texas, Austin, TX, USA Ben De Lacy Costello Unconventional Computing Centre, University of the West of England, Bristol, UK Jean-Pierre Dedieu Toulouse, France Paul Dellar OCIAM, Mathematical Institute, Oxford, UK Leszek F. Demkowicz Institute for Computational Engineering and Sciences (ICES), The University of Texas at Austin, Austin, TX, USA Luca Dieci School of Mathematics, Georgia Institute of Technology, Atlanta, GA, USA Bernard Ducomet Departement de Physique Theorique et Appliquee, CEA/DAM Ile De France, Arpajon, France Iain Duff Scientific Computing Department, STFC – Rutherford Appleton Laboratory, Oxfordshire, UK CERFACS, Toulouse, France Nira Dyn School of Mathematical Sciences, Tel-Aviv University, Tel-Aviv, Israel Bo Einarsson Link¨oping University, Link¨oping, Sweden Heinz W. Engl Johann Radon Institute for Computational and Applied Mathematics (RICAM), Austrian Academy of Sciences, Linz, Austria Charles L. Epstein Departments of Mathematics and Radiology, University of Pennsylvania, Philadelphia, PA, USA

Contributors

Contributors

xix

Maria J. Esteban CEREMADE, CNRS and Universit´e Paris-Dauphine, Paris, France Adel Faridani Department of Mathematics, Oregon State University, Corvallis, OR, USA Jean-Luc Fattebert Lawrence Livermore National Laboratory, Livermore, CA, USA Hans Georg Feichtinger Institute of Mathematics, University of Vienna, Vienna, Austria Yusheng Feng NSF/CREST Center for Simulation, Visulization and Real-Time Prediction, The University of Texas at San Antonio, San Antonio, TX, USA David V. Finch Department of Mathematics, Oregon State University, Corvallis, OR, USA Mathias Fink Institut Langevin, ESPCI ParisTech, Paris, France Michael S. Floater Department of Mathematics, University of Oslo, Oslo, Norway Aaron L. Fogelson Departments of Mathematics and Bioengineering, University of Utah, Salt Lake City, UT, USA A.S. Fokas DAMTP Centre for Mathematical Sciences, University of Cambridge, Cambridge, UK Massimo Fornasier Department of Mathematics, Technische Universit¨at M¨unchen, Garching bei M¨unchen, Germany Bengt Fornberg Department of Applied Mathematics, University of Colorado, Boulder, CO, USA Piero Colli Franzone Dipartimento di Matematica “F. Casorati”, Universit`a degli Studi di Pavia, Pavia, Italy Avner Friedman Department of Mathematics, Ohio State University, Columbus, OH, USA Dargan M.W. Frierson Department of Atmospheric Sciences, University of Washington, Seattle, WA, USA Gero Friesecke TU M¨unchen, Zentrum Mathematik, Garching, M¨unich, Germany Martin J. Gander Section de Math´ematiques, Universit´e de Gen`eve, Geneva, Switzerland Carlos J. Garc´ıa-Cervera Mathematics Department, University of California, Santa Barbara, CA, USA Edwin P. Gerber Center for Atmosphere Ocean Science, Courant Institute of Mathematical Sciences, New York University, New York, NY, USA Dimitrios Giannakis Center for Atmosphere Ocean Science (CAOS), Courant Institute of Mathematical Sciences, New York University, New York, NY, USA Amparo Gil Departamento de Matem´atica Aplicada y Ciencias de la Computaci´on, Universidad de Cantabria, E.T.S. Caminos, Canales y Puertos, Santander, Spain

xx

Vivette Girault Laboratoire Jacques-Louis Lions, UPMC University of Paris 06 and CNRS, Paris, France Ingrid Kristine Glad Department of Mathematics, University of Oslo, Oslo, Norway Dominik G¨oddeke Applied Mathematics, TU Dortmund, Dortmund, Germany Serdar G¨oktepe Department of Civil Engineering, Middle East Technical University, Ankara, Turkey Jerzy Gorecki Institute of Physical Chemistry and Warsaw University, Warsaw, Poland Nicholas Ian Mark Gould Scientific Computing Department, Rutherford Appleton Laboratory, Oxfordshire, UK Brian Granger Department of Physics, California Polytechnic State University, San Luis Obispo, CA, USA Frank R. Graziani Lawrence Livermore National Laboratory, Livermore, CA, USA Andrey Gritsun Institute of Numerical Mathematics, Moscow, Russia Martin Grohe Department of Computer Science, RWTH Aachen University, Aachen, Germany Nicola Guglielmi Dipartimento di Matematica Pura e Applicata, Universit`a dell’Aquila, L’Aquila, Italy ¨ Osman Guler Department of Mathematics and Statistics, University of Maryland Baltimore County, Baltimore, MD, USA Jeremy Gunawardena Department of Systems Biology, Harvard Medical School, Boston, MA, USA ¨ Michael Gunther Fachbereich Mathematik und Naturwissenschaften, Bergische Universit¨at Wuppertal, Wuppertal, Germany Max Gunzburger Department of Scientific Computing, Florida State University, Tallahassee, FL, USA Bertil Gustafsson Department of Information Technology, Uppsala University, Uppsala, Sweden Wolfgang Hackbusch Max-Planck-Institut f¨ur Mathematik in den Naturwissenschaften, Leipzig, Germany George A. Hagedorn Department of Mathematics, Center for Statistical Mechanics, Mathematical Physics, and Theoretical Chemistry, Virginia Polytechnic Institute and State University, Blacksburg, VA, USA Ernst Hairer Section de Math´ematiques, Universit´e de Gen`eve, Gen`eve, Switzerland Nicholas Hale Oxford Centre for Collaborative Applied Mathematics (OCCAM), Mathematical Institute, University of Oxford, Oxford, UK

Contributors

Contributors

xxi

Laurence Halpern Laboratoire Analyse, G´eom´etrie and Applications, UMR 7539 CNRS, Universit´e Paris, Villetaneuse, France John Harlim Department of Mathematics and Department of Meteorology, Pennsylvania State University, State College, PA, USA Fr´ed´eric Hecht Laboratoire Jacques-Louis Lions, UPMC University of Paris 06 and CNRS, Paris, France Dieter W. Heermann Institute for Theoretical Physics, Heidelberg University, Heidelberg, Germany Gabor T. Herman Department of Computer Science, The Graduate Center of the City University of New York, New York, NY, USA Jan S. Hesthaven Division of Applied Mathematics, Brown University, Providence, RI, USA Nicholas J. Higham School of Mathematics, The University of Manchester, Manchester, UK Helge Holden Department of Mathematical Sciences, Norwegian University of Science and Technology, Trondheim, Norway Jan Homann Department of Physics and Astronomy, University of Pennsylvania, Philadelphia, PA, USA Kai Hormann Universit`a della Svizzera italiana, Lugano, Switzerland Bei Hu Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, Notre Dame, IN, USA Arne Bang Huseby Department of Mathematics, University of Oslo, Oslo, Norway Daan Huybrechs Department of Computer Science, K.U. Leuven, Leuven, Belgium Victor Isakov Department of Mathematics and Statistics, Wichita State University, Wichita, KS, USA Arieh Iserles Department of Applied Mathematics and Theoretical Physics, Centre for Mathematical Sciences, University of Cambridge, Cambridge, UK Kazufumi Ito Center for Research in Scientific Computation and Department of Mathematics, North Carolina State University, Raleigh, NC, USA Yasushi Ito Aviation Program Group, Japan Aerospace Exploration Agency, Mitaka, Tokyo, Japan Zdzisław Jackiewicz Department of Mathematics and Statistics, Arizona State University, Tempe, AZ, USA Vincent Jacquemet Centre de Recherche, Hˆopital du Sacr´e-Coeur de Montr´eal, Montr´eal, QC, Canada Department of Physiology, Universit´e de Montr´eal, Institut de G´enie Biom´edical and Groupe de Recherche en Sciences et Technologies Biom´edicales, Montr´eal, QC, Canada

xxii

Laurent O. Jay Department of Mathematics, The University of Iowa, Iowa City, IA, USA Shi Jin Department of Mathematics and Institute of Natural Science, Shanghai Jiao Tong University, Shanghai, China Department of Mathematics, University of Wisconsin, Madison, WI, USA Christopher R. Johnson Scientific Computing and Imaging Institute, University of Utah, Warnock Engineering Building, Salt Lake City, UT, USA ¨ Ansgar Jungel Institut f¨ur Analysis und Scientific Computing, Technische Universit¨at Wien, Wien, Austria Rajiv K. Kalia Department of Computer Science, Department of Physics and Astronomy, and Department of Chemical Engineering and Materials Science, University of Southern California, Los Angeles, CA, USA Erich L. Kaltofen Department of Mathematics, North Carolina State University, Raleigh, NC, USA George Em Karniadakis Division of Applied Mathematics, Brown University, Providence, RI, USA Boualem Khouider Department of Mathematics and Statistics, University of Victoria, Victoria, BC, Canada Isaac Klapper Department of Mathematical Sciences and Center for Biofilm Engineering, Montana State University, Bozeman, MT, USA Rupert Klein FB Mathematik and Informatik, Freie Universit¨at Berlin, Berlin, Germany Peter Kloeden FB Mathematik, J.W. Goethe-Universit¨at, Frankfurt am Main, Germany Matthew G. Knepley Searle Chemistry Laboratory, Computation Institute, University of Chicago, Chicago, IL, USA ¨ Huseyin Koc¸ak Department of Computer Science, University of Miami, Coral Gables, FL, USA Jussi T. Koivum¨aki The Center for Biomedical Computing, Simula Research Laboratory, Lysaker, Norway The Center for Cardiological Innovation, Oslo University Hospital, Oslo, Norway Anatoly B. Kolomeisky Department of Chemistry-MS60, Rice University, Houston, TX, USA Natalia L. Komarova Department of Mathematics, University of California Irvine, Irvine, CA, USA Nikos Komodakis Ecole des Ponts ParisTech, Universite Paris-Est, Champs-surMarne, France UMR Laboratoire d’informatique Gaspard-Monge, CNRS, Champs-sur-Marne, France

Contributors

Contributors

xxiii

Alper Korkmaz Department of Mathematics, C¸ankiri Karatekin University, C¸ankiri, Turkey Gunilla Kreiss Division of Scientific Computing, Department of Information Technology, Uppsala University, Uppsala, Sweden Peter Kuchment Mathematics Department, Texas A&M University, College Station, TX, USA ´ M. Pawan Kumar Ecole Centrale Paris, Chˆatenay-Malabry, France ´ Equipe GALEN, INRIA Saclay, ˆIle-de-France, France Angela Kunoth Institut f¨ur Mathematik, Universit¨at Paderborn, Paderborn, Germany Thomas Kurtz University of Wisconsin, Madison, WI, USA Gitta Kutyniok Institut f¨ur Mathematik, Technische Universit¨at Berlin, Berlin, Germany Michael Kwok-Po Ng Department of Mathematics, Hong Kong Baptist University, Kowloon, Hong Kong Hans Petter Langtangen Simula Research Laboratory, Center for Biomedical Computing, Fornebu, Norway Department of Informatics, University of Oslo, Oslo, Norway Mats G. Larson Department of Mathematics and Mathematical Statistics, Ume˚a University, Ume˚a, Sweden Matti Lassas Department of Mathematics and Statistics, University of Helsinki, Helsinki, Finland Chun-Kong Law Department of Applied Mathematics, National Sun Yat-sen University, Kaohsiung, Taiwan Armin Lechleiter Zentrum f¨ur Technomathematik, University of Bremen, Bremen, Germany Sunmi Lee School of Human Evolution and Social Change, Arizona State University, Tempe, AZ, USA Department of Applied Mathematics, Kyung Hee University, Giheung-gu, Yongin-si, Gyeonggi-do, Korea ¨ Ors Legeza Theoretical Solid State Physics, Hungarian Academy of Sciences, Budapest, Hungary Benedict Leimkuhler Edinburgh University School of Mathematics, Edinburgh, Scotland, UK Melvin Leok Department of Mathematics, University of California, San Diego, CA, USA Randall J. LeVeque Department of Applied Mathematics, University of Washington, Seattle, WA, USA Adrian J. Lew Mechanical Engineering, Stanford University, Stanford, CA, USA

xxiv

Mathieu Lewin CNRS and D´epartement de Math´ematiques, Universit´e de CergyPontoise/Saint-Martin, Cergy-Pontoise, France Tien-Yien Li Department of Mathematics, Michigan State University, East Lansing, MI, USA Zhilin Li Center for Research in Scientific Computation and Department of Mathematics, North Carolina State University, Raleigh, NC, USA Knut-Andreas Lie Department of Applied Mathematics, SINTEF ICT, Oslo, Norway Guang Lin Fundamental and Computational Sciences Directorate, Pacific Northwest National Laboratory, Richland, WA, USA School of Mechanical Engineering, Purdue University, West Lafayette, IN, USA Department of Mathematics, Purdue University, West Lafayette, IN, USA Per L¨otstedt Department of Information Technology, Uppsala University, Uppsala, Sweden John S. Lowengrub Department of Mathematics, University of California, Irvine, CA, USA Benzhuo Lu Institute of Computational Mathematics and Scientific/Engineering Computing, Chinese Academy of Sciences, Beijing, China Christian Lubich Mathematisches Institut, Universit¨at T¨ubingen, T¨ubingen, Germany Franz Luef Department of Mathematics, University of California, Berkeley, CA, USA Li-Shi Luo Department of Mathematics and Statistics, Old Dominion University, Norfolk, VA, USA Beijing Computational Science Research Center, Beijing, China Mitchell Luskin School of Mathematics, University of Minnesota, Minneapolis, MN, USA Jianwei Ma Department of Mathematics, Harbin Institute of Technology, Harbin, China Yvon Maday Sorbonne Universit´es, UPMC Univ Paris 06, UMR 7598, Laboratoire Jacques-Louis Lions, Paris, France Institut Universitaire de France and Division of Applied Maths, Brown University, Providence, RI, USA Philip K. Maini Centre for Mathematical Biology, Mathematical Institute, University of Oxford, Oxford, UK Bradley T. Mallison Chevron Energy Technology Company, San Ramon, CA, USA Francisco Marcell´an Departamento de Matem´aticas, Universidad Carlos III de Madrid, Legan´es, Spain

Contributors

Contributors

xxv

Per-Gunnar Martinsson Department of Applied Mathematics, University of Colorado, Boulder, CO, USA Francesca Mazzia Dipartimento di Matematica, Universit`a degli Studi di Bari Aldo Moro, Bari, Italy Robert I. McLachlan Institute of Fundamental Sciences, Massey University, Palmerston North, New Zealand Joyce R. McLaughlin Department of Mathematical Sciences, Rensselaer Polytechnic Institute, Troy, NY, USA Volker Mehrmann Institut f¨ur Mathematik, MA 4-5 TU, Berlin, Germany Jens Markus Melenk Institute for Analysis and Scientific Computing, Vienna University of Technology, Wien, Austria Benedetta Mennucci Department of Chemistry, University of Pisa, Pisa, Italy Roeland Merks Life Sciences (MAC-4), Centrum Wiskunde and Informatica (CWI), Netherlands Consortium for Systems Biology/Netherlands Institute for Systems Biology (NCSB-NISB), Amsterdam, The Netherlands Aaron Meurer Department of Mathematics, New Mexico State University, Las Cruces, NM, USA Juan C. Meza School of Natural Sciences, University of California, Merced, CA, USA Owen D. Miller Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA, USA Graeme Milton Department of Mathematics, The University of Utah, Salt Lake City, UT, USA J.D. Mireles James Department of Mathematics, Rutgers, The State University of New Jersey, Piscataway, NJ, USA Konstantin Mischaikow Department of Mathematics, Rutgers, The State University of New Jersey, Piscataway, NJ, USA Nicolas Mo¨es Ecole Centrale de Nantes, GeM Institute, UMR CNRS 6183, Nantes, France Mohammad Motamed Division of Mathematics and Computational Sciences and Engineering (MCSE), King Abdullah University of Science and Technology, Thuwal, Saudi Arabia Hans Z. Munthe-Kaas Department of Mathematics, University of Bergen, Bergen, Norway Ander Murua Konputazio Zientziak eta A.A. Saila, Informatika Fakultatea, UPV/EHU, Donostia/San Sebasti´an, Spain Aiichiro Nakano Department of Computer Science, Department of Physics and Astronomy, and Department of Chemical Engineering and Materials Science, University of Southern California, Los Angeles, CA, USA

xxvi

Frank Natterer Department of Mathematics and Computer Science, Institute of Computational Mathematics and Instrumental, University of M¨unster, M¨unster, Germany Philip C. Nelson Department of Physics and Astronomy, University of Pennsylvania, Philadelphia, PA, USA Qing Nie Department of Mathematics, University of California, Irvine, CA, USA Harald Niederreiter RICAM, Austrian Academy of Sciences, Linz, Austria H. Frederik Nijhout Duke University, Durham, NC, USA Fabio Nobile EPFL Lausanne, Lausanne, Switzerland Dipartimento di Matematica “F. Brioschi”, Politecnico di Milano, Milan, Italy Sarah L. Noble School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA Clifford J. Nolan Department of Mathematics and Statistics, University of Limerick, Limerick, Ireland Ken-ichi Nomura Department of Computer Science, Department of Physics and Astronomy, and Department of Chemical Engineering and Materials Science, University of Southern California, Los Angeles, CA, USA Jan Martin Nordbotten Department of Mathematics, University of Bergen, Bergen, Norway W.L. Oberkampf Georgetown, TX, USA J. Tinsley Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin, Austin, TX, USA Roger Ohayon Structural Mechanics and Coupled Systems Laboratory, LMSSC, Conservatoire National des Arts et M´etiers (CNAM), Paris, France Luke Olson Department of Computer Science, University of Illinois at UrbanaChampaign, Urbana, IL, USA Sheehan Olver School of Mathematics and Statistics, The University of Sydney, Sydney, NSW, Australia Robert O’Malley Department of Applied Mathematics, University of Washington, Seattle, WA, USA Ahmet Omurtag Bio-Signal Group Inc., Brooklyn, NY, USA Department of Physiology and Pharmacology, State University of New York, Downstate Medical Center, Brooklyn, NY, USA Christoph Ortner Mathematics Institute, University of Warwick, Coventry, UK Alexander Ostermann Institut f¨ur Mathematik, Universit¨at Innsbruck, Innsbruck, Austria Jos´e-Angel Oteo Departament de F´ısica Te`orica, Universitat de Val`encia, Val`encia, Spain

Contributors

Contributors

xxvii

Hans G. Othmer Department of Mathematics, University of Minnesota, Minneapolis, MN, USA Gianluca Panati Dipartimento di Matematica, Universit di Roma “La Sapienza”, Rome, Italy Alexander Panfilov Department of Physics and Astronomy, Gent University, Gent, Belgium Mateusz Paprocki refptr.pl, Wroclaw, Poland Nikos Paragios Ecole des Ponts ParisTech, Universite Paris-Est, Champs-surMarne, France ´ Ecole Centrale Paris, Chˆatenay-Malabry, France ´ Equipe GALEN, INRIA Saclay, ˆIle-de-France, France Lorenzo Pareschi Department of Mathematics, University of Ferrara, Ferrara, Italy John E. Pask Lawrence Livermore National Laboratory, Livermore, CA, USA Geir K. Pedersen Department of Mathematics, University of Oslo, Oslo, Norway Michele Piana Dipartimento di Matematica, Universit`a di Genova, CNR – SPIN, Genova, Italy Olivier Pinaud Department of Mathematics, Colorado State University, Fort Collins, CO, USA Gernot Plank Institute of Biophysics, Medical University of Graz, Graz, Austria Oxford e-Research Centre, University of Oxford, Oxford, UK Gerlind Plonka Institute for Numerical and Applied Mathematics, University of G¨ottingen, G¨ottingen, Germany Aleksander S. Popel Systems Biology Laboratory, Department of Biomedical Engineering, School of Medicine, The Johns Hopkins University, Baltimore, MD, USA Jason S. Prentice Department of Physics and Astronomy, University of Pennsylvania, Philadelphia, PA, USA Luigi Preziosi Department of Mathematics, Politecnico di Torino, Torino, Italy Andrea Prosperetti Department of Mechanical Engineering, Johns Hopkins University, Baltimore, MD, USA Department of Applied Sciences, University of Twente, Enschede, The Netherlands ´ Serge Prudhomme Department of Mathematics and Industrial Engineering, Ecole Polytechnique de Montr´eal, Montr´eal, QC, Canada Amina A. Qutub Department of Bioengineering, Rice University, Houston, TX, USA Venkat Raman Aerospace Engineering, University of Michigan, Ann Arbor, MI, USA

xxviii

Ronny Ramlau Institute for Industrial Mathematics, Kepler University Linz, Linz, Austria Rakesh Ranjan NSF/CREST Center for Simulation, Visulization and Real-Time Prediction, The University of Texas at San Antonio, San Antonio, TX, USA Thilina Rathnayake Department of Computer Science, University of Moratuwa, Moratuwa, Sri Lanka Holger Rauhut Lehrstuhl C f¨ur Mathematik (Analysis), RWTH Aachen University, Aachen, Germany Stephane Redon Laboratoire Jean Kuntzmann, NANO-D – INRIA Grenoble – Rhˆone-Alpes, Saint Ismier, France Michael C. Reed Department of Mathematics, Duke University, Durham, NC, USA Peter Rentrop Technische Universit¨at M¨unchen, Fakult¨at Mathematik, Munich, Germany Nils Henrik Risebro Department of Mathematics, University of Oslo, Oslo, Norway Philip L. Roe Department of Aerospace Engineering, University of Michigan, Ann Arbor, MI, USA Thorsten Rohwedder Institut f¨ur Mathematik, Technische Universit¨at Berlin, Berlin, Germany Dana Ron School of Electrical Engineering, Tel Aviv University, Tel Aviv, Israel Einar M. Rønquist Department of Mathematical Sciences, Norwegian University of Science and Technology, Trondheim, Norway Jos´e Ros Departament de F´ısica Te`orica and IFIC, Universitat de Val`encia-CSIC, Val`encia, Spain Christopher J. Roy Aerospace and Ocean Engineering Department, Virginia Tech, Blacksburg, VA, USA ¨ Ulrich Rude Department of Computer Science, University Erlangen-Nuremberg, Erlangen, Germany Siegfried M. Rump Institute for Reliable Computing, Hamburg University of Technology, Hamburg, Germany Faculty of Science and Engineering, Waseda University, Tokyo, Japan Ann E. Rundell School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA Weldon School of Biomedical Engineering, Purdue University, West Lafayette, IN, USA Robert D. Russell Department of Mathematics, Simon Fraser University, Burnaby, BC, Canada Lenya Ryzhik Department of Mathematics, Stanford University, Stanford, CA, USA

Contributors

Contributors

xxix

Yousef Saad Department of Computer Science and Engineering, University of Minnesota, Minneapolis, MN, USA Paul E. Sacks Department of Mathematics, Iowa State University, Ames, IA, USA Mikko Salo Department of Mathematics and Statistics, University of Jyv¨askyl¨a, Jyv¨askyl¨a, Finland Bj¨orn Sandstede Division of Applied Mathematics, Brown University, Providence, RI, USA J.M. Sanz-Serna Departamento de Matem´atica Aplicada, Universidad de Valladolid, Valladolid, Spain Murat Sari Department of Mathematics, Pamukkale University, Denizli, Turkey Trond Saue Laboratoire de Chimie et Physique Quantiques, CNRS/Universit´e Toulouse III, Toulouse, France Robert Schaback Institut f¨ur Numerische und Angewandte Mathematik (NAM), Georg-August-Universit¨at G¨ottingen, G¨ottingen, Germany Otmar Scherzer Computational Science Center, University of Vienna, Vienna, Austria Johann Radon Institute for Computational and Applied Mathematics (RICAM), Austrian Academy of Sciences, Linz, Austria Tamar Schlick Department of Chemistry, New York University, New York, NY, USA Reinhold Schneider Institut f¨ur Mathematik, Technische Universit¨at Berlin, Berlin, Germany John C. Schotland Department of Mathematics and Department of Physics, University of Michigan, Ann Arbor, MI, USA Christoph Schwab Seminar for Applied Mathematics (SAM), ETH Z¨urich, ETH Zentrum, Z¨urich, Switzerland Javier Segura Departamento de Matem´aticas, Estad´ıstica y Computaci´on, Universidad de Cantabria, Santander, Spain ´ Eric S´er´e CEREMADE, Universit´e Paris-Dauphine, Paris, France James A. Sethian Department of Mathematics, University of California, Berkeley, CA, USA Mathematics Department, Lawrence Berkeley National Laboratory, Berkeley, CA, USA ¨ Rudiger Seydel Mathematisches Institut, Universit¨at zu K¨oln, K¨oln, Germany Lawrence F. Shampine Department of Mathematics, Southern Methodist University, Dallas, TX, USA Qin Sheng Department of Mathematics, Baylor University, Waco, TX, USA

xxx

Chi-Wang Shu Division of Applied Mathematics, Brown University, Providence, RI, USA Avram Sidi Computer Science Department, Technion – Israel Institute of Technology, Haifa, Israel David J. Silvester School of Mathematics, University of Manchester, Manchester, UK Bernd Simeon Department of Mathematics, Felix-Klein-Zentrum, TU Kaiserslautern, Kaiserslautern, Germany Kristina D. Simmons Department of Neuroscience, University of Pennsylvania, Philadelphia, PA, USA Mourad Sini Johann Radon Institute for Computational and Applied Mathematics (RICAM), Austrian Academy of Sciences, Linz, Austria Ralph Sinkus CRB3, Centre de Recherches Biom´edicales Bichat-Beaujon, Hˆopital Beaujon, Clichy, France Ian H. Sloan School of Mathematics and Statistics, University of New South Wales, Sydney, NSW, Australia Barry Smith Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL, USA Moshe Sniedovich Department of Mathematics and Statistics, The University of Melbourne, Melbourne, VIC, Australia Gustaf S¨oderlind Centre for Mathematical Sciences, Numerical Analysis, Lund University, Lund, Sweden Christian Soize Laboratoire Mod´elisation et Simulation Multi-Echelle, MSME UMR 8208 CNRS, Universite Paris-Est, Marne-la-Vall´ee, France Jan Philip Solovej Department of Mathematics, University of Copenhagen, Copenhagen, Denmark Erkki Somersalo Department of Mathematics, Applied Mathematics and Statistics, Case Western Reserve University, Cleveland, OH, USA Thomas Sonar Computational Mathematics, TU Braunschweig, Braunschweig, Germany Euan A. Spence Department of Mathematical Sciences, University of Bath, Bath, UK Samuel N. Stechmann Department of Mathematics, University of Wisconsin– Madison, Madison, WI, USA Plamen Stefanov Department of Mathematics, Purdue University, West Lafayette, IN, USA Gabriel Stoltz Universit´e Paris Est, CERMICS, Projet MICMAC Ecole des Ponts, ParisTech – INRIA, Marne-la-Vall´ee, France

Contributors

Contributors

xxxi

Arne Storjohann David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, ON, Canada ¨ Mathematical Institute, University of Oxford, Oxford, UK Endre Suli Joakim Sundnes Simula Research Laboratory, Lysaker, Norway Denis Talay INRIA Sophia Antipolis, Valbonne, France Martin A. Tanner Department of Statistics, Northwestern University, Evanston, IL, USA Vladimir Temlyakov Department of Mathematics, University of South Carolina, Columbia, SC, USA Steklov Institute of Mathematics, Moscow, Russia Nico M. Temme Centrum voor Wiskunde and Informatica (CWI), Amsterdam, The Netherlands ´ Tempone Division of Mathematics and Computational Sciences and EngineerRaul ing (MCSE), King Abdullah University of Science and Technology, Thuwal, Saudi Arabia Luis Tenorio Mathematical and Computer Sciences, Colorado School of Mines, Golden, CO, USA Kukatharmini Tharmaratnam Department of Mathematics, University of Oslo, Oslo, Norway Florian Theil Mathematics Institute, University of Warwick, Coventry, UK Franc¸oise Tisseur School of Mathematics, The University of Manchester, Manchester, UK Gaˇsper Tkaˇcik Department of Physics and Astronomy, University of Pennsylvania, Philadelphia, PA, USA Øystein Tr˚asdahl Department of Mathematical Sciences, Norwegian University of Science and Technology, Trondheim, Norway M.V. Tretyakov School of Mathematical Sciences, University of Nottingham, Nottingham, UK Yen-Hsi Tsai Department of Mathematics, Center for Numerical Analysis, Institute for Computational Engineering and Science, University of Texas, Austin, TX, USA Xuemin Tu Department of Mathematics, University of Kansas, Lawrence, KS, USA Stefan Turek Applied Mathematics, TU Dortmund, Dortmund, Germany Gabriel Turinici D´epartement MIDO, CEREMADE, Universit´e Paris-Dauphine, Paris, France Aslak Tveito Simula Research Laboratory, Center for Biomedical Computing, Fornebu, Norway Department of Informatics, University of Oslo, Oslo, Norway

xxxii

Gunther Uhlmann Department of Mathematics, University of Washington, Seattle, WA, USA Erik S. Van Vleck Department of Mathematics, University of Kansas, Lawrence, KS, USA Robert J. Vanderbei Department of Operations Research and Financial Engineering, Princeton University, Princeton, NJ, USA Priya Vashishta Department of Computer Science, Department of Physics and Astronomy, and Department of Chemical Engineering and Materials Science, University of Southern California, Los Angeles, CA, USA ´ Gilles Vilmart D´epartement de Math´ematiques, Ecole Normale Sup´erieure de Cachan, antenne de Bretagne, INRIA Rennes, IRMAR, CNRS, UEB, Bruz, France Gerhard Wanner Section de Math´ematiques, Universit´e de Gen`eve, Gen`eve, Switzerland Andy Wathen Mathematical Institute, Oxford University, Oxford, UK Christian Wieners Karlsruhe Institute of Technology, Institute for Applied and Numerical Mathematics, Karlsruhe, Germany Ragnar Winther Center of Mathematics for Applications, University of Oslo, Oslo, Norway Henryk Wo´zniakowski Department of Computer Science, Columbia University, New York, NY, USA Institute of Applied Mathematics, University of Warsaw, Warsaw, Poland Luiz Carlos Wrobel School of Engineering and Design, Brunel University London, Uxbridge, Middlesex, UK M. Wu Department of Mathematics, University of California, Irvine, CA, USA Christos Xenophontos Department of Mathematics and Statistics, University of Cyprus, Nicosia, Cyprus Dongbin Xiu Department of Mathematics and Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, UT, USA Eli Yablonovitch Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA, USA Chao Yang Computational Research Division, MS-50F, Lawrence Berkeley National Laboratory, Berkeley, CA, USA Robert Young Department of Mathematics, University of Toronto, Toronto, ON, Canada Harry Yserentant Institut f¨ur Mathematik, Technische Universit¨at Berlin, Berlin, Germany

Contributors

Contributors

xxxiii

Ya-xiang Yuan State Key Laboratory of Scientific/Engineering Computing, Institute of Computational Mathematics and Scientific/Engineering Computing, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, P.R. China Yong-Tao Zhang Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, Notre Dame, IN, USA Ding-Xuan Zhou Department of Mathematics, City University of Hong Kong, Hong Kong, China Ting Zhou Department of Mathematics, Northeastern University, Boston, MA, USA Tarek I. Zohdi Department of Mechanical Engineering, University of California, Berkeley, CA, USA Enrique Zuazua BCAM – Basque Center for Applied Mathematics, Bilbao, Basque Country, Spain Ikerbasque – Basque Foundation for Science, Bilbao, Basque Country, Spain

A

problems based on partial differential equations necessarily produces approximations that are in error when compared to the exact solutions. Methods to estimate discretization errors were proposed as early as the Serge Prudhomme 1970s [3] and initially focused on developing error estiDepartment of Mathematics and Industrial mators in terms of global (energy) norms (subdomain´ Engineering, Ecole Polytechnique de Montr´eal, residual methods, element residual methods, etc., see Montr´eal, QC, Canada [1,4,22] and references therein). One issue in those approaches is that they provide error estimates in abstract norms, which fail to inform the users about specific Synonyms quantities of engineering interest or local features of the solutions. It is only in the mid-1990s that a new type Adjoint-based method; Dual-weighted residual of error estimators was developed, usually referred to method; Goal-oriented error estimation as dual-weighted residual [6,8,9] or goal-oriented error estimators [15, 18], based on the solution of adjoint problems associated with user-defined quantities of inShort Description terest. In this case, the user is able to specify quantities A posteriori error estimation for quantities of interest of interest, written as functionals defined on the space is concerned with the development of computable es- of admissible solutions, and to assess the accuracy of timators of approximation errors (due to discretization the approximations in terms of these quantities. and/or model reduction) measured with respect to userdefined quantities of interest that are functionals of the Model Problem, Quantities of Interest, solutions to initial boundary-value problems.

A Posteriori Error Estimates of Quantities of Interest

and Adjoint Problem

Description A posteriori error estimation for quantities of interest is the activity in computational sciences and engineering that focuses on the development of computable estimators of the error in approximations of initialand/or boundary-value problems measured with respect to user-defined quantities of interest. The use of discretization methods (such as finite element and finite volume methods) to approximate mathematical

For the sake of simplicity in the exposition, we consider a linear boundary-value problem defined on an open bounded domain   Rd ; d D 1; 2, or 3, with boundary @. Assume that the boundary is decomposed into two parts, D and N , on which Dirichlet and Neumann boundary conditions are prescribed, respectively. Let U and V be two Hilbert spaces. The weak formulation of an abstract linear problem reads: Find u 2 U such that B .u; v/ D F .v/ ; 8v 2 V (1)

© Springer-Verlag Berlin Heidelberg 2015 B. Engquist (ed.), Encyclopedia of Applied and Computational Mathematics, DOI 10.1007/978-3-540-70529-1

2

A Posteriori Error Estimates of Quantities of Interest

where B.; / is a bilinear form on U  V and F ./ is a linear form on V . We suppose that B.; / and F ./ satisfy the hypotheses of the generalized Lax-Milgram Theorem to ensure that there exists a unique solution to the above problem. The goal of computer simulations is not necessarily to accurately approximate the solution u everywhere in the domain, but rather to predict certain quantities of the solution u. Quantities may be local averages of the solution, point-wise values (if u is sufficiently smooth), or local values of the gradient of u in some given direction. Let us suppose that the quantity of interest can be formulated as the linear functional Q: U ! R such that Z Z k .x/ u .x/ dx C kN .x/ u .x/ ds Q .u/ D 

N

and u0 denotes the first derivative of u, subjected to the Dirichlet boundary condition u D 0 on @. We suppose that one is interested in evaluating u.x0 /; x0 2 . In this case, U D V D H01 ./ and Z Q .u/ D u .x0 / D

ı .x  x0 / u .x/ dx

(6)



Z F .v/ D

f .x/ v .x/ dx

(7)



Z

u0 .x/ v 0 .x/ dx

B .u; v/ D

(8)



where ı is the Dirac function. For this quantity of interest, the adjoint solution z is called the Green’s function (often denoted by G.x; x0 /) and allows one to calculate u at point x0 in terms of the loading term (2) f , i.e.,

where k and kN represent two kernel functions (sometimes referred to as extractors) defined on  and N , respectively, that are introduced in order to be able to consider local quantities. For example, the local average of u in a subdomain !   can be evaluated by choosing k as the characteristic function: ( Z 1 1 if x 2 ! k .x/ D k .x/ dx D 1 with j!j 0 otherwise  (3)

Z u .x0 / D Q .u/ D F .z/ D

f .x/ G .x; x0 / dx (9) 

The strong form of the adjoint problem reads in this case: .K T z0 /0 D ı.x  x0 / in , subjected to the boundary condition z D 0 on @. We observe here that the differential operator associated with the adjoint problem is the same as that of the primal problem whenever the tensor K is symmetric.

Most quantities of interest frequently encountered in Goal-Oriented Error Estimation applications can be written in the above form. With a given quantity of interest, let us introduce the We suppose that the solution u of the primal problem following problem in weak form: cannot be computed exactly and must be approximated by a discretization method such as the finite difference Find z 2 V such that B .v; z/ D Q .v/ ; 8v 2 U (4) or finite element methods. Denoting by h and p the size and polynomial degree of the finite elements, let This problem is called the dual or adjoint problem U h;p  U and V h;p  V be conforming finite element and its solution z 2 V is referred to as the adjoint subspaces of U and V , respectively, with dim U h;p D solution, the dual solution, the influence function, or dim V h;p . Using the Galerkin method, a finite element the generalized Green’s function. We emphasize here approximation uh;p to the primal problem (1) is given that the adjoint solution z to Problem (4) is unique by the following discrete problem: as long as the linear quantity Q./ is bounded. A fundamental observation using the primal problem (1) Find uh;p 2 U h;p such that with v D z and the adjoint problem (4) with v D u is     B uh;p ; v h;p D F v h;p ; 8v h;p 2 V h;p (10) that Q .u/ D B .u; z/ D F .z/ (5) We denote by e 2 U the error in uh;p , i.e., e D u  Example 1 (Green’s function) Let  D .0; 1/  R. uh;p , and suppose that one is interested in evaluating We consider the problem of finding u that satisfies the quantity Q.u/: In other words, we aim at estimating .Ku0 /0 D f in , where K is a two-by-two tensor the error quantity:

A Posteriori Error Estimates of Quantities of Interest

  E D Q .u/  Q uh;p

3

(11) An estimate of the error is then provided by

Using the adjoint problem (4) and the primal problem (1), the error in the quantity of interest can be represented as

      E D R uh;p I zQ C R uh;p I z  zQ  R uh;p I zQ WD  (16)

Remark 1 Some error estimators have been proposed that consider the approximate solution zh;p to (14) h;p in order to get and estimate  h;p the error "  z  z h;p (12) E  R u I " or simply, E  B.u  u ; "/. See for example [15, 17–19].   where R uh;p I  is the residual functional associated Remark 2 The goal-oriented error estimation procewith the primal problem. From the discrete problem dure presented so far can be easily extended to linear (10), one can also straightforwardly derive the so- initial boundary-value problem. In this case, the adjoint called orthogonality property problem is a problem that is solved backward in time. 

 E D B .u; z/  B uh;p ; z     D F .z/  B uh;p ; z WD R uh;p I z

    R uh;p I v h;p D F uh;p    B uh;p ; v h;p D 0;

v h;p 2 V h;p (13)

which states that the solution uh;p is in some sense the “best approximation” of u in U h;p . From the error representation (12), it is clear that the error in the quantity of interest could be obtained if the adjoint solution z were known. Unfortunately, the adjoint problem (4) cannot be solved exactly and only on rare occasions is an analytical solution available. The idea is thus to compute a discrete (finite element) approximation zQ of the adjoint solution z. If one considers the finite element solution zh;p on space VQ D V h;p , with test functions vQ 2 UQ D U h;p (i.e., using the same finite element spaces as for uh;p ), i.e., by solving the problem

In order to capture errors due to the spatial and temporal discretization, the adjoint problem is approximated by halving the mesh size and time step.

Adaptive Strategies   The estimator  D R uh;p I zQ can be used for mesh adaptation. Let  h;p zQ be a projection of zQ on V h;p . Thanks to the orthogonality property,   the estimate is equivalent to  D R uh;p I zQ   h;p zQ . The objective is then to decompose  into element-wise contributions. Recalling the definition of the residual, one has    D R uh;p I zQ   h;p zQ     D F zQ   h;p zQ  B uh;p I zQ   h;p zQ

(17)

Because F ./ and B.; / are defined as integrals over the whole computational domain , they can be h;p h;p decomposed into a sum of contributions FK ./ and Find z 2 V such that BK .; / on each element of the mesh. It follows that  h;p   h;p h;p  DQ v ; 8v h;p 2 U h;p (14) B v ;z X     D FK zQ   h;p zQ  BK uh;p I zQ   h;p zQ then it is straightforward to show K  from the orthogonal X ity property that R uh;p I zh;p D 0. In other words, WD K (18) such an approximation of the adjoint solution would K fail to bring sufficient information about the error in the quantity of interest. It implies that the adjoint needs The quantities K define contributions on each element to be approximated on a discretization vector space K to the error in the quantity of interest. The represenfiner than V h;p . In practice, one usually selects VQ D tation of these contributions is not unique as one can V h=2;p ; VQ D V h;pC1 , or even VQ D V h=2;pC1 (and actually integrate by parts the terms BK to introduce similarly for UQ ) to get the approximation: interior residuals (with respect to the strong form of the differential equation inside each element) and jump Find zQ 2 VQ such that B .v; Q zQ/ D Q .v/ Q ; 8vQ 2 UQ residuals (with respect to solution fluxes across the (15) interfaces of the elements). We do not present here

A

4

A Posteriori Error Estimates of Quantities of Interest

the details of the different representations as these are where we have assumed that Q and B are differentiable problem-dependent. with respect to u, i.e., One can use the contributions K to determine     refinement indicators in an adaptive mesh refinement  h;p  Q uh;p C v  Q uh;p 0 Q u I v D lim (AMR) strategy. Actually, there exist several methods  !0  for element marking, such as the maximum strategy,  h;p   h;p  the fixed fraction strategy, the equidistribution strategy, B 0 uh;p I v; z D lim B u C vI z  B u I v  !0  etc. In the case of the maximum contribution method, (22) one considers the refinement indicator K on each element K; 0  K  1, as: and Q and B denote higher-order terms due to jK j K WD (19) the linearization of Q and B, respectively. In the maxK jK j above error representation, we have also introduced the All elements such that K  ˛ can then be marked for adjoint problem: refinement (with respect to h, p; or to h and p), where ˛ is a user-defined tolerance chosen between zero and Find z 2 V such that unity. In practice, the parameter is usually chosen to be (23)     B 0 uh;p I v; z D Q0 uh;p I v ; 8v 2 U ˛ D 0:5. One area of active research in AMR deals with the theoretical analysis of adaptive methods, the objective It is important to note that the adjoint problem is a being to show whether the adaptive methods ensure linear problem in z, which makes it easier to solve than convergence and provide optimal meshes [13, 14]. the primal problem. Proceeding as in the linear case, one can solve for an approximate solution zQ 2 VQ to the adjoint problem and derive the error estimator  Extension to Nonlinear Problems Goal-oriented error estimators, originally defined for linear boundary-value problems and linear quantities of interest, have been extended to the case of nonlinear problems and nonlinear quantities of interest. Let u 2 U be the solution of the nonlinear problem:

  E D Q .u/  Q uh;p     D R uh;p I zQ C R uh;p I z  zQ C     R uh;p I zQ WD 

(24)

As before, the estimator  can be decomposed into Find u 2 U such that B .uI v/ D F .v/ ; 8v 2 V element-wise contributions K for mesh adaptation. (20) where B.I / is a semilinear form, possibly nonlinear with respect to the first variable. Suppose also that one Concluding Remarks is interested in the nonlinear quantity of interest Q.u/ and that uh;p is a finite element approximation of the Goal-oriented error estimation is a topic that, to date, is solution u to (20). Then, by linearization, fairly well understood. It has actually been extended to modeling error estimation, where the modeling error  h;p  E D Q .u/  Q u is the difference between the solutions of two differ  ent models [10, 16], and has been applied to numer0 h;p h;p D Q u Iu  u C Q ous applications of engineering and scientific interests   (solid mechanics [21], fluid mechanics [19], wave D B 0 uh;p I u  uh;p ; z C Q (21) phenomena[5], Cahn-Hilliard equations [23], multi  D B.uI z/  B uh;p I z C Q  B scale modeling [7, 20], partial differential equations   with uncertain coefficients [2, 11, 12] etc.). The main D F .v/  B uh;p I z C Q  B challenge in goal-oriented error estimation essentially   WD R uh;p I z C Q  B lies in the determination of approximate solutions of

A Priori and A Posteriori Error Analysis in Chemistry

5

the adjoint problem that provide for accurate and re- 16. Oden, J.T., Prudhomme, S.: Estimation of modeling error in computational mechanics. J. Comput. Phys. 182, 496–515 liable error estimators while being cost-effective from (2002) a computational point of view. Another challenge in 17. Paraschivoiu, M., Peraire, J., Patera, A.T.: A posteriori finite the case of nonlinear problems is the design of adapelement bounds for linear-functional outputs of elliptic partial differential equations. Comput. Methods Appl. Mech. tive methods that simultaneously control discretization Eng. 150(1–4), 289–312 (1997) errors and linearization errors.

References 1. Ainsworth, M., Oden, J.T.: A Posteriori Error Estimation in Finite Element Analysis. Wiley, New York (2000) 2. Almeida, R.C., Oden, J.T.: Solution verification, goaloriented adaptive methods for stochastic advectiondiffusion problems. Comput. Methods Appl. Mech. Eng. 199(37–40), 2472–2486 (2010) 3. Babuˇska, I., Rheinboldt, W.C.: Error estimates for adaptive finite element computations. SIAM J. Numer. Anal. 15(4), 736–754 (1978) 4. Babuˇska, I., Strouboulis, T.: The Finite Element Method and Its Reliability. Oxford University Press, New York (2001) 5. Bangerth, W., Rannacher, R.: Adaptive finite element techniques for the acoustic wave equation. J. Comput. Acoust. 9(2), 575–591 (2001) 6. Bangerth, W., Rannacher, R.: Adaptive Finite Element Methods for Differential Equations. Lectures in Mathematics. ETH Z¨urich, Birkh¨auser (2003) 7. Bauman, P.T., Oden, J.T., Prudhomme, S.: Adaptive multiscale modeling of polymeric materials with Arlequin coupling and Goals algorithms. Comput. Methods Appl. Mech. Eng. 198, 799–818 (2009) 8. Becker, R., Rannacher, R.: A feed-back approach to error control in finite element methods: basic analysis and examples. East West J. Numer. Math. 4, 237–264 (1996) 9. Becker, R., Rannacher, R.: An optimal control approach to a posteriori error estimation in finite element methods. Acta Numer. 10, 1–102 (2001) 10. Braack, M., Ern, A.: A posteriori control of modeling errors and discretization errors. Multiscale Model. Simul. 1(2), 221–238 (2003) 11. Butler, T., Dawson, C., Wildey, T.: A posteriori error analysis of stochastic spectral methods. SIAM J. Sci. Comput. 33, 1267–1291 (2011) 12. Mathelin, L., Le Maˆıtre, O.: Dual-based a posteriori error estimate for stochastic finite element methods. Commun. Appl. Math. Comput. Sci. 2(1), 83–115 (2007) 13. Mommer, M., Stevenson, R.P.: A goal-oriented adaptive finite element method with convergence rates. SIAM J. Numer. Anal. 47(2), 861–886 (2009) 14. Nochetto, R.H., Siebert, K.G., Veeser, A.: Theory of adaptive finite element methods: An introduction. In: DeVore, R.A., Kunoth, A. (eds.) Multiscale, Nonlinear and Adaptive Approximation, pp. 409–542. Springer, Berlin (2009) 15. Oden, J.T., Prudhomme, S.: Goal-oriented error estimation and adaptivity for the finite element method. Comput. Math. Appl. 41, 735–756 (2001)

18. Prudhomme, S., Oden, J.T.: On goal-oriented error estimation for elliptic problems: application to the control of pointwise errors. Comput. Methods Appl. Mech. Eng. 176, 313–331 (1999) 19. Prudhomme, S., Oden, J.T.: Computable error estimators and adaptive techniques for fluid flow problems. In: Barth, T.J., Deconinck, H. (eds.) Error Estimation and Adaptive Discretization Methods in Computational Fluid Dynamics. Lecture Notes in Computational Science and Engineering, vol. 25, pp. 207–268. Springer, Heidelberg (2003) 20. Prudhomme, S., Chamoin, L., ben Dhia, H., Bauman, P.T.: An adaptive strategy for the control of modeling error in two-dimensional atomic-to-continuum coupling simulations. Comput. Methods Appl. Mech. Eng. 198(21–26), 1887–1901 (2001) 21. Stein, E., R¨uter, M.: Finite element methods for elasticity with error-controlled discretization and model adaptivity. In: Stein, E., de Borst, R., Hughes, T.J.R. (eds.) Encyclopedia of Computational Mechanics. Solids and Structures, vol. 2, chapter 2, pp. 5–58. Wiley (2004) 22. Verf¨urth, R.: A Posteriori Error Estimation Techniques for Finite Element Methods. Oxford University Press, Oxford (2013) 23. van der Zee, K.G., Oden, J.T., Prudhomme, S., HawkinsDaarud, A.J.: Goal-oriented error estimation for CahnHilliard models of binary phase transition. Numer. Methods Partial Differ. Equ. 27(1), 160–196 (2011)

A Priori and A Posteriori Error Analysis in Chemistry Yvon Maday Sorbonne Universit´es, UPMC Univ Paris 06, UMR 7598, Laboratoire Jacques-Louis Lions, Paris, France Institut Universitaire de France and Division of Applied Maths, Brown University, Providence, RI, USA

Synonyms Convergence analysis; Error estimates; Guaranteed accuracy; Refinement

A

6

Definition For a numerical discretization chosen to approximate the solution of a given problem or for an algorithm used to solve the discrete problem resulting from the previous discretization, a priori analysis explains how the method behaves and to which extent the numerical solution that is produced from the discretization/algorithm is close to the exact one. It also allows to compare the numerical method of interest with another one. With a priori analysis though, there is no definite certainty that a given computation provides a good enough approximation. It is only when the number of degrees of freedom and the complexity of the computation is large enough that the convergence of the numerical method can be guaranteedly achieved. On the contrary, a posteriori analysis provides bounds on the error on the solution or the output coming out of the simulation. The concept of a posteriori analysis can even contain an error indicator that informs the user on the strategy he should follow in order to improve the accuracy of its results by increasing the number of degrees of freedom in case where the a posteriori estimation is not good enough.

Overview Computational chemistry is a vast field including a variety of different approaches. At the root, the Schr¨odinger equation plays a fundamental role since it describes the behavior of matter, at the finest level, with no empirical constant or input. However, almost no simulation is based on the resolution of this equation since it is exceedingly expensive to solve for more than about ten atoms. The reason is that the wave function that describes at this level the state of matter is a time-dependent function of 3  .N C M / variable when a molecule with M nucleons and N electrons all around is to be simulated. Many approaches have been proposed to circumvent this impossibility to build a numerical simulation well suited for these equations. The first element takes into account the fact that the understanding of the state of the matter at the ground state, i.e., at the state of minimal energy, is already a valuable information out of which the calculation of excited states or unsteady solutions comes as a second step. This allows to get rid of the time dependency in these solution. The second element,

A Priori and A Posteriori Error Analysis in Chemistry

known as the Born-Oppenheimer approximation, is based on a dimensional analysis that allows somehow to decouple, among the particles that are in presence, the heaviest ones (the nucleons) from the lightest ones (the electrons); see the entry  Born–Oppenheimer Approximation, Adiabatic Limit, and Related Math. Issues. In this approximation, the behavior of the electrons is considered, given a fixed state of the nuclei, while the analysis of the behavior of the nucleons is done in a frame where the interaction with the electron is replaced by a potential in which the nucleons evolve. As for the analysis of the behavior of the electrons, the density functional theory is nowadays widely used since the seminal work of Hohenberg and Kohn [12] that establishes a one-toone correspondence between the ground state electron density and the ground state wave function of a manyparticle system. This is the archetype of ab initio approximations for electronic structures. Another approach is the Hartree-Fock and post-Hartree-Fock approximation, where the electronic wave function is sought as minimizing the ground state Schr¨odinger equation under the constraint of being a (sum of) Slater determinant(s) of one-particle orbitals. Invented by Dirac and Heisenberg, these determinants appear as a simple way to impose the antisymmetric property of the exact solution, resulting from the Pauli exclusion principle. Time dependance can be restored in these models to give rise to time-dependent Hartree or Hartree-Fock approximations. When the behavior of the electrons is understood, the analysis of the nucleons can then be based on molecular mechanics or dynamics, where the quantum electronic information is aggregated into force fields and the charged nuclei move in these force fields. Whatever model approximation is used, from ab initio approaches to empirical ones where models are added on the top of the Schr¨odinger equation, the problems are still challenging for the computation. Indeed, the complexity due to the number of variables of the N C M –body wave function solution of the linear Schr¨odinger model is replaced by complex and large nonlinearities in the resulting equations, the precise formula of which is actually not known in the DFT framework (we refer to  Density Functional Theory). The most common implementation is based on the Kohn-Sham model [14]. Note that the most accurate DFT calculations employ semi-empirical corrections, whose functional form is usually derived from a lot of

A Priori and A Posteriori Error Analysis in Chemistry

know-how and depends on parameters that need to be properly tuned up. The a priori and a posteriori analysis allows at this stage to quantify the quality of the actual model that is chosen as a surrogate to the Schr¨odinger equations. Let us denote as IF.U / D 0 the Schr¨odinger model and U the associated solution for conveniency and by F .u/ D 0 the chosen model with associated solution u, that is, a (set of) function(s) in three variables and from which informations about U can be reconstructed. This first stage of (modeling) approximation is complemented with a second step related to numerical implementations on computer that actually involves two associated stages of approximation. The first one, standard for approximating solution of partial differential equations, corresponds to the discretization of functions of three variables in IR by discrete functions depending on a finite (and generally high) number of degrees of freedom the most common candidates are found in the family of finite element or spectral (plane waves) discretizations when variational framework is present (generally preferred as it represents well the minimization of the associated energy traducing the ground state that is searched) or in the family of finite difference discretizations. We denote by Fı .uı / D 0 the associated discretization with uı being a function belonging to a finite dimensional space. The a priori and a posteriori analysis allows here to quantify the quality of the numerical discretization for the particular model that is considered. The second stage associated with approximation is related to the algorithms that are used in order to solve the finite dimensional problem that comes out of the discretization method. An example that illustrates this feature is provided by the way the nonlinearities of the model are treated since computers can only solve a linear system of finite dimension. Classically, this imply the use of iterative solvers, based on fixed point strategies. In order to illustrate this point, we first have to indicate that the problem to be solved is nonlinear. This can be done by writing the problem under the form Fı .uı / D F .uı I uı /; the iterative solver then consists in computing uı as the limit, when the iteration parameter k tends to infinity, of ukı solution of the fixed point iterative procedure F .ukı I uk1 ı / D 0. In order to provide a precise enough approximation in a small enough time, the algorithm that is used for these iterations must be smart enough and stopped at the right number of iterations so that the ukı is close enough to its limit uı . Here again, a priori

7

and a posteriori analysis allows to be confident in the convergence of the algorithm and the stopping criteria. Numerical analysis is involved in the three stages above in the approximation process and requires different pieces of analysis. All this is quite recent work and only very partially covered. We present in the following sections some details of the existing results. We refer also to  Numerical Analysis of Eigenproblems for Electronic Structure Calculations for a particular focus on a priori error analysis for nonlinear eigenproblems.

Error on the Model On the a priori side, the number of existing work is very small. In [11] for instance, the author considers the approximation by Hartree Fock (we refer to  Hartree–Fock Type Methods) and post-HartreeFock approaches of the Schr¨odinger equation (we refer to  Post-Hartree-Fock Methods and Excited States Modeling) – CI (configuration interaction) and MCSCF (multiconfiguration, self-consistent field) methods. It is proven that such an approach, even if it is based on an expansion on many Slater determinant, is never exact. However, MC-SCF methods approximate energies correctly and also wavefunctions, in the limit where the number K of Slater determinants goes to infinity. An actual quantification of the decay rate of the difference between the MC-SCF energy based on K Slater determinants and the exact quantum-mechanical ground state energy as K becomes large is quoted as an open question in this entry and is still as far as we know. For a more complete analysis, where the excited states are also considered, see [17]. Another piece of a priori analysis is presented in [9] and deals with the convergence of the timedependent Hartree problem, in which the solution to the Schr¨odinger equation is searched as a sum of K tensorial products known as the MCTDH approximation. The a priori analysis of the difference between the exact solution u.t/ (to the Schr¨odinger equation) and the discrete solution uK .t/ states that the error is upper bounded by the sum of the best approximation error (of u.t/ by a sum of K tensorial products) and a linear in time growing contribution ct " where " measures in some sense the best approximation error in the residual (Schr¨odinger equation) norm (close to a H 2 -type norm). We refer to [9] for the precise statement of the analysis.

A

8

As far as we know, there exists currently no result of a posteriori type in the literature on the evaluation of the error on the model. Yet, we can remind that such an a posteriori analysis exists in other contexts (see, e.g., [4] for a coupling of two models: one full and one degenerated that are used in different regions) leading to the idea that this is a feasible mathematical and numerical tool that would be very helpful in the present context.

Error on the Discretization The analysis of the discretization error is certainly the one that has been the most considered in the literature, even if the current results in the chemistry field are mostly very recent and still partial. There are essentially three types of discretizations differing from the basis sets that are chosen to approximate the molecular orbitals or the density functional and from the formulation of the discrete problem: (i) those based on a strong formulation of the equations (finite difference methods; see, e.g., the entry  Finite Difference Methods) where we are not aware of any full numerical analysis justification, (ii) those based on variational approximations either with universal complete basis sets (finite element; see, e.g., the entry  Finite Element Methods for Electronic Structure, plane wave, wavelets methods [13]) or (iii) with linear combination of ad hoc atomic orbitals (LCAO, e.g., of Gaussian basis sets, reduced basis methods [7,19]). For the two first approaches, the universality of the discrete approximation spaces allows to state that there exists a discrete function (e.g., the best fit, i.e., the projection in some appropriated norm) that is as close to the exact solution as required, at least whenever the dimension of the discrete space goes to infinity (known, e.g., for the Hartree-Fock approximation as the Hartree-Fock limit) and provided some regularity exists on the solution. The challenge is then to propose a discrete method able to select a unique solution in the discrete space that is almost as good as the best fit. This challenge is actually quite simple to face in case of a linear problem, but is very difficult if the problem is nonlinear – and as we explained above, the problem in the current context are almost always nonlinear. For the last type of discretization, the basis set is problem dependent and is only built up to approximate the solution of the very problem under consideration.

A Priori and A Posteriori Error Analysis in Chemistry

There are then two challenges: (i) does the best fit in this ad hoc discrete space eventually approximate well the exact solution and (ii) does the discrete method propose a fair enough approximation and again how does it compare with the best fit. Most of the works related to the a priori convergence analysis deal with the second type of approximation above. A summary of these results focusing on the particular case of the computation of the ground state for electronic structures (resulting in the approximation of eigenstates for a nonlinear eigenvalue problem) is presented in the above quoted entry  Numerical Analysis of Eigenproblems for Electronic Structure Calculations and states a complete enough convergence analysis with optimal rate on both the energies and wave functions, provided that the numerical integration rules that are used to compute the integrals stemming out of the variational formulation are computed with a good enough accuracy. We shall thus mainly focus here on the existing results that are not detailed in the above cited entry. For LCAO variational approximations, the choice of the basis defining the discrete space is generally taken as follows: (i) to any atom A of the periodic table, a of nA linearly independent AO is associated,  ˚collection nA 1nnA ; (ii) the discrete basis associated to a given molecule is built up by gathering all AO relative to the atoms in the  ˚ e.g., for the molecule A-B, one ˚ system, chooses

 D 1A .x  xN A /;    ; nAA .x  xN A /I 1B .x  xN B /;    ; nBB .x  xN B /g ; where xN A , xN B denote the respective positions in IR3 of the atoms A and B. After the paper of Boys’ [2], the polynomial Gaussian basis sets have become of standard use for the variational approximations of the solution of the Hartree-Fock equations. What is remarkable – even though a large amount of know-how is required in order to define the proper AOs – is that actually very few AOs are required to yield a very good approximation of the ground state of any molecular system. There exists very few papers dedicated to the a priori convergence; most of the current studies are restricted to the particular case of hydrogenoid solutions, i.e., the solution of the Hydrogen atom, whose analytic expression is known. An example is given by the papers [16] and [3] where exponential convergence is proven. By analogy, these results are extrapolated on molecules, looking at the shape of the cusps that the solution exhibit, and explain somehow the good behavior of these approximations; currently,

A Priori and A Posteriori Error Analysis in Chemistry

no complete analysis exists as far as we are aware of. Based on these results related to the best fit, the numerical analysis of the variational approximation of the ground state for Hartree-Fock or Kohn-Sham equations can proceed. This analysis uses the general paradigm of definition of explicit lower and upper bounds for outputs depending on the solution of a partial differential equation and is explained in [20] for the a posteriori analysis, and latter in [8] for the a priori analysis: optimal results are proven, under a hypothesis stating a kind of local uniqueness of the ground state solution. In a different direction, another problem recently analyzed deals with the approximation of the electronic structure on perturbed lattices of atoms. This model leads to the analysis of the spectrum of perturbations of periodic operators either for the Schr¨odinger equations or the Dirac equations. The periodic operator has a spectrum that is composed of bands (that can be analyzed by Bloch-Floquet theory); the perturbed operator has a spectrum that is composed of the same bands with possibly eigenvalues in the gaps. These localized eigenvalues are of physical interest, and the numerical methods should be able to approximate them well. The problem is that, very often, reasonable enough approximations produce discrete eigenvalues that have nothing to do with exact eigenvalues. These are named as spurious eigenvalues and analyzed in [1, 5, 18] from an a priori point of view. Some constructive approaches have been proposed in these papers to avoid the phenomenon of spurious eigenvalues and get optimal a priori convergence rate for the approximation of true eigenstates both for the wave functions and associated energies.

9

point of view, the correct number of iterations is not known and very few analysis is done in this direction. From the a priori point of view, the only analysis we are aware of is the self-consistent field (SCF) algorithms that has been, for years, the strategy of choice to solve the discretized Hartree-Fock equation (see e.g.,  Self-Consistent Field (SCF) Algorithms). In practice, though, the method has revealed successes and failures both in convergence and in convergence rates. A large amount of literature has proposed various tricks to overcome the lack of robustness of the original Roothaan algorithm. It is reported that this algorithm sometimes converges toward a solution to the HF equations but frequently oscillates between two states. The definitive answer to the questions raised by these convergence problems has been given by a series of papers of Canc`es and coauthors among which [6, 15]. An interesting cycle of order 2 has been identified in the behavior of the algorithm in frequent cases explaining why the algorithm oscillates between two states, none of which being a solution of the original nonlinear problem. The simple addition of a penalty term in the same spirit as a basic level shift or DIIS algorithms allows to avoid this oscillating behavior and corrects definitively the fixed point algorithm. This mathematical analysis has been a very important success in the community of computational chemists and has been implemented as a default method in classical softwares. The above results are almost the only ones existing in this category. We are still at the very beginning of this kind of analysis focussing on the algorithms, the variety of which is even larger than the variety of discrete schemes.

Conclusion Error on the Algorithm Most of the algorithms used for solving the above problems after a proper discretization has been implemented are iterative ones, either to solve a linear problem through a conjugate gradient algorithm or to solve an eigenvalue problem through a QR or a simple power algorithm (see, e.g., the entry  Fast Methods for Large Eigenvalues Problems for Chemistry) or finally to take into account the nonlinearities arising in the problem being solved by a fixed point procedure. These iterative algorithms may eventually converge (or not) after a large enough number of iterations. From the practical

We have presented some results on the a priori and a posteriori analysis focussing on approximations on the model, on the discretization strategy, and on the a chosen algorithm to simulate the solutions to problems in chemistry. Most of the results are partial only, and we are quite far from a full a posteriori analysis that would tell the user, after he performed a given discretization with a given number of degrees of freedom resulting in a discrete problem solved with a given algorithm using a fixed number of iterations, how far the discrete solution coming out from the computer is from the exact solution and we are even farther from a

A

10

procedure able to tell what should be done in order to improve the accuracy: either to increase the number of degrees of freedom or the increase number of iterations or change the model. This procedure though exists for a totally different context as is explained in [10]. This is certainly a direction of research and effort to be done by applied mathematicians that will lead to future and helpful progress for the reliability of approximations in this field.

References 1. Boulton, L., Boussa, N., Lewin, M.: Generalized Weyl theorem and spectral pollution in the Galerkin method. http://arxiv.org/pdf/1011.3634v2 2. Boys, S.F.: Electronic wavefunction I. A general method of calculation for the stationary states of any molecular system. Proc. R. Soc. A 200, 542–554 (1950) 3. Braess, D.: Asymptotics for the approximation of wave functions by sums of exponential sums. J. Approx. Theory 83, 93–103 (1995) 4. Brezzi, F., Canuto, C., Russo, A.: A self-adaptive formulation for the Euler/NavierStokes coupling. CMAME Arch. 73(3), 317–330 (1989) 5. Canc`es, E., Ehrlacher, V., Maday, Y.: Periodic Schrdinger operators with local defects and spectral pollution, arXiv:1111.3892 6. Canc`es, E., LeBris, C.: On the convergence of SCF algorithms for the HartreeFock equations. Math. Model. Numer. Anal. 34, 749–774 (2000) 7. Canc`es, E., LeBris, C., Maday, Y., Turinici, G.: Towards reduced basis approaches in ab initio electronic structure computations. J. Sci. Comput. 17(1), 461–469 (2002) 8. Canc`es, E., LeBris, C., Maday, Y.: M´ethodes Math´ematiques en chimie quantique: une Introduction (in French). Math´ematiques and Applications (Berlin), vol. 53. Springer, Berlin (2006) 9. Conte, D., Lubich, C.: An error analysis of the multiconfiguration time-dependent Hartree method of quantum dynamics. ESAIM M2 AN 44, 759–780 (2010) 10. El Alaoui, L., Ern, A., Vohralk, M.: Guaranteed and robust a posteriori error estimates and balancing discretization and linearization errors for monotone nonlinear problems. Comput. Method Appl. Mech. Eng. 200, 2782–2795 (2011) 11. Friesecke, G.: The multiconfiguration equations for atoms and molecules: charge quantization and existence of solutions. Arch. Ration. Mech. Anal. 169, 35–71 (2003) 12. Hohenberg, P., Kohn, W.: Inhomogeneous electron gas. Phys. Rev. 136(3B), B864–B871 (1964) 13. Kobus, J., Quiney, H.M., Wilson, S.: A comparison of finite difference and finite basis set Hartree-Fock calculations for the N2 molecule with finite nuclei. J. Phys. B Atomic Mol. Opt. Phys. 34, 10 (2001) 14. Kohn, W., Sham, L.J.: Self-consistent equations including exchange and correlation effects. Phys. Rev. 140(4A), A1133–A1138 (1965)

Absorbing Boundaries and Layers 15. Kudin, K., Scuseria, G.E., Canc`es, E.: A black-box selfconsistent field convergence algorithm: one step closer. J. Chem. Phys. 116, 8255–8261 (2002) 16. Kutzelnigg, W.: Theory of the expansion of wave functions in a Gaussian basis. Int. J. Quantum Chem. 51, 447–463 (1994) 17. Lewin, M.: Solution of multiconfiguration equations in quantum chemistry. Arch. Ration. Mech. Anal. 171, 83–114 (2004) ´ Spectral pollution and how to avoid 18. Lewin, M., S´er´e, E.: it (with applications to Dirac and periodic Schr¨odinger operators). Proc. Lond. Math. Soc. 100(3), 864–900 (2010) 19. Maday, Y., Razafison, U.: A reduced basis method applied to the restricted HartreeFock equations. Comptes Rendus Math. 346(3–4), 243–248 (2008) 20. Maday, Y., Turinici, G.: Error bars and quadratically convergent methods for the numerical simulation of the Hartree-Fock equations. Numer. Math. 94, 739–770 (2003)

Absorbing Boundaries and Layers Laurence Halpern Laboratoire Analyse, G´eom´etrie and Applications, UMR 7539 CNRS, Universit´e Paris, Villetaneuse, France

Synonyms Artificial; Computational; Free-Space; Nonreflective; Open or Far-Field Boundary Conditions; Sponge Layers

Summary Absorbing boundaries and layers are used to limit the computational domain in the numerical approximation of partial differential equations in infinite domains, such as wave propagation problems or computational fluid dynamics. In a typical seismic problem, the wave equation Lu D f must be solved in the subsurface with data g on the surface; the solution u is sought in the domain DS in magenta in Fig. 1. The domain in blue is a computational layer LC ; their union is the computational domain DC .

Absorbing Boundaries and Layers

11

Absorbing Boundaries and Layers, Fig. 1 Absorbing layer (courtesy of L. M´etivier [19])

A DS

In the theory of absorbing boundaries, the original equation is solved in DC , and a special boundary condition is imposed on its boundary to simulate the entire subsurface, which amounts to reducing as much as possible the reflection of waves inside the domain of interest DS . Note that in the early history of the theory, the computational domain and the domain of interest were the same. The layer strategy is to modify the equation inside the computational layer, using LC v D f instead, with simple boundary condition at the exterior border. In both cases, the modified problem in the computational domain is required to fulfill important properties: 1. Well posedness: For any data g, there exists a unique solution v, with estimates in some norms: kvk1  C.kf k2 C kgk3 /: 2. Transparency: For a given ", one can choose either the absorbing boundary conditions or the size of the layer, such that ku  vkDS  ": 3. Simplicity: The additional amount of code writing due to the layer should be limited. 4. Cost: The layer should be as small as possible. Regarding item 4, note that Dirichlet boundary conditions for the wave equation would act as a wall, thus producing a 100 % error after a time T equal to twice the size of the domain.

LC

Absorbing Boundaries The question emerged in the mid-seventies, with an illustrating idea for the wave equation by the geophysicist from Berkeley W.D. Smith [23]. It relies on the plane wave analysis, which will be a useful tool all throughout. Consider the wave equation in R2 , @t t u  @11 u  @22 u D 0:

(1)

Suppose one wants to reduce the computational domain to R2 D fx; x1 < 0g. The plane waves are solutions of (1), of the form u D Ae i.!t k  x / , with the dispersion relation ! 2 D jkj2 D k12 C k22 . The waves propagating toward x1 > 0 are such that their group velocity rk ! e 1 is positive, i.e., k!1 > 0. Place a fixed boundary  at x1 D 0 (Dirichlet boundary condition u D 0), and launch a plane wave from x1 < 0 toward . By the Descartes’ law, it is reflected into uR D Ae i.!t Ck1x1 k2 x2 / : the reflection coefficient is equal to 1. Replace the fixed boundary by a free one (Neumann boundary condition @1 u D 0). Now the reflected wave is uR D CAe i.!t Ck1x1 k2 x2 / : the reflection coefficient is equal to 1. Perform the computation twice – once with Dirichlet, then with Neumann – and add the results to eliminate the reflection. This ingenious idea is of course too simple: if more than one boundary is required to be nonreflecting, more elementary computations have to be made. For instance, eight computations have to be made at a three-dimensional corner. Furthermore, the argument is no longer exact when the velocity is variable in

12

Absorbing Boundaries and Layers

N the domain. However, it launched the Holy Grail for X aj X  12 ; .1  X /  a0 C 30 years. 1  bj X j D1 The breakthrough came with the plasma physicist E.L. Lindmann [18], who paved the way for much N X aj k22 of the subsequent work in the subject. His analysis : (4) G.i k2 ; i !/ D a0 C ! 2  bj k22 was purely discrete but will be presented below at the j D1 continuous level. Consider again a plane wave traveling to the right, Substituting into (2) leads to the absorbing boundary impinging the boundary , where a boundary condi- condition tion is defined via an operator G.@2 ; @t /: N X @t u C @1 u C Gj @1 u D 0; (2) @t u C G.@2 ; @t /@1 u D 0: j D1

The reflected wave is RAe i.!t Ck1x1 k2 x2 / , where R is where each Gj operates on functions ' defined on   defined by the boundary condition, .0; T /, Gj ' D j solution of i ! i k1 G.i k2 ; i !/CR.i ! Ci k1 G.i k2 ; i !// D 0:  ! 12 k2 2 . Then by the Define G0 .i k2 ; i !/ D 1  ! dispersion relation, the reflection coefficient is equal to G  G0 RD . If G 1, the boundary operator is the G C G0 transport operator in the x1 direction and is transparent to the waves at normal incidence, i.e., if k2 D 0, R D 0. This condition will be referred to as first-order absorbing and can be generalized in 

@t u C

1 @1 u D 0; sin 0

(3)

which is transparent to the waves impinging the boundary at incidence angle 0 . With some more caution, G0 would be the symbol of the Neumann to Dirichlet map for the wave equation and the half-space R2C . Choosing G to be G0 eliminates all reflections. A similar analysis can be made for spherical boundaries as well and led, together with fast integral methods, to some successful tools by H.B. Keller and followers (see [11] for a review). However, in the physical variables, G0 is an integral operator in time and space, therefore numerically costly. It is then worthwhile trying to get approximations to G0 , which starts by an approximation of the square root. E.L. Lindmann was the first to notice that a Taylor series expansion would lead to an unstable boundary condition and proposed a continuous fraction approximation:

@t t

j

 bj @22

j

D aj @22 ';

with the initial conditions j D @t j D 0. The coefficients .aj ; bj / were found such as to optimize the reflection coefficient over all angles of incidence. The idea of approximation was developed further, but with a Pad´e approximation of G01 instead of G0 , by B. Engquist and A. Majda in [7, 8]. Their first two approximations, named 15ı and 45ı , respectively, in the geophysical literature, at the boundary x1 D 0 for the half-space x1 < 0, were @t u C c@1 u D 0;

1 @t t u C c@t1 u  @22 u D 0: (5) 2

In those two papers, an entire theory, in the frame of the theory of reflection of singularities, permitted an extension to hyperbolic systems (elastodynamics, shallow water, Euler), variable coefficients, and curved boundaries. For instance, the extension of (3) to a circle of radius R is, with a derivative @r in the radial direction, @t u C @r u C

1 u D 0 in 2D; 2R

@t u C @r u C

1 u D 0 in 3D: R

Among the approximations, those that generate well-posed initial boundary value problems were identified in [25]. Various extensions to other type

Absorbing Boundaries and Layers

13

of problems, mainly parabolic (advection-diffusion, Stokes, Navier-Stokes), followed in the 1990s; see, for instance, [13, 24]. There is not much extra cost compared to Dirichlet since the boundary conditions are local in time and space. R. Higdon [15] found an expression for the absorbing boundary conditions as a product of first orderoperators (3) that is very useful on the discrete level. Another concept of far-field boundary condition was developed by A. Bayliss and E. Turkel, based on asymptotic expansions of the solution at large distances [3]. They proposed a sequence of radiation operators for the wave equation in the form  n  Y 2j  1 u; LC r j D1

Absorbing Layers M. Israeli and S. Orszag introduced and analyzed in 1981 the sponge layers for the one-dimensional wave equation [17]. They added in a layer of width ı, what they called a Newtonian cooling .x/.@t u C @x u/ to the equation. The right-going waves cross the interface without seeing it, while the left-going waves, reflected by the exterior boundary of the layer, are damped. This strategy was coupled with an absorbing boundary condition at the end of the layer. For more complicated equations, the numerical performance of such layers with discontinuous coefficient is not as good as one would hope. One reduces reflections at the interface for the right-going waves by choosing .x/ > 0, vanishing to order k > 0 at the origin:

.x/ D A.k C 1/.k C 2/

and showed well-posedness and error estimates. At these early times, no layer was used. It is only in [10, 12] that optimized layers were introduced. The evanescent waves are damped in the layer of width ı, and the coefficients in (4) are chosen to minimize the error in the domain of interest over the time length T , generalizing (5) to @t u C ˛0 @1 D 0;

u@t t u C ˛1 @t1 u  ˇ1 @22 u D 0: (6)

The coefficients are given by

D

p 1 2 4 2ı 1 1 ; ˛0 D p ; ˛1 D p ; ˇ1 D : T 1C 1C

Only a small and balanced error remains when optimized absorbing conditions are used, as shown in Table 1 where the error in the domain of interest caused by the truncation is measured in the L2 norm in space at time t D T .

 x k ı  x : ı ı

(7)

That reduces the rate of absorption and thereby increases the width of the layer required. A WKB analysis shows that the leading reflection by such layers of incoming wave packets of amplitude O.1/ and wavelength " is O."kC1 / [17]. In practice k is chosen equal to 3. At that time it was difficult to see how to extend the strategy to higher dimension, so absorbing boundary conditions were more widely employed than sponge layers. . . whose revenge came 15 years later. In 1994, the electrical engineer B´erenger found the Grail, the perfectly matched layers for Maxwell’s system [5, 6]. The striking idea was to design a “lossy medium” in the layer, so that no reflection occurs at the interface, for all frequencies and all angles of incidence, and furthermore the transmitted wave is damped in the layers. Interesting interpretations involving a complex change of coordinates are very useful for harmonic problems [21]. The first paper deals with the transverse electric modes (supposing no source term):

Absorbing Boundaries and Layers, Table 1 Comparison of the error caused by the truncation of the domain by the various methods Error at t D T 2

L

First Order

Second Order

Dirichlet

Orthogonal

Optimized

Orthogonal

Optimized

100 %

22 %

21 %

10 %

5%

A

14

Absorbing Boundaries and Layers

"0 @t E1  @2 H3 C E1 D 0; "0 @t E2 C @1 H3 C E2 D 0; 0 @t H3 C @1 E2  @2 E1 C  H3 D 0:

(8)

In the last equation, the magnetic component H3 is broken into two subcomponents, H31 and H32 , and the equation itself is doubled, giving rise to a 4  4 system: "0 @t E1  @2 .H31 C H32 / C 2 E1 D 0; "0 @t E2 C @1 .H31 C H32 / C 1 E2 D 0; 0 @t H31 C @1 E2 C 1 H31 D 0; 0 @t H32  @2 E1 C 2 H32 D 0:

(9)

This method experienced a huge success in the computational world due not only to the features presented above but also to the ease of implementation, especially at the corner, where absorbing boundary conditions were difficult to implement. These properties offset the extra cost due to the addition of a new equation in 2D (and even more in 3D). An intense activity followed, concerning the mathematical analysis of the PML and the extension to other systems such as Euler equations and elastodynamics. The perfectly matched model for the general hyperbolic problem below is built in two steps. L.@t ; @x / U WD @t U C

The parameters . 1 ; 2 ; 1 ; 2 / characterize the PML. They are in this first stage, piecewise constant, and assembled along the famous picture in Fig. 2 (noting that the vacuum is PML(0; 0; 0; 0)). B´erenger showed by an argument resembling a plane wave analysis, which under the assumptions

j ="0 D j =0 , the interfaces were transparent (perfectly matched) to waves at all incidences and wavenumbers. Furthermore, he exhibits an apparent reflection coefficient (due to the perfect conductor condition at the exterior border of the layer), R./ D e 2. cos ="0 c/ı , showing that the layer is indeed absorbing. He however noticed that in practical computations, the absorption coefficient needs to be continuous, and he uses polynomial values (7) of Israeli and Orszag. Figure 3 with the same colorbar as in Fig. 4, shows the superior performances of the Berenger’s layers.

d X

Ak @k U D 0 ;

kD1

.t; x/ 2 R1Cd ; U.t; x/ 2 CN :

The first step is a splitting of the system. The split system involves the unknown .U 1 ; U 2 ; : : : ; U d / 2 .CN /d : @t U j C Aj @j .U 1 C U 2 C : : : U d / D 0 ; j D 1; : : : ; d :

(11)

The second step is the insertion of a damping j .xj / supported in fXj  xj  Xj C ıj g, in the j equation: @t U j C Aj @j .U 1 CU 2 C: : : U d / C j .xj / U j D 0 : (12) Note, for comparison, that the sponge layer of Israeli and Orszag generalizes into +

+∗

PML (0, 0, s 2 , s 2 ) +

(10)

+∗

−∗ PML (s − 1 , s 1 , s2 , s2 )

+

+∗

+

+∗

PML (s 1 , s 1 , s 2 , s 2 ) vacuum

−∗ PML (s − 1 , s 1 , 0, 0)

Outgoing wave +

+∗

PML (s 1 , s 1 , 0, 0)

Wave source

Perfect conductor

+

−∗ − −∗ PML (s − 1, s1 , s2, s2 )

+∗

−∗ PML (s 1 , s 1 , s − 2 , s2 ) −∗ PML (0, 0, s − 2 , s2 )

Absorbing Boundaries and Layers, Fig. 2 The PML technique by B´erenger [5]

15 1

1

0.9

0.9

0.9

0.8

0.8

0.8

0.7

0.7

0.7

0.6

0.6

0.6

0.5

0.5

0.5

y

1

y

y

Absorbing Boundaries and Layers

0.4

0.4

0.4

0.3

0.3

0.3

0.2

0.2

0.2

0.1

0.1

0.1

0

0 0.1

0.2

0.3

0.4

x Exact solution

A

0

0.5

0.1

0.2

0.3

0.4

0.5

d

d x Sponge layer (13) of [17]

0.1

0.2

0.3

0.4

0.5

d x Berenger layer (9)

1

1

0.9

0.9

0.9

0.8

0.8

0.8

0.7

0.7

0.7

0.6

0.6

0.6

0.5

0.5

0.5

y

1

y

y

Absorbing Boundaries and Layers, Fig. 3 Snapshots of the solution of the wave equation with various layer strategies and polynomial absorption

0.4

0.4

0.4

0.3

0.3

0.3

0.2

0.2

0.2

0.1

0.1

0.1

0

0 0 0.1

0.2

0.3

x

0.4

0.5

d

second order condition (5)

0 0 0.1

0.2

0.3

x

0.4

0.5

0 0.1

0.2

first order optimized in (6)

0.3

x

d

0.4

0.5

d

second order optimized in (6)

Absorbing Boundaries and Layers, Fig. 4 Error caused by the truncation with various boundary conditions

16

@t U C

Absorbing Boundaries and Layers d X lD1

Al @l U C

X

j .xj /C .Aj /U D 0 ;

j

(13) introducing the notation C .Aj / for the spectral projector on the eigenspace corresponding to strictly positive eigenvalues of Aj . S. Abarbanel and D. Gottlieb realized quite soon in [1] that for Maxwell’s system, (11) was a weakly hyperbolic problem, which means that the Cauchy problem has a unique solution but with a loss of derivae .0; /kH 1 .R2 / . e .t; /kL2 .R2 /  .1 C C t/kU tives, i.e., kU It is then possible that a zero-order perturbation of (11) would lead to ill-posed Cauchy problems. An example of such a situation is given in [14]. Alternatively, if L is strongly well posed (i.e., the Cauchy problem has a unique solution in L2 with no loss of derivative) and if L1 .0; @/ is elliptic (i.e., det L1 .0; / ¤ 0 for all real ), then e L is strongly well posed for any absorption . 1 .x1 /; : : : ; d .xd // in .L1 .R//d . This is the case for the wave equation or the linear elastodynamic system. The analysis of the Maxwell PML relies on the construction of an augmented system and the construction of a symmetrizer. In two dimensions, it requires each absorption to be in W 1;1 .R/ [20] and in three dimensions in W 2;1 .R/ [14]. In the latter, there e0 exists a constant C such that, for any initial data U in H 2 .R3 /, the system (12) has a unique solution in L2 .R3 / with e .t; /k.L2 .R3 //9  C e C t e kU U 0 .H 2 .R3 //9 : Even though discontinuous damping coefficients are replaced in the computation by polynomials, it is amazing to notice that the question of well-posedness for the problem described in Fig. 2 with discontinuous coefficients is still an open problem in three dimensions. Once well-posedness is proved, the perfect matching follows by a change of variables. A good definition of perfection was given in [2]. Coming to absorption, the situation is contrasted. For Maxwell’s equations, B´erenger proved its layer to be absorbing. Other cases were shown to exhibit amplification: Euler equations linearized around a nonzero mean flow in [16], anisotropic elasticity in [4]. In those papers, an analysis involving group and phase velocity was performed, and other layers were proposed. These analyses were extended to nonconstant absorption in [14].

Other Issues and Applications The need to bound the computational domain arises for stationary problems, elliptic problems in mechanics, and harmonic problems in scattering, for instance. In that case, other tools are available, like integral equations, coupling between finite elements and boundary elements, multipoles, and infinite elements. The concept of absorbing boundary condition is closely related to paraxial equations used in geophysics or underwater acoustics to approximate waves in a preferred direction [25]. It is also related to optimized Schwarz domain decomposition methods [9,10]. Perfectly matched layers were also used for domain decomposition in relation with harmonic problems [22].

References 1. Abarbanel, S., Gottlieb, D.: A mathematical analysis of the PML method. J. Comput. Phys. 134, 357–363 (1997) 2. Appel¨o, D., Hagstr¨om, T., Kreiss, G.: Perfectly matched layers for hyperbolic systems: general formulation, wellposedness and stability. SIAM J. Appl. Math. 67, 1–23 (2006) 3. Bayliss, A., Turkel, E.: Radiation boundary conditions for wave-like equations. Commun. Pure Appl. Math. 32, 313–357 (1979) 4. B´ecache, E., Fauqueux, S., Joly, P.: Stability of perfectly matched layers, group velocities and anisotropic waves. J. Comput. Phys. 188, 399–433 (2003) 5. B´erenger, J.P.: A perfectly matched layer for the absorption of electromagnetic waves. J. Comput. Phys. 114, 185–200 (1994) 6. B´erenger, J.P.: Three-dimensional perfectly matched layer for the absorption of electromagnetic waves. J. Comput. Phys. 127, 363–379 (1996) 7. Engquist, B., Majda, A.: Absorbing boundary conditions for the numerical simulation of waves. Math. Comput. 31(139), 629–651 (1977) 8. Engquist, B., Majda, A.: Radiation boundary conditions for acoustic and elastic wave calculations. Commun. Pure Appl. Math. 32, 313–357 (1979) 9. Engquist, B., Zhao, H.K.: Absorbing boundary conditions for domain decomposition. Appl. Numer. Math. 27, 341–365 (1998) 10. Gander, M.J., Halpern, L.: Absorbing boundary conditions for the wave equation and parallel computing. Math. Comput. 74(249), 153–176 (2004) 11. Hagstrom, T.: Radiation boundary conditions for the numerical simulation of waves. Acta Numer. 8, 47–106 (1999) 12. Halpern, L.: Absorbing boundary conditions and optimized Schwarz waveform relaxation. BIT 46, 21–34 (2006) 13. Halpern, L., Rauch, J.: Artificial boundary conditions for general parabolic equations. Numer. Math. 71, 185–224 (1995)

Actin Cytoskeleton, Multi-scale Modeling

17

are generated by utilizing the chemical free energy in ATP to build actin networks and power myosin motor contraction. In solution, actin monomers (G-actin) assemble into two-stranded filaments (F-actin), bundles of filaments and gels. The helical F-actin filament is asymmetric, with a barbed or plus end and a pointed or minus end, and this leads to asymmetric reaction kinetics at the two ends. In solution, G-actin primarily contains ATP, but a G-ATP monomer that is incorporated in a filament subsequently hydrolyzes its bound ATP into ADP-Pi-actin and releases the phosphate Pi to yield G-ADP. As a result maintenance of actin structures at steady state requires a constant energy supply in the form of ATP. All three G-actin types bind to filament tips, but with different kinetic rates. In vivo, the structures formed range from microspikes and filopodia, to larger pseudopodia and broad lamellipodia, and their type is tightly controlled by intracellular regulatory molecules and extracellular mechanical and chemical signals. Three basic processes involved in cellular movement are (1) controlled spatio-temporal remodeling of the actin network, (2) construction and destruction of “traction pads” – called focal complexes or focal adhesions (FAs) – that are complexes of integrins and other proteins that transiently assemble for force transmission to the substrate, and (3) generation of forces to move the cell body over these traction pads. Four zones of actin networks that occur in motile cells are (1) the lamellipodium (LP), a region of rapid actin turnover that extends 3–5 m from the leading edge of the cell, (2) the lamellum (LM), a contractile network that Actin Cytoskeleton, Multi-scale Modeling extends from behind the leading edge to the interior of the cell, (3) the convergence zone, in which the Hans G. Othmer retrograde flow in the LP meets the anterograde flow Department of Mathematics, University of Minnesota, in the cell body, and (4) the actin network in the cell Minneapolis, MN, USA body, which contains the major organelles (cf. Fig. 1). To produce directed cell movement, spatio-temporal control of the structure of the actin subnetworks and Synonyms their interaction with the membrane and FAs is necessary, and as many as 60 actin-binding proteins, grouped Actin polymerization; Cell motility; Force generation by their function as follows, may be involved. 1. Sequestering proteins that sequester actin monomers to prevent spontaneous nucleation and end-wise polymerization of filaments (thymosin-ˇ4) or Introduction interact with actin monomers to affect nucleotide exchange (profilin, cofilin, and twinfilin). Cell locomotion is essential in numerous processes such as embryonic development, the immune response, 2. Crosslinking proteins such as ˛-actinin that crosslink the actin filaments. Others such as vinculin, and wound healing. Movement requires forces, which

14. Halpern, L., Petit-Bergez, S., Rauch, J.: The analysis of matched layers. Conflu. Math. 3, 159–236 (2011) 15. Higdon, R.: Absorbing boundary conditions for difference approximations to the multidimensional wave equation. Math. Comput. 47, 437–459 (1986) 16. Hu, F.: On absorbing boundary conditions of linearized Euler equations by a perfectly matched layer. J. Comput. Phys. 129, 201–219 (1996) 17. Israeli, M., Orszag, S.: Approximation of radiation boundary conditions. J. Comput. Phys. 41, 115–135 (1981) 18. Lindmann, E.L.: Free-space boundary conditions for the time dependent wave equation. J. Comput. Phys. 18, 16–78 (1975) 19. M´etivier, L.: Une m´ethode d’inversion non lin´eaire pour l’imagerie sismique haute r´esolution. Ph.D. thesis, Universit´e Paris 13 (2009) 20. M´etral, J., Vacus, O.: Caract`ere bien pos´e du probl`eme de Cauchy pour le syst`eme de B´erenger. C. R. Math. Acad. Sci. Paris 328, 847–852 (1999) 21. Rappaport, C.M.: Interpreting and improving the PML absorbing boundary condition using anisotropic lossy mapping of space. IEEE Trans. Magn. 32, 968–974 22. Sch¨adle, A., Zschiedrich, L.: Additive Schwarz method for scattering problems using the PML method at interfaces. In: Domain Decomposition Methods in Science and Engineering XVI. Lecture Notes in Computational Science and Engineering, vol. 55, pp. 205–212. Springer (2011) 23. Smith, W.D.: A nonreflecting plane boundary for wave propagation problems. J. Comput. Phys. 15, 492–503 (1974) 24. Szeftel, J.: Absorbing boundary conditions for non linear partial differential equations. Comput. Methods Appl. Mech. Eng. 195, 3760–3775 (2006) 25. Trefethen, L.N., Halpern, L.: Well-posedness of one-way wave equations and absorbing boundary conditions. Math. Comput. 47(167), 421–435 (1986)

A

18

Actin Cytoskeleton, Multi-scale Modeling, Fig. 1 The lamellipodium and lamellum at the leading edge of a cell. The convergence zone and cell body lie proximal to the lamellum. F-actin is stained green and activated myosin is stained blue. Sites of adhesion to the substrate are in red (From [1], with permission)

talin, and zyxin link the cell cortex to the plasma membrane. 3. Severing proteins such as ADF/cofilin and gelsolin that sever F-actin to generate more filament ends for network growth or disassembly. 4. Other proteins cap filament ends to regulate addition or loss of actin subunits (capping protein, gelsolin, Arp2/3), nucleate filament growth (Arp2/3, formin), or enhance subunit dissociation (cofilin). The LP actin network consists of short, branched F-actin whose formation is promoted by Arp2/3, which nucleates new filaments de novo or initiates branches from preexisting filaments [2], while filament severing and depolymerization at the pointed end is promoted by ADF/cofilin. Sequestering proteins such as twinfilin or ˇ-thymosins bind G-actin to prevent filament growth, while others such as profilin enhance nucleotide exchange. These and capping proteins control the G-actin pool so as to produce short, stiff filaments at the leading edge. The dynamic balance between these processes produces a zone 1–3 m wide of rapid network formation at the leading edge, followed by a narrow band of rapid actin depolymerization [3]. The LM is a zone immediately distal to the LP containing longer, less dense, and more bundled filaments. Actin polymerization and depolymerization in the LM is localized at spots that turn over randomly, and retrograde flow is slow [4]. Lamella also contain myosin II and tropomyosin, which are absent from the

Actin Cytoskeleton, Multi-scale Modeling

LP [3], and the actin and myosin form bundles called stress fibers. Contraction of stress fibers attached to the FAs at the junction between the LP and LM generates the force to move the cell body forward in the third step of migration [5]. To understand the interplay between the various subnetworks and the regulatory proteins governing the dynamics of the cytoskeleton, which is the name given to the actin network and all its associated proteins, mathematical models that link molecular-level behavior with macroscopic observations on forces exerted, cell shape, and cell speed are needed. How to formulate tractable models presents a significant challenge. The major subproblems involved are (1) understanding the dynamic control of the different structures in the actin network, (2) modeling the construction and disassembly of FAs and stress fibers, and analyzing how the level of motor activity and the adhesiveness of the substrate determine the cell speed, and (3) analyzing whole-cell models of simple systems to understand how the mechanical balance between components produces stable steady states of actin turnover, motor activity, and cell shape and how these respond to chemical and mechanical signals. Models for actin dynamics and the cytoskeleton, some of which are discussed below, range from stochastic models of single filaments to continuum models for whole cell movement. (See [6,7] for a more detailed explanation of cell movement, [8, 9] for texts on the mechanics of polymers and motors, [10, 11] for further details about the cytoskeleton, [12] for the biophysics of force transduction via integrins, and [13] for a more comprehensive review of mathematical modeling.)

Single Filament Dynamics To understand some of the issues we consider a simplified description of a single filament in a pool of fixed monomer concentration, ignoring the distinction between monomer types, and the fact that a growing filament has two strands. The rate of monomer addition is higher at the plus or barbed end than at the minus or pointed end. At each end the reaction

F1 C Fn1

kC ! F n  k

(1)

Actin Cytoskeleton, Multi-scale Modeling

19

occurs, where Fn is the filament consisting of n actin where  D k  ;  D k C =.NA  Vo / and NA is subunits and F1 the G-actin monomer. If we neglect Avogadro’s number. The steady-state monomer distriall processes but addition or release at the ends, the bution is time rate change of filament length (l) measured in monomers is governed by the equation (3) q1 .n/ D lim q.n; t/ t !1

dl D kCc  k; dt

(2)

where the rates are now the sum of the rates at the two ends. Therefore, there is steady state length if the monomer concentration is c D k  =k C Kd . There is also a critical concentration c˙ for each end of a filament at which the on- and off-rates for a given form of the monomer are equal. Above this the end grows, while below that it shrinks. G-ATP has a much higher on-rate at the plus end than at the minus end, and therefore the critical concentration cC is lower than the critical concentration c for the minus end. The equilibrium constants are the same at both ends for G-actin-ADP (G-ADP), and hence the critical concentrations are the same, but since G-ATP is the dominant form of the monomer in vivo, its kinetics dominate the growth or decay of the filament. At the overall critical concentration there is net addition of monomers at the plus end and loss at the minus end, and the filament is said to “treadmill.” The preceding description is simplified in that the structure of the filament is ignored and the dynamics are treated deterministically. We remedy this in two steps: first we describe the stochastic analog of the preceding and then consider the filament structure. Suppose that a single filament of initial length l0 polymerizes in a solution of volume Vo , free monomer mo , and total monomer count N D lo C mo . Let q.n; t/ be the probability of having n monomers in the monomer pool at time t, and let p.n; t/ be the probability of the filament being of length n at time t. The evolution equation for q.n; t/ is dq.0; t/ D  q.0; t/ C  q.1; t/ dt dq.n; t/ D  q.n  1; t/  . C n/ q.n; t/ dt C .n C 1/ q.n C 1; t/ .1  n  N  1/ dq.N; t/ D  q.N  1; t/  N q.N; t/ dt

1 D nŠ

 n   ! N X  1  k =  kŠ  kD0

and p1 .n/ D q1 .N  n/. The mean is

M1 D

N X

n q1 .n/

(4)

nD0

"     !# N X  1  k 1  N D = 1  NŠ  kŠ  kD0

and as N ! 1 this tends to a Poisson distribution with mean =, which is the monomer number at the critical concentration in the preceding deterministic analysis. Next suppose that the monomers are of two types – G-ATP and G-ADP – and consider the initial condition of a pure G-ADP filament tethered at the pointed end. When the G-ATP is below the critical concentration, the filament tip switches between the ATPcapped state and the ADP-exposed state. The filament comprises two parts: the ATP-cap and the ADP-core portion of length m and n, respectively, and we let p.m; n; t/ be the probability of this state. Denote the ATP on- and off-rates as ˛ and ˇ, and the ADP off-rate by  . The master equation for the filament state is then dp.m; n; t/ D ˛ p.m  1; n; t/ C ˇ p.m C 1; n; t/ dt  .˛ C ˇ/ p.m; n; t/

.m  1/

dp.0; n; t/ D ˛ p.0; n; t/ C ˇ p.1; n; t/ dt C  p.0; m C 1; t/   p.0; n; t/: Analysis of this model leads to the length distribution, various moments, and the distribution of the lifetime of an ATP cap. It also leads to an explanation of two experimental observations: (1) the elongation rate curve is piecewise linear in that the slopes are different

A

20

Actin Cytoskeleton, Multi-scale Modeling

below and above the critical concentration; and (2) the measured in vitro diffusion constant is 30–45 times higher than the prediction according to earlier deterministic analysis [14].

Bulk Filaments in Solution Actin Cytoskeleton, Multi-scale Modeling, Fig. 2 A schematic of the network for nucleation and filament growth (From [15] with permission)

New phenomena arise when new filaments can be generated from monomers via nucleation in a closed system. The filaments now interact via the monomer pool, and because nucleation of a new filament is energetically less favorable than addition to an existing filament population, individual filaments elongate until one, the tendency is to produce longer rather than more the monomer pool equilibrates with filaments ( 30 s). filaments. The evolution starting with a pure monomer The dynamics of bulk filaments can be rewritten as pool involves the sequence Nucleation ! Growth ! Monomer/polymer equilibrium ! Length redistribution: The system is constrained by a maximal length, N , for the filaments, and thus, a deterministic description of filament growth leads to

dci D k C c1 ci 1  k  ci  k C c1 ci C k  ci C1 dt

(8)

k C c1 C k  2 .i  4/ (9)

D .k C c1  k  /.ci  ci 1 / C .ci C1  2ci C ci 1 /

and from this one sees that the dynamics involves N X dc1 two processes: convection of the filament distribution, D 2.k1C c12  k1 c2 /  .kiC1 c1 ci 1  ki1 ci / dt represented in the first term, which dominates when i D3  C (5) c1 >> k =k , and diffusion represented by the second term. Before establishment of the monomer– dcn C  polymer equilibrium, convection dominates diffusion, D .kn1 c1 cn1  kn1 cn / dt and one observes in the computational results that C  the maximum of the length distribution increases at  .kn c1 cn  kn cnC1 / for n 2 .3; N  1/ (6) a predictable speed [15]. Later in the evolution, the unimodal distribution eventually evolves, albeit very dcN D kNC1 c1 cN 1  kN 1 cN : (7) slowly, into an exponential steady state distribution. If dt one assumes that the monomer pool is approximately constant in this phase one has the linear system C The on-rates, ki .i D 1; 2; 3:::/, are equal and denoted k C , and the off-rates, ki .i D 3; 4; 5:::/ dc are equal and denoted k  , but each differs from those D Ac C  (10)   dt of nucleation steps, k1 ; k2 . The flow of monomers between the different pools is shown in Fig. 2. The first two nucleation steps are fast reactions, and equi- where c D .c2 ; c3 ; : : : ; cN /T ,  D .k1C c12 ; 0; : : : ; 0/T , librate on time scales estimated as .k1 C 4k1C c1 /1 and A is an .N 1/.N 1/ matrix. A spectral analysis O.106 s/ and .k2 C 9.k1 =k2C /k2C c1 /1 O.103 s/, of A shows that the slowest mode relaxes on a time respectively [15]. The trimers then elongate and in this scale of order of N 2 , which for N D 2; 000 is of order stage, the actin flux is via the trimer to filament step. 106 s [15]. This exceptionally slow relaxation provides Because of the high nucleation off-rates, one finds that a possible explanation for why different experiments the monomer pool is above the critical concentration lead to different conclusions concerning the steady when nucleation stalls. After the establishment of the state distribution.

Actin Cytoskeleton, Multi-scale Modeling

Models of Force Generation and Membrane Protrusion The basic problem in polymerization models is to understand how a monomer can attach to a filament abutting a surface. Mogilner and Oster [16] proposed the elastic Brownian ratchet (EBR) model in which the thermal motion of the polymerizing filaments collectively produce a directed force. This model requires untethered filament ends at a surface for the free energy of monomer addition to generate force, and treats the actin filament as a flexible wire, whose end is fluctuating due to thermal fluctuations (see Fig. 3). When bent away from the surface, a subunit can bind to the filament and lengthen it. The restoring force of the filament straightening against the surface delivers the propulsive force. Given the measured stiffness of actin filaments, it was found that the length of the flexible actin filament (i.e., the “free” length beyond the last crosslinking point) must be quite short, in the range 30–150 nm. Beyond this length, thermal energy is taken up in internal bending modes of the filament, and pushing is ineffective. These considerations imply a requirement for the cell to balance the relative rates of branching, elongation, and capping. Theoretical calculations suggest that the cell tunes these parameters to obtain rapid motility and that it uses negative feedback

Actin Cytoskeleton, Multi-scale Modeling, Fig. 3 A model for the thermal vibration of a filament anchored in a rigid network (From [16])

21

via capping to dynamically maintain the number of barbed ends close to optimal levels [17]. The EBR model assumes filaments are untethered at the tip to permit the intercalation of monomers. However, most biomimetic assays on beads indicate a strong attachment of the actin tail to the moving surface, and suggest that at least a portion of filaments should be tethered to the surface. To account for these experimental findings, the EBR model was modified to allow tethering of filaments, and in this model, three classes of filaments are involved in force generation [18]. It is assumed that new filaments are nucleated via the NPF-Arp2/3 complex pathway, and that initially, these newly formed filaments attach to the surface and allow no polymerization. In addition, the attachment may be under stress and thus exert retarding force at the surface. After detachment, the filaments polymerize at the surface and exert forces on the surface as in the EBR model, or they can be capped. On the other hand, Dickinson and colleagues proposed an alternative mechanism for the force generation [19]. In this model surface-bound, clamp motors anchor the filament to the surface and promote the processive elongation of the filament. The detailed mechanism is described as the “Lock, Load, and Fire” mechanism, in which an end-tracking protein remains tightly bound (“locked” or clamped) onto the end of one subfilament of the double-stranded growing actin filament. After binding to specific sequences on tracker proteins, profilin-ATP-actin is delivered (“loaded”) to the unclamped end of the other strand, whereupon ATP within the currently clamped terminal subunit of the bound strand is hydrolyzed (“fired”), providing the energy needed to release that arm of the endtracker, which then can bind another profilin-ATP-actin to begin a new monomer-addition round. The last step triggers advance of the clamp to the terminal subunit in preparation for a new polymerization cycle. Both models capture two main features of actindriven motility. First, filament growth against an obstacle can produce substantial protrusive forces, and in both the filament network is attached to the surface. However, there also are clear differences between them. First, the tethered ratchet model uses the free energy released by monomer polymerization as the sole energy source for protrusion, whereas the LLF model utilizes both energy released by ATP-hydrolysis and the monomer binding energy for propulsion. As a result, the LLF model can produce work at

A

22

a lower monomer concentration. Second, the first model predicts a force-velocity relationship in which the velocity is force-sensitive at small loads but less sensitive at larger loads, whereas the reverse obtains for the LLF model. It is quite likely that both models apply to cellular motility – the tethered Brownian ratchet model may be closer to reality in lamellipodia containing dendritic actin networks, whereas the LLF may better describe the extension of filopodia composed of bundles of parallel filaments – only further experiments can determine this.

Actin Cytoskeleton, Multi-scale Modeling

5. The complex of ADC and ADP-actin that dissociates from the filament ends is in equilibrium with ADC and G-ADP. 6. The nucleotide exchange on actin monomer is a slow process, further inhibited by ADC, whereas profilin enhances this rate. 7. ATP-actin monomers are sequestered by ˇthymosins to prevent spontaneous nucleation, but provide a pool of ATP-actin for assembly. Details can be found in the original literature.

Summary Integration of Signaling, Network Dynamics, and Force Generation The dendritic nucleation model has been proposed as a unified description of actin dynamics in whole cells [20]. The model involves elongation at free barbed ends, ATP-hydrolysis and Pi release, capping of barbed ends as the filament array moves away from the leading edge, and pointed-end uncapping and disassembly, presumably from the pointed end. The network dynamics are regulated by several actin-binding proteins as described below [21, 22]. 1. The Arp2/3 complex is activated upon binding to WASP that is activated by the small GTPase CDC42. 2. Active Arp2/3 nucleates actin-filament assembly and caps the free pointed end of the filaments, or it binds to the side of a filament and then nucleates filament growth or captures the barbed ends of a preexisting filament. Growth of filaments is rapid and the lag in Pi dissociation leads to filaments in the leading edge that are composed predominantly of ATP- and ADP-Pi-actin and do not bind to ADF/cofilin (ADC). 3. Capping of the barbed ends by capping proteins prevents their further elongation. At the rear of lamellipodia, two mechanisms, filament severing and uncapping of pointed ends by removal of Arp2/3, could contribute to the rapid depolymerization. Severing by ADC is likely to occur at junctions between regions of filaments that are saturated with ADC and naked F-actin. The depolymerization is enhanced by Aip1. 4. ADC enhances depolymerization of ADP-actin from free filament ends in the rear of lamellipodia.

The foregoing gives a brief glimpse into the complexity of actin networks, but these are just one component of the machinery involved in cell motility. Two basic modes of movement are used by eukaryotic cells – the mesenchymal mode and the amoeboid mode. The former, which was described earlier, can be characterized as “crawling” in fibroblasts or “gliding” in keratocytes. This mode dominates in cells such as fibroblasts when moving on a 2D substrate. In the amoeboid mode, which does not rely on strong adhesion, cells are more rounded and employ shape changes to move – in effect “jostling through the crowd” or “swimming.” Leukocytes use this mode for movement through the extracellular matrix (ECM) in the absence of adhesion sites. Recent experiments have shown that numerous cell types display enormous plasticity in locomotion in that they sense the mechanical properties of their environment and adjust the balance between the modes accordingly [23]. Thus pure crawling and pure swimming are the extremes on a continuum of locomotion strategies, but many cells can sense their environment and use the most efficient strategy in a given context. Heretofore, mathematical modeling has primarily focused on the mesenchymal mode, but a unified description for locomotion in a 3D ECM that integrates signaling and mechanics is needed. As others have stated – “the complexity of cell motility and its regulation, combined with our increasing molecular insight into mechanisms, cries out for a more inclusive and holistic approach, using systems biology or computational modeling, to connect the pathways to overall cell behavior” [24]. This remains a major challenge for the future, and some early steps to address this challenge are given in [25, 26] and references therein.

Adaptive Mesh Refinement Acknowledgments Research supported by NSF grants DMS0517884 and DMS-0817529.

References 1. Gardel, M.L., Sabass, B., Ji, L., Danuser, G., Schwarz, U.S., Waterman, C.M.: Traction stress in focal adhesions correlates biphasically with actin retrograde flow speed. J. Cell Biol. 183, 999–1005 (2008) 2. Svitkina, T.M., Borisy, G.G.: Arp2/3 complex and actin depolymerizing factor/cofilin in dendritic organization and treadmilling of actin filament array in lamellipodia. J. Cell Biol. 145, 1009–1026 (1999) 3. Ponti, A., Machacek, M., Gupton, S.L., Waterman-Storer, C.M., Danuser, G.: Two distinct actin networks drive the protrusion of migrating cells. Science 305, 1782–1786 (2004) 4. Ponti, A., Matov, A., Adams, M., Gupton, S., WatermanStorer, C.M., Danuser, G.: Periodic patterns of actin turnover in lamellipodia and lamellae of migrating epithelial cells analyzed by quantitative fluorescent speckle microscopy. Biophys. J. 89, 3456–3469 (2005) 5. Hotulainen, P., Lappalainen, P.: Stress fibers are generated by two distinct actin assembly mechanisms in motile cells. J. Cell Biol. 173, 383–394 (2006) 6. Bray, D.: Cell Movements: From Molecules to Motility. Garland Pub., New York (2001) 7. Ananthakrishnan, R., Ehrlicher, A.: The forces behind cell movement. Int. J. Biol. Sci. 3, 303–317 (2007) 8. Howard, J.: Mechanics of Motor Proteins and the Cytoskeleton. Sinauer Associates, Inc, Sunderland (2001) 9. Boal, D.: Mechanics of the Cell. Cambridge University Press, Cambridge (2002) 10. Li, S., Guan, J.L., Chien, S.: Biochemistry and biomechanics of cell motility. Annu. Rev. Biomed. Eng. 7, 105–150 (2005) 11. Mofrad, M.R.K.: Rheology of the cytoskeleton. Annu. Rev. Fluid Mech. 41, 433–453 (2009) 12. Geiger, B., Spatz, J.P., Bershadsky, A.D.: Environmental sensing through focal adhesions. Nat. Rev. Mol. Cell Biol. 10, 21–33 (2009) 13. Mogilner, A.: Mathematics of cell motility: have we got its number? J. Math. Biol. 58, 105–134 (2009) 14. Hu, J., Othmer, H.G.: A theoretical analysis of filament length fluctuations in actin and other polymers. J. Math. Biol., DOI 10.1007/S00285-010-0400-6 (2011) 15. Hu, J., Matzavinos, A., Othmer, H.G.: A theoretical approach to actin filament dynamics. J. Stat. Phys. 128, 111–138 (2007) 16. Mogilner, A., Oster, G.: Cell motility driven by actin polymerization. Biophys. J. 71, 3030–3045 (1996) 17. Mogilner, A., Edelstein-Keshet, L.: Regulation of actin dynamics in rapidly moving cells: a quantitative analysis. Biophys. J. 83, 1237–1258 (2002) 18. Mogilner, A., Oster, G.: Force generation by actin polymerization II: the elastic ratchet and tethered filaments. Biophys. J. 84, 1591–1605 (2003) 19. Dickinson, R.B., Purich, D.L.: Clamped-filament elongation model for actin-based motors. Biophys. J. 82, 605–617 (2002)

23 20. Mullins, R.D., Heuser, J.A., Pollard, T.A.: The interaction of Arp2/3 complex with actin: nucleation, high affinity pointed end capping, and formation of branching networks of filaments. Proc. Natl. Acad. Sci. U.S.A. 95, 6181–6186 (1998) 21. Chen, H., Bernstein, B.W., Bamburg, J.R.: Regulating actinfilament dynamics in vivo. Trends Biochem. Sci. 25, 19–23 (2000) review 22. Pollard, T.D., Borisy, G.G.: Cellular motility driven by assembly and disassembly of actin filaments. Cell 112, 453–465 (2003) 23. Renkawitz, J., Schumann, K., Weber, M., L¨ammermann, T., Pflicke, H., Piel, M., Polleux, J., Spatz, J.P., Sixt, M.: Adaptive force transmission in amoeboid cell migration. Nat. Cell Biol. 11, 1438–1443 (2009) 24. Insall, R.H., Machesky, L.M.: Actin dynamics at the leading edge: from simple machinery to complex networks. Dev. Cell 17, 310–322 (2009) 25. Gracheva, M.E., Othmer, H.G.: A continuum model of motility in ameboid cells. Bull. Math. Biol. 66, 167–194 (2004) 26. Stolarska, M.A., Kim, Y., Othmer, H.G.: Multi-scale models of cell and tissue dynamics. Philos. Trans. R. Soc. A 367, 3525 (2009)

Adaptive Mesh Refinement Robert D. Russell Department of Mathematics, Simon Fraser University, Burnaby, BC, Canada

Synonyms Adaptive grid refinement; Adaptive regridding; Adaptive remeshing

Short Definition A major challenge when solving a PDE in numerical simulation is the need to improve the quality of a given computational mesh. A desired mesh would typically have a high proportion of its points in the subdomain(s) where the PDE solution varies rapidly, and to avoid oversampling, few points in the rest of the domain. Given a mesh, the goal of an adaptive mesh refinement or remeshing process is to locally refine and coarsen it so as to obtain solution resolution with a minimal number of mesh points, thereby achieving economies in data storage and computational efficiency.

A

24

Basic Principles of Adaptive Refinement A ubiquitous need for mesh adaptivity for a wide array of science and engineering problems has led to the development of a profusion of methods. This has made mesh adaptivity both an extremely active, multifaceted area of research and a common stumbling block for the potential user looking for a method which matches their particular needs. Standardization of techniques and terminology has only occurred within individual problem domains. Nevertheless, certain basic principles often apply in describing various adaptivity approaches. Fundamental to any mesh refinement algorithm is the strategy for specifying the individual mesh elements (or cells). More precisely, the size, shape, and orientation of each element must be specified. Two complementary adaptive principles for doing this are equidistribution and alignment. While the first has been used in the mesh adaptivity community since first introduced by de Boor for solving ODEs in 1973, an understanding of alignment in multidimensions is relatively recent [3]. Normally, the element sizes are determined by explicitly using some sort of equidistribution process: sizes are equalized relative to a user-specified density function , i.e., mesh sizes are inversely proportional to the magnitude of  in the elements. Common choices of  are some measure of the approximate solution error or some physical solution features such as arc length or curvature. There are a wide range of algorithmic approaches for determining element shape and orientation. Some generate isotropic meshes, where the element aspect ratio (the ratio of the radii of its circumscribed and inscribed spheres) is kept close to one. These can be preferred when they can resolve the solution without using an undue number of mesh points. Anisotropic meshes are favored when there is a need for better alignment of the mesh with certain solution directions, such as those arising due to boundary or interior layers and sharp interfaces. A convenient mathematical framework for analyzing many mesh adaptivity methods utilizes a solution-dependent, positive definite monitor function M (whether or not M is explicitly used in the adaptive process or not). M imposes a metric on the physical domain, and one views the goal of the mesh adaptation to be to generate an M -uniform mesh [3].

Adaptive Mesh Refinement

For such a mesh, the element sizes are (i) constant, and 1 consequently equidistributed in  D det.M / 2 , and (ii) equilateral. The latter condition can be precisely stated in terms of an alignment condition, with the elements’ orientations along directions of the eigenvectors of M and lengths reciprocally proportional to the square root of the singular values of M .

Some Refinement Strategies M. Berger and her co-workers pioneered development of some of the first sophisticated dynamic regridding algorithms, which they called local adaptive mesh refinement or AMR methods. Employed originally for computational fluid dynamics applications, usage has expanded into many other application areas. Beginning with a coarsely resolved base-level regular Cartesian grid, individual elements are tagged for refinement using a user-supplied criterion (such as requiring equidistribution of a mass density function over cells) and structured remeshing done to preserve local uniformity of grids. It is an adaptive strategy well suited for use with finite difference methods. These and the subsequent structured adaptive mesh refinement (SAMR) techniques aim at preserving the high computational performance achievable on uniform grids on a hierarchically adapted nonuniform mesh [4]. Another popular class of adaptive methods for solving PDEs is hp-methods. These finite element and finite volume methods naturally handle irregularly shaped physical domains and compute on an unstructured, or irregular, mesh (where the mesh elements cover the domain in an irregular pattern). With h-refinement, elements are added and deleted but in an unstructured manner, normally using an a posteriori error estimate as the density function. The p-refinement feature permits the approximation order on individual elements to change. There is an extensive literature on these well-studied methods (e.g., see [2] and the article hp-version FEMs). Moving mesh methods, which are designed specifically for time-dependent PDEs, use a fixed number of mesh points and dynamically relocate them in time so as to adapt to evolving solution structures. Since mesh connectivity does not change, they are often amenable to use with finite difference methods. When used with finite element or finite volume methods, they are called r-adaptive (or relocation) methods.

ADI Methods

The adaptivity strategy and analysis focus on how to optimally choose mesh point locations. In [3] they are viewed within a framework of M -uniform meshes, and Huang’s strategy for choosing M to minimize the standard interpolation error bound is developed for important standard cases. This treatment applies for both isotropic error bounds, where M is basically a scalarvalued matrix function I , and anisotropic bounds, where the level of mesh anisotropy and mesh alignment are precisely tied together. Fortuitously, there are often close relationships between this analysis for moving mesh methods and the other important types of adaptive mesh methods, particularly h-adaptive methods. Final comments: For computational PDEs, adaptive mesh refinement is part and parcel of a successful algorithm since computing an accurate solution is codependent upon computing a suitable mesh. In other areas, such as geometric modeling in computer graphics and visualization where a surface mesh is used to represent a given shape, adaptive remeshing is often subordinate to fast processing techniques for the modeling, editing, animation, and simulation process. (For a survey of recent developments in remeshing of surfaces focusing mainly on graphics applications, see [1].) One of the many remaining challenges in the field is ultimately to find common features of adaptive mesh refinement and common terminology between such disparate areas.

Cross-References  Step Size Control

25

ADI Methods Qin Sheng Department of Mathematics, Baylor University, Waco, TX, USA

Mathematics Subject Classification 65M06; 65N06

Synonyms Alternating direction implicit method; Splitting method

Description Finite-difference methods have been extremely important to the numerical solution of partial differential equations. An ADI method is one of them with extraordinary features in structure simplicity, computational efficiency, and flexibility in applications. The original ADI idea was proposed by D. W. Peaceman and H. H. Rachford, Jr., [12] in 1955. Later, J. Douglas, Jr., and H. H. Rachford, Jr., [3] were able to implement the algorithm by splitting the timestep procedure into two fractional steps. The strategy of the ADI approach can be readily explained in a contemporary way of modern numerical analysis. To this end, we let D be a two-dimensional spacial domain and consider the following partial differential equation:

References 1. Alliez, P., Ucelli, G., Gotsman, C., Attene, M.: Recent advances in remeshing of surfaces. In: De Floriani, L., Spagnuolo, M. (eds.) Shape Analysis and Structuring, Mathematics and Visualization, pp. 53–82. Springer, New York (2008) 2. Bangerth, W., Rannacher, R.: Adaptive Finite Element Methods for Differential Equations. Lectures in Mathematics, p. 207. Birkhauser Verlag, Basel (2003) 3. Huang, W., Russell, R.D.: Adaptive Moving Meshes, p. 432. Springer, New York (2011) 4. Plewa, T., Linde, T., Weirs, V.G. (eds.): Adaptive Mesh Refinement – Theory and Applications. Series in Lecture Notes in Computational Science and Engineering, vol. 41, p. 554. Springer, New York (2005)

@u D F u C Gu; .x; y/ 2 D; t > t0 ; @t

(1)

where F ; G are linear spacial differential operators. Assume that an appropriate semidiscretization of (1) yields the following system: v 0 D Av C Bv; t > t0 ;

(2)

where A; B 2 Cnn ; AB ¤ BA in general, and v 2 Cn approximates u on D. Let v.t0 / D v0 be an

A

26

ADI Methods

initial vector given. Then for arbitrary > 0; the exact and homogeneous Dirichlet boundary conditions are solution of (2) can be provided by the variation-of- used. Let Dh be a uniform mesh region superimposed over D with a constant step size h > 0 and w˛;ˇ D constant formula [7] w.x˛ ; yˇ / be a mesh function defined on Dh : If finite Z differences e .  /A Bv.t C /d ; t  t0 : v.t C / D e A v.t/C 0

uj C1;k  2uj;k C uj 1;k uj;kC1  2uj;k C uj;k1 ; h2 h2

An application of the left-point rule and [0/1] Pad´e approximant to the above equation, dropping all are used for approximating spacial derivatives introtruncation errors, offers the fully discretized scheme duced by (7), then the ADI method for solving (1) is a collection of w.t C / D .I  A/1 w.t/ `C1=2

C .I  A/1 Bw.t/

.1 C 2j;k /wj;k

D .I  A/1 .I C B/w.t/; t  t0 ;

where w approximates v: By the same token, from the exact solution of (2), v.t C 2 / D e B v.t C / Z C e .  /B Av.t C C /d ; t C  t0 ; we acquire that

`C1=2

D .1  2j;k /w`j;k C j;k w`j;kC1 (3)

0

`C1=2

 j;k wj C1;k  j;k wj 1;k

C j;k w`j;k1 ; .xj ; yk / 2 Dh I

(8)

`C1 `C1 `C1 .1 C 2j;k /wj;k  j;k wj;kC1  j;k wj;k1 `C1=2

D .1  2j;k /wj;k

`C1=2

C j;k wj C1;k

`C1=2

C j;k wj 1;k ; .xj ; yk / 2 Dh ;

(9)

where j;k D

t t aj;k ; j;k D 2 bj;k ; t  t0 ; 2 2h 2h

w.t C2 / D .I  B/1 .I C A/w.t C /; t C  t0 : are generalized CFL numbers. Note that, while (8) (4) is implicit in the x direction and explicit in the y Let t D 2 be the temporal step and denote w` D direction, (9) is alternatively implicit in the y direction w.t/; w`C1=2 D w.t C /; w`C1 D w.t C 2 /: The and explicit in the x direction. In fact, both equations ADI method for solving (1) follows immediately from can be solved as tridiagonal systems since there exists a permutation matrix P such that A D PBP > for (3), (4), corresponding matrices A; B in (2). This alternative     direction, or split, procedure has significantly reduced t t A w`C1=2 D I C B w` ; (5) the complexity and amount of computations of the I 2 2 given problem [3, 17].     t t Further, in circumstances when D is rectangular, A `C1 `C1=2 B w A w I D IC ; t  t0 : is block diagonal with same-sized tridiagonal blocks 2 2 (6) and B is block tridiagonal with same-sized diagonal blocks under a properly formulated v. Furthermore, A; B commute if a; b are constants. Example 1 Suppose that Finally, we may have noticed that with (7), (1) connects to many important partial differential equations in applications. For instance, it is a diffusion equation @2 @2 F D a.x; y/ 2 ; G D b.x; y/ 2 ; .x; y/ 2 D; when a; b > 0 or a paraxial @x @y p Helmholtz equation if a D (7) b D i=.2/ with i D 1 and   1; .x; y/ 2 D:

ADI Methods

27

Example 2 Consider (7) and homogeneous Dirichlet boundary conditions. Let Dp;q be a nonuniform mesh region superimposed over D: If we approximate the spacial derivatives introduced by finite differences 

2 pj C pj 1 2 qk C qk1



uj C1;k  uj;k uj;k  uj 1;k  pj pj 1

uj;kC1  uj;k uj;k  uj;k1  qk qk1

 ;

where r is the gradient vector and a > 0 is a function of x and y: There is no need to expand the equation to meet the format of (1). Instead, it is often more appropriate to semidiscretize (12) directly for (2). Assume that homogeneous Dirichlet boundary conditions are used. A continuing central difference operation on a uniform spacial mesh Dh yields

 ;

0 vk;j D

1

ak1=2;j vk1;j C ak;j 1=2 vk;j 1 h2

where p˛ D x˛C1  x˛ ; qˇ D yˇC1  yˇ ; then  .1 C C j;k C j;k /wj;k

`C1=2

CakC1=2;j vkC1;j C ak;j C1=2 vk;j C1   ak1=2;j C ak:j 1=2 C akC1=2;j

  C j;k wj C1;k  j;k wj 1;k `C1=2

`C1=2

  C ak;j C1=2 vk;j

 ` C ` D .1  C j;k  j;k /wj;k C j;k wj;kC1 ` C  j;k wj;k1 ; .xj ; yk / 2 Dp;q I

(10)

D

 `C1 C `C1  `C1 .1 C C j;k C j;k /wj;k  j;k wj;kC1  j;k wj;k1  D .1  C j;k  k;j /wj;k

`C1=2

C C j;k wj C1;k

`C1=2

C  j;k wj 1;k ; .xj ; yk / 2 Dp;q ; `C1=2

C (11)

  1

ak1=2;j vk1;j  ak1=2;j C akC1=2;j vk;j 2 h  C akC1=2;j vkC1;j   1

ak;j 1=2 vk;j 1  ak:j 1=2 C ak;j C1=2 vk;j 2 h  C ak;j C1=2 vk;j C1 k; j D 1; 2; : : : ; n:

are equivalent to (5) and (6), where C j;k D

t aj;k ; pj .pj C pj 1 /

 j;k D

t aj;k ; .pj C pj 1 /pj 1

C j;k D

t bj;k ; qk .qk C qk1 /

 j;k D

t bj;k ; t  t0 ; .qk C qk1 /qk1

The arrangement leads to the semidiscretized system (2) where block tridiagonal matrices A; B contain the contribution of the differentiation in the x and y variables, respectively [7]. Remark 1 The same ADI method can be used if a linear nonhomogeneous equation, or nonhomogeneous boundary conditions, is anticipated. In the situation, we only need to replace (2) by

are again generalized CFL numbers. Note that (10) is v 0 D Av C Bv C ; t > t0 ; again implicit in the x direction and explicit in the y direction and (11) is implicit in the y direction and explicit in the x direction. which has the solution Example 3 Consider the two-dimensional advectiondiffusion equation @2 u D r.aru/; 0  x; y  1; @t

(12)

v.t C / D e A v.t/ Z e .  /A ŒBv.t C / C .t C /d ; t  t0 : C 0

A

28

ADI Methods

By the same token, we have

Integrals 1 ; 2 can be evaluated exactly in many cases, say, when  is a constant vector.

v.t C 2 / D e B v.t C / Z C e .  /B ŒAv.t C C / 0

C.t C C /d ; t C  t0 : The integral equations lead to the ADI method w.t C / D .I  A/1 .I C B/w.t/ C

1;

Remark 3 The same ADI strategy can be used for solving partial differential equations consisting of multiple components, such as multidimensional problems. As an illustration, we consider

t  t0 ;

w.t C 2 / D .I  B/1 .I C A/w.t C / C

2;

t C  t0 ;

@u D F u C Gu C Hu; .x; y/ 2 D; t > t0 : @t

where Z 1

Z 2



D

Note that operators F ; G; H need not to be dimensional. Given any > 0; its solution can be written as

e .  /A .t C /d ;

0

D

Remark 2 The same ADI method can be extended for the numerical solution of certain nonlinear, or even singular, partial differential equations. In the particular case if semilinear equations are considered, then the only additional effort needed is perhaps to employ suitable numerical quadratures for (13) (Fig. 1).

e .  /B .t C C /d :

(13)

0

Z



v.t C / D e A v.t/ C

e .  /A .B C C /v.t C /d ; t  t0 ;

0

Z



v.t C 2 / D e v.t C / C B

e .  /B .C C A/v.t C C /d ; t C  t0 ;

0

Z

v.t C 3 / D e

C



v.t C 2 / C

e .  /C .A C B/v.t C 2 C /d ; t C 2  t0 ;

0

where A; B; C are due to semidiscretizations involv- Maxwell’s equations, and stochastic differential equaing F ; G, and H: The rest of discussions is similar to tions in various applications. A large number of investigations and results can be found in the latest those before. publications [7, 15]. Remark 4 The ADI method can be used together with some highly effective numerical strategies, such as Remark 6 We note that A; B in (2) are not necessary temporal and spacial adaptations, and compact finite- matrices. They can be more general linear or nonlinear difference schemes. Detailed discussions can be found operators. This leads to an exciting research field of in numerous recent publications [15, 16]. operator splitting, in which important mathematical Remark 5 The ADI method can be modified for solv- tools, such as semigroups, Hopf algebra, and symplectic integrations [1, 7, 10], play fundamental roles. ing other types of differential equations such as @2 u D F u C Gu; @t 2 F u C Gu D ;

Remark 7 Basic ideas of the ADI method have been extended well beyond the territory of traditional finitedifference methods. The factored ADI strategy for Sylvester equations in image processing is a typical

ADI Methods

29

100

γ

A

0 0.5 0.5

ADI Methods, Fig. 1 The blow-up profile of a singular nonlinear reaction-diffusion equation solution [15] obtained by using the ADI method (16) on exponentially graded nonuniform spa-

2.5

1.5

3.5

4.5

5.5

cial meshes with a temporal adaptation (Courtesy of Q. Sheng and M. A. Beauregard, Baylor University)

example. Interested reader may wish to continue ex- For a local error analysis, we set w.t/ D v.t/ in (14). ploring the latest discussions for the ADI finite element It follows from (14), (15) that method, ADI spectral and collocation methods, ADI finite-difference time-domain (ADI-FDTD) methods, as well as domain decompositions. ".t C / D w.t C /  v.t C / D

Accuracy and Stability Because of the ways of derivations, it is natural to conjecture that each of (5) and (6), or, equivalently, (3) and (4), is of first-order accuracy locally. The prediction can in fact be verified by using the following straightforward analysis. Consider (3). Recall that

 2  2 A C AB  BA  B 2 v.t/ 2 3  3 5A C 5A2 B  ABA  BA2 C 3Š  AB 2  B 2 A  BAB  B 3 v.t/ CO. 4 /:

Therefore, (3) is of the first order [13]. Note that the accuracy remains as is even when A; B commute. .I  A/1 D I C A C 2 A2 C 3 A3 C 4 A4 C    However, (3) becomes a second-order scheme if B D cA; where c 2 C; since in the case it reduces to a under proper constraints. Substituting the above into Crank-Nicolson method [7]. Verifications of (4) and variations are similar. (3), we acquire that Nevertheless, a continuing operation of (3) and (4) pair is of second order. To see this, we substitute (3) 

 w.t C / D I C .A C B/ C 2 A2 C AB (14) into (4) to yield    C 3 A3 C A2 B C    w.t/; t  t0 : (16) w.t C 2 / D .I  B/1 .I C A/

On the other hand, a direct integration of the linear Schr¨odinger equation (2) yields v.t C / D e .ACB/ v.t/; t  t0 :

.I  A/1 .I C B/w.t/; t  t0 ;

which is the standard Peaceman-Rachford splitting (15) [7, 12, 13]. Thus,

30

ADI Methods



 w.t C 2 / D I C .B C A/ C 2 B 2 C BA    C 3 B 3 C B 2A C    

  I C .A C B/ C 2 A2 C AB    C 3 A3 C A2 B C    w.t/

D I C 2 .A C B/ C 2 2 .A C B/2  C 2 3 A3 C A2 B C BA2   C B 2 A C BAB C B 3 C    w.t/: Let w.t/ D v.t/ and replace by 2 in (15). It follows immediately from the above that ".t C 2 / D w.t C 2 /  v.t C 2 / D

2 3  3 A C A2 B  2ABA C BA2 3  C B 2 A C BAB  2AB 2 C B 3 v.t/ CO. 4 /; t  t0 :

Closely Related Issues I. The LOD Method. The introduction and original analysis of this method are due to E. G. D’Yakonov, G. I. Marchuk, A. A. Samarskii, and N. N. Yanenko [4, 9, 20]. Recall the semidiscretized system (2). Needless to say, its solution (15) can be approximated via the exponential splitting [13]: e 2 .ACB/ D e 2 A e 2 B C O. 2 /; ! 0;

Therefore, the Peaceman-Rachford splitting is second order. This accuracy cannot be further improved even when B D cA: Similar to other numerical methods, the numerical stability of the ADI method depends on the properties of A; BI the functional space; and the norm used. All these can be traced back to the original differential equation problem involving (1) and the discretization utilized. In a general sense, the stability of an ADI scheme is secured if the inequality, kM k  1;

spectrum or Fourier analysis. A typical proof of the unconditional stability when (7) is involved can be found in [6]. Stability analysis of the ADI method for boundary value problems is in general more sophisticated. Approaches via the two aforementioned major tools are also different [7]. Asymptotic and nonconventional stability definitions have also been proposed for particular ADI applications, such as highly oscillatory wave equations and nonlinear problems [15, 16]. If the linear partial differential equation problem is well posed, then the convergence of its ADI numerical solution follows from the Lax equivalence theorem [7].

(18)

that is, v.t C 2 /  e 2 A e 2 B v.t/; t  t0 : Thus, an application of the [1/1] Pad´e approximant leads immediately to the local one-dimensional, or LOD, method: w.t C 2 / D .I  A/1 .I C A/

(19)

.I  B/1 .I C B/w.t/; t  t0 : (17)

holds for certain norms, where M D .I  A/1 .I C B/; .I  B/1 .I C A/, or .I  B/1 .I C A/.I  A/1 .I C B/ in case if (3), (4), or (16) is utilized. If a Cauchy problem is considered, then verifications of (17) can be straightforward via either matrix

The above may look like another Peaceman-Rachford splitting by a first glance, but they are fundamentally different. Nevertheless, (19) can be reformu lated to     t t `C1=2 B w B w` I I D IC 2 2

(20)

ADI Methods

31

    II. The Exponential Splitting. Let A 2 Cnn : t t `C1 A w A w`C1=2 ; t  t0 ; I D IC Assume that A D A1 C A2 C    C Am and Ak Aj ¤ 2 2 (21) Aj Ak ; 1  k; j  m; k ¤ j: We wish to approximate e A by a convex combination of the products of matrix in a same way for (5) and (6). Although each of the exponentials, above is a second-order Crank-Nicolson method in its respective direction, their combination, (19), is of firstorder accuracy due to the limitation of (18). In other words, while in the ADI method, two first-order onedimensional solvers (3) and (4) form a second-order two-dimensional solver (16); two second-order onedimensional solvers in the LOD approach generate a first-order two-dimensional method (19). However, this does not reduce any popularity of the LOD method, partially due to its distinguished computational advantages as demonstrated in (20) and (21). It can be readily proven that the LOD method is unconditionally stable if all eigenvalues of A; B lie in the left half of the complex plane [7, 13]. This also leads to the convergence. Naturally, many questions about the LOD method emerge. One of them is: can it be of a higher order? The answer is affirmative. But how? This leads to the fascinating field of exponential splitting.

R. / D

K X

Y

J.k/

k

e ˛k;j Ar.k;j / ;

j D1

kD1

1  r.k; j /  m; k  0;

(22)

without tapping into the Baker-Campbell-Hausdorff [5] or Lie-Trotter [19] formula. The function R serves as a foundation to a number of extremely important splitting methods, in addition to the LOD scheme (19) based on (18). The most well-known specifications of (22) include the Strang’s splitting [10, 17], e 2 .A1 CA2 CCAm / D e A1 e A2 e A3    e Am1 e 2 Am e Am1    e A2 e A1 C O. 3 /; (23)

100 0.025 10−5 10−10

0.02

10−15

ε

ε

0.015

0.01

10−20 10−25 10−30

0.005

10−35 0 10−2

10−1

100

101

τ

ADI Methods, Fig. 2 Global error estimate [14] of the first-order exponential splitting with five distinctive pairs of random matrices whose eigenvalues lie in the left half of the complex plane. These matrices are frequently used in statistical

10−2

10−1

100

101

τ

computations. A logarithmic scale is used in the y direction in the second frame (spectral norm used, courtesy of Q. Sheng, Baylor University)

A

32

ADI Methods

and the parallel splitting [13, 14], e 2 .A1 CA2 CCAm / D

1  2 A1 2 A2 e e    e 2 Am 2 C e 2 Am e 2 Am1    e 2 A1 CO. 3 /;

 (24)

as ! 0: Both formulas are of second order. For more general investigations, we define   ". / D R. /  e .A1 CA2 CCAm / D O pC1 ; ! 0: It is shown by Q. Sheng [13] in 1989 that p  2 if ˛k;j  0; 1  j  J.k/; 1  k  K: (25) This result was later reconfirmed by M. Suzuki [18] and several others [1, 10, 11] independently. The statement (25) reveals an accuracy barrier for all

splitting methods based on (22). More specifically speaking, the maximal order of accuracy for exponential splitting methods is two, as far as the positivity constraints need to be observed. These constraints, unfortunately, are often necessary for the stability whenever diffusion operators A; A1 ; A2 ; : : : ; Am are participated. Inspired by a huge potential in applications including the HPC and parallel computations, a wave of studies for more effective local and global error estimates of the exponential splitting has entered a new era since the preliminary work in 1993 [2, 8, 14]. Now, let us go back to the LOD method. A straightforward replacement of (18) by either (22) or (23) yields a second-order new scheme. The additional computational, as well as programming, cost incurred is only at a minimum (Fig. 2). III. Connections Between ADI and LOD Methods. Recall (16). Applying the formula twice, we acquire that

2 w.t C 4 / D .I  B/1 .I C A/.I  A/1 .I C B/ w.t/ D .I  B/1 .I C A/.I  A/1 .I C B/.I  B/1 .I C A/.I  A/1 .I C B/w.t/; t  t0 : Thus,

.I C B/w.t C 4 / D .I C B/.I  B/1 .I C A/.I  A/1 .I C B/.I  B/1 .I C A/.I  A/1 .I C B/w.t/; t  t0 :

Since .I C A/.I  A/1 ; .I C B/.I  B/1 are second-order [1/1] Pad´e approximants of e 2 A ; e 2 B , respectively, denote w0 . / D .I C B/w. / and drop all truncation errors. We obtain that

w1 .t C 4 / D e A e 2 B e 2 A e 2 B e A w1 .t/    D e A e B e B e 2 A e B  B A  e e w1 .t/; t  t0 :

(27) w0 .t C 4 / D e 2 B e 2 A e 2 B e 2 A w0 .t/ 2  D e 2 B e 2 A w0 .t/; t  t0 : (26) Both (26) and (27) indicate repeated applications of LOD strategies or particular settings of R in (22). The above exponential splitting can be comprised in These formulas are important to not only applied and a symmetric way. To this end, we may let w1 . / D computational mathematics but also modern quantum e A w0 . /: It follows from (26) immediately that and statistical physics [10, 14, 18].

Adjoint Methods as Applied to Inverse Problems

References 1. Chin, S.A.: A fundamental theorem on the structure of symplectic integrators. Phys. Lett. A 354, 373–376 (2006) 2. Descombes, S., Thalhammer, M.: An exact local error representation of exponential operator splitting methods for evolutionary problems and applications to linear Schr¨odinger equations in the semi-classical regime. BIT 50, 729–749 (2010) 3. Douglas, J. Jr., Rachford, H.H. Jr.: On the numerical solution of heat conduction problems in two and three space variables. Trans. Am. Math. Soc. 82, 421–439 (1956) 4. D’Yakonov, E.G.: Difference schemes with splitting operator for multi-dimensional nonstationary problems. Zh. Vychisl. Mat. i Mat. Fiz. 2, 549–568 (1962) 5. Hausdorff, F.: Die symbolische Exponentialformel in der Gruppentheorie. Ber Verh Saechs Akad Wiss Leipzig 58, 19–48 (1906) 6. Hundsdorfer, W.H., Verwer, J.G.: Stability and convergence of the Peaceman-Rachford ADI method for initial-boundary value problems. Math. Comput. 53, 81–101 (1989) 7. Iserles, A.: A First Course in the Numerical Analysis of Differential Equations, 2nd edn. Cambridge University Press, London/New York (2011) 8. Jahnke, T., Lubich, C.: Error bounds for exponential operator splitting. BIT 40, 735–744 (2000) 9. Marchuk, G.I.: Some applications of splittingup methods to the solution of problems in mathematical physics. Aplikace Matematiky 1, 103–132 (1968) 10. McLachlan, R.I., Quispel, G.R.W.: Splitting methods. Acta Numer. 11, 341–434 (2002) 11. Moler, C., Van Loan, C.: Nineteen dubious ways to compute the exponential of a matrix, twenty-five years later. SIAM Rev. 45, 3–46 (2003) 12. Peaceman, D.W., Rachford, H.H. Jr.: The numerical solution of parabolic and elliptic differential equations. J. Soc. Ind. Appl. Math. 3, 28–41 (1955) 13. Sheng, Q.: Solving linear partial differential equations by exponential splitting. IMA J. Numer. Anal. 9, 199–212 (1989) 14. Sheng, Q.: Global error estimate for exponential splitting. IMA J. Numer. Anal. 14, 27–56 (1993) 15. Sheng, Q.: Adaptive decomposition finite difference methods for solving singular problems-a review. Front. Math. China (by Springer) 4, 599–626 (2009) 16. Sheng, Q., Sun, H.: On the stability of an oscillationfree ADI method for highly oscillatory wave equations. Commun. Comput. Phys. 12, 1275–1292 (2012) 17. Strang, G.: On the construction and comparison of difference schemes. SIAM J. Numer. Anal. 5, 506–517 (1968) 18. Suzuki, M.: General theory of fractal path integrals with applications to many body theories and statistical physics. J. Math. Phys. 32, 400–407 (1991) 19. Trotter, H.F.: On the product of semi-groups of operators. Proc. Am. Math. Soc. 10, 545–551 (1959) 20. Yanenko, N.N.: The Method of Fractional Steps; the Solution of Problems of Mathematical Physics in Several Variables. Springer, Berlin (1971)

33

Adjoint Methods as Applied to Inverse Problems

A

Frank Natterer Department of Mathematics and Computer Science, Institute of Computational Mathematics and Instrumental, University of M¨unster, M¨unster, Germany

Synonyms Adjoint differentiation; Back propagation; Time reversal

Definition Adjoint methods are iterative methods for inverse problems of partial differential equations. They make use of the adjoint of the Fr´echet derivative of the forward map. Applying this adjoint to the residual can be viewed as time reversal, back propagation, or adjoint differentiation.

Overview Inverse problems for linear partial differential equations are nonlinear problems, but they often have a bilinear structure; see [9]. This structure can be used for iterative methods. As an introduction, see [11].

Wave Equation Imaging As a typical example that has all the relevant features, we consider an inverse problem for the wave equation. Let ˝ be a domain in Rn ; n > 1 and T D Œ0; t1 . Let uj be the solution of @2 uj D f uj in ˝  T; @2 t @uj D qj on @˝  T; @ u D 0 for t < 0:

(1) (2) (3)

34

Adjoint Methods as Applied to Inverse Problems

Here qj represents a source, and  is the exterior normal on @˝, and j D 0; : : : ; p  1. The problem is to recover f from the values gj D uj j@˝T . Such problems come up, e.g., in ultrasound tomography; see, e.g., [11], chapt. 7.4. Let uj D uj .f / be the solution of (1,2,3) and put Rj .f / D uj .f /j@˝T . Then the problem amounts to solving the nonlinear system

For this to make sense, we have to consider Rj as an operator between suitable Hilbert spaces. In [7] it has been shown that under natural assumptions (e.g., f positive), Rj is a differentiable operator from H 2 .˝/ into H 1=2 .@˝  T / with H s the usual Sobolev spaces. In order to compute Rj0 .f /, we replace f; u in (1,2,3) by f C h; u C w with h; w small and ignore higher order terms. We get for w

Rj .f / D gj ; j D 0; : : : ; p  1

@2 w D f w C hu in @˝  T; @t 2 @w D 0 on @˝  T; @ w D 0 for t < 0:

(8)

Rj0 .f /h D wj@˝T

(9)

(4)

for f . A natural way to solve this system is the Kaczmarz method; see [11], chapter 7. For linear problems such as X-ray CT, it is known as ART; see [8]. The Kaczmarz method is an iterative method with the update And we have f  ˛Rj0 .f / .Rj .f /  gj /:

f

(5)

(6) (7)

Here j is taken mod p, and ˛ is a relaxation parameter. For the computation of the adjoint – as an operator Going through all the p equations once is called one from L2 .˝  T / into L2 .@˝  T / – we make use of sweep. Rj0 is the Fr´echet derivative and Rj0 its adjoint. Green’s second identity in the form

Z Z  ˝

T

  Z Z  1 @2 w 1 @2 z  w zdtdx   z wdtdx D f @t 2 f @t 2 ˝ T  Z   t1 Z Z  @w @z @z @w wz zw dtdx C dx : @ @ @t @t @˝ T ˝ 0

This holds for any functions w; z on ˝  T . Choosing for w the solution of (6,7,8) and for z the solution of the final value problem @2 z D f z in ˝  T; @t 2 @z D g on @˝  T; @ z D 0 for t < t1 we obtain Z ˝

h f

Z

Z

Z uj zdtdx D T

or, using inner products,

wgdtdx; @˝

T

(13)

(11)

  Z 1 uj zdt D .Rj0 .f /h; g/L2 .@˝T / : h; f T L2 .˝/ (16) Hence

(12)

(10)

.Rj0 .f // g D

1 f

Z uj zdt:

(17)

T

It is this final value structure of the adjoint which suggests the names time reversal and back propagation. (14) In order to show what can be achieved, we reconstruct a breast phantom [2] that is frequently used in ultrasound tomography. We simulated data for 32 sources with a central frequency of 100 kHz and reconstructed (15) by the adjoint method with 3 resp. 6 sweeps; see Fig. 1. The method can also be used in the frequency domain, i.e., for the inverse problem of the Helmholtz equation [10]. For the corresponding electromagnetic

Adjoint Methods as Applied to Inverse Problems

35

A

Adjoint Methods as Applied to Inverse Problems, Fig. 1 Reconstruction of breast phantom from 32 sources at 100 kHz. Left: after 3 sweeps. Middle: after 6 sweeps. Right: original

inverse problem which involves the Maxwell equa- an operator from H 2 .˝/  H 1 .˝/ into L2 .@˝/; see tions, the method has been used in [6, 12]. [3]. Exactly as in the ultrasound case, we see that the adjoint of Rj0 .f /, as an operator from L2 .˝/  Ł2 .˝/ into L2 .@˝/, is given by .Rj0 .f // .g./ D .r uN j  Optical and Impedance Tomography rz; uN j z/> where z is the solution of The adjoint method as explained above can be used for a variety of inverse problems. In optical tomography [1] the forward problem is d iv. ruj /  . C i !/uj D 0 in ˝; @u D qj on @˝: @

(18)

(22)

z D g on @˝:

(23)

A special case is impedance tomography [4, 5]( D 0; ! D 0).

(19)

is the diffusion coefficient,  the attenuation, and ! the frequency of the source qj . The problem is to recover ;  from gj D uj j@˝ , all other quantities being known. We put f D . ; /> and Rj .f / D uj j@˝ . We then have to solve the nonlinear system Rj .f / D gj ; j D 0; : : : ; p  1 for f . Very much in the same way as in the case of ultrasound tomography we find that Rj0 .f /h D wj@˝ ; h D .h1 ; h2 /> , where w is the solution of d iv. rw/  . C i !/w D d iv.h1 ruj / C h2 uj in ˝; (20) @w D 0 on @˝: @

d iv. rz/  .  i !/z D 0 in ˝;

(21)

Under natural assumptions, such as > 0, it follows from the elliptic regularity of the problem that this is in fact the Fr´echet derivative of Rj .f / viewed as

References 1. Arridge, S.R.: Optical tomography in medical imaging. Inverse Probl. 15, R41–R93 (1999) 2. Borup, D.T., Johnson, S.A., Kim, W.W., Berggren, M.J.: Nonperturbative diffraction tomography via Gauss-Newton iteration applied to the scattering integral equation. Ultrason. Imaging 14, 69–85 (1992) 3. Connolly, T.J., Wall, J.N.: On Fr´echet differentiability of some nonlinear operators occurring in inverse problems: an implicit function theorem approach. Inverse Probl. 6, 949– 966 (1990) 4. Cheney, M., Isaacson, D., Newell, J.C.: Electrical impedance tomography. SIAM Rev. 41, 85–101 (1999) 5. Dobson, D.: Convergence of a reconstruction method for the inverse conductivity problem. SIAM J. Appl. Math. 52, 442–458 (1992) 6. Dorn, O., Bertete-Aguirre, H., Berryman, J.G., Papanicolaou, G.C.: A nonlinear inversion method for 3D electromagnetic imaging using adjoint fields. Inverse Probl. 15 1523–1558 (1999) 7. Dierkes, T., Dorn, O., Natterer, F., Palamodov, V., Sielschott, H.: Fr´echet derivatives for some bilinear inverse problems. Siam J. Appl. Math. 62, 2092–2113 (2002)

36 8. Hermann, G.: Image Reconstruction From Projections. Academic Press, San Francisco (1980) 9. Natterer, F.: Numerical methods for bilinear inverse problems. Preprint, University of M¨unster, Department of Mathematics and Computer Science (1996) 10. Natterer, F., W¨ubbeling, F.: A propagation-backpropagation algorithm for ultrasound tomography. Inverse Probl. 11, 1225–1232 (1995) 11. Natterer, F., W¨ubbeling, F.: Mathematical Methods of Image Reconstruction, p. 275. SIAM, Philadelphia (2001) 12. V¨ogeler, M.: Reconstruction of the three-dimensional refractive index in electromagnetic scattering by using a propagation-backpropagation method. Inverse Probl. 19, 739–753 (2003)

Advancing Front Methods Yasushi Ito Aviation Program Group, Japan Aerospace Exploration Agency, Mitaka, Tokyo, Japan

Mathematics Subject Classification 32B25

Synonyms Advancing Front Techniques (AFT)

Short Definition Advancing front methods are commonly used for triangulating a given domain in two dimensions (2D) or three dimensions (3D).

Description Basic Algorithm The concept of the advancing front methods was first proposed by Lo [1], Peraire et al. [2], and L¨ohner [3]

Advancing Front Methods

for discretizing 2D domains with isotropic triangles. It was then extended to 3D domains for creating isotropic tetrahedra. The typical advancing front methods require the following steps [4]: 1. Discretize the boundaries of the domain to be meshed as a set of edges in 2D (faces in 3D), which is called as the initial front in the advancing front methods (Fig. 1a). The initial front forms closed curve(s) (surface(s) in 3D) that encloses the domain to be triangulated. Elements will be created one by one on the front by adding new points in the interior of the domain. 2. Select the shortest edge (the smallest face in 3D) as a base of the triangle (tetrahedron in 3D) to be generated. 3. Determine the ideal position for the vertex of the triangle (tetrahedron in 3D) based on several local and global parameters including user-specified parameters. This enables the generation of elements in variable size with desired stretching. 4. Select other possible candidates for the vertex from the points already generated by defining a searching circle (sphere in 3D) that contains the base and the ideal point. 5. Select the best candidate passing several validity and quality criteria, such as no intersection of new edges (faces in 3D) with the front and positive area (volume) of the element to be created. Create a new triangle (tetrahedron in 3D) using the point and then update the front so that it surrounds the remainder of the domain that needs to be meshed. 6. Continue steps 2 through 5 until the front becomes empty (Fig. 1e). Consequently, the size and quality of elements near the initial front can be controlled easily by the advancing front methods. The connectivity of the boundary edges (and the boundary faces in 3D) is naturally preserved, while the Delaunay triangulation does not. However, the advancing front methods tend to create lower-quality elements than Delaunay triangulation methods. Node smoothing and edge or face swapping are essential at the end of the mesh generation process to improve the quality of elements (Fig. 1f). The advancing front method can also be combined with a Delaunay triangulation method to produce better-quality elements (see the entry on Delaunay triangulation).

Advancing Front Methods Advancing Front Methods, Fig. 1 Various stages of advancing front method applied for letter i in 2D: (a) boundary points and edges (initial front); (b) two, (c) 99, (d) 116 and (e) 156 triangles created; (f) final mesh after node smoothing

Advancing Front Methods, Fig. 2 Mesh generation around a two-element airfoil: (a) triangular, (b) hybrid, and (c) anisotropic meshes

37

A

38

Advancing Layers Method The advancing front methods can be extend to create hybrid (Fig. 2b) or anisotropic (Fig. 2c) meshes for high Reynolds number viscous flow simulations, which require high spatial resolution in the direction normal to no-slip walls, to resolve boundary layers well [5–10]. This type of the advancing front methods is usually called as the advancing layers method [5] or advancing normals method [9]. Hybrid meshes in 3D consist of triangular-prismatic and/or hexahedral layers on the no-slip walls, tetrahedra in the remainder of the domain, and a small number of pyramids to connect quadrilateral faces with triangular ones. Hybrid meshes can be easily converted to anisotropic meshes by subdividing elements into right-angle simplexes.

References 1. Lo, S.H.: A new mesh generation scheme for arbitrary planar domains. Int. J. Numer. Methods Eng. 21, 1403–1426 (1985). doi:10.1002/nme.1620210805 2. Peraire, J., Vahdati, M., Morgan, K., Zienkiewicz O.: Adaptive remeshing for compressible flow computations. J. Comput. Phys. 72, 449–466 (1987). doi:10.1016/0021-9991(87)90093-3 3. L¨ohner, R.: Some useful data structures for the generation of unstructured grids. Commun. Appl. Numer. Methods 4, 123–135 (1988). doi:10.1002/cnm.1630040116 4. Morgan, K., Peraire, J., Peir´o, J.: Unstructured grid methods for compressible flows. In: Special Course on Unstructured Grid Methods for Advection Dominated Flows. Advisory Group for Aerospace Research and Development (AGARD) Report 787, pp. 5-1–5-39. AGARD, France (1992) 5. Pirzadeh, S.: Unstructured viscous grid generation by the advancing-layers method. AIAA J. 32(2), 1735–1737 (1994). doi:10.2514/3.12167 6. L¨ohner, R.: Matching semi-structured and unstructured grids for Navier–Stokes calculations. In: Proceedings of the AIAA 11th CFD Conference, Orlando, FL, AIAA Paper 933348-CP, pp. 555–564 (1993) 7. Kallinderis, Y., Khawaha, A., McMorris, H.: Hybrid prismatic/tetrahedral grid generation for complex geometries. In: 33rd Aerospace Sciences Meeting and Exhibit, Reno, NV, AIAA paper 95-0211 (1995) 8. Connell, S.D., Braaten, M.E.: Semistructured mesh generation for three-dimensional Navier–Stokes calculations. AIAA J 33(6), 1017–1024 (1995). doi:10.2514/3.12522 9. Hassan, O., Morgan, K., Probert, E.J., Peraire, J.: Unstructured tetrahedral mesh generation for threedimensional viscous flows. Int. J. Numer. Methods Eng. 39(4), 549–567 (1996). doi:10.1002/(SICI)10970207(19960229)39:4549::AID-NME868>3.0.CO;2-O

Agent-Based Models in Infectious Disease and Immunology 10. Ito, Y., Murayama, M., Yamamoto, K., Shih, A.M., Soni, B.K.: Efficient hybrid surface and volume mesh generation for viscous flow simulations. In: 20th AIAA Computational Fluid Dynamics Conference, Honolulu, HI, AIAA Paper 2011–3539 (2011)

Agent-Based Models in Infectious Disease and Immunology Catherine A.A. Beauchemin Department of Physics, Ryerson University, Toronto, ON, Canada

Synonyms Cellular automata (or automaton); Individual-based models

Acronyms ABM: Abs: CTL: IR: NK: ODE: Th:

Agent-based model Antibodies Cytotoxic T lymphocyte Immune response Natural killer cells Ordinary differential equation Helper T cell

Short Definition Agent-based models (or ABMs) of infectious diseases and/or immunological systems are computer models for which the key units of the modeled system – pathogens, target cells, immune cells – are explicitly represented as discrete, autonomous agents characterized by a set of states (e.g., infected, activated). Transition of an agent between states or its movement through space (if applicable) obeys a set of rules based on the agent’s current internal states (e.g., if the cell has been alive for 3 h then it transitions to the dead state), that of its neighbors (e.g., if the cell’s neighbor is infected, then it has a 20 % chance of transitioning to the infected state), and/or its perception of its external environment (e.g., if the concentration of

Agent-Based Models in Infectious Disease and Immunology

interferon around the cell is above some threshold, it will not produce virus). The discrete, rule-based, and agent-centric nature of ABMs makes them the most natural representation of disease and immune systems elements. The value of the application of ABMs to the study of infectious diseases and immunology is in the insights it can provide on the impact of discrete, few, localized interactions on the overall course and outcome of a disease and/or the activation decision of an immune response.

Description ABMs as Natural Descriptions of the Immune Response and Infectious Diseases Agent-based models (or ABMs) are computer models whose formulation consists of a set of rules controlling the local interactions of explicitly represented autonomous agents with each other and/or with their environment. Because the formulation of the ABM is done only at the level of individual agents, the overall dynamics of the system is not explicitly specified in the model. Rather, it is an emergent property which comes about as a result of the cumulative local interactions between the many individual agents of the system, acting autonomously, with little or no information about the global state of the system. For this reason, the overall dynamics of an ABM are sometimes surprising. An ABM approach is not suitable for all problems, but the immune system, and its interaction with pathogens, lends itself naturally to an ABM description. The immune response (IR) to an infectious disease comprises a variety of agents (e.g., cytotoxic T lymphocytes, dendritic cells, plasma B cells), each with different rules governing their interactions with each other (e.g., a helper T cell activating a B cell) and/or with their environment (e.g., local cytokine concentrations upregulating the expression of receptors on the surface of T cells). The IR to a pathogen is not centrally controlled, and information about the system-wide damage inflicted by the pathogen or its overall control by the IR is not reported and analyzed in a single centralized decision-making location. Instead, the IR is initiated through the local, stochastic interactions of a few agents based on partial local information, which can trigger a cascade of events that will activate different sets, or branches, of the IR (e.g., a cellular versus a humoral adaptive IR). Despite its

39

decentralized nature, the immune system can mount an efficient, organism-wide response to a pathogen, and is capable of complex behavior such as learning (i.e., the recognition of new pathogens) and memory (i.e., the more rapid and effective response to previously encountered pathogens). While ABMs may be the most natural way to describe the IR to an infectious disease, they are not always the simplest or wisest choice. The ease with which one can exhaustively incorporate biological details in an ABM can lead to complicated models with a large number of parameters. Ordinary differential equation (ODE) models, usually requiring a smaller number of parameters, have been more widely applied to the study of the IR and infectious diseases (a good overview is presented in Nowak and May [7]). Ultimately, the modeling approach and the level of detail in implementation should be determined by the question one wishes to address. Typically, ABMs are best suited to address questions where the dynamics of interest is driven by the local, stochastic interactions of a discrete number of agents. This is often the case in modeling the very early and very late events of an infection where cell and pathogen numbers are small, mean-field kinetics does not apply, and small, local fluctuations can make the difference between infection resolution and exponential growth. The Main Agents of the Immune Response A brief overview of the major agents of the IR is presented here to facilitate comprehension of their implementation in an ABM. For a comprehensive account of the key elements of the IR, their function, and their interactions, one should consult an immunology textbook (see for example, Kindt et al. [5]). The IR is divided into two types of responses: the innate and the adaptive response. The cells and molecules that make up the innate IR include the physical barriers comprising the skin and the mucus linings, the danger or damage signals secreted by affected cells in the form of eicosanoid and cytokines, and a wide range of cells such as natural killer (NK) cells, mast cells, dendritic cells, macrophages, and neutrophils, which express receptors that recognize patterns common to a wide range of microorganisms. Because the innate IR reacts to all pathogens in the same generic way, it does not recognize specific pathogens, and therefore cannot recall previous encounters. In contrast, the adaptive immune response is designed to specifically recognize

A

40

a pathogen, and previous exposures lead to a more rapid and effective response against the pathogen. The main cellular components of the adaptive IR are the B (for bone) and T (for thymus) lymphocytes, and their specificity for a restricted set of pathogens is dictated by the specific antigen receptors they express. Not all receptors of the adaptive IR are constructed in the same way, but the recognizable parts of the antigen (the epitopes) and the recognizing part of the receptors consist of a string of amino acids whose sequence and environment defines their folded shape. The compatibility between an antigen’s epitope and a B or T cell receptor is called the affinity, and is a function of how well the antigen and receptor fit together, as with a lock and key. The B cells expressing receptors that recognize the pathogen respond to it by secreting large amounts of their receptors, called antibodies (Abs), which can bind the pathogen and both mark it for removal by the innate IR, and block it from causing further infection (e.g., by forming a virusAb complex). The T lymphocytes that recognize the pathogen are responsible for either secreting cytokines that direct the type of IR which will be made to the pathogen (this is done by helper T lymphocytes, or Th cells), or for killing pathogen-infected cells (this is done by cytotoxic T lymphocytes, or CTLs). The CTL response is referred to as the cellular adaptive response, whereas the response made by the Abs, and the B cells which secreted them, is called the humoral adaptive response. B and T cells with a sufficient affinity for the pathogen will be activated by its presence and undergo a rapid expansion to control the pathogen. After the infection is cleared, a portion of these cells will differentiate into memory cells which stay around to make a more rapid and effective response to the pathogen when it is encountered again. Thus, memory of previously encountered pathogens is stored in a distributed fashion by our immune system via the memory B and T lymphocyte populations. The memory is encoded as a bias in numbers favoring those cells able to recognize the more commonly encountered pathogens. ABM Representation of the Immune Response and Infectious Diseases In choosing a representation for the interactions between the different agents of the IR, or their interaction with a pathogen, the level of details required is set by the problem to be addressed. If one wishes to look at the issue of immune evasion of a virus through

Agent-Based Models in Infectious Disease and Immunology

mutation, the dynamics of T cells competing for antigen recognition, the process of immunodominance, or affinity maturation, it can be necessary to explicitly represent the affinity between, for example, the antigen and a T cell receptor. The explicit representation of the amino acids which make up the antigen’s epitope or the receptor, and their explicit folding and shape matching would be too numerically intensive to permit larger-scale simulations of several interactions, and this level of sophistication is typically unnecessary. Instead, for simplicity and efficiency, epitopes and receptors are often represented using strings of binary digits. Within the framework of a binary string representation, the affinity of a receptor for the antigen’s epitope can be expressed in a number of ways as a function of the number of similar (binary XNOR) or dissimilar (binary XOR) sites between the two strings, and can also consider different alignments, i.e., the number of similar or dissimilar bits as the epitope is shifted with respect to the receptor. An example of the representation of the different agents of the adaptive IR and antigen as well as the computation of their affinity is illustrated in Fig. 1. If fluctuations in quantities such as infected cells or virus concentration over time is the kinetic of interest, there is likely no need to explicitly represent the affinity of the adaptive IR for the pathogen. Instead, an easy simplification is to represent in the ABM only those cells which will actively participate in, and affect, infection kinetics, i.e., those whose affinity satisfies a minimum threshold. The affinity of each cell for the pathogen can then either be randomly assigned from, say, a normal distribution of affinities, or fixed to the average value of affinity for all cells. There is also much flexibility in choosing the appropriate level of details to represent smaller molecular agents such as pathogens, antibodies, and cytokines. These can be represented as individual agents performing a random walk (akin to lattice Boltzmann diffusion) on the simulation grid, able to individually interact, bind with other agents, and form complexes. Alternatively, these entities can be modeled as discrete quantities or continuous concentrations over space whose amount at each site changes locally as more are released or taken up by various agents, and which diffuse through space according to a finite-difference approximation to the diffusion equation.

Agent-Based Models in Infectious Disease and Immunology

41

Antigen 198 218

Receptor 57

Receptor 57

Endocytosis and processing of Ag peptide by APC

B MHC II 137

B MHC II 137 MHC II-peptide 141

Receptor 186

T Agent-Based Models in Infectious Disease and Immunology, Fig. 1 Example of bit string representation of receptors and epitopes. Here, T and B cell receptors and the B cell’s MHC II are represented by 8-bit strings. The MHC II (or major histocompatibility complex II) is found on antigen-presenting cells and is responsible for binding and presenting processed antigens to T cells for recognition. The antigen (Ag) is represented by

Example: Influenza Infection Within a Host Let us consider the implementation of an ABM for the spread of an influenza infection within a host. We will focus our attention on a small patch of the upper respiratory tract. Since the lung predominantly consists of a single layer of cells everywhere except in the trachea, we will represent this patch as a two-dimensional, hexagonal grid with each lattice site corresponding to one susceptible epithelial cell. Furthermore, since the respiratory tract epithelium is folded into a cylinder, we will apply toroidal boundary conditions to our grid such that cells moving (or molecules diffusing) off one edge appear at the opposite edge. We will also implement a simplified cellular IR in the form of CTLs which can move from site to site, activate and proliferate upon encounter with infected cells, kill infected cells, and undergo programmed contraction. We will represent influenza virions (virus particles) as a field across the simulation grid by storing the local virus concentration at each grid site, and solving the diffusion equation over each grid site, Vi;j , using the finite-difference approximation t Ct Vi;j

  4DV t 2DV t X t t V D 1 C Vnei ; i;j .x/2 3 .x/2

two 8-bit strings, with short and long blocks representing 0s and 1s, respectively. The antigen’s bare 8-bit string is used as an epitope for recognition by the B cell receptor. The antigen’s boxed 8-bit string is used to represent an antigen peptide which is processed, and presented on the MHC II of the B cell. (This example representation is modeled after the IMMSIM simulator and the image is modified from Seiden and Celada [8].)

P t is the sum of the virion concentration at where Vnei all six honeycomb neighbors of site .i; j / at time t, DV is the virions’ diffusion coefficient, t the size of your time steps, and x the diameter of one cell or grid site (this equation is derived in Beauchemin et al. [2]). Thus, the model comprises two types of discrete agents, the susceptible epithelial cells and the CTLs, and one continuous field, the influenza virions. The evolution of the system will be governed by the set of rules illustrated in Fig. 2. Once agents and fields have been defined, and rules have been established for their evolution and interactions, it is a matter of implementing the model, and letting it evolve according to the rules. Gaining a better understanding of the dynamics of this system as the parameter values characterizing each rule (e.g., p, c, vCTL ) are varied over their biologically plausible range is achieved by running a large number of simulations, paying attention to both the average behavior, and outlier cases. One may wish to consider performing a sensitivity analysis to characterize how choices in parameter values can affect the model’s kinetics, as discussed in Bauer et al. [1].

A

Agent-Based Models in Infectious Disease and Immunology, Fig. 2 Set of rules for an example ABM of an influenza infection under the control of a CTL response. This decision tree is to be traversed at each time t , for each site .i; j /. Epithelial cells can be in any of three states: uninfected (U ), infected (I ), and dead (D), and are assigned a time of death (tdeath ) chosen at random from a normal distribution of mean D and standard deviation D . CTLs can be in any of two states: naive (N ) or effector (E), and carry three

internal properties: their time of transfer to the effector state (teffector ) chosen randomly from a normal distribution, the number of divisions they have undergone since entering the effector state (Ndiv ), and the time of their next scheduled division (tdiv ). The execution of some of the rules is controlled by parameters such as: the viral production rate (p), the clearance rate of virus (c), the infectivity of the virus (ˇ), and the speed of CTLs (vCTL )

42 Agent-Based Models in Infectious Disease and Immunology

Algorithms for Low Frequency Climate Response

Platforms for the Implementation of ABMs of Immunology and Infectious Diseases The implementation of an ABM can be accomplished in any programming language, but there are also a number of environments specifically designed for this type of model. The construction of a comprehensive immune simulator is a laborious affair, and interested researchers may wish to utilize one of many which already exist. Ultimately, what constitutes an ideal platform will depend on the problem at hand, the desired information, and the available time and computational power. Thankfully, there are many options available. For those interested in immune simulators, IMMSIM and its derivatives are a good place to start. In IMMSIM, receptors and epitopes are represented as binary strings. In its original version IMMSIM implements Th and B cells, antigen presenting cells, Abs, and antigens, representing only the humoral adaptive response (see [8] for a comprehensive description of the simulator). It can simulate the expansion of Th and B cell populations based on a simulated continuous input or a set initial quantity of antigen. It was later extended to include CTLs, epithelial cells, generic cytokines, as well as a more virus-like antigen capable of infecting cells, and replicating within them. Since its creation, IMMSIM has been translated into different languages, and expanded in various directions by several researchers, resulting in IMMSIM23, IMMSIM3, IMMSIM++, IMMSIM-C, C-IMMSIM, and ParIMM. Other immune simulators followed, such as PathSim Visualizer, CAFISS, and SIMMUNE. These simulators were used to investigate a wide range of IR processes such as affinity maturation, hypermutation, rheumatoid factor, the transition process between immune and disease states, vaccine efficiency, and HIV escape mutant selection. For the implementation of the spread of an infectious disease within an individual with no particular emphasis on receptor specificity, many simulation platforms are also readily available, including MASyV, CyCells, and SIS-I and SIS-II. For those wishing to develop their own ABM simulation of immunology or infectious disease, some programming environments can greatly facilitate the task, with some particularly suitable for those with limited programming experience. Among the most popular are NetLogo, StarLogo, and Repast. For reviews of some of these simulators, and the studies to which they were applied, interested readers

43

may wish to refer to Bauer et al. [1], Forrest and Beauchemin [4], Louzoun [6], and Chavali et al. [3].

A References 1. Bauer, A.L., Beauchemin, C.A., Perelson, A.S.: Agentbased modeling of host-pathogen systems: the successes and challenges. Info. Sci. 179(10), 1379–1389 (2009). doi:10.1016/j.ins.2008.11.012 2. Beauchemin, C., Forrest, S., Koster, F.T.: Modeling influenza viral dynamics in tissue. In: Bersini, H., Carneiro, J. (eds.) Proceedings of the 5th International Conference on Artificial Immune Systems (ICARIS 06), Lecture Notes in Computer Science, vol. 4163, pp. 23–36. Springer, Berlin Heidelberg (2006). doi:10.1007/11823940 3 3. Chavali, A.K., Gianchandani, E.P., Tung, K.S., Lawrence, M.B., Peirce, S.M., Papin, J.A.: Characterizing emergent properties of immunological systems with multi-cellular rulebased computational modeling. Trends Immunol. 29(12), 589–599 (2008). doi:10.1016/j.it.2008.08.006 4. Forrest, S., Beauchemin, C.: Computer immunology. Immunol. Rev. 216(1), 176–197 (2007). doi:10.1111/j.1600065X.2007.00499.x 5. Kindt, T.J., Goldsby, R.A., Osborne, B.A., Kuby, J.: Kuby Immunology, 6th edn. W. H. Freeman and Company, New York (2007) 6. Louzoun, Y.: The evolution of mathematical immunology. Immunol. Rev. 216(1), 9–20 (2007). doi:10.1111/j.1600065X.2006.00495.x; 7. Nowak, M.A., May, R.M.: Virus Dynamics: Mathematical Principles of Immunology and Virology. Oxford University Press, Oxford (2000) 8. Seiden, P.E., Celada, F.: A model for simulating cognate recognition and response in the immune system. J. Theor. Biol. 158(3), 329–357 (1992). doi:10.1016/S00225193(05)80737-4

Algorithms for Low Frequency Climate Response Andrey Gritsun Institute of Numerical Mathematics, Moscow, Russia

Definition Terms/Glossary The climate is the ensemble of states passed by the climate system in sufficiently long time period The climate response is the change of statistical characteristics of the climate system in response to the action of external perturbation

44

The fluctuation response relation is the relationship between statistical characteristics of an unperturbed system and its response to external perturbation Chaotic system is the system that shows exponential sensitivity to initial conditions

Algorithms for Low Frequency Climate Response

quality of a model and tune a model so that it will approximate f˚ K .t/g better. This strategy is widely accepted in climate research (as an example, one can mention AMIP, CMIP, and many other projects devoted to the model intercomparison [29]). Now let us suppose that the system is subject to some additional forcing due to the natural (volcanic eruptions, solar variations, etc.) or anthropogenic imDescription pact. As a result, instead of (1) we will have to deal By general definition, “the climate” is the ensemble with perturbed climate system of states passed by the Earth climate system (system d˚ 0 consisting of the atmosphere, the hydrosphere, the D G.˚ 0 ; t/ C ıF .t/ ˚ 2 RN ; (3) cryosphere, the land surface, and the biosphere) in dt sufficiently long time period. In particular, the World Meteorological Organization (www.wmo.int) uses 30- and perturbed model year time interval in the definition of the Earth climate. d' 0 Because of the complexity, the Earth climate system D g.' 0 ; t/ C ıf .t/ ' 0 2 Rn : (4) could not be replicated in the laboratory and the major dt way of its exploration is based on numerical modeling. The major question is how we can estimate new This leads to an assumption that there exists some ideal climate f˚ 0K .t/g. When the question is to evaluate the model of the climate system. Let us write equations of system response to a known forcing, the most straightthis ideal model in the form of dynamical system forward way is to run a model with ıf .t/ D ıF .t/ and analyze model output using some statistical method. d˚ D G.˚; t/; ˚.0/ D ˚0 ; ˚ 2 RN : (1) This strategy, for instance, is widely used by IPCC dt [12] for the prediction of climate changes due to the In practice, the nonlinear operator Gdescribing the human activity. When we are interest in the problem of evolution of the real climate system is unknown (due to finding most dangerous impact on the system resulting the system complexity and lack of the physical knowl- in the largest possible system response, the above edge) and the system dimensionalityN is supposed to approach may not be practical as we need to check a be very large. System (1) with initial condition ˚0 huge number of test forcings. For a specific climate defines a trajectory ˚.t/ D S.t; ˚0 /: The full system model and limited set of perturbations, this crude force state at a given moment and system trajectory ˚.t/ are method could be successful though (an example of also unknown (because of the limited resolution of the this approach aimed to study the sensitivity properties observing system), and we have at our disposal just its of the HADCM3 climate model is the “climateprediclower dimensional projection ˚ K .t/; .k N / avail- tion.net” project). able in the form of different dataset products covering In the same time, it should be pointed out that in approximately 60 years of the system evolution. general there is no advance guarantee that the sensitivInstead of the ideal climate system (1), scientists ity of the system (2) could be any close to that of the deal with some climate model based on the laws of system (1) (equations of (1) are unknown and could the hydro- and thermodynamics and parameterizations be very different from equations of (2)). The simple of physical processes that could not be resolved in a example in [19] shows that predictability barriers bemodel explicitly: tween (1) and (2) could exist despite the fact that approximating system reproduces some basic statistical d' characteristics of the true model exactly. This is why D g.'; t/ ' 2 Rn : (2) the question of what properties a climate model should dt have to predict the Earth climate system sensitivity Comparing statistical characteristics of f˚ K .t/; t 2 correctly is very nontrivial from mathematical point Œtini ; tfin g with that of f'.t/g, one can estimate the of view [5, 19, 20, 22]. Results described in this entry

Algorithms for Low Frequency Climate Response

45

suggest that to get a good approximation of the climate system sensitivity, models should reproduce not only basic statistical characteristics but also lagged covariances, important periodic processes, and the structure of unstable and stable manifolds of the climate system.

Deterministic Dynamics, Fluctuation-Response relation Let us consider a case of constant in time forcing f .u; t/ f .u/: This is a typical numerical experiments when atmospheric model is used with constant boundary conditions (i.e., “perpetual January” experiment). Instead of (2) we now have a system of autonomous ODE: du D f .u/: dt

(5)

calculation of rst because neither system stationary density nor its gradient is known. A possible approach suggested in [3] is to use nonparametric estimate to approximate rst . In other words one may try to m 1 P approximate st as st  N.u; ui ; h/ using m i D1 available data fui g. In a simplest case one may use isotropic Gaussian kernels (i.e., N.u; ui ; h/ D   u  ui .u  ui /T .u  ui / ; rN D  2 N ). c exp  2 2h h The choice for bandwidth parameter h depends on the system dimensionality, and amount of available data and should be made experimentally. As a result one may estimate (6) without any assumption on the form of the system equilibrium density. This strategy may not work however for highly dimensional systems because of the data requirements. Response relation (6) could be simplified further if the form of the equilibrium density is known. In an important case of Gaussian probability density st .u/ D c0 exp..C.0/1 u; u// (assuming zero average state for simplicity), equation (6) could be rewritten as

The useful way to describe the ensemble of the system states is to introduce probability density .u; t/ giving the probability of the R system to be inside some Z t set A as P .u.t/ 2 A/ D A .u; t/d u. From the physR.s/dsıf ; ı <  > .t/ D ical viewpoint it is natural to expect that there exists 0 Z an equilibrium probability density st .u/ giving a timeR.s/ D . .u.s C //u. /T st .u/d u/C.0/1: independent way for estimation of the system statistical characteristics. Let .u/ be some state-dependent (7) characteristic of the system. R Its average value, by the definition, is <  >D .u/st .u/d u. Response of In (7) C.0/ D R uuT  d u is the covariance matrix of st the <  > to a perturbation of the system right-hand the system and c is density normalization. 0 side ıf could be determined [22] as The relation (7) was first obtained by Kraichnan [15] for the so-called regular systems (systems preZ t serving phase volume and having quadratic integral of ı <  > .t/ D R.t  s/dsıf ; 0 motion). As it will be shown later, the relations (6) Z and (7) are valid for much wider class of systems. R.t  s/ D  d u.u.t//Œrst .u.t  s//T : C. Leith suggested that the Earth’s atmosphere dynam(6) ics is reasonably close to the dynamics of the regular system and proposed to use (7) for the estimation of the R.t  s/ is called the response operator and it climate system sensitivity [17]. Method, indeed, was relates changes of the observable to the changes of successfully used for a number of systems including external forcing. It is important that the system is atmospheric general circulation models [11, 24]. perturbed around its stationary density st .u/ and st .u/ Practical implementation of (7) is very straightforis differentiable. ward. One has to calculate a long system trajectory (or From (6) it is clear that response of the system take a long enough dataset) and construct the system depends on equilibrium density and dynamics of covariance matrix C.0/ and cross-covariance between unperturbed system only. As a result a data-driven the system state and a system observable. The only algorithm for the estimation of the climate sensitivity sources of difficulties here is the inversion of C.0/ that could be constructed. The largest problem here is a could be numerically unstable when C.0/ is estimated

A

46

Algorithms for Low Frequency Climate Response

with numerical error. If the calculation of C.0/ is based on a data sample of length T , then according to the central limitptheorem, the error in C.0/ will be proportional to 1= T . From this we see that the shorter the dataset, the greater the errors will be. Furthermore, as suggested by the division by C.0/ in (7), the smaller the eigenvalues of C.0/, the larger will be the errors in R. Given these considerations, it is not surprising that one can expect a much more accuracy when using the state vectors of reduced dimensionality for calculations in (7). Further discussion on the dimension reduction procedure could be found in [11, 23].

1 R.s/ D lim !1

The widely accepted fact now is that a typical atmospheric system based on Navier-Stokes type of equations is dissipative (i.e., it is contracting phase space) and chaotic (system trajectories are sensitive with respect to initial conditions) [6]. The system energy is bounded and system evolves in a bounded set. These two facts guarantee (for finite-dimensional systems) the existence of a nontrivial invariant attracting fractal set (that is called “the attractor”) inside the system phase space. This type of behavior is well known for the Lorenz’63 system [18]. Many atmospheric systems also have this attractor existence property [5]. In spite of the attractor fractality, a physical invariant measure on the attractor still could be introduced in similar way [6] as Z

Z

t

.u0 / D limt !1

.S. ; u0 //d =t

where S. ; u0 / is a system trajectory with initial condition u0 . In an important ergodic case, .u0 / does not depend on u0 for almost all initial conditions and the averages over all typical trajectories are the same coinciding with the average over the measure. Response formula (6) does not work for dissipative chaotic systems (system contracts phase space volume and support of the measure is a fractal). Instead, it should be replaced [1] by Z

t

R.s/dsıf ; 0

0

@.S.s; u.t 0 /// 0 dt : @u.t 0 /



@f .u/ dT D T; dt @u

T .t0 / D E

T; E 2 Rnxn :

Finally we have t

ı <  > .t/ D R.s/ D lim !1

1

R.s/dsıf ; Z

0

r.u.s// 0

@S.s; u.t 0 // 0 dt : @u.t 0 / (8)

Relation (8) is called the short-time fluctuationdissipation relation and could be used to estimate response of the system for sufficiently small t. Because of the system chaoticity and existence of positive Lyapunov exponents, norm of T .t/ grows exponentially and calculations are unstable at large t. Example of this approach could be found in [1,2]. Similar approach based on the adjoint integrations of the tangent linear system was proposed in [7] with the same drawback of numerical instability at large t.

Chaotic Dissipative Dynamics, Axiom A Response Formula

d ;

0

ı <  > .t/ D



Calculation of @.S.s; u.t 0 ///[email protected] 0 / requires the knowledge of the system tangent propagator T .t; t0 / Œ@S.t; u/=@ut0 being the fundamental solution of the linearized dynamics

Z

Chaotic Dissipative Dynamics, Short-Time FDR

Z

The linear relationship between a system response and an external forcing means that the system invariant measure must be smooth with respect to changes of the system parameters (forcing). For a general dissipative chaotic system, it cannot be guaranteed in advance. However, there exists a special class of systems having this property – the Axiom A systems [14]. The tangent space of an Axiom A system at every point u0 could be divided into the sum of expanding .E u .u0 //, contracting .E s .u0 //, and neutral .E n .u0 // subspaces invariant with respect to the tangent propagator T .t/. T .t/ exponentially expands vectors from E u .u0 / and exponentially contracts vectors from E s .u0 /. Subspaces E u .u0 / and

Algorithms for Low Frequency Climate Response

47

E s .u0 / correspond to positive and negative Lyapunov directions, respectively. Neutral subspace E n .u0 / is one dimensional and is parallel to the direction of motion f .u/ (direction responsible for the only zero Lyapunov exponent of the system). E u .u0 /; E s .u0 /, and E n .u0 / must have nonzero angles and be smooth with respect to u0 . For models of atmospheric dynamics, it is not possible to verify Axiom A property directly. More likely, atmospheric systems have much weaker chaotic properties being the systems with nonzero Lyapunov exponents. Nevertheless, it is reasonable to expect that for typical multidimensional chaotic atmospheric systems, elements of Axiom A theory may still work (at least for macroscopic observables). This is the statement of the so-called chaotic hypothesis [8]. For Axiom A systems, a generalized response formula could be obtained. Decomposing the tangent space into expanding and neutral-contracting subspaces on can split ıf at every point of the phase space as ıf D ıf u C ıf s where ıf u 2 E u .u0 /; ıf s 2 E n .u0 / [ E s .u0 /. It could be shown [26, 27] that R.t/ D Ru .t/ C Rs .t/ W Ru .t/ D  lim !1

1

Chaotic Dissipative Dynamics, Shadowing Because the tangent space of the Axiom A system is divided into strongly expanding and strongly contracting subspaces, the system trajectories are structurally stable with respect to the small perturbations of the system parameters [14]. Every typical trajectory of perturbed system could be closely traced (or shadowed) by the trajectory of original system. It is possible to determine shadowing trajectory numerically. This gives another method for the construction of the system response operator [28]. Let u.t; u0 / be a trajectory of unperturbed system. Let us first apply a smooth transformation to the system phase space u1 .u/ D u C " .u/ with small " and smooth .u/. It could be shown [28] that u1 .u.t; u1 .u0 /// will satisfy to the modified system of equations d u1 D f .u1 / C "ıf .u1 / C O."2 / dt

(11)

with

Z



r.u.s//divu .Pu .u.t 0 ///dt 0 ;

0

(9)

ıf .u1 / D d .u1 /= dt  @f =@u1 .u1 / ˝. .u1 //: (12)

and R .s/ D lim !1 Z 1 @S.s; u.t 0 // s P .u.t 0 //dt 0 ; r.u.s// 0 @u.t 0 / (10) s

where Pu .u0 /; Ps .u0 / are correspondent projectors and divu denotes divergence operator acting on expanding part of the tangent space. Now (9) does not contain numerical instability at large t as short-time FDT (8) and (9, 10) give a stable way for determination of the response operator. It should be pointed out however that (10) requires numerically expensive differentiation of the Pu .u0 /. More practical approach would be to approximate (10) via quasi-Gaussian expression (7) that leads to hybrid response algorithm of [1] (provided, of course, that the system measure on expanding directions is close to a Gaussian one). This hybrid approach was successfully applied to geophysical systems in [2].

Operator ˝ relates a change of the system righthand side with a change of coordinates in the phase space. Difference between < .u/ > and < .u1 / > may be estimated as follows (note that u1 .t/ and u.t/ stay close all the time): Z ı <  >D lim

t !1 0

t

..u/  .u1 //d =t

 " < @ =@u .u/ >D " < @ =@u ˝ 1 ıf .u/ > : (13) Recall that u1 .t/ is the solution of (11), so up to the second order in ıf (13) gives an expression for the response operator provided that ˝ is invertable. Corresponding algorithm consists in calculation of coordinate transformation .u/ for given ıf from (12). Let us decompose .u/ and ıf using covariant Lyapunov basis ei .u/:

A

48

Algorithms for Low Frequency Climate Response

ıf .u/ D

n X

fi .u/ei .u/ .u/ D

i D1

n X

i .u/ei .u/;

i D1

Chaotic Dissipative Dynamics, UPO Expansion

(14)

Another useful property of Axiom A system consists in the fact that the set of unstable periodic orbits (UPOs) where, by the definition of covariant Lyapunov vectors is dense on the attractor of the system [14]. Any arbitrary trajectory of the system can be approximated [16], with any prescribed accuracy by periodic orbits, and all system statistical characteristics could be calculated d @f ei .u.t// D ei .u.t//  i ei .u.t// (15) using UPOs. Let us rewrite trajectory average for dt @u discrete time as and i are correspondent Lyapunov exponents. Substitution of (14) and (15) into (12) gives the following:

ıf .u/ D

n X

fi .u.t//ei .u.t//

i D1

D

n  X d i .u.t// i D1

dt

  i i .u.t// ei .u.t//: (16)

Now i .u/ corresponding to a positive and negative Lyapunov exponents could be obtained by integrating the system d i .u.t// D i i .u.t// C ıfi .u.t// dt

(17)

backward and forward in time, respectively. In the direction of the zero Lyapunov exponent, one has d i .u.t// D to solve “time compressed” equation dt ıfi .u.t// < ıfi .u.t// > [28]. Correspondent numerical strategy could be described as follows [28]. One has to calculate covariant Lyapunov vectors and Lyapunov exponents (several effective methods are described in [16]), produce decomposition of ıf along trajectory into positive and negative covariant Lyapunov subspaces, and solve correspondent system (17). The method requires correct inversion of “shadowing” operator, meaning that covariant Lyapunov vectors must form a basis in the tangent space of the system in every point (i.e., no zero tangencies between Lyapunov vectors are allowed).



I 1 1 X vi .S.i; u0 //; V i D0

V D

I 1 X

vi :

(18)

i D0

Here I is the number of measurements assumed to be large enough, vi is the weight of the measurement at time i (in most typical cases, all weights are equal to one), and V is the total weight of the measurements (V is equal to I when all measurements are equivalent). The UPOs of the system are embedded into the system attractor and the system trajectory evolves in the phase space passing from one orbit to another. As a result trajectory average (18) could be obtained by the averaging along the orbits. From the physical point of view, it is clear that orbits visited by a system trajectory more frequently must have larger weights in this approximation. Consequently, we obtain approximation formula (19) that looks almost the same as (18) [27]:



I.T / 1 X wi .ui /; W i D1

X

I.T /

W D

wi :

(19)

i D1

Here ui is a periodic point of the system with period T (i.e., ui D S.T; ui //I UPO with period T has T periodic points; I.T / is the total number of periodic points of the system with period equal to T and wi is the weight of a periodic point depending on orbit instability characteristics (all periodic points of the same UPO have the same weights). The number of periodic points I.T / should not be small to get decent approximation. For the Axiom A systems, this procedure could be theoretically justified [14, 27], so the trajectory averaging (18) and the UPO expansion (19) are equivalent. According to the theory [27], weights wi must be calculated as

Algorithms for Low Frequency Climate Response

  X wi D 1= exp T ijC ; j

49

(20) Stochastic Dynamics,

Fluctuation-Response Relation

where are positive Lyapunov exponents of the Another way to describe climate system is to represent orbit containing i th periodic point. Relations (19) it with the help of dynamical-stochastical equation and (20) give another method for calculation of the [19, 22] system response operator. d' Indeed, for the perturbed system, we have similar D g.'; t/ C .'/WP ˚ 2 RN : (22) dt UPO expansion formula ijC

I.T / I.T / X 1 X 0  0 .ıf /  0 wi .u0i /; W 0 D w0i ; W i D1 i D1

(21)

WP is a Gaussian white noise and <   T >D 2 . Nonlinear operator of the system is supposed to be a constant in time or time periodic with period T .g.'; t/ D g.'; t C T // reflecting annual and diurnal cycles. Noise term is responsible for unresolved small-scale processes or stochastic parameterizations in the model. Probability density of the system could be obtained from the corresponding Fokker-Plank equation [21,25]

where “prime” symbols denote periodic points and its weights for the perturbed system. Since the system is assumed to be structurally stable, no bifurcations appear under small changes of the external forcing and orbits of the perturbed system depend implicitly @ on the forcing perturbation ıf: As a result one can D div.g.'; t// C divr. /: (23) @t numerically estimate the changes of all periodic points with respect to ıf . In the same way, the expression per Let st .'; t/ be a time-periodic solution of the for the approximate response operatorV is obtained by formal differentiation of (21) with respect to ıf at Fokker-Plank equation. Let us estimate the average value of .'/ for times t D t 0 C iT; t 0 2 Œ0; T ; i D ıf D 0: 1 : : : 1. By the definition we have 0 1 I.T / PI 0 @ @ 1 X 0 0i A i D0 .'.t C iT // ı  w .u / ıf V ıf i 0 D limI !1 < .'/ > 0 t @.ıf / W i D1 I Z per N @W 0 D .'/st .'; t 0 /d' V D 2 W @.ıf /  I.T /  1 X @w0i @.u0i / Similar to the case of deterministic dynamics con.ui / C wi C sidered in Chapter 1, one can obtain response relation W i D1 @.ıf / @.ıf / [21]. To calculate operator V , it is necessary to approximate numerically all the derivatives entering this expression. Most numerically expensive part of this procedure constitutes in determination of the changes of UPO Lyapunov exponents (appearing in (20)) due to the change of the forcing. Method also requires the knowledge of the system tangent propagator. Set of UPOs giving decent approximation of the system trajectory also must be known. In spite of its complexity, this method could be used for the atmospheric and oceanic models of intermediate complexity [9, 13].

Z

t

ı <  > .t/ D Z D

R.t  s/dsıf;R.t  s/

0 per

d u.'.t//Œrst .'.t  s/; t  s/T : (24)

Obviously, (24) has exactly the same form as (6) in a constant-in-time forcing case. The derivation of (24) requires an existence of attracting timeperiodic or stationary solution for the FokkerPlank equation (23). In a quasi-Gaussian case

A

50

Algorithms for Low Frequency Climate Response

sides, Izvestiya. Atmos. Ocean Phys. 46(N6), 748–756 st .'; s/  c0 exp..Cs .0/1 '; '// giving a time(2010) periodic version of fluctuation-dissipation relation (7) 11. Gritsun, A.S., Branstator, G.: Climate response using 0 .t D .t  s/ mod T / as: a three-dimensional operator based on the fluctuationper

Z

t

ı <  > .t/ D Z D

R.t  s/dsıf ;

R.t  s/

12.

T

13.

0

st .'; t 0 /d u.'.t//'.t  s/ Ct 0 .0/1 : per

(25) Practical implementation of (24) and (25) is analogous to that of (6) and (7). One needs to calculate a long system trajectory, estimate a set of covariance matrices Ct .0/ for every date of the system period, and calculate correspondent cross-covariances. The timeperiodic case requires more data for a comparable accuracy (as we need to calculate Ct .0/ for every date): still, it could be successfully applied for complex atmospheric models [10].

14.

15. 16.

17. 18. 19.

20.

References 1. Abramov, R.V., Majda, A.J.: Blended response algorithms for linear fluctuation-dissipation for complex nonlinear dynamical systems. Nonlinearity 20(N12), 2793–2821 (2007) 2. Abramov, R.V., Majda, A.J.: Low frequency climate response of quasigeostrophic winddriven ocean circulation. J. Phys. Oceanogr. 42, 243–260 (2011) 3. Cooper, F.C., Haynes, P.H.: Climate sensitivity via a nonparametric fluctuation–dissipation theorem. J. Atmos. Sci. 68, 937–953 (2011) 4. Dymnikov, V., Filatov, A.: Mathematics of Climate Modelling. Birkh¨auser, Boston (1996) 5. Dymnikov, V.P., Gritsun, A.S.: Current problems in the mathematical theory of climate, Izvestiya. Atmos. Ocean Phys. 41(N3), 263–284 (2005) 6. Eckmann, J.-P., Ruelle, D.: Ergodic theory of chaos and strange attractors. Rev. Mod. Phys. 57, 617-656 (1985) 7. Eyink, G.L., Haine, T.W.N., Lea, D.J.: Ruelle’s linear response formula, ensemble adjoint schemes and L’evy flights Nonlin. 17, 1867–1889 (2004) 8. Gallavotti, G.: Chaotic hypotesis: onsanger reciprocity and fluctuation-dissipation theorem. J. Stat. Phys. 84, 899–926 (1996) 9. Gritsun, A.: Unstable periodic orbits and sensitivity of the barotropic model of the atmosphere. Russ. J. Numer. Anal. Math. Model. 25(N4), 303–321 (2010) 10. Gritsun, A.: Construction of the response operators onto small external forcings for the general circulation atmospheric models with time-periodicright hand

21.

22.

23.

24.

25. 26.

27.

28.

29.

dissipation theorem. J. Atmos. Sci. 64(N7), 2558–2575 (2007) IPCC Fifth Assessment Report: Climate Change (AR5), 2013, http://www.ipcc.ch/publications and data/ publications and data reports.htm Kazantsev, E.: Sensitivity of the barotropic ocean model to external influences: approach by unstable periodic orbits. Nonlinear Process. Geophys. 8, 281–300 (2001) Katok, A., Hasselblatt, B.: Introduction to the Modern Theory of Dynamical Systems, 767 p. Cambridge university press, Cambridge (1995) Kraichnan, R.: Classical fluctuation-relaxation theorem. Phys. Rev. 113, 1181–1182 Kuptsov, P.V., Parlitz, U.: Theory and computation of covariant lyapunov vectors. J. Nonlinear Sci 22(5), 727–762 (2012) Leith, C.E.: Climate response and fluctuation dissipation. J. Atmos. Sci. 32(N10), 2022–2025 (1975) Lorenz, E.N.: Deterministic nonperiodic flow. J. Atmos. Sci. 20, 130–141 (1963) Majda, A.: Challenges in climate science and contemporary applied mathematics. Commun. Pure Appl. Math. 65(I7), 920–948 (2012) Majda, A., Gershgorin, B.: Quantifying uncertainty in climate change science through empirical information theory. Proc. Natl. Acad. Sci. U.S.A. 107(34), 14958–14963 (2010) Majda, A.J., Wang, X.: Linear response theory for statistical ensembles in complex systems with timeperiodic forcing. Commun. Math. Sci. 8(N1), 145–172 (2010) Majda, A.J., Abramov, R.V., Grote, M.J.: Information theory and stochastics for multiscale nonlinear systems. CRM Monograph Series, vol. 25. American Mathematical Society, Providence (2005) Majda, A.J., Gershgorin, B., Yuan, Y.: , Low-frequency climate response and fluctuation dissipation theorems: theory and practice. J. Atmos. Sci. 67(N4), 1186–1201 (2010) Ring, M.J., Plumb, R.A.: The response of a simplified GCM to axisymmetric forcings: applicability of the fluctuation – dissipation theorem. J. Atmos. Sci. 65, 3880–3898 (2008) Risken, H.: The Fokker-Planck Equation, 2nd edn. Springer, Berlin (1989) Ruelle: General linear response formula in statistical mechanics, and the fluctuation-dissipation theorem far from equilibrium. Phys. Lett. A 245, 220 (1998) Ruelle, D.: Smooth dynamics and new theoretical ideas in nonequilibrium statistical mechanics. J. Stat. Phys. 95, 393 (1999) Wang, Q.: Forward and adjoint sensitivity computation of chaotic dynamical systems. J Comput. Phys. 235, 1–13 (2013) World climate research program (WCRP), 1980, http:// www.wcrp-climate.org/index.php/unifying-themes/unifyingthemes-modelling/modelling-wgcm

Analysis and Computation of Hyperbolic PDEs with Random Data

Analysis and Computation of Hyperbolic PDEs with Random Data 1

2;3

Mohammad Motamed , Fabio Nobile , and Ra´ul Tempone1 1 Division of Mathematics and Computational Sciences and Engineering (MCSE), King Abdullah University of Science and Technology, Thuwal, Saudi Arabia 2 Dipartimento di Matematica “F. Brioschi”, Politecnico di Milano, Milan, Italy 3 EPFL Lausanne, Lausanne, Switzerland

Introduction Hyperbolic partial differential equations (PDEs) are mathematical models of wave phenomena, with applications in a wide range of scientific and engineering fields such as electromagnetic radiation, geosciences, fluid and solid mechanics, aeroacoustics, and general relativity. The theory of hyperbolic problems, including Friedrichs and Kreiss theories, has been well developed based on energy estimates and the method of Fourier and Laplace transforms [8, 16]. Moreover, stable numerical methods, such as the finite difference method [14], the finite volume method [17], the finite element method [6], the spectral method [4], and the boundary element method [11], have been proposed to compute approximate solutions of hyperbolic problems. However, the development of the theory and numerics for hyperbolic PDEs has been based on the assumption that all input data, such as coefficients, initial data, boundary and force terms, and computational domain, are exactly known. There is an increasing interest in including uncertainty in these models and quantifying its effects on the predicted solution or other quantities of physical interest. The uncertainty may be due to either an intrinsic variability of the physical system (aleatoric uncertainty) or our ignorance or inability to accurately characterize all input data (epistemic uncertainty). For example, in earthquake modeling, seismic waves propagate in a geological region where, due to soil spatial variability and the uncertainty of measured soil parameters, both kinds of uncertainties are present. Consequently, the field of uncertainty quantification (UQ) has arisen as a new scientific discipline. UQ is the

51

science of quantitative characterization, reduction, and propagation of uncertainties and the key for achieving validated predictive computations. The numerical solution of hyperbolic problems with random inputs is a new branch of UQ, relying on a broad range of mathematics and statistics groundwork with associated algorithmic and computational development and aiming at accurate and fast propagation of uncertainties and the quantification of uncertain simulation-based predictions. The most popular method for solving PDEs in probabilistic setting is the Monte Carlo sampling; see, for instance, [7]. It consists in generating independent realizations drawn from the input distribution and then computing sample statistics of the corresponding output values. This allows one to reuse available deterministic solvers. While being very flexible and easy to implement, this technique features a very slow convergence rate. In the last few years, other approaches have been proposed, which in certain situations feature a much faster convergence rate. They exploit the possible regularity that the solution might have with respect to the input parameters, which opens up the possibility to use deterministic approximations of the response function (i.e., the solution of the problem as a function of the input parameters) based on global polynomials. Such approximations are expected to yield a very fast convergence. Stochastic Galerkin [3, 10, 22, 32, 38] and stochastic collocation [2, 26, 27, 37] are among these techniques. Such new techniques have been successfully applied to stochastic elliptic and parabolic PDEs. In particular, it is shown that, under particular assumptions, the solution of these problems is analytic with respect to the input random variables [2, 25]. The convergence results are then derived from the regularity results. For stochastic hyperbolic problems, the analysis was not well developed until very recently; see [1,23,24]. In the case of linear problems, there are a few works on the one-dimensional scalar advection equation with a timeand space-independent random wave speed [13,31,36]. Such problems also possess high regularity properties provided the data live in suitable spaces. The main difficulty however arises when the coefficients vary in space or time and are possibly non-smooth. In this more general case, the solution of linear hyperbolic problems may have lower regularity than those of elliptic, parabolic, and hyperbolic problems with constant

A

52

Analysis and Computation of Hyperbolic PDEs with Random Data

random coefficients [1, 23, 24]. There are also recent works on stochastic nonlinear conservation laws; see, for instance, [18, 19, 28, 34, 35]. In the present notes, we assume that the uncertainty in the input data is parameterized either in terms of a finite number of random variables or more generally by random fields. Random fields can in turn be accurately approximated by a finite number of random variables when the input data vary slowly in space, with a correlation length comparable to the size of the physical domain. A possible way to describe such random fields is to use the truncated KarhunenLo´eve [20,21] or polynomial chaos expansion [39]. We will address theoretical issues of well-posedness and stochastic regularity, as well as efficient treatments and computational error analysis for hyperbolic problems with random inputs. We refer to [1, 23, 24] for further details.

Problem Statement In this section, for simplicity, we consider the linear second-order scalar acoustic wave equation with random wave speed and deterministic boundary and initial conditions. We motivate and describe the source of randomness and address the well-posedness of the problem. For extensions to the system of elastic wave equations, see [1, 23]. Let D be a convex bounded polygonal domain in Rd , d D 2; 3, and .; F ; P / be a complete probability space. Here,  is the set of outcomes, F  2 is the algebra of events, and P W F ! Œ0; 1 is a probability measure. Consider the stochastic initial boundary value problem (IBVP): find a random function u W Œ0; T   DN   ! R, such that P -almost everywhere in , i.e., almost surely (a.s), the following holds   ut t .t; x; !/  r  a2 .x; !/ru.t; x; !/ D f .t; x/ in Œ0; T   D  ; u.0; x; !/ D g1 .x/; ut .0; x; !/ D g2 .x/ onft D 0g  D  ; u.t; x; !/ D 0 on Œ0; T   @D  : (1)

f 2 L2 ..0; T /I L2 .D//; g1 2 H01 .D/; g2 2 L2 .D/; (2) are compatible. The only source of randomness is the wave speed a which is assumed to be bounded and uniformly coercive, 0 < amin  a.x; !/  amax < 1; almost everywhere in D;

a.s:

(3)

Assumption (3) guarantees that the energy is conserved and therefore the stochastic IBVP (1) is well posed [24]. In many wave propagation problems, the source of randomness can be described or approximated by only a small number of uncorrelated random variables. For example, in seismic applications, a typical situation is the case of layered materials where the wave speeds in the layers are not perfectly known and therefore are described by uncorrelated random variables. The number of random variables is therefore the number of layers. In this case, the randomness is described by a finite number of random variables. Another situation is when the wave speeds in layers are given by random fields, which in turn are approximated by a truncated Karhunen-Lo´eve expansion. Hence, the number of random variables corresponds to the number of layers as well as the number of terms in the expansion. In this case, the randomness is approximated by a finite number of random variables. This motivates us to make the following finite dimensional noise assumption on the form of the wave speed, a.x; !/ D a.x; Y1 .!/; : : : ; YN .!//; almost everywhere in D;

a.s;

(4)

where N 2 NC and Y D ŒY1 ; : : : ; YN  2 RN is a random vector. We denote by n Yn ./ the image of Y and assume that Yn is bounded. We let  D QN n nD1 n and assume further that the random vector Y has a bounded joint probability density function  W  ! RC with  2 L1 . /. We note that by using a similar approach to [2, 5], we can also treat unbounded random variables, such as Gaussian and exponential Here, the solution u is the displacement, and t and x D variables. The finite dimensional noise assumption (4) im.x1 ; : : : ; xd /> are the time and location, respectively, and the data plies that the solution of the stochastic IBVP (1)

Analysis and Computation of Hyperbolic PDEs with Random Data

can be described by only N random variables, i.e., u.t; x; !/ D u.t; x; Y1 .!/; : : : ; YN .!//. This turns the original stochastic problem into a deterministic IBVP for the wave equation with an N -dimensional parameter, which allows the use of standard finite difference and finite element methods to approximate the solution of the resulting deterministic problem u D u.t; x; Y /, where t 2 Œ0; T , x 2 D, and Y 2  . Note that the knowledge of u D u.t; x; Y / fully determines the law of the random field u D u.t; x; !/. Here, as an example of form (4), we consider a random speed a given by a.x; !/ D a0 .x/ C

N X

an .x; !/ IDn .x/;

nD1

an .x; !/ D Yn .!/ ˛n .x/;

˛n 2 C 1 .Dn /; (5)

where I is the indicator function. In this case, D is a heterogeneous medium consisting of N nonN overlapping sub-domains fDn gN nD1 , fYn gnD1 are N independent random variables, and f˛n gnD1 are smooth functions defined on sub-domains. The boundaries of sub-domains, which are interfaces of speed discontinuity, are assumed to be smooth. The ultimate goal is the prediction of statistical moments of the solution u or statistics of some given quantities of physical interest. As an example, we consider the following quantity, Z TZ Q.Y / D u.t; x; Y / .x/ d x dt D

0

Z

C

u.T; x; Y / .x/ d x;

(6)

D

where u solves (1) and the mollifiers  and functions of x.

are given

53

collocation, are sample-based approaches. They rely on a set of deterministic models corresponding to a set of realizations and hence require no modification to the existing deterministic solvers. Finally, hybrid methods are a mixture of both intrusive and nonintrusive approaches. Nonintrusive (or sample-based) methods are attractive in the sense that they require only a deterministic solver for computing deterministic models. In addition, since the deterministic models are independent, it is possible to distribute them onto multiple processors and perform parallel computation. The most popular nonintrusive technique is the Monte Carlo method. While being very flexible and easy to implement, this technique features a very slow convergence rate. The multilevel Monte Carlo method has been proposed to accelerate the slow convergence of Monte Carlo sampling [12]. Quasi-Monte Carlo methods can be considered as well that aim at achieving higher rates of convergence [15]. The stochastic collocation (SC) method is another nonintrusive technique, in which regularity features of the quantity of interest with respect to the input parameters can be exploited to obtain a much faster or possibly spectral convergence. In SC, the problem (1) is first discretized in space and time, using a deterministic numerical method. The obtained semidiscrete problem is then collocated in a set of  col location points fY .k/ gkD1 2  to compute the approximate solutions uh .t; x; Y .k/ /, where h represent the discretization mesh/grid size. A global polynomial approximation is then built upon those evaluations, uh; .t; x; Y / D

 X

uh .t; x; Y .k/ / Lk .Y /;

kD1 

for suitable multivariate polynomial bases fLk gkD1 such as Lagrange polynomials. Finally, using the Gauss quadrature formula, we can easily approximate the Nonintrusive Numerical Methods statistical moments of the solution. For instance, Z There are in general three types of methods for EŒu.:; Y /  EŒu .:; Y / D uh; .:; Y / .Y / dY h; propagating uncertainty in PDE models with random  inputs: intrusive, nonintrusive, and hybrid methods. Z  X Intrusive methods, such as perturbation expansion, .k/ D uh .:; Y / Lk .Y / .Y / dY intrusive polynomial chaos [39], and stochastic  kD1 Galerkin [10], require extensive modifications in  X existing deterministic solvers. On the contrary, nonuh .:; Y .k/ / k :  intrusive methods, such as Monte Carlo and stochastic kD1

A

54

Analysis and Computation of Hyperbolic PDEs with Random Data

A key point in SC is the choice of the set of collocation points fY .k/ g, i.e., the type of computational grid in the N -dimensional stochastic space. A full tensor grid, based on Cartesian product of mono-dimensional grids, can only be used when the number of stochastic dimensions N is small, since the computational cost grows exponentially fast with N (curse of dimensionality). Alternatively, sparse grids can reduce the curse of dimensionality. They were originally introduced by Smolyak for high-dimensional quadrature and interpolation computations [30]. In the following, we will briefly review the sparse grid construction. Let j 2 ZN C be a multi-index containing nonnegative integers. For a nonnegative index jn in j, we introduce a sequence of one-dimensional polynomial interpolant operators U jn W C 0 .n / ! Pp.jn / .n / of degree p.jn / on p.jn / C 1 suitable knots. With U 1 D 0, we define the detail operator jn WD U jn  U jn 1 :

which may be the solution or some given functional of the solution. For example, a fast spectral convergence is possible for highly regular outputs. We will discuss this issue in more details in the next section.

Stochastic Regularity and Convergence of Stochastic Collocation As extensively discussed in [1, 23, 24], the error in the stochastic collocation method is related to the stochastic regularity of the output quantity of interest (the solution or a given functional of the solution). It has been shown that, under particular assumptions, the solution of stochastic, elliptic, and parabolic PDEs is analytic with respect to the input random variables [2,25]. In contrast, the solution of stochastic hyperbolic PDEs may possess lower regularity. In this section, first, we will convey general stochastic regularity properties of hyperbolic PDEs through simple examples. Then, we state the main regularity results followed by the convergence results for SC.

Finally, introducing a sequence of index sets I.`/  ZN C , the sparse grid approximation of u W  ! V at level ` reads Examples: General Stochastic Regularity Properties N X O Example 1 Consider the 1D Cauchy problem for the u.`;N / .:; Y / D jn Œu.:; Y /: (7) scalar wave equation, j2I.`/ nD1

2 in Œ0; T   R; Furthermore, in order for the sum (7) to have some utt .t; x; y/  y uxx .t; x; y/ D 0; telescopic properties, which are desirable, we impose u.0; x; y/ D g.x/; ut .0; x; y/ D 0; on ft D 0g  R; an additional admissibility condition on the set I [9]. An index set I is said to be admissible if 8 j 2 I, with g 2 C01 .R/. The wave speed is constant and given by a single random variable y. We want to j  en 2 I for 1  n  N; jn  1; investigate the regularity of the solution u with respect to y. For this purpose, we extend the parameter y into holds. Here, en is the n-th canonical unit vector. the complex plane and study the extended problem in To fully characterize the sparse approximation op- the complex plane. It is well known that if the exerator in (7), we need to provide the following: tended problem is well posed and the first derivative of • A level ` 2 N and a function p.j / representing the resulting complex-valued solution with respect to the relation between an index j and the number of the parameter satisfies the so-called Cauchy-Riemann points in the corresponding one-dimensional poly- conditions, the solution can analytically be extended nomial interpolation formula U j . into the complex plane, and hence u will be analytic • A sequence of sets I.`/. Typical examples include with respect to y. We therefore let y D yR C i yI , total degree and hyperbolic cross grids. where yR ; yI 2 R, and apply the Fourier transform • The family of points to be used, such as Gauss or with respect to x to obtain Clenshaw-Curtis abscissae, [33]. The rate of convergence for SC depends on the  i y k t  g.k/ O e C ei y k t ; uO .t; k; y/ D stochastic regularity of the output quantity of interest, 2

Analysis and Computation of Hyperbolic PDEs with Random Data

where uO and gO are the Fourier transforms of u and g with respect to x, respectively. Hence, jOu.t; k; y/j jg.k/j O e

jyI j jkj t

:

Therefore, the Fourier transform of the solution uO .t; k; y/ grows exponentially fast in time, unless the Fourier transform of the initial solution g.k/ O decays faster than e jyI j jkj t . In order for the Cauchy problem to be well posed, g needs to belong to the Gevrey space G q .R/ of order q < 1 [29]. For such functions, 1=q we have jg.k/j O  C e  jkj for positive constants C and , and hence the problem is well posed in the complex strip †r D f.yR C i yI / 2 C W jyI j  rg. Note that G 1 .R/ is the space of analytic functions. If g 2 G 1 .R/, the problem is well posed only for a finite time interval when t  =r. This shows that even if the initial solution g is analytic, the solution u is not analytic with respect to y for all times in †r .

55

8 y y g.y t C x/; g.y t  x/ C yCyC ˆ ˆ C ˆ ˆ < x < 0;   (9) u.t; x; Y / D 2 y y   ˆ g .y t  x/ ; ˆ C y Cy y ˆ  C C ˆ : x > 0: Clearly, the solution (9) is infinitely differentiable with respect to both parameters y and yC in .0; C1/. Note that the smooth initial solution u.0; x; Y /, which is contained in one layer with zero value at the interface, automatically satisfies the interface conditions (8) at time zero. Otherwise, if for instance the initial solution crosses the interface without satisfying (8), a singularity is introduced in the solution, and the high regularity result does not hold any longer. In the more general case of multidimensional heterogeneous media consisting of sub-domains, the interface jump conditions on a smooth interface ‡ between two sub-domains DI and DII are given by

Œu.t; :; Y /‡ D 0; Œa2 .:; Y / un .t; :; Y /‡ D 0: Example 2 Consider the 1D Cauchy problem for the (10) scalar wave equation in a domain consisting of two Here, the subscript n represents the normal derivative, homogeneous half-spaces separated by an interface at and Œv.:/ is the jump in the function v across the ‡ x D 0, interface ‡. In this general case, the high regularity with respect to parameters holds provided the smooth  2  initial solution satisfies (10). The jump conditions are ut t .t; x; Y / a .x; Y / ux .t; x; Y / x D 0; satisfied for instance when the initial data are contained in Œ0; T   R; within sub-domains. This result for Cauchy problems can easily be extended to IBVPs by splitting the probu.0; x; Y / D g.x/; ut .0; x; Y / D a g 0 .x/; lem to one pure Cauchy and two half-space problems. on ft D 0g  R; See [24] for more details. We note that the above high stochastic regularity result is valid only for particular with g 2 C01 .0; 1/. The wave speed is piecewise types of smooth data. In real applications, the data constant and a function of a random vector of two are not smooth. Let us now consider a more practical datum. variables Y D Œy ; yC ,  a.x; Y / D

y ; yC ;

Example 3 Consider the 1D Cauchy problem for the scalar wave equation in a domain consisting of two homogeneous half-spaces separated by an interface at x D 0,

x < 0; x > 0:

  By d’Alembert’s formula and the interface jump conutt .t; x; y/ a2 .x; y/ ux .t; x; y/ x D 0; in Œ0; T   R; ditions at x D 0, u.0; x; y/ D g.x/; ut .0; x; y/ D 0; on ft D 0g  R; u.t; 0 ; Y / D u.t; 0C ; Y /;

y2 ux .t; 0 ; Y /

2 D yC ux .t; 0C ; Y /;

the solution reads

(8)

with g 2 H01 .R/ being a hat function with a narrow support supp g D Œx0  ˛; x0 C ˛, with 0 < ˛ 1, located at x0 < ˛. The wave speed is piecewise constant and a function of a random variable y,

A

56

Analysis and Computation of Hyperbolic PDEs with Random Data

 a.x; y/ D

1; y;

x < 0; x > 0:

Similar to Example 2, we can obtain the closed form of the solution as

81 ˆ 2 g.t C x/; ˆ ˆ < 12 g.t C x/ C 1y g.t  x/;  2 .1Cy/  u.t; x; y/ D 1 x ˆ g t C y ; ˆ ˆ : 1Cy 0; Since g 2 H01 , there is only one bounded derivative @y u, and higher bounded y-derivatives do not exist. However, if we consider a quantity of interest Q.y/ as in (6) with mollifiers ; 2 C01 .R/ vanishing at x < 0, we can employ integration by parts and shift the derivatives on g to the mollifiers. Therefore, although the solution u has only one bounded y-derivative, the quantity of interest Q is smooth with respect to y.

x < x0  t x0  t < x < 0 0 < x < x0 C y t x > x0 C y t

Theorem 1 For the solution of the stochastic IBVP (1) with data given by (2) and a random piecewise smooth wave speed satisfying (3) and (5), we have @Y u 2 C 0 .0; T I L2 .D//;

8 Y 2 :

Theorem 2 Consider the quantity of interest Q.Y / in (6) and let the smooth mollifiers ; 2 C01 .D/ vanish at the discontinuity interfaces. Then with the Remark 1 Immediate results of the above three examassumptions of Theorem 1, we have ples are the following: 1. For the solution of the 1D Cauchy problem for the Q 2 C 1 . /: wave equation to be analytic with respect to the random wave speed at all times in a given complex In practical applications, quantities of interest, such strip †r with r > 0, the initial datum needs to live as Arias intensity, spectral acceleration, and von Mises in a space strictly contained in the space of analytic functions, which is the Gevrey space G q .R/ with stress, are usually nonlinear in u. In such cases, the high 0 < q < 1. Moreover, if the problem is well stochastic regularity property might not hold. However, posed and the data are analytic, the solution may one can perform a low-pass filtering (LPF) on the low be analytic with respect to the parameter in †r only regular solutions or quantities of interest by convolving them with smooth kernels such as Gaussian functions, for a short time interval.   2. In a 1D heterogeneous medium with piecewise 1 jxj2 smooth wave speeds, if the data are smooth and x 2 Rd : Kı .x/ D p exp  2 ; d 2 ı . 2  ı/ the initial solution satisfies the interface jump con(11) ditions (10), the solution to the Cauchy problem is Here, the standard deviation ı is inversely proportional smooth with respect to the wave speeds. If the initial to the maximum frequency that is allowed to pass. For solution does not satisfy (10), the solution is not instance, the filtered solution is given by the convolusmooth with respect to the wave speeds. tion 3. In general, the solution u has only finite stochastic regularity. The stochastic regularity of functionals Qı .u/.t; x; Y / D .u ? Kı /.t; x; Y / of the solution (such as mollified quantities of inZ terest) can however be considerably higher than the D Kı .x  xQ / u.t; xQ ; Y / d xQ : (12) regularity of the solution. D The filtered solution (12) is of a type similar to the quantity of interest (6) with smooth mollifiers. General Results: Stochastic Regularity We now state some regularity results in the more However, the main difference here is the boundary general case when the data satisfy the minimal assump- effects introduced by the convolution. Therefore, in the presence of a compactly supported smooth tions (2). See [23, 24] for proofs.

Analysis and Computation of Hyperbolic PDEs with Random Data

kernel K 2 C01 .D/ as mollifier, the quantity (12) will have high stochastic regularity on a smaller domain x 2 Dı  D with dist.@Dı ; @D/  d > 0. We choose d so that Kı .x0 / with jx0 j D d is essentially zero. This implies that for any x 2 Dı , the support of Kı .x  xQ / is essentially vanishing at @D. We note that by performing filtering, we introduce an error ju  Qı .u/j which is proportional to O.ı 2 /. This error needs to be taken into account. We refer to [1] for a more rigorous error analysis of filtered quantities.

57

C D Cs

1  CsN jjjj1=2 1 1  Cs

max

max

d D1;:::;N 0k1 ;:::;kd s

jjDYk11 : : : DYkdd uh jjL2 . IW / ; depends on uh , s, and N , but not on `.

Remark 2 It is possible to show that the semi-discrete solution uh can analytically be extended on the whole region †.; / D fZ 2 CN ; dist.n ; Zn /  ; n D 1; : : : ; N g, with the radius of analyticity D O.h/ [24]. This can be used to show that for both full tensor and Smolyak interpolation, we will have a Convergence Results for Stochastic Collocation fast exponential decay in the error when the product In order to obtain a priori estimates for the total error h p.`/ is large. As a result, with a fixed h, the error jju  uh; jjW ˝L2 . / , with W WD L2 .0; T I L2 .D//, we convergence is slow (algebraic) for a small ` and split it into two parts fast (exponential) for a large `. Moreover, the rate of convergence deteriorates as h gets smaller. " WD jju  uh; jj  jju  uhjj C jjuh  uh; jj DW "I C "II : (13) Remark 3 The main parameters in the computations include h, .`/, and possibly ı if filtered quantities The first term "I controls the convergence of the de- are used. In order to find the optimal choice of the terministic numerical scheme with respect to the mesh parameters, we need to minimize the computational size h and is of order O.hr /, where r is the minimum complexity of the SC method, subject to the total error between the order of accuracy of the finite element constraint " D TOL. We refer to [1] for more details. or finite difference method used and the regularity of the solution. Notice that the constant in the term O.hr / is uniform with respect to Y . The second term "II is derived as follows from the stochastic regularity References results. See [24] for proofs.

Theorem 3 Consider the isotropic full tensor product interpolation. Then "II  C s=N :

(14)

P where the constant C D Cs N nD1 maxkD0;:::;s jjDYkn uh;` jjL1 . IW / does not depend on `. Theorem 4 Consider the Smolyak sparse tensor product interpolation based on Gauss-Legendre abscissae, and let uh; be given by (7). Then for the solution uh with s  1 bounded mixed Y -derivatives, "II  C

 log 2  2 N s Clog N ; 1 C log2  N

D 1 C log 2 .1 C log2 1:5/  2:1; where the constant

(15)

1. Babuˇska, I., Motamed, M., Tempone, R.: A stochastic multiscale method for the elastodynamic wave equations arising from fiber composites (2013, preprint) 2. Babuˇska, I., Nobile, F., Tempone, R.: A stochastic collocation method for elliptic partial differential equations with random input data. SIAM J. Numer. Anal. 45, 1005–1034 (2007) 3. Babuˇska, I., Tempone, R., Zouraris, G.E.: Solving elliptic boundary value problems with uncertain coefficients by the finite element method: the stochastic formulation. Comput. Methods Appl. Mech. Eng. 194, 1251–1294 (2005) 4. Boyd, J.P.: Chebyshev and Fourier Spectral Methods. Springer, Berlin/New York (1989) 5. Charrier, J.: Strong and weak error estimates for elliptic partial differential equations with random coefficients. IMA J. Numer. Anal. 50, 216–246 (2012) 6. Eriksson, K., Estep, D., Hansbo, P., Johnson, C.: Computational Differential Equations. Studentlitteratur, Lund (1996) 7. Fishman, G.S.: Monte Carlo: Concepts, Algorithms, and Applications. Springer, New York (1996) 8. Friedrichs, K.O.: Symmetric hyperbolic linear differential equations. Commun. Pure Appl. Math. 7, 345–392 (1954) 9. Gerstner, T., Griebel, M.: Dimension-adaptive tensorproduct quadrature. Computing 71, 65–87 (2003)

A

58 10. Ghanem, R.G., Spanos, P.D.: Stochastic Finite Elements: A Spectral Approach. Springer, New York (1991) 11. Gibson, W.: The Method of Moments in Electromagnetics. Chapman and Hall/CRC, Boca Raton (2008) 12. Giles, M.B.: Multilevel Monte Carlo path simulation. Oper. Res. 56, 607–617 (2008) 13. Gottlieb, D., Xiu, D.: Galerkin method for wave equations with uncertain coefficients. Commun. Comput. Phys. 3, 505–518 (2008) 14. Gustafsson, B., Kreiss, H.O., Ologer, J.: Time Dependent Problems and Difference Methods. Wiley-Interscience, New York (1995) 15. Hou, T.Y., Wu, X.: Quasi-Monte Carlo methods for elliptic PDEs with random coefficients and applications. J. Comput. Phys. 230, 3668–3694 (2011) 16. Kreiss, H.-O.: Initial boundary value problems for hyperbolic systems. Commun. Pure Appl. Math. 23, 277–298 (1970) 17. Leveque, R.J.: Finite Volume Methods for Hyperbolic Problems. Cambridge University Press, Cambridge/New York (2002) 18. Lin, G., Su, C.-H., Karniadakis, G.E.: Predicting shock dynamics in the presence of uncertainties. J. Comput. Phys. 217, 260–276 (2006) 19. Lin, G., Su, C.-H., Karniadakis, G.E.: Stochastic modeling of random roughness in shock scattering problems: theory and simulations. Comput. Methods Appl. Mech. Eng. 197, 3420–343 (2008) 20. Lo´eve, M.: Probability Theory I. Graduate Texts in Mathematics, vol. 45. Springer, New York (1977) 21. Lo´eve, M.: Probability Theory II. Graduate Texts in Mathematics, vol. 46. Springer, New York (1978) 22. Matthies, H.G., Kees, A.: Galerkin methods for linear and nonlinear elliptic stochastic partial differential equations. Comput. Methods Appl. Mech. Eng. 194, 1295–1331 (2005) 23. Motamed, M., Nobile, F., Tempone, R.: Analysis and computation of the elastic wave equation with random coefficients (2013, submitted) 24. Motamed, M., Nobile, F., Tempone, R.: A stochastic collocation method for the second order wave equation with a discontinuous random speed. Numer. Math. 123, 493–536 (2013) 25. Nobile, F., Tempone, R.: Analysis and implementation issues for the numerical approximation of parabolic equations with random coefficients. Int. J. Numer. Meth. Eng. 80, 979–1006 (2009) 26. Nobile, F., Tempone, R., Webster, C.G.: An anisotropic sparse grid stochastic collocation method for partial differential equations with random input data. SIAM J. Numer. Anal. 46, 2411–2442 (2008) 27. Nobile, F., Tempone, R., Webster, C.G.: A sparse grid stochastic collocation method for partial differential equations with random input data. SIAM J. Numer. Anal. 46, 2309–2345 (2008) 28. Poette, G., Despr´es, B., Lucor, D.: Uncertainty quantification for systems of conservation laws. J. Comput. Phys. 228, 2443–2467 (2009) 29. Rodino, L.: Linear Partial Differential Operators in Gevrey Spaces. World Scientific, Singapore (1993)

Angiogenesis, Computational Modeling Perspective 30. Smolyak, S.A.: Quadrature and interpolation formulas for tensor products of certain classes of functions. Doklady Akademii Nauk SSSR 4, 240–243 (1963) 31. Tang, T., Zhou, T.: Convergence analysis for stochastic collocation methods to scalar hyperbolic equations with a random wave speed. Commun. Comput. Phys. 8, 226–248 (2010) 32. Todor, R.A., Schwab, C.: Convergence rates for sparse chaos approximations of elliptic problems with stochastic coefficients. IMA J. Numer. Anal. 27, 232–261 (2007) 33. Trefethen, L.N.: Is Gauss quadrature better than ClenshawCurtis? SIAM Rev. 50, 67–87 (2008) 34. Tryoen, J., Le Maitre, O., Ndjinga, M., Ern, A.: Intrusive projection methods with upwinding for uncertain nonlinear hyperbolic systems. J. Comput. Phys. 229, 6485–6511 (2010) 35. Tryoen, J., Le Maitre, O., Ndjinga, M., Ern, A.: Roe solver with entropy corrector for uncertain hyperbolic systems. Comput. Appl. Math. 235, 491–506 (2010) 36. Wang, X., Karniadakis, G.E.: Long-term behavior of ploynomial chaos in stochastic flow simulations. Comput. Methods Appl. Mech. Eng. 195, 5582–5596 (2006) 37. Xiu, D., Hesthaven, J.S.: High-order collocation methods for differential equations with random inputs. SIAM J. Sci. Comput. 27, 1118–1139 (2005) 38. Xiu, D., Karniadakis, G.E.: Modeling uncertainty in steady state diffusion problems via generalized polynomial chaos. Comput. Methods Appl. Mech. Eng. 191, 4927–4948 (2002) 39. Xiu, D., Karniadakis, G.E.: The Wiener-Askey polynomial chaos for stochastic differential equations. SIAM J. Sci. Comput. 24, 619–644 (2002)

Angiogenesis, Computational Modeling Perspective Amina A. Qutub1 and Aleksander S. Popel2 1 Department of Bioengineering, Rice University, Houston, TX, USA 2 Systems Biology Laboratory, Department of Biomedical Engineering, School of Medicine, The Johns Hopkins University, Baltimore, MD, USA

Abstract Modulating capillary growth, or therapeutic angiogenesis, has the potential to improve treatments for many conditions that affect the microvasculature including cancer, peripheral arterial disease, and ischemic stroke. The ability to translate angiogenesis-targeting therapies to the clinic as well as tailor these treatments

Angiogenesis, Computational Modeling Perspective

to individuals hinges on a detailed mechanistic understanding of how the process works in the human body. Mathematical and computational modeling of angiogenesis have emerged as tools to aid in treatment design, drug development, and the planning of therapeutic regimes. In this encyclopedia entry, we describe angiogenesis models that have been developed across biological scales and outline how they are being used in applications to human health.

Introduction Angiogenesis, the growth of capillaries from existing microvasculature, arises in dozens of physiological and pathological conditions. It is critical to wound healing and response to exercise. Angiogenesis also plays a pivotal role in the progression of cancer and tissue recovery after stroke. As such, angiogenesis has been the target of numerous therapies. However, so far, clinical trials and approved drugs targeting angiogenesis have performed below their potential [1, 2]. In part this is because drugs have been limited in their targets. Emerging angiogenic treatments are more likely to be multimodal, targeting multiple molecules or cells [3]. In addition, the need for tighter control over the degree, duration, and efficacy of new capillary growth by proper timing and dosing of therapies has been recognized. To harness the full therapeutic use of angiogenesis, a greater quantitative understanding of the process is needed. Modeling becomes a vital tool to capture the complexity of angiogenesis and aid in drug design and development. Modeling of angiogenesis first emerged in the 1970s, with differential equation-based models of capillary network formation [4, 5]. In the following decades, models considered generic growth factors as stimuli for vessel growth and captured key characteristics that subsequent models have maintained: including the ability for vessels to form as a function of biochemical, mechanical, and genetic properties [6–8]. As more knowledge of the molecular players in angiogenesis appeared from experimental studies, ligand-receptor and signaling models began to probe the effects of factors like vascular endothelial growth factor [9, 10], fibroblast growth factor [11], matrix metalloproteinases [12], and hypoxia-inducible factor 1 [13] on angiogenic potential. Cell-based models arose to address the need to understand how single cell behavior and cell-

59

matrix interactions give rise to emergent properties of whole capillary networks [14–16]. Bioinformatic approaches have since been harnessed to help predict molecular compounds to target and analyze highthroughput imaging data [17–19]. Multiscale modeling emerged to bridge the gap between models and allow predictions across time and spatial scales [20, 21]. Broadly, computational modeling of angiogenesis can be divided into these five methodology categories, which we discuss in general chronological order of their appearance in the modeling literature: networklevel modeling, molecular-level modeling, cell-based modeling, bioinformatics, and multiscale modeling.

Angiogenesis Modeling Methodology Network-Level Modeling The first computational models of angiogenesis focused on the formation of new capillaries at the tissue or network level. Experimental observations in the early 1970s showed that capillary formation in tumors was a function of an uncharacterized molecule coined “tumor angiogenic factor” or TAF [22]. The effect of this vascular stimulus on the growth of capillaries in melanoma was modeled in 1976 by Deakin [23]. The following year, Liotta et al. published a partial differential equation model of the diffusion of vessels and tumor cells within a growing tumor [5]. Differential equations lent themselves well to characterizing the observed phenomenon, as researchers recognized that as the tumor grew, the concentration of vessels changed in time as a function of factors in the microenvironment and location within the tumor. Continuous, deterministic approaches captured the main features being quantified experimentally: vessel growth, tumorigenesis as a function of vessel density, and vessel growth as a function of TAF levels. As experimental knowledge grew in the following decades, angiogenesis models become more mechanistically detailed. Current capillary network models frequently contain three main categories of stimuli guiding directional changes in neovascularization: chemotaxis, haptotaxis, and a random and/or baseline contribution. They also allow vasculature density to increase through growth and decrease through apoptosis. Mathematically, the motility stimuli can be expressed through basic governing equations that conserve mass [6], e.g.,

A

60

Angiogenesis, Computational Modeling Perspective

@n D Jchemotaxic C Jhaptotaxic C Jrandom @t @n D .c/ nrc C ”.m/nrm  Dn rn @t where n is the endothelial cell or vessel density per unit area; c is the TAF concentration; is the chemotactic constant or function denoting the sensitivity of the cells to TAF;  is the haptotactic constant or function; m is the concentration of the matrix fibers (e.g., fibrin or collagen), and dn is the cell random motility coefficient. Along with capillary growth dynamics, scaling factors and scale-independent properties of capillary networks have also been explored computationally [24]. Fractal analysis is one technique researchers have used, with applications to correlating tumor grades to fractal dimensions of the vasculature network [25–30]. Molecular-Level Modeling: Reaction Diffusion Modeling

to the cell membrane is often considered. By the mid1990s, it was broadly accepted that a balance of many proangiogenic and antiangiogenic factors determined the degree of angiogenesis [37], and these compounds are present in varying levels based on tissue location and disease state. Furthermore, multiple receptors and ligand combinations in the same family yield divergent effects on angiogenesis [38]. Modeling at the receptorligand level added the quantitative predictions of when the angiogenic balance was tipped and how targeting a particular receptor or ligand changed the balance of growth factors present in tissues [11]. Results of ligand-receptor models have predicted how intracoronary delivery of bFGF distributes in the myocardium [33]: how VEGF signals in healthy skeletal muscle [39], peripheral arterial disease [40], and breast cancer [41]; and how matrix metalloproteinases degrade the local collagen matrix surrounding activated vascular cells [12, 42].

Intracellular signaling models Discovery of intracellular pathways associated with hypoxia-induced angiogenesis led to computational models at the subcellular level. Hypoxia-inducible factor 1 (HIF1), discovered in 1992 by Gregg Semenza [43], is a potent transcription factor that activates hundreds of genes including vascular endothelial growth factor and other angiogenic-associated molecules. Accumulation of HIF1 at low levels of oxygen has been referred to as an angiogenic switch – rapidly turning on the signaling pathways associated with neovascularization [44–47]. Subsets of this pathway have been modeled computationally [45, 46, 48–50, 100]. Like the receptor-ligand interaction models, the intracellular signaling models have predominantly taken k1 the form of chemical kinetic reactions, modeled by a Dll4 C Notch $ Dll4  Notch series of ordinary differential equations, where kinetic k1 rates are obtained from experimental data or estimates. Michaelis-Menten kinetics with substrate saturation d ŒDll4 D k1 ŒNotch ŒDll4 C k1 ŒDll4  Notch and reversibility assumptions help simplify the set of dt possible equations and kinetic rates. Results of these k1 and k1 are the forward and reverse binding rates, models have predicted the effect of reactive oxygen respectively, and brackets indicate concentration. This species on HIF1 levels in cancer and ischemia [51], the is also bounded by the limit that the total amounts of induction of VEGF signaling by HIF1 in tumor cells Dll4 and Notch remain constant. Factors may present [50], and the transcriptional changes during hypoxia in the interstitial fluid surrounding a tissue or cell or [46, 49]. be bound to the local matrix. Transport by diffusion Additionally, models have looked at integrating of the growth factors throughout the microenvironment receptor-ligand binding on the cell membrane

Receptor-ligand models In the 1980s, experimental studies identified the molecule vascular endothelial growth as a potent angiogenic or TAF factor [31, 32], and new models incorporated this knowledge. The first angiogenic growth factors explored experimentally were also those that were first studied computationally: FGF [11,33,34] and VEGF [9,35,36]. Most of these models employ reaction-diffusion equations. A series of ODEs describe the binding of multiple ligands to their respective receptors. Kinetic rates are estimated from literature where available. A simplified example of a receptor-ligand kinetic model is shown for the Dll4 ligand, its receptor Notch, and their bound complex:

Angiogenesis, Computational Modeling Perspective

with intracellular signaling. A stochastic Boolean representation has been employed to study signaling transduction and crosstalk between the VEGF, integrin, and cadherin receptors [35]. Proliferation and migration signaling pathways have also begun to be modeled in the context of microvascular formation, as has the molecular regulation of endothelial cell precursor differentiation activity during angiogenesis [101, 102]. Cell-Based Modeling Cell-based modeling covers a range of computational techniques, all focusing on the cell as the main functional unit of the model. One of the first cell-based computational models relevant to angiogenesis was a 1991 stochastic differential equation model by Stokes, Lauffenburger, and Williams that described the migration of individual endothelial cells, including chemotaxis, persistence, and random motion [52]. Subsequent models have considered how the microvascular cells interact with each other and their environment to form capillaries and eventually a full capillary network. Here, we briefly present three approaches to model angiogenesis processes at the cell level: agent-based programming, cellular automata, and cellular Potts modeling. Agent-based programming Agent-based programming originated in the 1940s and sprung from areas as diverse as game theory, ecology, and economics [53]. Agents are defined conceptually as individual entities or objects. In the computer code, they are represented as data structures with a number of unique characteristics: Agents interact with their environment, and they are capable of modifying their surroundings. They also interact with one another and can influence each other’s behaviors. They carry a computational genome, or a sequence of instructions (referred to as rules). Rules guide the agents’ response to other agents and their environment. Agents may move on discrete or continuous grids. In the larger context of biology, agent-based programming has been applied to study diverse phenomena including angiogenesis in mesenteric tissue [54], membrane transport [55, 56], inflammatory response [57, 58], and tumor growth [59, 60]. Angiogenesis models employing agents have allowed the exploration of cell behavioral patterns, chemotaxis, receptor

61

signaling, and network phenotype. In these models, each cell is represented by a single agent or multiple agents. Recent work has used agents to represent sections of cells and to allow for detailed movement, filopodia extension, elongation, or directional cues for cells [15, 61, 62]. Agent rules require listing the factors that influence cell behavior as events, with direct counterparts in biology. Rules may be algebraic or differential equations, Boolean, or a series of logical statements. An example rule for cell behavior would be the extent of cell migration as a function of a specific growth factor or whether a cell changes state from quiescent to active [15]. Developing rules is an iterative process, much like perfecting in vitro or in vivo experiments. As more knowledge is gained, the current assumptions may change, and a cycle of improvements is needed to keep pace with current biological information. Agentbased microvasculature models have been applied diversely, including a study assessing the effect of stem cell trafficking through the endothelium [63]; an experimentally coupled model of the formation and selection of tip cells in angiogenesis [61, 64]; and a simulation studying the coupled processes of endothelial migration, elongation, and proliferation [15]. Cellular automata Cellular automata are classified as a subset of agents. They also follow rules. However, unlike with agents, the rules are restricted to simple discrete changes – i.e., they are state changes of the automaton and refer only to individual automaton, and not grouped entities. While agents can adapt, change form and state, interact with each other, and modify their environment, cellular automata can change state but generally do not adapt in form or function. When a rule embodies a continuous algebraic or differential equation, then agents are generally used rather than cellular automata. That said, in the literature there is a blurry line between agents and automata, and in angiogenesis models, they have sometimes been used interchangeably [54, 65, 66]. Potts model A third means of cell-based modeling, the cellular Potts model, was developed in 1992 by James Glazier and Francois Graner. A main principle behind the cellular Potts model is the idea of energy minimization. Analogous to agents or automata, in the cellular Potts model, a cell lattice or grid is updated one pixel at a

A

Cardiac Peripheral

Lung Brain

Macular degeneration Inflammation

Ischemia

Breast

Brain

Pathological condition Cancer General

Hybrid agent-based models and bioinformatics Reaction-diffusion models Coupled convection, reaction-diffusion and agent-based models; compartmental models

Molecular

Convection-diffusion models Agent-based modeling Reaction-diffusion models Reaction-diffusion models Agent-based modeling; network models

Network Cell Molecular Molecular Multiscale

Molecular Multiscale

Agent-based modeling; reaction model Agent-based modeling

Multiscale Cell

Molecular

Fractals, cellular automaton model; compartmental Reaction-diffusion models, bioinformatics

Fractals Agent-based modeling Reaction-diffusion models

Network Cell Molecular Multiscale

Algebraic relationships, diffuse interface

Angiogenesis modeling methods Reaction-diffusion, pharmacokinetics, compartmental modeling; hybrid cellular automaton model; finite element, level set method Diffuse interface modeling; fractals; convection-diffusion modeling; invasion percolation Agent-based modeling, cellular Potts approach Reaction-diffusion models, bioinformatics

Multiscale

Molecular

Cell

Network

Biological level Multiscale

Monocyte trafficking

Genetics, metabolism, VEGF signaling, protein-protein interactions Tumor growth, hypoxia, tumor invasion, heterogeneity Vascular pattern formation Tumor growth, radiation effects, hypoxia Metabolism, delivery of antiangiogenic therapy; hypoxic response signaling Vascular pattern formation with Gompetzian tumor growth; VEGF whole body distribution Delivery of anti-VEGF therapy, differentiating breast cancer patients by race, and angiogenic genetic pathways EGFR-TGF signaling Cell-cell, cell-environmental interactions; capillary formation Protein signatures underlying cell behaviors; hypoxic response signaling FGF2 signaling and therapeutic delivery Exercise effects on sprouting angiogenesis, hypoxic response, VEGF effects; whole-body VEGF distribution O2 , blood flow, membrane permeability Stromal cell trafficking VEGF signaling Ocular drug delivery

Main processes studied VEGF tissue distribution, delivery of antiVEGF therapy, effects of blood flow on cancer cell colony formation; tumor invasion Tumor growth, hypoxia, nutrient consumption; vascular pattern formation; incorporation of progenitor cells; vascular normalization Tumor growth, hypoxia, nutrient consumption

Angiogenesis, Computational Modeling Perspective, Table 1 Angiogenesis models with therapeutic application, across biological scales

[58, 77]

[98] [63] [97] [99]

[33, 96] [40,62,97]

[51]

[94] [95]

[87, 93]

[65,78,92]

[27, 28] [59,90,91] [51]

[83, 89]

[17,87,88]

[86]

[26,29,30, 81–85]

Ref. [78–80]

62 Angiogenesis, Computational Modeling Perspective

Angiogenesis, Computational Modeling Perspective

63

A

Angiogenesis, Computational Modeling Perspective, Fig. 1 Molecular, cellular, and capillary network processes explored by angiogenesis computational and mathematical models. During

angiogenesis hypoxic cells secrete growth factors that drive the migration and proliferation of microvascular endothelial cells

time in response to rules. The rules are probabilistic and always include a calculation and minimization of the effective energy function (Hamiltonian) that governs lattice updates. In application to a vascular cell during angiogenesis, energy minimization can be based on properties such as adhesion, proliferation, chemotaxis, cell state, and signaling. Cellular Potts models have recently been applied to study sprouting, contact inhibition and endothelial migration patterns [67–70]. Agents, automata, and Potts models each has benefits for modeling cell behavior. The latter two apply more restrictions to what can be modeled, either by energy or state changes – however, generally they are

easier to define and characterize. Agents have few restrictions on design and provide a versatile framework for modeling biology. However, universal methods to define and analyze features of biological agent-based models and their rules have yet to be well established, as they have for differential equations [71]. Bioinformatics With the advent of systems biology experimental techniques like gene and protein arrays, and the development of public databases, comes the ability to obtain quantitative data in a high-throughput manner. This has provided high-dimensional preclinical and clinical data on molecules that regulate angiogenesis [103]. Making

64

Angiogenesis, Computational Modeling Perspective

use of this knowledge, angiogenesis researchers have References developed algorithms to study matrix-related proteins 1. Cai, J., Han, S., Qing, R., Liao, D., Law, B., Boulton, M.E.: [17, 18] and probe intracellular pathways in patient In pursuit of new anti-angiogenic therapies for cancer cell samples [72]. While the bioinformatic algorithms treatment. Front. Biosci. 16, 803–814 (2011) vary considerably, they generally involve a search for a 2. Ebos, J.M., Kerbel, R.S.: Antiangiogenic therapy: impact pattern: e.g., a string of amino acid sequences, a degree on invasion, disease progression, and metastasis. Nat. Rev. Clin. Oncol. 8, 316 (2011) of connectivity, or pairs of expression levels. They also 3. Quesada, A.R., Medina, M.A., Alba, E.: Playing only one must compensate for a false discovery rate and retain instrument may be not enough: limitations and future of the the statistical relevancy of the original experimental antiangiogenic treatment of cancer. Bioessays 29, 1159– data. 1168 (2007) Multiscale Modeling Together, the above models have helped illuminate how diverse stimuli induce angiogenesis, on multiple biological levels. Angiogenesis involves thousands of proteins, hundreds of genes, multiple cell types, and many tissues involving changes in both space and time [21]. To accurately characterize how these factors interact, multiscale modeling becomes essential [73]. Integrating models in a flexible manner is an ongoing challenge for angiogenesis modelers. Techniques to parameterize [74], coarse-grain, modularize [21], and distribute [75] models have helped advance the field. Recent multiscale models have been applied to characterize angiogenesis in a heterogeneous brain cancer environment [76], effects of exercise on skeletal muscle angiogenesis [21,62], and monocyte trafficking [77].

Conclusions and Therapeutic Applications All of the five methods described have been employed to predict the effects of existing therapy or test a new mechanistic hypothesis that could be a target for therapeutic development. We highlight a few examples in Table 1. Together these techniques offer a suite of tools to study physiological and pathological angiogenesis in silico [104, 105]. Increasingly, angiogenesis modeling is being integrated with wet lab experiments or clinical work, with the ultimate goal of better understanding cellular function and human health. In summary, mathematical and computational modeling allow researchers to characterize the complex signaling, biological function, and structures unique to angiogenesis – offering a powerful tool to understand and treat diseases associated with the microvasculature (Fig. 1).

4. Deakin, A.S.: Model for initial vascular patterns in melanoma transplants. Growth 40, 191–201 (1976) 5. Liotta, L.A., Saidel, G.M., Kleinerman, J.: Diffusion model of tumor vascularization and growth. Bull. Math. Biol. 39, 117–128 (1977) 6. Anderson, A.R., Chaplain, M.A.: Continuous and discrete mathematical models of tumor-induced angiogenesis. Bull. Math. Biol. 60, 857–899 (1998) 7. Chaplain, M.A., Anderson, A.R.: Mathematical modelling, simulation and prediction of tumour-induced angiogenesis. Invasion Metastas. 16, 222–234 (1996) 8. Sun, S., Wheeler, M.F., Obeyesekere, M., Patrick, C.W., Jr.: A deterministic model of growth factor-induced angiogenesis. Bull. Math. Biol. 67, 313–337 (2005) 9. Mac Gabhann, F., Popel, A.S.: Systems biology of vascular endothelial growth factors. Microcirculation 15, 715–738 (2008) 10. Mac Gabhann, F., Qutub, A.A., Annex, B.H., Popel, A.S.: Systems biology of pro-angiogenic therapies targeting the VEGF system. Wiley Interdiscip. Rev. Syst. Biol. Med. 2, 694–707 (2010) 11. Filion, R.J., Popel, A.S.: A reaction-diffusion model of basic fibroblast growth factor interactions with cell surface receptors. Ann. Biomed. Eng. 32, 645–663 (2004) 12. Karagiannis, E.D., Popel, A.S.: A theoretical model of type I collagen proteolysis by matrix metalloproteinase (MMP) 2 and membrane type 1 MMP in the presence of tissue inhibitor of metalloproteinase 2. J. Biol. Chem. 279, 39105–39114 (2004) 13. Nathan, J.C., Qutub, A.A.: Patient-specific modeling of hypoxic response and microvasculature dynamics. In: Kerckhoffs, R.C. (ed.) Patient-Specific Modeling of the Cardiovascular System, pp. 183–201. Springer, New York (2010) 14. Peirce, S.M.: Computational and mathematical modeling of angiogenesis. Microcirculation 15, 739–751 (2008) 15. Qutub, A.A., Popel, A.S.: Elongation, proliferation & migration differentiate endothelial cell phenotypes and determine capillary sprouting. BMC Syst. Biol. 3, 13 (2009) 16. Das, A., Lauffenburger, D., Asada, H., Kamm, R.D.: A hybrid continuum-discrete modelling approach to predict and control angiogenesis: analysis of combinatorial growth factor and matrix effects on vessel-sprouting morphology. Philos. Transact. A Math. Phys. Eng. Sci. 368, 2937–2960 (2010) 17. Rivera, C.G., Bader, J.S., Popel, A.S.: Angiogenesisassociated crosstalk between collagens, CXC chemokines, and thrombospondin domain-containing proteins. Ann. Biomed. Eng. 39, 2213–2222 (2011)

Angiogenesis, Computational Modeling Perspective 18. Karagiannis, E.D., Popel, A.S.: A systematic methodology for proteome-wide identification of peptides inhibiting the proliferation and migration of endothelial cells. Proc. Natl. Acad. Sci. U.S.A. 105, 13775–13780 (2008) 19. Kharait, S., Hautaniemi, S., Wu, S., Iwabu, A., Lauffenburger, D.A., Wells, A.: Decision tree modeling predicts effects of inhibiting contractility signaling on cell motility. BMC Syst. Biol. 1, 9 (2007) 20. Deisboeck, T.S., Wang, Z., Macklin, P., Cristini, V.: Multiscale cancer modeling. Annu. Rev. Biomed. Eng. 13, 127– 155 (2011) 21. Qutub, A., Gabhann, F., Karagiannis, E., Vempati, P., Popel, A.: Multiscale models of angiogenesis. IEEE Eng. Med. Biol. Mag. 28, 14–31 (2009) 22. Folkman, J., Merler, E., Abernathy, C., Williams, G.: Isolation of a tumor factor responsible for angiogenesis. J. Exp. Med. 133, 275–288 (1971) 23. Carey, K.A., Farnfield, M.M., Tarquinio, S.D., CameronSmith, D.: Impaired expression of notch signaling genes in aged human skeletal muscle. J. Gerontol. A Biol. Sci. Med. Sci. 62, 9–17 (2007) 24. Baish, J.W., Stylianopoulos, T., Lanning, R.M., Kamoun, W.S., Fukumura, D., Munn, L.L., Jain, R.K.: Scaling rules for diffusive drug delivery in tumor and normal tissues. Proc. Natl. Acad. Sci. U.S.A. 108, 1799–1803 (2011) 25. Seaman, M.E., Peirce, S.M., Kelly, K.: Rapid analysis of vessel elements (RAVE): a tool for studying physiologic, pathologic and tumor angiogenesis. PLoS One 6, e20807 (2011) 26. Baish, J.W., Gazit, Y., Berk, D.A., Nozue, M., Baxter, L.T., Jain, R.K.: Role of tumor vascular architecture in nutrient and drug delivery: an invasion percolation-based network model. Microvasc. Res. 51, 327–346 (1996) 27. Di Ieva, A., Grizzi, F., Ceva-Grimaldi, G., Aimar, E., Serra, S., Pisano, P., Lorenzetti, M., Tancioni, F., Gaetani, P., Crotti, F., Tschabitscher, M., Matula, C., Rodriguez, Y.B.R.: The microvascular network of the pituitary gland: a model for the application of fractal geometry to the analysis of angioarchitecture and angiogenesis of brain tumors. J. Neurosurg. Sci. 54, 49–54 (2010) 28. Risser, L., Plouraboue, F., Steyer, A., Cloetens, P., Le Duc, G., Fonta, C.: From homogeneous to fractal normal and tumorous microvascular networks in the brain. J. Cereb. Blood Flow Metab. 27, 293–303 (2007) 29. Grizzi, F., Russo, C., Colombo, P., Franceschini, B., Frezza, E.E., Cobos, E., Chiriva-Internati, M.: Quantitative evaluation and modeling of two-dimensional neovascular network complexity: the surface fractal dimension. BMC Cancer 5, 14 (2005) 30. Amyot, F., Camphausen, K., Siavosh, A., Sackett, D., Gandjbakhche, A.: Quantitative method to study the network formation of endothelial cells in response to tumor angiogenic factors. IEE Proc. Syst. Biol. 152, 61–66 (2005) 31. Ferrara, N.: Vascular endothelial growth factor. Arterioscler. Thromb. Vasc. Biol. 29, 789–791 (2009) 32. Senger, D.R., Galli, S.J., Dvorak, A.M., Perruzzi, C.A., Harvey, V.S., Dvorak, H.F.: Tumor cells secrete a vascular permeability factor that promotes accumulation of ascites fluid. Science 219, 983–985 (1983)

65 33. Filion, R.J., Popel, A.S.: Intracoronary administration of FGF-2: a computational model of myocardial deposition and retention. Am. J. Physiol. Heart Circ. Physiol. 288, H263–H279 (2005) 34. Stokes, C.L., Lauffenburger, D.A.: Analysis of the roles of microvessel endothelial cell random motility and chemotaxis in angiogenesis. J. Theor. Biol. 152, 377–403 (1991) 35. Bauer, A.L., Jackson, T.L., Jiang, Y., Rohlf, T.: Receptor cross-talk in angiogenesis: mapping environmental cues to cell phenotype using a stochastic, Boolean signaling network model. J. Theor. Biol. 264, 838–846 (2010) 36. Mac Gabhann, F., Popel, A.S.: Model of competitive binding of vascular endothelial growth factor and placental growth factor to VEGF receptors on endothelial cells. Am. J. Physiol. Heart Circ. Physiol. 286, H153–H164 (2004) 37. Folkman, J.: Angiogenesis in cancer, vascular, rheumatoid and other disease. Nat. Med. 1, 27–31 (1995) 38. Cao, Y., Linden, P., Shima, D., Browne, F., Folkman, J.: In vivo angiogenic activity and hypoxia induction of heterodimers of placenta growth factor/vascular endothelial growth factor. J. Clin. Invest. 98, 2507–2511 (1996) 39. Mac Gabhann, F., Popel, A.S.: Interactions of VEGF isoforms with VEGFR-1, VEGFR-2, and neuropilin in vivo: a computational model of human skeletal muscle. Am. J. Physiol. Heart Circ. Physiol. 292, H459–H474 (2007) 40. Ji, J.W., Mac Gabhann, F., Popel, A.S.: Skeletal muscle VEGF gradients in peripheral arterial disease: simulations of rest and exercise. Am. J. Physiol. Heart Circ. Physiol. 293, H3740–H3749 (2007) 41. Stefanini, M.O., Wu, F.T., Mac Gabhann, F., Popel, A.S.: A compartment model of VEGF distribution in blood, healthy and diseased tissues. BMC Syst. Biol. 2, 77 (2008) 42. Karagiannis, E.D., Popel, A.S.: Distinct modes of collagen type I proteolysis by matrix metalloproteinase (MMP) 2 and membrane type I MMP during the migration of a tip endothelial cell: insights from a computational model. J. Theor. Biol. 238, 124–145 (2006) 43. Wang, G.L., Semenza, G.L.: General involvement of hypoxia-inducible factor 1 in transcriptional response to hypoxia. Proc. Natl. Acad. Sci. U.S.A. 90, 4304–4308 (1993) 44. Semenza, G.L.: Oxygen-dependent regulation of mitochondrial respiration by hypoxia-inducible factor 1. Biochem. J. 405, 1–9 (2007) 45. Qutub, A.A., Popel, A.S.: A computational model of intracellular oxygen sensing by hypoxia-inducible factor HIF1 alpha. J. Cell. Sci. 119, 3467–3480 (2006) 46. Kohn, K.W., Riss, J., Aprelikova, O., Weinstein, J.N., Pommier, Y., Barrett, J.C.: Properties of switch-like bioregulatory networks studied by simulation of the hypoxia response control system. Mol. Biol. Cell. 15, 3042–3052 (2004) 47. Semenza, G.L.: HIF-1: using two hands to flip the angiogenic switch. Cancer Metastas. Rev. 19, 59–65 (2000) 48. Dayan, F., Monticelli, M., Pouyssegur, J., Pecou, E.: Gene regulation in response to graded hypoxia: the nonredundant roles of the oxygen sensors PHD and FIH in the HIF pathway. J. Theor. Biol. 259, 304–316 (2009) 49. Dayan, F., Roux, D., Brahimi-Horn, M.C., Pouyssegur, J., Mazure, N.M.: The oxygen sensor factor-inhibiting hypoxia-inducible factor-1 controls expression of distinct

A

66

50.

51.

52.

53. 54.

55.

56.

57.

58.

59.

60.

61.

62.

63.

64.

65.

Angiogenesis, Computational Modeling Perspective genes through the bifunctional transcriptional character of hypoxia-inducible factor-1alpha. Cancer Res. 66, 3688–3698 (2006) Yucel, M.A., Kurnaz, I.A.: An in silico model for HIFalpha regulation and hypoxia response in tumor cells. Biotechnol. Bioeng. 97, 588–600 (2007) Qutub, A.A., Popel, A.S.: Reactive oxygen species regulate hypoxia-inducible factor HIF1alpha differentially in cancer and ischemia. Mol. Cell. Biol. 28, 5106–5119 (2008) Stokes, C.L., Lauffenburger, D.A., Williams, S.K.: Migration of individual microvessel endothelial cells: stochastic model and parameter measurement. J. Cell. Sci. 99(Pt 2), 419–430 (1991) Maes, P.: Modeling adaptive autonomous agents. Artif. Life 135–162 (1993/1994) Peirce, S.M., Van Gieson, E.J., Skalak, T.C.: Multicellular simulation predicts microvascular patterning and in silico tissue assembly. Faseb J. 18, 731–733 (2004) Qutub, A.A., Hunt, C.A.: Glucose transport to the brain: a systems model. Brain Res. Brain Res. Rev. 49, 595–617 (2005) Liu, Y., Hunt, C.A.: Mechanistic study of the cellular interplay of transport and metabolism using the synthetic modeling method. Pharm. Res. 23, 493–505 (2006) An, G., Mi, Q., Dutta-Moscato, J., Vodovotz, Y.: Agentbased models in translational systems biology. Wiley Interdiscip. Rev. Syst. Biol. Med. 1, 159–171 (2009) Tang, J., Hunt, C.A.: Identifying the rules of engagement enabling leukocyte rolling, activation, and adhesion. PLoS Comput. Biol. 6, e1000681 (2010) Mansury, Y., Diggory, M., Deisboeck, T.S.: Evolutionary game theory in an agent-based brain tumor model: exploring the ‘Genotype-Phenotype’ link. J. Theor. Biol. 238, 146–156 (2006) Enderling, H., Anderson, A.R., Chaplain, M.A., Beheshti, A., Hlatky, L., Hahnfeldt, P.: Paradoxical dependencies of tumor dormancy and progression on basic cell kinetics. Cancer Res. 69, 8814–8821 (2009) Bentley, K., Mariggi, G., Gerhardt, H., Bates, P.A.: Tipping the balance: robustness of tip cell selection, migration and fusion in angiogenesis. PLoS Comput. Biol. 5, e1000549 (2009) Liu, G., Qutub, A.A., Vempati, P., Mac Gabhann, F., Popel, A.S.: Module-based multiscale simulation of angiogenesis in skeletal muscle. Theor. Biol. Med. Model. 8, 6 (2011) Bailey, A.M., Lawrence, M.B., Shang, H., Katz, A.J., Peirce, S.M.: Agent-based model of therapeutic adiposederived stromal cell trafficking during ischemia predicts ability to roll on P-selectin. PLoS Comput. Biol. 5, e1000294 (2009) Jakobsson, L., Franco, C.A., Bentley, K., Collins, R.T., Ponsioen, B., Aspalter, I.M., Rosewell, I., Busse, M., Thurston, G., Medvinsky, A., Schulte-Merker, S., Gerhardt, H.: Endothelial cells dynamically compete for the tip cell position during angiogenic sprouting. Nat. Cell. Biol. 12, 943–953 (2010) Sedivy, R., Thurner, S., Budinsky, A.C., Kostler, W.J., Zielinski, C.C.: Short-term rhythmic proliferation of human breast cancer cell lines: surface effects and fractal growth patterns. J. Pathol. 197, 163–169 (2002)

66. Lee, Y., Kouvroukoglou, S., McIntire, L.V., Zygourakis, K.: A cellular automaton model for the proliferation of migrating contact-inhibited cells. Biophys. J. 69, 1284– 1298 (1995) 67. Merks, R.M., Perryn, E.D., Shirinifard, A., Glazier, J.A.: Contact-inhibited chemotaxis in de novo and sprouting blood-vessel growth. PLoS Comput. Biol. 4, e1000163 (2008) 68. Bauer, A.L., Jackson, T.L., Jiang, Y.: Topography of extracellular matrix mediates vascular morphogenesis and migration speeds in angiogenesis. PLoS Comput. Biol. 5, e1000445 (2009) 69. Graner, F., Glazier, J.A.: Simulation of biological cell sorting using a two-dimensional extended Potts model. Phys. Rev. Lett. 69, 2013–2016 (1992) 70. Shirinifard, A., Gens, J.S., Zaitlen, B.L., Poplawski, N.J., Swat, M., Glazier, J.A.: 3D multi-cell simulation of tumor growth and angiogenesis. PLoS One 4, e7190 (2009) 71. Long, B., Qutub, A.A.: Promoting behavioral rules to agents in modeling angiogenesis. In: Annual Conference on Biomedical Engineering Society (BMES), Austin (2010) 72. Kornblau, S.M., Qiu, Y.H., Zhang, N., Singh, N., Fader, S., Ferrajoli, A., York, H., Qutub, A.A., Coombes, K.R., Watson, D.K.: Abnormal expression of friend leukemia virus integration 1 (Fli1) protein is an adverse prognostic factor in acute myeloid leukemia. Blood 118, 5604–5612 (2011) 73. Frieboes, H.B., Chaplain, M.A., Thompson, A.M., Bearer, E.L., Lowengrub, J.S., Cristini, V.: Physical oncology: a bench-to-bedside quantitative and predictive approach. Cancer Res. 71, 298–302 (2011) 74. Southern, J., Pitt-Francis, J., Whiteley, J., Stokeley, D., Kobashi, H., Nobes, R., Kadooka, Y., Gavaghan, D.: Multiscale computational modelling in biology and physiology. Prog. Biophys. Mol. Biol. 96, 60–89 (2008) 75. Ropella, G.E., Hunt, C.A.: Cloud computing and validation of expandable in silico livers. BMC Syst. Biol. 4, 168 (2010) 76. Zhang, L., Wang, Z., Sagotsky, J.A., Deisboeck, T.S.: Multiscale agent-based cancer modeling. J. Math. Biol. 58, 545–559 (2009) 77. Bailey, A.M., Thorne, B.C., Peirce, S.M.: Multi-cell agentbased simulation of the microvasculature to study the dynamics of circulating inflammatory cell trafficking. Ann. Biomed. Eng. 35, 916–936 (2007) 78. Stefanini, M.O., Qutub, A.A., Gabhann, F.M., Popel, A.S.: Computational models of VEGF-associated angiogenic processes in cancer. Math. Med. Biol. 29, 85–94 (2012) 79. Zheng, X., Wise, S.M., Cristini, V.: Nonlinear simulation of tumor necrosis, neo-vascularization and tissue invasion via an adaptive finite-element/level-set method. Bull. Math. Biol. 67, 211–259 (2005) 80. Alarcon, T., Byrne, H.M., Maini, P.K.: A cellular automaton model for tumour growth in inhomogeneous environment. J. Theor. Biol. 225, 257–274 (2003) 81. Bearer, E.L., Lowengrub, J.S., Frieboes, H.B., Chuang, Y.L., Jin, F., Wise, S.M., Ferrari, M., Agus, D.B., Cristini, V.: Multiparameter computational modeling of tumor invasion. Cancer Res. 69, 4493–4501 (2009)

Angiogenesis, Mathematical Modeling Perspective 82. Frieboes, H.B., Jin, F., Chuang, Y.L., Wise, S.M., Lowengrub, J.S., Cristini, V.: Three-dimensional multispecies nonlinear tumor growth-II: tumor invasion and angiogenesis. J. Theor. Biol. 264, 1254–1278 (2010) 83. Lowengrub, J.S., Frieboes, H.B., Jin, F., Chuang, Y.L., Li, X., Macklin, P., Wise, S.M., Cristini, V.: Nonlinear modelling of cancer: bridging the gap between cells and tumours. Nonlinearity 23, R1–R9 (2010) 84. Stoll, B.R., Migliorini, C., Kadambi, A., Munn, L.L., Jain, R.K.: A mathematical model of the contribution of endothelial progenitor cells to angiogenesis in tumors: implications for antiangiogenic therapy. Blood 102, 2555– 2561 (2003) 85. Jain, R.K., Tong, R.T., Munn, L.L.: Effect of vascular normalization by antiangiogenic therapy on interstitial hypertension, peritumor edema, and lymphatic metastasis: insights from a mathematical model. Cancer Res. 67, 2729–2735 (2007) 86. Bauer, A.L., Jackson, T.L., Jiang, Y.: A cell-based model exhibiting branching and anastomosis during tumorinduced angiogenesis. Biophys. J. 92, 3105–3121 (2007) 87. Mac Gabhann, F., Popel, A.S.: Targeting neuropilin-1 to inhibit VEGF signaling in cancer: comparison of therapeutic approaches. PLoS Comput. Biol. 2, e180 (2006) 88. Jain, H.V., Nor, J.E., Jackson, T.L.: Modeling the VEGFBcl-2-CXCL8 pathway in intratumoral agiogenesis. Bull. Math. Biol. 70, 89–117 (2008) 89. Zhang, L., Athale, C.A., Deisboeck, T.S.: Development of a three-dimensional multiscale agent-based tumor model: simulating gene-protein interaction profiles, cell phenotypes and multicellular patterns in brain cancer. J. Theor. Biol. 244, 96–107 (2007) 90. Deisboeck, T.S., Zhang, L., Yoon, J., Costa, J.: In silico cancer modeling: is it ready for prime time? Nat. Clin. Pract. Oncol. 6, 34–42 (2009) 91. Mansury, Y., Kimura, M., Lobo, J., Deisboeck, T.S.: Emerging patterns in tumor systems: simulating the dynamics of multicellular clusters with an agent-based spatial agglomeration model. J. Theor. Biol. 219, 343–370 (2002) 92. Stefanini, M.O., Wu, F.T., Mac Gabhann, F., Popel, A.S.: Increase of plasma VEGF after intravenous administration of bevacizumab is predicted by a pharmacokinetic model. Cancer Res. 70, 9886–9894 (2010) 93. Martin, D.N., Boersma, B.J., Yi, M., Reimers, M., Howe, T.M., Yfantis, H.G., Tsai, Y.C., Williams, E.H., Lee, D.H., Stephens, R.M., Weissman, A.M., Ambs, S.: Differences in the tumor microenvironment between African-American and European-American breast cancer patients. PLoS One 4, e4531 (2009) 94. Wang, Z., Bordas, V., Sagotsky, J., Deisboeck, T.S.: Identifying therapeutic targets in a combined EGFRTGFfbetagR signalling cascade using a multiscale agentbased cancer model. Math. Med. Biol. 29, 95–108 (2012) 95. Rekhi, R., Long, B., Arevalos, C.A., Jiwon, J., Qutub, A.: Modeling the angiogenic response of the neurovasculature in ischemia. In: Annual Conference on Biomedical Engineering Society (BMES), Austin (2010) 96. Waters, S.L., Alastruey, J., Beard, D.A., Bovendeerd, P.H., Davies, P.F., Jayaraman, G., Jensen, O.E., Lee, J., Parker, K.H., Popel, A.S., Secomb, T.W., Siebes, M., Sherwin, S.J., Shipley, R.J., Smith, N.P., van de Vosse, F.N.:

67

97.

98.

99.

100.

101.

102.

103.

104.

105.

Theoretical models for coronary vascular biomechanics: progress & challenges. Prog. Biophys. Mol. Biol. 104, 49– 76 (2010) Wu, F.T., Stefanini, M.O., Mac Gabhann, F., Kontos, C.D., Annex, B.H., Popel, A.S.: VEGF and soluble VEGF receptor-1 (sFlt-1) distributions in peripheral arterial disease: an in silico model. Am. J. Physiol. Heart Circ. Physiol. 298, H2174–H2191 (2010) Ji, J.W., Tsoukias, N.M., Goldman, D., Popel, A.S.: A computational model of oxygen transport in skeletal muscle for sprouting and splitting modes of angiogenesis. J. Theor. Biol. 241, 94–108 (2006) Mac Gabhann, F., Demetriades, A.M., Deering, T., Packer, J.D., Shah, S.M., Duh, E., Campochiaro, P.A., Popel, A.S.: Protein transport to choroid and retina following periocular injection: theoretical and experimental study. Ann. Biomed. Eng. 35, 615–630 (2007) Nguyen, L.K., Cavadas, M.A., Scholz, C.C., Fitzpatrick, S.F., Bruning, U., Cummins, E.P., Tambuwala, M.M., Manresa, M.C., Kholodenko, B.N., Taylor, C.T., Cheong, A.: A dynamic model of the hypoxia-inducible factor 1˛ (HIF-1˛) network. J. Cell. Sci. 126, 1454–1463 (2013) Logsdon, E.A., Finley, S.D., Popel, A.S., Mac Gabhann, F.: A systems biology view of blood vessel growth and remodelling. J. Cell. Mol. Med. 18, 1491–1508 (2014) Clegg, L.E., Mac Gabhann, F.: Systems biology of the microvasculature. Integr. Biol. (Camb.) (2015) (Epub ahead of print) Chu, L.H., Rivera, C.G., Popel, A.S., Bader, J.S.: Constructing the angiome: a global angiogenesis protein interaction network. Physiol. Genomics 44, 915–924 (2012) Finley, S.D., Chu, L.H., Popel, A.S.: Computational systems biology approaches to anti-angiogenic cancer therapeutics. Drug Discov. Today 20, 187–197 (2015) Finley, S.D., Popel, A.S.: Effect of tumor microenvironment on tumor VEGF during anti-VEGF treatment: systems biology predictions. J. Natl. Cancer Inst. 105, 802– 811 (2013)

Angiogenesis, Mathematical Modeling Perspective M. Wu and John S. Lowengrub Department of Mathematics, University of California, Irvine, CA, USA

Angiogenesis Modeling Angiogenesis is the process of the development of new blood vessels from a preexisting vasculature. In early development, angiogenesis occurs both in the yolk sac (Fig. 1) and in the embryo after the primary vascular

A

68

Angiogenesis, Mathematical Modeling Perspective, Fig. 1 Sprouting angiogenesis (white arrows) in the 3-day-old quail yolk sac (Reprinted by permission from Macmillan Publishers Ltd: Nature [13], copyright (2010))

plexus is formed by vasculogenesis [13]. Later, angiogenesis continues to support the development of organs after birth [5]. In adults, sprouting angiogenesis takes place during wound healing and pathological conditions, such as tumor-induced angiogenesis and ocular and inflammatory disorders [4]. Angiogenesis is a very complex process involving proliferation and movement of endothelial cells (ECs), degradation and creation of extracellular matrix, fusion (anastomosis) of vessels, as well as pruning and remodeling [13], with all of these features being driven by complex signaling processes and nonlinear interactions. The neovasculature eventually remodels itself into a hierarchical network system in space, which may take on abnormal characteristics such as tortuous, leaky vessel distributions under pathological conditions (e.g., tumor growth). Readers are referred to Figg and Folkman [4] for biological details. To better understand angiogenesis, both continuum, fully discrete, and continuum-discrete mathematical models have been developed. Generally, there are two modeling approaches. One approach focuses on blood vessel densities rather than vessel morphology. In this case, continuum conservation laws are introduced to describe the dynamics of the vessel densities and angiogenic factors. Alternatively, the other approach involves modeling the vessel network directly with the

Angiogenesis, Mathematical Modeling Perspective

vessels consisting of cylindrical segments connected at a set of nodes. Mechanisms such as branching and anastomosis have been modeled as well as vascular endothelial cell (EC) proliferation and migration via chemotaxis up gradients of angiogenic factors. Models have been developed to obtain the details of blood flow and network remodeling. In 1998, Anderson and Chaplain [1] developed a continuum-discrete mathematical model that describes vessel sprouting, branching, and anastomosis in the context of tumor-induced angiogenesis. Later, Levine et al. [7] modeled capillary formation involving proteolytic enzymes, angiogenesis factors, and different states of receptors on the ECs using reinforced random walks. Godde and Kurz [6] computed the blood flow rate inside a discrete vessel model and simulated vascular remodeling during angiogenesis responding to biophysical, chemical, and hemodynamic factors (i.e., wall shear stress). McDougall et al. [10] also modeled blood flow and vascular remodeling but in the context of the Anderson-Chaplain 1998 model. These studies relied on the fundamental vessel physiology studies by Pries et al. [12]. At the microscopic level, Bentley et al. [2] studied tip cell selection using discrete cell-based models. Zheng et al. [16] developed the first model coupling tumor growth and angiogenesis by combining the continuum model analyzed by Cristini et al. [3] with the Anderson-Chaplain angiogenesis model, treating neovasculature as a source of oxygen regardless of the flow rate. Later, Macklin et al. [9] extended this work by incorporating the effects of blood flow and vessel remodeling. Welter et al. [15] studied vascular tumor growth accounting for both venous and arteriole vessels. Owen et al. [11] developed a multiscale model of angiogenesis and tumor growth combining a cellular automaton model for tumor growth that incorporates intracellular signaling of the cell cycle and production of proangiogenic factors with a model for a developing vascular network. See the review by Lowengrub et al. [8] for a more complete collection of references.

Movement of ECs The neovasculature grows by the development of sprouts, movement of tip ECs, and proliferation of ECs behind the tip cell. Accordingly, Anderson and Chaplain assumed that the motion of an individual

Angiogenesis, Mathematical Modeling Perspective

69

a

b Parent vessel

Parent vessel

A j−1

P0

j

P2

P1 P3

j+1

i−1

i

TAF gradient

P4=0

Q (i,j)

j

i+1

i

Tumour

Tumour

Angiogenesis, Mathematical Modeling Perspective, Fig. 2 Schematic of the development of a neovascular network. (a) EC movement is constrained on the grid. The sprout endothelial tip cell located at a node in a Cartesian grid may move to one of the four orthogonal neighbors (filled circles) with a probability of P1 ; P2 ; P3 ; P4 or remain at the current node with a probability of P0 . (b) The vessel network formed by tip cell movement; EC proliferation and anastomosis form a network of straight

segments connected at nodes. The flow through the network is obtained by solving the mass conservation equations at the nodes (From McDougall et al. [10], reprinted from Vol 241, Mathematical modelling of dynamic adaptive tumour-induced angiogenesis: Clinical implications and therapeutic targeting strategies, Pages 26, Copyright (2010), with permission from Elsevier)

endothelial cell located at the tip of a capillary sprout governs the motion of the whole sprout. This was later confirmed by an experimental study [5]. In the Anderson-Chaplain model, the vasculature is defined on a Cartesian grid (Fig. 2), and the tip cell movement is governed by three factors: random movement, chemotaxis up gradients of angiogenic promoters (e.g., vascular endothelial cell growth factor, VEGF), and haptotaxis up gradients of cellular adhesion sites, which are assumed to be proportional to the density of extracellular matrix. In this model, a continuum EC density, n, is introduced and obeys the mass conservation equation:

obtain an explicit vessel representation by discretizing Eq. (1) and introducing probabilities for EC motion: P0 for remaining stationary, P1 for moving right, P2 for moving left, P3 for moving up, and P4 for moving down. See Fig. 2. The probabilities Pi arise from the finite difference approximation of Eq. (1). For example,

@n D Dn r 2 n  r  . T nrT /  r  .nrE/ @t

(1)

where Dn is the random motility coefficient. The parameter T is the chemotactic coefficient depending on the proangiogenesis factor VEGF concentration T ,  is the haptotactic coefficient, and E is the density of the extracellular matrix (ECM) density. Anderson and Chaplain then developed a stochastic model to

P1 D

1 P



  Dt t  C

T .Ti;j / Ti C1;j  Ti 1;j 2 2 x .2x/   C Ei C1;j  Ei 1;j

(2)

where PN is a normalizing factor. Similar formulas are obtained for Pi D 2; 3; 4 and P0 D 1  .P1 C P2 C P3 C P4 /. The movement of individual ECs now follows this stochastic equation. EC proliferation is modeled by division of cells trailing the tip EC. Further, branching at sprout tips and loops formed by anastomosis are incorporated via rules in the discrete system.

70

Angiogenesis, Mathematical Modeling Perspective

Blood Flow and Vascular Remodeling During angiogenesis, new sprouts anastomose and expand the vascular network. At the same time, the newly formed networks remodel themselves, stimulated by mechanical forces and chemical factors. The wall shear stress, which depends on the blood viscosity, the blood flow rate, and the interaction of pro- and antiangiogenic proteins are vital factors for both the development and maintenance of the vascular bed. Furthermore, these effects are nonlinearly coupled. In most angiogenesis models, the vessels are assumed to consist of cylindrical segments connected at vessel nodes. Between two connected vessel nodes p and q, a Poiseuille-like expression can be determined for the flow rate: Qpq D

4 Ppq Rpq

8Lpq

(3)

where Rpq is the radius of the vessel segment, Ppq D .Pp  Pq / is the pressure drop,  is the apparent viscosity, and Lpq is the length of the vessel segment. Assuming that blood is a non-Newtonian fluid, the apparent viscosity  D .R; H / where H is the hematocrit, which is the volume fraction of red blood cells (RBC) in the blood [12]. The pressure and flow rate in the system are determined by solving the conservation equations: 0D

X

Qpq D

4 X Rpq .Pp  Pq /

8L

to cell-cell signaling from the ECs in the network along vessel wall. The coefficient ks is the shrinking tendency, and kw , kp , km , kc are the sensitivities’ responses to the different stimuli. Furthermore, vessels with small flow rates or under extremely high pressures may be pruned from the system [6, 9–12].

Application to Vascular Tumor Growth Angiogenesis models can be used to study vascular tumor growth (see references given earlier) and treatment of chemotherapy [14]. Given the flow rate in the interconnected vascular network, the distribution of RBCs (hematocrit), which carry oxygen, nutrients, and chemical species transported in the blood, may be computed by solving transport equations in the vascular network. The oxygen and nutrients supplied by the developing neovasculature provide sources of growthpromoting factors for the tumor. The hypoxic tumor cells release proangiogenic factors (e.g., VEGF) which provide a source of T and thus affect the development and remodeling of the vascular network. Zheng et al. [16] and later Macklin et al. [9] modeled this by solving quasi-steady diffusion equations for T 0 Dr  .DT rT /  Tdecay T  .Tbinding T /Btips .x; t/ C Tprod ;

(7)

(4)

where DT is the diffusion coefficient of proangiogenic factors, Btip is the indicator function of the sprout The wall shear stress can be calculated by tips (i.e., =1 at sprout tips), and Tdecay , Tbinding , and Tprod denote the natural decay, binding, and produc4 pq D jQpq j (5) tion rates of proangiogenic factors. Typically, Tprod D 3 Rpq N Tprod Bhypoxic .x; t/ where Bhypoxic is the indicator funcNT which is strongly related to the sprouting and remod- tion for the hypoxic tumor cells and prod is a rate. eling of vascular network. In particular, over a time Analogously, the oxygen concentration can be assumed to satisfy interval t, the radius is adapted according to q

q

  R D kw Swss C kp Sp C km Sm C kc Sc  ks Rt (6)

0 D r  .Dr /   . / C  pre C  neo ;

where Swss represents the stimulation by wall shear stress, Sp represents the stimulation by intravascular pressure, Sm represents the stimulation of metabolites (e.g., oxygen) which is related to blood flow rate and hematocrit, and Sc represents the stimulation due

where D is the diffusion coefficient of oxygen and  . / is the uptake rate. Oxygen is supplied by preexisting,  pre , and by newly developing vessels,  neo . The extravasation of oxygen by the neovasculature can be modeled as [9]:

(8)

Angiogenesis, Mathematical Modeling Perspective

71

A

Angiogenesis, Mathematical Modeling Perspective, Fig. 3 Tumor-induced angiogenesis and vascular tumor growth. The tumor regions (red-proliferating region, blue-hypoxic region with produce VEGF, brown-necrotic region) and the oxygen, mechanical pressure, and hematocrit level are shown. Around day 48, the hypoxic regions in the tumor release proangiogenic factors which trigger the beginning of angiogenesis; the new vessels from the boundary at the top of the domain grow down-

 neo D neo Bneo .x; t/.

H HD

Future Directions for Angiogenesis Modeling

H min /C

.1 c.Pvessel ; P //.1  /;

ward to supply the tumor with oxygen and growth-promoting factors. At the later stages (around day 82.5), the extravasation of oxygen decreases because of the increased mechanical pressure generated by the growing tumor (From Macklin et al. [9], reprinted with kind permission from Springer Science+Business Media: Journal of Mathematical Biology, Multiscale modelling and nonlinear simulation of vascular tumour growth, Vol 58, Page 787, 2008)

(9)

Thus far, angiogenesis models have tended to focus on the mesoscale, with limited description of the biophysical details of cell-cell interactions, biochemical signaling and mechanical forces, and mechanotransduction-driven signaling processes. An important future direction for angiogenesis modeling involves the development of multiscale models that are capable of describing the nonlinear coupling among intracellular signaling processes, cell-cell interaction, and the development and functionality of a neovascular network.

where neo is a constant, Bneo .x; t/ is the indicator function of the neovasculature (i.e., = 1 at vessel locations), H is the hematocrit, H D and H min reflect the normal and minimum levels of H , P is the solid pressure, and Pvessel is the intravascular pressure. The function c.Pvessel ; P / models the suppression of extravasation by the stress generated by the solid tumor. A sample simulation of the progression of a vascularized tumor shown in Fig. 3 illustrates the development of a complex vascular network, the heterogenous de- Acknowledgements The authors thank Hermann Frieboes livery of oxygen, and the morphological instability of for the valuable discussions. The authors are grateful for the partial funding from the National Science Foundation, the growing tumor.

72 Division of Mathematical Sciences, and the National Institutes of Health through grants NIH-1RC2CA148493-01 and NIHP50GM76516 for a National Center of Excellence in Systems Biology at UCI.

Applications to Real Size Biological Systems

Applications to Real Size Biological Systems

Christophe Chipot Laboratoire International Associ´e CNRS, UMR 7565, Universit´e de Lorraine, Vandœuvre-l`es-Nancy, France References Theoretical and Computational Biophysics Group, Beckman Institute for Advanced Science and 1. Anderson, A.R.A., Chaplain, M.A.J.: Continuous and disTechnology, University of Illinois at crete mathematical model of tumour-induced angiogenesis. Urbana-Champaign, Urbana-Champaign, IL, USA Bull. Math. Biol. 60, 857–899 (1998) 2. Bentley, K., Gerhardt, H., Bates, P.A.: Agent-based simulation of notch-mediated tip cell selection in angiogenic sprout initialisation. J. Theor. Biol. 250, 25–36 (2008) 3. Cristini, V., Lowengrub, J., Nie, Q.: Nonlinear simulation of tumor growth. J. Math. Biol. 46, 191–224 (2003) 4. Figg, W.D., Folkman, J.: Angiogenesis: An Integrative Approach from Science to Medicine. Springer, New York (2008) 5. Gerhardt, H., Golding, M., Fruttiger, M.: Vegf guides angiogenic sprouting utilizing endothelial tip cell filopodia. J. Cell Biol. 161, 1163–1177 (2003) 6. Godde, v., Kurz, H.: Structural and biophysical simulation of angiogenesis and vascular remodeling. Dev. Dyn. 220, 387–401 (2001) 7. Levine, H.A., Sleeman, B.D., Nilsen-Hamilton, M.: Mathematical modeling of the onset of capillary formation initiating angiogenesis. Math. Biol. 42, 195–238 (2001) 8. Lowengrub, J.S., Frieboes, H.B, Jin, F., Chuang, Y.-L., Li, X., Macklin, P., Wise, S.M., Cristini, V.: Nonlinear modeling of cancer: bridging the gap between cells and tumors. Nonlinearity 23, R1–R91 (2010) 9. Macklin, P., McDougall, S., Anderson, A.R.A., Chaplain, M.A.J., Cristini, V., Lowengrub, J.: Multiscale modelling and nonlinear simulation of vascular tumour growth. J. Math. Biol. 58, 765–798 (2009) 10. McDougall, S.R., Anderson, A.R.A., Chaplain, M.A.J.: Mathematical modelling of dynamic adaptive tumourinduced angiogenesis: clinical implications and therapeutic targeting strategies. J. Theor. Biol. 241, 564–589 (2006) 11. Owen, M.R., Alarc´on, T., Maini, P.K., Byrne, H.M.: Angiogenesis and vascular remodelling in normal and cancerous tissues. J. Math. Biol. 58, 689–721 (2009) 12. Pries, A.R., Secomb, T.W., Gaehtgens, P.: Structural adaptation and stability of microvascular networks: theory and simulations. Am. J. Physiol. Heart Circ. Physiol. 275, H349–H360 (1998) 13. Risau, W.: Mechanisms of angiogenesis. Nature 386 (1997) 14. Sinek, J.P., Sanga, S., Zheng, X. Frieboes, H.B., Ferrari, M., Cristini, V.: Predicting drug pharmacokinetics and effect in vascularized tumors using computer simulation. Math. Biol. 58, 485–510 (2009) 15. Welter, M., Barthab, K., Riegera, H.: Vascular remodelling of an arterio-venous blood vessel network during solid tumour growth. J. Theor. Biol. 259, 405–422 (2009) 16. Zheng, X., Wise, S.M., Cristini, V.: Nonlinear simulation of tumor necrosis, neo-vascularization and tissue invasion via an adaptive finite-element/level-set method. Bull. Math. Biol. 67, 211–259 (2005)

Mathematics Subject Classification 65Y05; 68U20; 70F10; 81V55; 82-08; 82B05; 92-08

Synonyms High-Performance Computer Simulations of Molecular Assemblies of Biological Interest

Short Definition Arguably enough, structural biology and biophysics represent the greatest challenge for molecular dynamics, owing to the size of the biological objects of interest and the time scales spanned by processes of the cell machinery wherein they are involved. Here, molecular dynamics (MD) simulations are discussed from a biological perspective, emphasizing how the endeavor to model increasingly larger molecular assemblies over physiologically relevant times has shaped the field. This entry shows how the race to dilate the spatial and temporal scales has greatly benefitted from groundbreaking advances on the hardware, computational front, as well as on the algorithmic front. The current trends in the field, boosted by cutting-edge achievements, provide the basis for a prospective outlook into the future of biologically oriented numerical simulations.

Description Grasping the function of sizable molecular objects, like those of the cell machinery, requires at its core the

Applications to Real Size Biological Systems

knowledge of not only the structural aspects of these organized systems but also their dynamical signature. Yet, in many circumstances, the intrinsic limitations of conventional experimental techniques thwart access to the microscopic detail of these complex molecular constructs. The so-called computer revolution that began some 40 years ago considerably modified the perspectives, enabling their investigation by means of numerical simulations that rely upon first principles – this constitutes the central idea of the computational microscope [22], a concept coined by Klaus Schulten, and subsequently borrowed by others [11], of an emerging instrument for cell biology at atomic resolution. In reality, as early as the end of the 1950s, Alder and Wainwright [1] had anticipated that such computational experiments, initially performed on small model systems, in particular on a collection of hard spheres, could constitute a bridge between macroscopic experimental observations and its microscopic counterpart. This tangible link between two distinct size scales requires a periodic spatial replication of the simulated sample, thus emancipated of undesirable, spurious edge effects. Ideally, the complete study of any molecular assembly would necessitate that the time-dependent Schr¨odinger equation be solved. In practice, however, the interest is focused primarily on the trajectory of the nuclei, which can be generated employing the classical equations of motion by virtue of the Born-Oppenheimer approximation. Ten years after the first MD simulation of Alder and Wainwright, the French physicist Loup Verlet [36] put forth a numerical integration scheme of the Newtonian equations, alongside with an algorithm for the generation and the bookkeeping of pair lists of neighboring atoms, which facilitates the computation of interatomic interactions – both are still utilized nowadays under various guises (see entry  Sampling Techniques for Computational Statistical Physics). Cornerstone of molecular-mechanics simulations, the potential energy function is minimalist and limited in most cases to simple harmonic terms and trigonometric series to describe the geometric deformation of the molecule, and to the combination of Coulombic and Lennard-Jones potentials for computing the interaction of atoms that are not bonded chemically [6]. The underlying idea of a rudimentary force field is to dilate the time scales by reducing to the bare minimum the cost incurred in the evaluation of the energy at each

73

time step – this is certainly true for atomic force fields; this is even more so for the so-called coarse-grained approaches. Keeping this idea in mind is of paramount importance when one ambitions to reach the microand possibly the millisecond time scale (Fig. 1). Whether in the context of thermodynamic equilibrium or of nonequilibrium, numerous developments have boosted molecular dynamics to the status of a robust theoretical tool, henceforth an unavoidable complement to a large range of experimental methods. The reader is referred to entries  Sampling Techniques for Computational Statistical Physics and  LargeScale Computing for Molecular Dynamics Simulation, which outline the algorithms that arguably represent milestones in the history of numerical simulations. The true revolution in the race for expanding time and size scales remains, however, that which accompanied the advent of parallel architectures and spatialdecomposition algorithms [6], reducing linearly the computer time with the number of available processors (see entry  Large-Scale Computing for Molecular Dynamics Simulation). This unbridled race for the longest simulation or that of the largest molecular construct has somewhat eclipsed the considerable amount of work invested in the enhanced representation of interatomic forces, notably through the introduction of polarizabilities or distributed multipoles, or possibly both – increasing accordingly the cost of the computation, the inevitable ransom of a greater precision. It also outshined the tremendous effort spent in characterizing the error associated to the numerical integration of the equations of motion [13], a concept generally ignored in simulations nearing the current limits of molecular dynamics, either of time or of size nature. MD Simulations of Biological Systems: A Computational Challenge MD simulations require a discrete integration of the classical equations of motion with a time step limited by the fastest degrees of freedom of the molecular assembly – in fully atomistic descriptions of biological systems, the vibration of those chemical bonds involving hydrogen atoms imposes increments on the order of 1 fs to guarantee energy conservation (see entry  Molecular Geometry Optimization: Algorithms). Millions to billions of integration steps are, therefore, necessary to access biologically relevant time scales. As has been alluded to in the introduction, it is crucial that the cost incurred by an energy evaluation be as

A

74

Applications to Real Size Biological Systems

Applications to Real Size Biological Systems, Fig. 1 Inversely related time and size scales currently amenable to classical molecular dynamics. Up-to-date massively parallel, possibly dedicated architectures combined with scalable MD programs open the way to new frontiers in the exploration of biological systems. From left to right – water, the fundamental ingredient of the cell machinery; the bovine pancreatic trypsin inhibitor, an enzyme, the breathing of which can be monitored on the millisecond time scale; the ˇ2 -adrenergic receptor, a G protein-coupled receptor, target of adrenaline, involved in vasoconstriction and vasodilation processes; the ribosome, the

intricate biological device that decodes the genetic material and produces proteins; a 64-million atom model of the human immunodeficiency virus-1 (HIV-1) capsid formed by about 1,300 proteins. By combining cryo-electron-microscopy data with molecular dynamics simulation, a full atomic-resolution structure of the capsid was obtained, which is currently the largest entry of the protein data bank. Shown here are the hexameric (gold) and pentameric (green) assembly units of the HIV-1 capsid. For clarity, the aqueous or the membrane environment of the latter biological objects has been omitted

low as possible. On the other hand, in the framework of pairwise additive potentials, the theoretical computational effort varies as N 2 , the square of the number of particles forming the molecular construct. In practice [33], using a spherical truncation for the shortrange component of electrostatic interactions and a discretization scheme to solve the Poisson equation and, thus, determine their long-range contribution in the reciprocal space, the actual cost reduces to N log N . Concomitant with the emergence of parallel computer architectures, the need of greater computational efficiency to tackle large biological systems prompted the alteration of serial, possibly vectorial MD codes and the development of novel strategies better suited to the recent advances on the hardware front. This impetus can be traced back as early as the late 1980s, with, among other pioneering endeavors, the construction of a network of transputers and the writing of one of the first scalable MD programs, EGO [17]. As novel, shared-memory and nonuniformmemory-access architectures became increasingly available to the modeling community, additional effort was invested in harnessing the unprecedented computational power now at hand. The reader is referred to entry  Large-Scale Computing for

Molecular Dynamics Simulation for further detail on parallelization paradigms, chief among which spatial-decomposition schemes form the bedrock of such popular scalable MD codes as NAMD [28], LAMMPS [29], Desmond [4], and GROMACS [20]. From the perspective of numerical computing, the millions of idle personal computers disseminated in households worldwide represent a formidable computational resource, which could help address important challenges in science and technology. Under these premises, the Space Sciences Laboratory at the University of California, Berkeley, realized that if the computer cycles burnt by flying toasters and other animated screen savers were to be channeled to scientific computing, a task that would require hundreds of years to complete could be executed within a handful of hours or days. The successful distributed computing effort incepted by the Search for ExtraTerrestrial Intelligence (SETI) program in 1999, SETI@home, relies in large measure upon the latter assumption. Following the seminal idea of SETI@home, Graham Richards [30] proposed in 2001 to utilize dormant computers for drug discovery in an endeavor coined Screensaver Lifesaver, which applies grid computing over 3.5 million personal computers to find new leads

Applications to Real Size Biological Systems

of cancer-fighting potential. Extension to molecular dynamics of these early experiments of distributed computing forms the conceptual basis for such popular ventures as folding@home endowed with a worldwide array of personal computers donating computing cycles to tackle challenging problems of biological relevance, like ab initio protein folding [32]. Some 20 years down the road after the inception of massively parallel MD programs targeted at large arrays of central processing units (CPUs), numerical simulations are, yet again, the theater of another revolution triggered by the advent of general-purpose graphics processing units (GPUs). The latter have become inexpensive multiple-core, generic processors capable of handling floating-point operations in parallel, and produced at a reduced cost as a consequence of their massive use in consumer electronics, notably in videogaming consoles. While blockbuster programs like NAMD [28] and GROMACS [20] have been rapidly adapted to hybrid, CPU/GPU architectures, novel MD codes like HOOMD-blue [2] are being designed for graphics cards only. Since the pioneering simulations carried out on an array of transputers [17], new attempts to contrive a dedicated supercomputer consisting of specialized processors for molecular dynamics are put forth. The boldest and probably most successful endeavor to this date is that of D. E. Shaw Research and the development of the Anton supercomputer [31] in their Manhattan facilities. In its first inception, Anton featured 128 processors, sufficient to produce a trajectory of 17 s for a hydrated protein consisting of about 24,000 atoms. In its subsequent version harboring 512 processors, Anton can tackle significantly larger molecular assemblies, like membrane proteins. It is fair to recognize that the introduction of this new generation of dedicated supercomputers has cornered the competition of brute-force simulations by accessing through unprecedented performance time scales hitherto never attained. A crucial aspect of MD simulations is the analysis of the trajectories, from whence important conclusions on the function of the biological system can be drawn. Generating configurations at an unbridled pace over increasingly longer time scales for continuously growing molecular assemblies raises a heretofore unsuspected problem – the storage of a massive amount of data and their a posteriori treatment [14]. As an illustration, the disk space necessary to store on a picosecond basis the

75

Cartesian coordinates of a 1-ms simulation of a small protein immersed in a bath of water – representing a total of about 10,000 atoms, would roughly amount to 240 TB. An equivalent disk space would be required to store the velocities of the system at the same frequency. A practical option to circumvent this difficulty consists in performing on the fly a predefined series of analyses, which evidently implies that the trajectory ought to be regenerated, should a different set of analyses be needed. More than the computational speed, storage and handling of gigantic collections of data have become the rate-limiting step of MD simulations of large biological objects over physiologically relevant time scales.

Brute-Force Molecular Dynamics to Address Biological Complexity Advances on both the hardware and the software fronts have opened the way to the unbiased, realistic description through time and at atomic resolution of complex molecular assemblies. In practice, novel computer architectures endowed with large arrays of processors have virtually abolished the obstacle that the size of the biological objects amenable to molecular dynamics represents. Over the past decade, access to appreciably larger, faster computers has pushed back the sizescale limit to a few millions atoms, thereby allowing such intricate elements of the living world as the ribosome or the capsid of a virus to be modeled. The size scale is quickly expanding to about 100 million atoms, notably with the unprecedented simulation of a chromatophore [26] – a pseudo-organelle harboring the photosynthetic apparatus of certain bacteria and one of the most intricate biological systems ever investigated by means of numerical simulations, or the capsid of the human immunodeficiency virus-1 (HIV-1) formed by about 1,300 proteins [38]. From the perspective of large molecular constructs, the time scale remains a major methodological and technological lock-in. Current MD simulations span the micro- and, under favor circumstances, the millisecond time scale. Though the number of atoms forming the biological object and the accessible time scale are inversely related, recent computing platforms have made size scales of tens to hundreds of thousand atoms and time scales on the order of 100 s concomitantly attainable – an unprecedented overlap of scales compatible with the biological reality.

A

76

Among the many successes gleaned from numerical simulations, deciphering from first-principle key events in the folding pathway of small proteins undoubtedly represents the largest step toward a longstanding holy grail of modern structural biology [24]. While brute-force molecular simulations have proven suitable to fast-folders self-organizing into simple secondary-structure motifs, like the triple ˛-helix pattern of the renowned villin headpiece [14], they have also pinpointed noteworthy shortcomings in common, multipurpose macromolecular force fields. Of particular interest are the so-called ˇ-sheet proteins, the final fold of which results from the lateral association of ˇ-strand by means of a network of hydrogen bonds. Central to the formation of this network are the directionality and the strength of the hydrogen bonds, two facets of minimalist potential energy functions that are often believed to be inaccurately parameterized. Tweaking and finetuning the force field has helped overcome the barrier raised by the inherently approximate and, hence, incomplete nature of the latter, albeit at the risk of perturbing what appears to be a subtle, intertwined construct, likely to result in undesirable butterfly effects (Fig. 2). Membrane transport is yet another area of structural biology that has greatly benefited from advances in MD simulations. Whereas experiment by and large only offers a fragmentary, static view of the proteins responsible for conveying chemical species across the cell membrane, numerical simulations provide a detailed, atomic-level description of the complete transport pathway. Repeated brute-force, unbiased simulations of the spontaneous binding of adenosine diphosphate to the mitochondrial ADP/ATP carrier [9] – the commonly accepted preamble to the conformational change of the latter, have supplied a consistent picture of the association pathway followed by the nucleotide toward its host protein. They have also revealed how the electrostatic signature of the internal cavity of the carrier funnels the substrate to the binding site [9] and how subtle, single-point mutations could impair the function of the protein, leading in turn to a variety of severe pathologies. Expanding their time scale to the microsecond range, MD simulations have, among others, shed light on the key events that underlie the closure of the voltage-gated potassium channel Kv1.2 [21]. They have also offered a detailed view of the conformations

Applications to Real Size Biological Systems

Applications to Real Size Biological Systems, Fig. 2 Typical time scales spanned by biological processes of the cell machinery. From top to bottom – translational and orientational relaxation of water on the picosecond time scale, that is, about three orders of magnitude slower than bond vibrations in the molecule; spontaneous binding of zanamivir or Relenza, an inhibitor of neuraminidase in the A/H1N1 virus; folding of the villin headpiece subdomain, a fast-folding protein formed by 35 amino acids; conformational transition in the mitochondrial transporter of adenosine di- and triphosphate. Binding of the nucleotide to the internal cavity of the membrane protein is believed to be the antechamber to the transition [9], wherein the carrier closes on one side of the mitochondrial membrane, as it opens to the other side. For clarity, the aqueous or the membrane environment of the latter biological objects has been omitted. An alternative to brute-force MD simulations, which still cannot reconcile the biological time and size scales, consists in addressing the collectivity of the process at hand through a selection of relevant order parameters, or collective variables

that are essential for the opening and closure of the bacterial Gloeobacter violaceus pentameric pH-gated ion, or GLIC channel involved in signal-transduction processes [27]. Closely related to the spontaneous binding assays in the mitochondrial carrier, extensive, microsecondtime-scale simulations have been performed to map the association pathway delineating the formation of

Applications to Real Size Biological Systems

protein/ligand complexes. Among the different examples published recently, two studies [5, 12] provide a cogent illustration of the current limitations of MD and of what one can expect to extract from brute-force simulations. Central to drug discovery is the question of how a ligand binds to its target protein. The first study [12] delves into the mechanism whereby known therapeutic agents associate with the ˇ1 – and ˇ2 –adrenergic receptors, two membrane proteins pertaining to the family of G protein-coupled receptors, a class of proteins involved in a wide variety of physiological processes. While long trajectories undoubtedly help dissect how binding proceeds from the recognition of the drug by the protein, its entry into the internal cavity, to the moment it is eventually locked in place, they also provide little information about the thermodynamics of the phenomenon. In sharp contrast, the complete sequence of events that describe the paradigmatic benzamidine to trypsin binding process was reconstructed by generating numerous 100-ns trajectories, about one third of which were reactive, leading to the native enzyme/inhibitor complex [5]. On the basis of this collection of simulations, not only thermodynamic but also kinetic data were inferred, quantifying with an appreciable precision an otherwise qualitative picture of the binding process. The reader is referred to entry  Calculation of Ensemble Averages for an overview of the methods aimed at the determination of averages from numerical simulations. That definitive conclusions cannot be drawn from a single observation justifies the generation of an ensemble of trajectories, from whence general trends can be inferred. In the inexorable race for longer MD simulations, it ought to be reminded that it is sometimes preferable to have access to n trajectories of length l rather than to a single trajectory of length nl. The former scenario supports a less questionable, more detailed view of the slow processes at play while offering a thermodynamic basis for quantifying it. Yet, should the purpose of the ensemble of trajectories be the determination of thermodynamic, possibly kinetic observables, brute-force simulations may not be the bestsuited route and alternate approaches, either of perturbative nature or relying upon collective variables [19], ought to be preferred (see entry  Computation of Free Energy Differences).

77

Preferential Sampling and the Simulation of Slow Processes Over the past 30 years, an assortment of methods has been devised to enhance, or possibly accelerate, sampling in numerical simulations [7,23]. These methods can be roughly divided into two main classes, the first of which addresses the issue of slow processes by acting on the entire molecular assembly with the objective to accelerate sampling of its lowenergy regions. In the second class of methods, coined importance-sampling methods [7, 23], biases are applied to a selection of order parameters, or collective variables, to improve sampling in the important regions, relevant to the slow process of interest, and at the expense of other regions of configurational space. Detail of these two classes of methods can be found in entry  Computation of Free Energy Differences. Success of collective-variable-based methods depends to a large extent upon the validity of the underlying hypothesis, which supposes a time-scale separation of the slow degrees of freedom, in connection with the reaction coordinates, and all other, hard, fast degrees of freedom. In other words, the modeler is left with the complicated task of either selecting a few relevant order parameters, or including a large number of variables to describe the transition space. The key here is intuition and the astute choice of an appropriate reaction-coordinate model, appreciably close to the true committor function. Intuition may, however, turn out to be insufficient. In the prototypical example of a wide-enough channel lined with amino moieties, translocation of negative ions is intuitively described by a single order parameter – the distance separating the anion from the center of mass of the channel projected onto the long axis of the latter. Though intuitive, this model of the reaction coordinate is ineffective, as the negative charges are likely to graze the wall of the pore and bind tightly to the amino groups, thereby impeding longitudinal diffusion. The epithet ineffective ought to be understood here as prone to yield markedly uneven sampling along the chosen order parameter – a manifestation of quasi-non-ergodicity, which remains of utmost concern for finite-length simulations. Under such circumstances, the common remedies consist in increasing the dimensionality and, hence, the collectivity of the model reaction coordinate, and forcibly sampling the slow degrees of freedom, which act as

A

78

barriers orthogonal to the chosen order parameter. Spawning replicas that will explore the reaction coordinate in parallel valleys represents a possible route to achieve the latter option. The reader is referred to entry  Computation of Free Energy Differences for additional detail. Merging intuition and practical considerations, in particular cost-effectiveness, explains why the vast majority of preferential-sampling computations resort to the crudest, one-dimensional representation of the committor function. Under a number of circumstances, however, it might be desirable to augment the dimensionality of the model reaction coordinate. As early as 2001, for instance, employing a stratification version of umbrella sampling, Bern`eche and Roux [3] tackled the mechanism of potassium-ion conduction in the voltage-gated channel KcsA. Their free-energy calculations brought to light two prevailing states, wherein either two or three potassium ions occupy the selectivity filter. The highest free-energy barrier toward ion permeation was estimated to be on the order of 2–3 kcal/mol, suggesting that conduction is limited by self-diffusion in the channel. Over 10 years later, it was cogently demonstrated in a proof of concept involving a simple peptide that multidimensionality represents a compelling, albeit costly, option to lift conformational degeneracy and discriminate between key states of the freeenergy landscape [19]. In an adaptive-biasing-force calculation [8, 18] of the folding and diffusion of a nascent-protein-chain model, this idea was applied to describe by means of highly collective variables two processes that are inherently concomitant [15]. Armed with the appropriate toolkit of order parameters with the desired degree of collectivity [28], modeling the complex, concerted movements of elements of the cell machinery is now within reach. Not too surprisingly, the current trend is to model collective motions and possibly quantify thermodynamically the latter – even though the order parameter utilized is of rather low collectivity – a general tendency expected to pervade in the coming years with increasingly more sophisticated biological objects. The investigation of the free-energy cost of translocon-assisted insertion of membrane proteins [16] cogently illustrates how the choice of the method impacts our perception of the problem at hand and how it ought to be addressed. To understand why

Applications to Real Size Biological Systems

experiment and theory supply for a variety of amino acids appreciably discrepant free energies of insertion – for instance, membrane insertion of arginine requires 14–17 kcal/mol [10] according to MD simulations, but only 2–3 kcal/mol according to experiment – a two-stage mechanism invoking the translocon, an integral membrane protein that conveys the nascent peptide chain as it is produced by the ribosome, has been put forth. In an attempt to reconcile theoretical and experimental estimates, the thermodynamics of translocating from water to the hydrophobic core of a lipid bilayer an arginine residue borne by an integral, pseudo-infinite poly-leucine ˛-helix, was measured, yielding a net free-energy change of about 17 kcal/mol [10]. The latter calculation raises two issues of concern – a conceptual one and a fundamental one. Conceptually, since the focus is primarily on the end points of the free-energy profile delineating translocation, a tedious, poorly converging potential-of-mean-force calculation is unwarranted and ought to be replaced by a perturbative one [7] (see entry  Computation of Free Energy Differences). Fundamentally, aside from the potential function of the translocon, the translocation process ignores the role of the background ˛-helix, the contribution of which should be measured in a complete thermodynamic cycle. Doing so, it is found that the translocon not only reduces the cost incurred by charged amino-acid translocation but also reduces the gain of hydrophobic-amino-acid insertion [16]. Not a preferential-sampling method per se, MD flexible fitting [34] is a natural extension of molecular dynamics to reconcile crystallographic structures with their in vivo conformation usually observed at low resolution with electron microscopy. The algorithm incorporates the map supplied by the latter as an external potential defined on a three-dimensional grid, ensuring that high-density regions correspond to energy minima. Atoms of the biological object of interest, thus, undergo forces that are proportional to the gradient of the electron-microscopy map. This method has recently emerged as popular, promising complement to cryoelectron-microscopy experiments fueled by a series of success stories, which began with atomic models of the Escherichia coli ribosome [37] in different functional states imaged at various resolutions by means of cryoelectron microscopy.

Applications to Real Size Biological Systems

A Turning Point in Structural Biology With 55 years of hindsight since the pioneering simulation of Alder and Wainwright [1], it is rather legitimate to wonder about the future of molecular dynamics – which new, uncharted frontiers can we reasonably expect to attain within the coming years? What the recent, avant-garde simulations, either brute-force, unbiased, or sampling selected collective variables, have taught us is that nothing is set in stone forever. What lies today at the bleeding edge will undoubtedly become obsolete tomorrow. The future of MD simulations and, more generally, of numerical simulations is intimately connected to that of computer architectures and the underlying technological challenges that their improvement represents. It still remains that between the first simulation of a protein, the bovine pancreatic trypsin inhibitor, by McCammon et al. [25], and the recent simulations of protein NTL9 over a handful of milliseconds [24], the time scale has dilated by a factor of a billion within a matter of less than 35 years. Notwithstanding its duration of 8 ps, which one might find anecdotal today in view of the current standards banning subnanosecond investigations, the former simulation [25] did represent a turning point in the field of computational structural biology by establishing an everlasting connection between theoretical and computational chemistry and biology. Conversely, the latter simulation [24] demonstrates the feasibility of in silico folding of complex tertiary structures by dissecting each and every step of the folding pathway. Collateral effect of an artifactual behavior of the force fields or biological reality, micro- and millisecond simulations suggest that proteins breathe, unfolding and refolding unceasingly. While this result per se is not revolutionary, it, nonetheless, paves the way to the ab initio prediction of the three-dimensional structure of proteins and conceivably protein assemblies – an endeavor that is still today subservient to possible inaccuracies of the potential energy function, but will be associated tomorrow to a more rigorous description of interatomic interactions. The 58 residues of the bovine pancreatic trypsin inhibitor simulated for the first time in 1977 [25] represent less than a 1,000 atoms. Also less than 35 years later, with its first simulation of a chromatophore, the research group of Klaus Schulten has expanded

79

the size scale amenable to brute-force molecular dynamics by a factor of 100,000. Some might question, beyond the unprecedented technological prowess, the true biological relevance of a computation limited to a few nanoseconds for such a complex molecular construction – what take-home message can we possibly draw on the basis of thermal fluctuations around a structure presumably at thermodynamic equilibrium? However questionable to some, this heroic effort ought to be viewed beyond a mere computational performance, heralding the forthcoming accessibility to the atomistic description of complete organelles by means of numerical simulations. It is, however, interesting to note that over an identical period of time, time and size scales have not evolved similarly. Even with a modest number of processors, tackling the micro- and, armed with patience, the millisecond time scale has been feasible for several years, assuming a sufficiently small molecular assembly. Yet, only with the emergence of massively parallel architectures, in particular the petascale supercomputer Blue Waters, can large molecular assemblies of several millions of atoms be tackled. Molecular dynamics on the millisecond time scale or on the million-atom size scale still constitutes at this stage a formidable technological challenge, which, nonetheless, reveals a clear tendency for the years to come. As the first petaflop supercomputers become operational, the scientific community speculates on the forthcoming exascale machines and the nature of the molecular systems these novel architectures will be capable of handling. Assuming that in the forthcoming decades the evolution of molecular dynamics will closely follow the trend imparted by Moore’s law, Wilfred van Gunsteren [35] optimistically predicts that in about 20 years, simulating on the nanosecond time scale a complete bacteria, Escherichia coli, will be within reach, and a full mammal cell some 20 years further down the road. However encouraging, these extrapolations based on current performances on the hardware and software fronts bring to light an appreciable gap between time and size scales, which will be evidently difficult to fill, hence, suggesting that there is still a long way to go before brute-force, unbiased simulations of biological macro-objects are able to supply a chronologically and dynamically relevant information.

A

80

While there is a consensus that the potential role of molecular dynamics in structural biology was rather poorly prognosticated by the most pessimistic biologists – then and to a certain extent still now, bridging the gap between experimentally observed macroscopic and numerically simulated microscopic objects admittedly remains out of reach. In lieu of bleeding-edge brute-force, unbiased molecular dynamics, perhaps should alternate, preferential-sampling approaches be favored, capable of reconciling time and size scales, provided that the key degrees of freedom of the biological process at play are well identified. Perhaps is the massive increase of the computational resources envisioned in the coming years also an opportunity for a top-down revision of minimalist macromolecular force fields – are less approximate and, hence, more general and more reliable potential energy functions conceivable, even if this implies revising our ambitions in terms of time and size scales? Or is the discovery of new, uncharted frontiers in spatial and temporal scales, at the expense of accuracy, our ultimate objective? Acknowledgements Image of the HIV virus capsid courtesy of Juan R. Perilla and Klaus J. Schulten, Theoretical and Computational Biophysics Group, University of Illinois UrbanaChampaign.

References 1. Alder, B.J., Wainwright, T.E.: Phase transition for a hard sphere systems. J. Chem. Phys. 27, 1208–1209 (1957) 2. Anderson, J.A., Lorenz, C.D., Travesset, A.: General purpose molecular dynamics simulations fully implemented on graphics processing units. J. Comput. Phys. 227(10), 5342– 5359 (2008) 3. Bern`eche, S., Roux, B.: Energetics of ion conduction through the KC channel. Nature 414(6859), 73–77 (2001) 4. Bowers, K.J., Chow, E., Xu, H., Dror, R.O., Eastwood, M.P., Gregersen, B.A., Klepeis, J.L., Kolossvary, I., Moraes, M.A., Sacerdoti, F.D., Salmon, J.K., Shan, Y., Shaw, D.E.: Scalable algorithms for molecular dynamics simulations on commodity clusters. In: Proceedings of the ACM/IEEE SC 2006 Conference, Tampa (2006) 5. Buch, I., Giorgino, T., Fabritiis, G.D.: Complete reconstruction of an enzyme-inhibitor binding process by molecular dynamics simulations. Proc. Natl. Acad. Sci. USA 108, 10184–10189 (2011)

Applications to Real Size Biological Systems 6. Chipot, C.: Molecular dynamics: observing matter in motion. In: Boisseau, P., Lahmani, M., Houdy, P. (eds.) Nanoscience: Nanobiotechnology and Nanobiology. Springer, Heidelberg/Dordrecht/London/New York (2009) 7. Chipot, C., Pohorille, A. (eds.): Free Energy Calculations: Theory and Applications in Chemistry and Biology. Springer, Berlin/Heidelberg/New York (2007) 8. Darve, E., Pohorille, A.: Calculating free energies using average force. J. Chem. Phys. 115, 9169–9183 (2001) 9. Dehez, F., Pebay-Peyroula, E., Chipot, C.: Binding of adp in the mitochondrial adp/atp carrier is driven by an electrostatic funnel. J. Am. Chem. Soc. 130, 12725–12733 (2008) 10. Dorairaj, S., Allen, T.W.: On the thermodynamic stability of a charged arginine side chain in a transmembrane helix. Proc. Natl. Acad. Sci. USA 104, 4943–4948 (2007) 11. Dror, R.O., Jensen, M.Ø., Borhani, D.W., Shaw, D.E.: Exploring atomic resolution physiology on a femtosecond to millisecond timescale using molecular dynamics simulations. J. Gen. Physiol. 135, 555–562 (2010) 12. Dror, R.O., Pan, A.C., Arlow, D.H., Borhani, D.W., Maragakis, P., Shan, Y., Xu, H., Shaw, D.E.: Pathway and mechanism of drug binding to g-protein-coupled receptors. Proc. Natl. Acad. Sci. USA 108, 13118–13123 (2011) 13. Engle, R.D., Skeel, R.D., Drees, M.: Monitoring energy drift with shadow hamiltonians. J. Comput. Phys. 206, 432–452 (2005) 14. Freddolino, P.L., Harrison, C.B., Liu, Y., Schulten, K.: Challenges in protein folding simulations: timescale, representation, and analysis. Nat. Phys. 6, 751–758 (2010) 15. Gumbart, J.C., Chipot, C., Schulten, K.: Effects of the protein translocon on the free energy of nascent-chain folding. J. Am. Chem. Soc. 133, 7602–7607 (2011) 16. Gumbart, J.C., Chipot, C., Schulten, K.: Free-energy cost for translocon-assisted insertion of membrane proteins. Proc. Natl. Acad. Sci. USA 108, 3596–3601 (2011) 17. Heller, H., Grubm¨uller, H., Schulten, K.: Molecular dynamics simulation on a parallel computer. Mol. Simul. 5, 133–165 (1990) 18. H´enin, J., Chipot, C.: Overcoming free energy barriers using unconstrained molecular dynamics simulations. J. Chem. Phys. 121, 2904–2914 (2004) 19. H´enin, J., Forin, G., Chipot, C., Klein, M.L.: Exploring multidimensional free energy landscapes using time-dependent biases on collective variables. J. Chem. Theor. Comput. 6, 35–47 (2010) 20. Hess, B., Kutzner, C., van der Spoel, D., Lindahl, E.: Gromacs 4: algorithms for highly effcient, load-balanced, and scalable molecular simulation. J. Chem. Theor. Comput. 4, 435–447 (2008) 21. Jensen, M.Ø., Jogini, V., Borhani, D.W., Leffler, A.E., Dror, R.O., Shaw, D.E.: Mechanism of voltage gating in potassium channels. Science 336, 229–233 (2012) 22. Lee, E.H., Hsin, J., Sotomayor, M., Comellas, G., Schulten, K.: Discovery through the computational microscope. Structure 17, 1295–1306 (2009)

Applied Control Theory for Biological Processes 23. Leli`evre, T., Stoltz, G., Rousset, M.: Free Energy Computations: A Mathematical Perspective. Imperial College Press, London/Hackensack (2010) 24. Lindorff-Larsen, K., Piana, S., Dror, R.O., Shaw, D.E.: How fast-folding proteins fold. Science 334, 517–520 (2011) 25. McCammon, J.A., Gelin, B.R., Karplus, M.: Dynamics of folded proteins. Nature 267, 585–590 (1977) 26. Mei, C., Sun, Y., Zheng, G., Bohm, E. J., Kale, L.V., Phillips, J.C., Harrison, C.: Enabling and scaling biomolecular simulations of 100 million atoms on petascale machines with a multicore-optimized message-driven runtime. In Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’11, (New York, NY, USA), pp. 61:1–61:11, ACM, 2011. 27. Nury, H., Poitevin, F., Renterghem, C.V., PChangeux, J., Corringer, P.J., Delarue, M., Baaden, M.: One-microsecond molecular dynamics simulation of channel gating in a nicotinic receptor homologue. Proc. Natl. Acad. Sci. USA 107, 6275–6280 (2010) 28. Phillips, J.C., Braun, R., Wang, W., Gumbart, J., Tajkhorshid, E., Villa, E., Chipot, C., Skeel, R.D., Kal´e, L., Schulten, K.: Scalable molecular dynamics with NAMD . J. Comput. Chem. 26, 1781–1802 (2005) 29. Plimpton, S.: Fast parallel algorithms for short–range molecular dynamics. J. Comput. Phys. 117, 1–19 (1995) 30. Richards, W.G.: Virtual screening using grid computing: the screensaver project. Nat. Rev. Drug Discov. 1, 551–555 (2002) 31. Shaw, D.E., Deneroff, M.M., Dror, R.O., Kuskin, J.S., Larson, R.H., Salmon, J.K., Young, C., Batson, B., Bowers, K.J., Chao, J.C., Eastwood, M.P., Gagliardo, J., Grossman, J.P., Ho, C.R., Ierardi, D.J., Kolossv´ary, I., Klepeis, J.L., Layman, T., McLeavey, C., Moraes, M.A., Mueller, R., Priest, E.C., Shan, Y., Spengler, J., Theobald, M., Towles, B., Wang, S.C.: Anton, a special-purpose machine for molecular dynamics simulation. SIGARCH Comput. Archit. News 35, 1–12 (2007) 32. Shirts, M., Pande, V.: Screen savers of the world unite! Science 290, 1903–1904 (2000) 33. Toukmaji, A.Y., Board, J.A., Jr.: Ewald summation techniques in perspective: a survey. Comput. Phys. Commun. 95, 73–92 (1996) 34. Trabuco, L.G., Villa, E., Mitra, K., Frank, J., Schulten, K.: Flexible fitting of atomic structures into electron microscopy maps using molecular dynamics. Structure 16, 673–683 (2008) 35. van Gunsteren, W.F., Bakowies, D., Baron, R., Chandrasekhar, I., Christen, M., Daura, X., Gee, P., Geerke, D.P., Gl¨attli, A., H¨unenberger, P.H., Kastenholz, M.A., Oostenbrink, C., Schenk, M., Trzesniak, D., van der Vegt, N.F.A., Yu, H.B.: Biomolecular modeling: goals, problems, perspectives. Angew. Chem. Int. Ed. Engl. 45, 4064–4092 (2006) 36. Verlet, L.: Computer “experiments” on classical fluids. i. thermodynamical properties of lennard–jones molecules. Phys. Rev. 159, 98–103 (1967) 37. Villa, E., Sengupta, J., Trabuco, L.G., LeBarron, J., Baxter, W.T., Shaikh, T.R., Grassucci, R.A., Nissen, P.,

81 Ehrenberg, M., Schulten, K., Frank, J.: Ribosome-induced changes in elongation factor tu conformation control gtp hydrolysis. Proc. Natl. Acad. Sci. USA 106, 1063–1068 (2009) 38. Zhao, G., Perilla, J.R., Yufenyuy, E.L., Meng, X., Chen, B., Ning, J., Ahn, J., Gronenborn, A.M., Schulten, K., Aiken, C., Zhang, P.: Mature HIV-1 capsid structure by cryoelectron microscopy and all-atom molecular dynamics. Nature 497, 643–646 (2013)

Applied Control Theory for Biological Processes Sarah L. Noble1 and Ann E. Rundell1;2 1 School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA 2 Weldon School of Biomedical Engineering, Purdue University, West Lafayette, IN, USA

Synonyms Applications; Cellular processes; Model predictive control

Short Definition Controlling biological processes involves unique challenges not encountered in the control of traditionally engineered systems. Real-time continuous feedback is generally not available and realizable control actions are limited; in addition, mathematical models of biological processes are highly uncertain, often simple abstractions of complex biochemical and gene regulatory networks. As a result, there have been minimal efforts to apply control theory at the cellular level. Model predictive control is especially well suited to control cellular processes as it naturally accommodates the slow system sampling and controller update rates necessitated by experimental limitations. This control strategy is also known to be robust to model uncertainties, measurement noise, and output disturbances. The application of model predictive control to manipulate the behavior of cellular processes is an emerging area of research and may help rationalize the design and development of new cell-based therapies.

A

82

Description Over the last half century, cell-based therapies have found increasing use in the design of substitute biologics to treat and cure disease. While organ and tissue transplantation has been a successful method for improving the health and quality of life for tens of thousands of Americans, the number of patients requiring a transplant outpace the donor availability such that the transplant waiting list grows by approximately 300 people every month [8]. One goal of tissue engineering is to produce biologically engineered substitutes for donor-supplied organs and tissue. Of particular importance to the design of substitute biologics are strategies to direct and control cell fate. Currently, most cellbased therapy research approaches the control of cell fate via hypothesis-driven experiments [5] which are generally expensive and exhaustive. A more rational approach will be vital to accelerate the development of these potentially curative therapies, shifting the paradigm from experimentally interrogating cell responses to rationally engineering cell population behavior [5].

Model Predictive Control The inherent complexity of the intracellular signaling events that direct cell function limits the ability of intuition and exploratory experimental approaches to efficiently control cell fate. A control-theoretic, model-based approach is better equipped to handle this complexity by capitalizing on data from measurable cell states to inform quantitative predictions regarding the state of the entire cell system. However, most of the modern theoretical control strategies that have been developed address very different types of problems than those associated with biological systems. Model predictive control (MPC) is well suited for solving control problems relating to cellular processes as it can be applied to uncertain systems with infrequent sampling and explicit consideration of constraints. In general, MPC determines the optimal input sequence for a linear system [1]. However, because mathematical models of cellular processes are rarely linear, nonlinear MPC (NMPC) is frequently employed. With NMPC, the nonlinear system model can be directly utilized in the control input calculation. The general NMPC problem can be formulated as a constrained optimization problem (see (1)), with the mathematical

Applied Control Theory for Biological Processes

model of the process dx=dt D f .x .t/ ; u .t/ I p/ used to predict the plant behavior over a finite prediction horizon; the controller action, u .t/, is selected so that the difference between the predicted system trajectory and the desired (reference) trajectory is minimized subject to constraints on the controller and state dynamics. min '.x.t /; u.t /; p/subject to: u.t/

dx dt

g.x.t /; u.t /I p/ D 0 h.x.t /; u.t /I p/ < 0

D f .x.t /; u.t /I p/

uL  u.t /  uH

(1)

xL  x.t /  xH

where x 2 0, then the method

a3 D 1  2.a0 C a1 C a2 /; b1 D b6 D 0:209515106613362; b2 D b5 D 0:143851773179818; b3 D b4 D

1  .b1 C b2 /: 2

(10)

Another high-order composition, which yields good methods of orders 8 and 10, is

'a1 h ı    ı 'am h ı    ı 'a1 h

(11)

 1 where 'h is any time-symmetric method. See [17] ın ın '˛h 'ˇh '˛h ; ˛ D 2n  .2n/1=.2kC1/ ; ˇ D 12n˛ for the best-known high-order methods of this type. (7) Methods of orders 4, 6, 8, and 10 require at least 3, is time symmetric and has order 2k C 2. This yields 7, 15, and 31 's. These have the further advantage that methods of order 4 containing 3 applications of Eq. (6) they can be used with any time-symmetric method 'h , when n D 1, of order 6 containing 9 applications not just (6). Examples are (i) the midpoint rule xk 7! of Eq. (6) when n D 2, and so on [5, 18, 21]. These xkC1 D xk C hf ..xk C xkC1 /=2/, which, although no are not the most accurate high-order methods known, longer explicit, does preserve any constant symplectic although they are simple to implement, and the fourth- or Poisson structure and any linear symmetries, is time reversible with respect to any linear reversing order method with n D 2 and k D 1 is satisfactory. There are good methods of orders 4 and 6 of symmetry, and is unconditionally linearly stable, none type (2). For example, a good fourth-order method [3] of which are true for (6); (ii) the partitioned symplectic Runge–Kutta method .qk ; pk / 7! .qkC1 ; pkC1 / dehas m D 6 and fined by d1 D c6 D 0:0792036964311957; d2 D c5 D 0:2228614958676077; d3 D c4 D 0:3246481886897062; d4 D c3 D 0:1096884778767498;

pkC1=2 D pk  h2 H  q .qk ; pkC1=2 / qkC1 D qk C h2 Hp .qk ; pkC1=2  / CHp .qkC1 ; pkC1=2 / pkC1 D pk  h2 Hq .qkC1 ; pkC1=2 /;

d5 D c2 D 0:3667132690474257; d6 D c1 D 0:1303114101821663:

(8) which is symplectic for canonical Hamiltonian systems and can be cheaper than the midpoint rule (and reduces In the case of a 2-term splitting f D f1 C f2 , (2) to leapfrog when H.q; p/ D T .p/ C V .q/); and becomes (iii) the RATTLE method for Hamiltonian systems with holonomic (position) constraints [12]. (9) eam hf1 ı ebm hf2 ı    ı ea1 hf1 ı eb1 hf2 ı ea0 hf1 : The disadvantage that order greater than 2 requires negative time steps can be overcome if one allows comand the parameters (8) become plex coefficients in (2) [4]. Method (7) with n D k D 1 and complex cube root is an example. Composition methods of order up to 14 whose coefficients have posa0 D a6 D 0:0792036964311957; itive real part are found in [6], and these can be stable a1 D a5 D 0:353172906049774; on dissipative systems such as (semidiscretizations of) a2 D a4 D 0:0420650803577195; parabolic PDEs.

C

240

Composition Methods

Correctors The use of a “corrector” (also known as processing or effective order) [1, 20] can reduce the local error still further. Suppose the method ' can be factored as ' D ı

ı 1 :

(12)

Then to evaluate n time steps, we have ' ın D ı ın ı 1 , so only the cost of is relevant. Moreover, the corrector does not need to be evaluated exactly; it can be rapidly approximated to any desired accuracy. The maps ' and are conjugated by the map , which can be regarded as a change of coordinates. Many dynamical properties of interest are invariant under changes of coordinates; in this case we can even omit entirely and simply use the method . For example, calculations of Lyapunov exponents, phase space averages, partition functions, existence and periods of periodic orbits, etc., fall into this class. Initial conditions are not invariant under changes of coordinates, so applying is important if one is interested in a particular initial condition, such as one determined experimentally. The order of ' is called the effective order of . Methods of effective order 4, 6, 8, and 10, with given in (11), require 3, 5, 9, and 15 factors in ; the corrector can greatly reduce the local truncation error at fixed computational work [1].

possibilities. The function T .q; p/ can also include part of the potential, a famous example being the solar system which can be treated by including all sun– planet interactions in T and the (much smaller) planet– planet interactions in V . A striking example is the Takahashi–Imada method 1 2 h V V. which is (1) with modified potential V  24 It has effective order 4 yet uses essentially a singleforce evaluation with a positive time step [19]. A good fourth-order method for simple mechanical systems with no modified potential and no corrector [3] is given by Eq. (9) together with m D 7 and a0 D a7 D 0; a1 D a6 D 0:245298957184271; a2 D a5 D 0:604872665711080; a3 D a4 D

1  .a2 C a3 /; 2

b1 D b7 D 0:0829844064174052; b2 D b6 D 0:396309801498368; b3 D b5 D 0:0390563049223486; b4 D 1  2.b1 C b2 C b3 /:

(13)

Nearly Integrable Systems Simple Mechanical Systems These are canonical Hamiltonian systems with Hamiltonians of the form kinetic plus potential energy, i.e., H D T .q; p/ C V .q/, T D 12 p T N.q/p. If T is integrable, then highly accurate high-order compositions are available, sometimes called Runge–Kutta–Nystr¨om methods. These exploit the fact that (i) many higherorder Poisson brackets of T and V , which would normally contribute to the truncation error, are identically zero, and (ii) the potential V may be explicitly modified to increase the accuracy. Define a product of two functions of q by W  V WD fW; fV; T gg D rW T N rV . A modified potential is a function generated from V by  and linear combinations, i.e., VQ D c0 V Cc1 V  V Cc2 .V  V / V C: : : : The modified force r VQ often (e.g., in N -body problems) costs little more than rV itself. The use (or not) of modified potentials and the use (or not) of a corrector gives many

Composition methods are superb for near-integrable systems with f D f1 C "f2 , for the error is automatically O."/ and vanishes with ". It is also possible to expand the error as a Taylor series in h and " and preferentially eliminate error terms with small powers of ", which is advantageous if " h. This gives, for example, a 2-stage method of order O."2 h4 C "3 h3 /, a 3-stage method of order O."2 h6 C "3 h4 /, and so on [2,9,13]. This idea combines particularly well with the use of correctors. For any composition, even standard leapfrog, for all n there is a corrector that eliminates the O."hp / error terms for all 1 < p < n. Thus, any splitting method is “really” O."2 / accurate on nearintegrable problems. This approach is widely used in solar system studies [10, 20]. Pseudospectral semidiscretization of PDEs such as qR D qxx C f .q/ leads to ODEs of the form qR D Lq C f .q/. Although the linear part qP D p, pP D Lq could be split as in the standard leapfrog, it is also

Composition Methods

241

possible to split the system into linear and nonlinear parts, solving the linear part exactly. This is the approach traditionally used for the nonlinear Schr¨odinger equation. If the nonlinearity is small, then the system is near integrable, and often the highly accurate methods of sections “General Systems”, “Correctors”, and “Simple Mechanical Systems” can be used.

is time symmetric and preserves S to order p C 2 (and has a, b > 0). In contrast, there is a general procedure to preserve reversing symmetries. For a reversing symmetry R of order 2, for any integrator 'h the composition .R ı ' 1 ı R1 / ı ' 1 h 1 h 2

Preserving Symmetries and Reversing Symmetries A diffeomorphism S W M ! M of phase space is a symmetry of the map (or flow) ' if S ı ' D ' ı S. A diffeomorphism RW M ! M is a reversing symmetry of the map ' if Rı' D ' 1 ıR. Symmetries map orbits to orbits, while reversing symmetries map orbits to time-reversed orbits; they both have strong effects on the dynamics of ' and should be preserved where possible. Maps with a given symmetry form a group, so symmetries are preserved by composition. Factors with the required symmetries can sometimes be found by splitting into integrable pieces with all the required symmetries. For canonical Hamiltonian systems, a common case is that of the standard splitting H D T .p/ C V .q/ which respects symmetries that are cotangent lifts of symmetries of the position variables, i.e., S.q; p/ D .g.q/; .Dg.q//1T p/. Linear and affine symmetries are automatically preserved by all Runge–Kutta methods. There is no general procedure to preserve arbitrary groups of diffeomorphisms (such as symplectic structure combined with a given symmetry group.) Despite this, composition can improve the accuracy of symmetry preservation [8]. Let 'h W M ! M be an integrator for xP D f .x/ which has a finite group G of symmetries. Let 'h preserve G to order p. Then the composition Y g ı 'h ı g 1 g2G

preserves G to order p C 1. This composition may be iterated to achieve any desired order. If S is a symmetry of order 2 (i.e., S ı S D id) and 'h is a time-symmetric method preserving S to even order p, then the composition 'ah ıS ı'bhıS ı'ah ;

2aCb D 1; 2a2pC1 b 2pC1 D 0

2

(14)

is R-reversible. If 'h is given by (3) and each fi is reversible, then (14) reduces to (6).

References 1. Blanes, S., Casas, F., Murua, A.: Computational methods for differential equations with processing. SIAM J. Sci. Comput. 27, 1817–1843 (2006) 2. Blanes, S., Casas, F., Ros, J: Processing symplectic methods for near-integrable Hamiltonian systems. Celest. Mech. Dyn. Astr. 77, 17–35 (2000) 3. Blanes, S., Moan, P.C.: Practical symplectic partitioned Runge–Kutta and Runge–Kutta–Nystr¨om methods. J. Comput. Appl. Math. 142, 313–330 (2002) 4. Chambers, J.E.: Symplectic integrators with complex time steps. Astron. J. 126, 1119–1126 (2003) 5. Creutz,M., Gocksch, A.: Higher-order hybrid Monte Carlo algorithms. Phys. Rev. Lett. 63, 9–12 (1989) 6. Hansen, E., Ostermann, A.: High order splitting methods for analytic semigroups exist. BIT 49, 527–542 (2009) 7. Hairer, E., Lubich, C., Wanner, G.: Geometric Numerical Integration: Structure-Preserving Algorithms for Ordinary Differential Equations, 2nd edn. Springer, Berlin (2006) 8. Iserles, A., McLachlan, R.I., Zanna, A.: Approximately preserving symmetries in the numerical integration of ordinary differential equations. Eur. J. Appl. Math. 10, 419–445 (1999) 9. Laskar, J., Robutel, P.: High order symplectic integrators for perturbed Hamiltonian systems. Celest. Mech. 80, 39– 62 (2001) 10. Laskar, J., Robutel, P., Joutel, F., Gastineau, M., Correia, A.C.M., Levrard, B.: A long-term numerical solution for the insolation quantities of the Earth. Astron. Astrophys. 428, 261–285 (2004) 11. Leimkuhler, B., Reich, S.: Simulating Hamiltonian Dynamics. Cambridge University Press, Cambridge (2004) 12. Leimkuhler, B.J., Skeel, R.D.: Symplectic numerical integration in constrained Hamiltonian systems. J. Comput. Phys. 112, 117–125 (1994) 13. McLachlan, R.I.: Composition methods in the presence of small parameters. BIT 35, 258–268 (1995) 14. McLachlan, R.I., Quispel, G.R.W.: Splitting methods. Acta Numer. 11, 341–434 (2002) 15. McLachlan, R.I., Quispel, G.R.W.: Explicit geometric integration of polynomial vector fields. BIT 44, 515–538 (2004) 16. McLachlan, R.I., Quispel, G.R.W.: Geometric integrators for ODEs. J. Phys. A 39, 5251–5285 (2006)

C

242 17. Sofroniou,M., Spaletta, G.: Derivation of symmetric composition constants for symmetric integrators. Optim. Methods Softw. 20, 597–613 (2005) 18. Suzuki, M.: Fractal decomposition of exponential operators with applications to many-body theories and Monte Carlo simulations. Phys. Lett. A 146, 319–323 (1990) 19. Takahashi, M., Imada, M.: Monte Carlo calculations of quantum systems. II. Higher order correction. J. Phys. Soc. Jpn. 53, 3765–3769 (1984) 20. Wisdom, J., Holman, M., Touma, J.: Symplectic correctors. In: Marsden, J.E., Patrick, G.W., Shadwick, W.F. (eds.) Integration Algorithms and Classical Mechanics, pp. 217–244. AMS, Providence (1996) 21. Yoshida, H.: Construction of higher order symplectic integrators. Phys. Lett. A 150, 262–268 (1990)

Compressible Flows Timothy Barth NASA Ames Research Center, Moffett Field, CA, USA

Synonyms Compressible hydrodynamics; Gas dynamic flow

Glossary/Definition Terms Boltzmann transport equation A statistical description of particles in thermodynamic nonequilibrium that exchange momenta and energy via collisions. Caloric equation of state A state equation that describes the temperature dependence of internal energy or heat capacity. Contact wave A linearly degenerate wave family arising in the Riemann problem of gasdynamics and more general solutions of the Euler equations of gasdynamics that is characterized by continuous velocity and pressure fields and piecewise discontinuous density and temperature fields. Chapman-Enskog expansion A perturbed Maxwellian distribution expansion devised [5] and [9] utilizing Knudsen number as a perturbation parameter. Compressible potential A simplification of the Euler equations of gasdynamics for isentropic irrotational

Compressible Flows

flow wherein velocity is represented as the gradient of a potential function. Clausius-Duhem inequality A statement of the second law of thermodynamics that expresses the transport, production due to heat flux, and dissipation of specific entropy. Classical solution A solution that possesses enough differentiability so that all derivatives appearing in the partial differential equation exist pointwise. Entropy A thermodynamic quantity representing the unavailability of energy for conversion into work in a closed system. Euler equations of gasdynamics A continuum system of conservation laws representing the conservation of mass, linear momenta, and energy of a compressible fluid while neglecting the effects of viscosity and heat conduction. Fourier heat conduction A heat conduction model that expresses the heat flux as the product of a material’s heat conductivity and the negated temperature gradient. Homentropic flow A fluid flow that has a uniform and constant entropy. Isentropic flow A fluid flow that has constant entropy associated with particles that may vary from particle to particle. Kinetic description of gases Description of a gas as a large collection of particles. Knudsen number A dimensionless number defined as the ratio of the molecular mean free path length to a representative physical length scale. Maxwellian distribution A statistical description of particle velocities in idealized gases where the particles move freely inside a closed system without interacting with one another except for collisions in which they exchange energy and momentum with each other or with their thermal environment. Navier-Stokes equations A continuum system of conservation laws representing the conservation of mass, linear momenta, and energy of a compressible fluid including the effects of viscosity and heat conduction. Newtonian fluid A fluid in which the viscous stresses arising from its flow are linearly proportional to the local strain rate. Rarefaction wave A wave family with smooth solution data arising in the Riemann problem of gasdynamics and more general solutions of the Euler equation of gasdynamics. Rarefaction waves have

Compressible Flows

243

a constant entropy and constant Riemann invariants for all except one Riemann invariant when traversing across the wave. Riemann problem of gasdynamics A one-dimensional solution of the Euler equations of gasdynamics that begins with piecewise constant initial data. Shock wave A wave family with piecewise discontinuous solution data arising in the Riemann problem of gasdynamics and more general solutions of the Euler equations of gasdynamics. Shock waves satisfy the Rankine-Hugoniot jump relations and an entropy inequality. Symmetric hyperbolic form A first-order system of conservation laws is in symmetric hyperbolic form if the quasi-linear form (possibly after a change of independent variables) has all coefficient matrices that are symmetric. Thermodynamic equation of state A state relationship between thermodynamic variables.

C

Description Compressible flow describes a fluid in motion with density that can vary in space and time. The equations of compressible flow express the conservation of mass, momentum, and energy. A consequence of the variable fluid density is a finite propagation speed for information signals in the flow field. Propagating waves in a compressible flow can coalesce in both space and time resulting in steep gradients and the formation of shock wave discontinuities. Propagating waves can also diverge in space and time resulting in rarefaction wave phenomena.

Overview The mathematical study of compressible flow and shock waves dates back to the nineteenth century with hodograph transformation methods for the nonlinear equations already in use around the beginning of the twentieth century [16]. Considerable research activity was initiated during and after World War II motivated by the emergence of jet aircraft, high-speed missiles, and modern explosives. Early textbooks on the subject of compressible fluid dynamics [7, 15, 23], and [16] discuss the compressible Euler equations of gas dynamics for an inviscid fluid as well as forms of the compressible Navier-Stokes equations for viscous

Compressible Flows, Fig. 1 NASA space shuttle compressible flow simulation

fluids. Nevertheless, the majority of significant advancements occurred through the use of various simplifying approximations, e.g., steady-state flow, selfsimilarity, irrotationality, homentropic fluid, isentropic fluid, and adiabatic fluid. With present-day high-speed computers, the direct numerical approximation of the compressible Euler and Navier-Stokes equations is now routinely carried out for a wide variety of engineering and scientific applications such as automobile engine combustion, explosive detonation, nuclear physics, astrophysics, and aerodynamic performance prediction (see Fig. 1). A mathematical understanding of compressible flow has evolved from a number of different perspectives that are fundamentally related: • Symmetrization structure. Recast the compressible flow equations in symmetric hyperbolic form via a change of dependent variables • Kinetic Boltzmann moment structure. Derive the compressible flow equations as moments of kinetic approximations from statistical mechanics

244

• Wave structure. Represent the structure of compressible flow in terms of fundamental wave decompositions Each of these perspectives is briefly recounted in later sections of this entry. The symmetrization structure of the Euler and Navier-Stokes equations plays a central role in energy analysis and global stability of numerical methods for approximating them [12]. The existence of an entropy function and entropy flux pair is sufficient to guarantee that a change of variable can be found that symmetrizes these systems [17]. It then becomes straightforward to verify that these systems satisfy the second law of thermodynamics as expressed by the Clausius-Duhem inequality [25]. The kinetic moment theory provides a derivation of the compressible flow equations as moments of the Boltzmann transport equation [3, 4]. In addition, the moment construction provides a linkage between stability of kinetic systems understood in the sense of Boltzmann’s celebrated H -theorem and stability as understood from continuum analysis. The kinetic moment theory provides a systematic approach for extending compressible flow to include gas mixtures, rarefied gas regimes, and extended physical models such as needed in fluid plasma modeling. Understanding the wave structure of compressible flow has played an enormous role in the development of numerical methods for systems of conservation laws. In particular, the Riemann problem of gas dynamics discussed below is extensively used as a fundamental building block in finite-volume methods pioneered by Godunov [10] and extended to high-order accuracy by van Leer [13]. Numerical flux functions constructed from approximate solutions of the Riemann problem also arise in the discontinuous Galerkin finite element method of Reed and Hill [20] as extended to compressible flow by Cockburn et al. [6].

Models of Compressible Flow

Compressible Flows

vectors and g 2 R.d C2/d the viscous flux vectors for the compressible Navier-Stokes equations. The fluid density, temperature, pressure, and total energy are denoted by , T , p, and E, respectively. The Cartesian velocity components are denoted by ui for i D 1; : : : ; d . The compressible Navier-Stokes equations for a viscous Newtonian fluid and the compressible Euler equations (g 0) of an inviscid fluid are given by u;t C f .i;x/i D g .i;x/i

(1)

and 0

1  u D @uj A ; E

1 ui D @ui uj C ıij p A ; ui .E C p/ 0

f .i /

1 0 A; D @  ij uk i k  qi 0

g .i /

j D 1; : : : ; d

with a comma subscript denoting differentiation and an implied sum on repeated indices. For polytropic  -law gases, the previous equations may be closed using the following standard gas models and relations: • Caloric equation of state for the internal energy eint assuming a constant specific heat at constant volume Cv : eint .T / D Cv T • Thermal equation of state of an ideal gas: p.; T / D  R T D .  1/eint .T / • Total energy:   1 p 1 C juj2 E D  eint C juj2 D 2  1 2

• Fourier heat conductivity model with thermal diffuCompressible Euler and Navier-Stokes sivity  0: Equations qi D  T;xi The compressible Euler and Navier-Stokes equations in d space dimensions and time express the conserva- • Isotropic Newtonian fluid shear stress with viscosity tion of mass, momentum, and energy of an inviscid and parameters   0 and  C 2=3  0: viscous fluid, respectively. Let u 2 Rd C2 denote a vector of conserved variables for mass, linear momenta,  ij D  .ui;xj C uj;xi / C  uk;xk ıij and energy. Let f 2 R.d C2/d denote the inviscid flux

Compressible Flows

where R denotes the specific gas constant and  the adiabatic index. Compressible Potential Equation The compressible potential equation is derived from the compressible Euler equations under the assumption of irrotational flow. Expressing the velocity as the gradient of a potential, u D rˆ, insures that the continuity equation is identically satisfied. The pressure and density terms in the Euler equations are combined using the perfect gas law and isentropic flow relations, thus resulting in the compressible potential equation

245

.u21 c 2 /

2 @2  @2  2 2 @  C.u c / 2u u D 0: (5) 1 2 2 @u1 @u2 @u22 @u21

An even simpler form referred to as Chaplygin’s equation is obtained in terms of the velocity magnitude q and the turning angle  by introducing u1 D q cos  and u2 D q sin : @2  q 2 c 2 @2  @ D 0: C Cq 2 2 2 2 @ c  q @q @q

(6)

The function .u/ does not have direct physical interpretation, so an alternative form using the (2) two-dimensional compressible stream function, ‰;x1 D u2 , ‰;x2 D u1 , is constructed using the with c the local sound speed. For steady two- Mohlenbrock-Chaplygin transformation [26]: dimensional flow, this equation reduces to @ juj2 @2 ˆ D c 2 r 2 ˆ; C juj2 C .u  r/ 2 @t @t 2

 2  2 @2 ˆ 2 @ ˆ C u  c C 2u u D0 1 2 2 @x1 @x2 @x12 @x22 (3) with the local sound speed c calculated from the energy 2 2 2 equation, c 2 D c02   1 > 0 2 juj . If juj  c is everywhere satisfied, then the flow is supersonic and this equation is hyperbolic. If juj2  c 2 < 0 is everywhere satisfied, then the flow is subsonic and this equation is elliptic. When both conditions exist in a flow field, the equation undergoes a type change and the flow is called transonic. 

u21  c 2

 @2 ˆ

@2 ‰ q 2 c 2 @2 ‰ c 2 C q 2 @‰ D 0: C C q @ 2 c 2  q 2 @q 2 c 2  q 2 @q

(7)

Analytical solutions of (7) may be obtained using a separation of variables. Unfortunately, the hodographtransformed equations are very difficult to use for practical engineering problems owing to singularities arising from the hodograph transformation (e.g., uniform flow is mapped to a single point) and the unwieldy nonlinearity introduced by physically motivated boundary conditions. Consequently, the hodographtransformed equations have been used primarily to Hodograph Transformation of Compressible generate analytical solutions for verifying the accuracy Flow of numerical approximations of the Euler equations. Even under the simplifying assumptions of two- As a brief example, a particular analytical solution dimensional irrotational isentropic steady-state flow, of (7) using a separation of variables is given by Eq. (3) remains nonlinear. The task of obtaining exact solutions of (3) is generally not feasible without ‰.I q; / D  =2 .q/F .a ; b I  C 1I .q// e i  ; (8) some additional transformation of the equation to alleviate the complication of nonlinearity. The goal  2 of hodograph transformation is to convert a nonlinear q , with F the hypergeometric function, .q/ D qlim partial differential equation into a linear differential .C1/ 1 equation by inverting the roles of dependent and a C b D    1 , a b D  2. 1/ , and qlim a independent variables. Specifically, the following limiting velocity magnitude. An analytical solution for the specific value  D 1 was derived by Ringleb [21] Legendre transformation: that corresponds to transonic isentropic irrotational .u/ C ˆ.x/ D x  u; (4) flow in a turning duct. Streamline pairs may be chosen as boundaries of the domain such that a small region of the two-dimensional full potential equation (3) of supersonic flow occurs that smoothly transitions to subsonic flow as depicted in Fig. 2. yields a linear differential equation for .u/:

C

246

Compressible Flows

1.5 M

1

1

0

v  .u;t C f .i;x/i / D U;t C F;x.ii/ :

−0.5 −1 −1.5 −1.5

Dotting the conservation law system (9) with the symmetrization variables yields the entropy extension equation (10) for smooth solutions

streamlines sonic line

0.5

x2

with u;v symmetric positive definite (SPD) and f .i;v/ symmetric.

M

−1

−0.5

0 x1

0, Eq. (13) suggests u;t C f .i;x/i D 0; u; f .i / 2 Rm (9) that suitable entropy pairs for the Euler equations are given by Compressible Flows, Fig. 2 Ringleb transonic flow

with implied sum on repeated indices, i D 1; : : : ; d . fU.u/; F .i / .u/g D fc0 .s0  s/; c0 ui .s0  s/g In addition, the theory assumes the existence of a scalar entropy inequality equation in divergence form with uniformly convex entropy and entropy flux pairs for chosen constants s0 and c0 > 0. This choice is not fU.u/; F .i / .u/g W Rm  Rm 7! R  R such that unique and other entropy pairs for the Euler equations can be found [11]. Under the change of variable u 7! v, .i / U;t C F  0: (10) the compressible Euler equations are symmetrized: ;xi

For this system, we have the following theorem concerning the symmetrization of (9):

u;v v;t C f .i;v/ v;xi D 0: „ƒ‚… „ƒ‚… SPD

(14)

Symm

Theorem 1 (Mock [17]) Existence of uniformly conThe symmetrization variables for the compressible vex entropy pairs fU; F i g implies that the change of Euler equations are readily calculated, i.e., for c0 D 1 T variable u 7! v with v D .U;u / symmetrizes the and s0 D 0: conservation law system (9) 1 0 1 juj2   s  2T .i / u;v v;t C f ;v v;xi D 0; v D .U;u /T D @ .  1/ uTi A : (15) „ƒ‚… „ƒ‚… 1 SPD .  1/ Symm T

Compressible Flows

247

Finally, dotting the compressible Euler equations definite (SPSD). Choosing c0 D 1 and s0 D 0, the with the symmetrization variables yields the negated symmetrization variables for the compressible Navierentropy equation (13) as required by the general theory Stokes equations are identical to the symmetrization variables already given for the Euler equations in   .i / .i / v  u;t C f ;xi D U;t C F;xi D .s/;t C .ui s/;xi : Eq. (15). Finally, dotting the compressible Navier(16) Stokes equations with the symmetrization variables Use of this identity arises naturally in the energy   (21) v  u;t C f .i;x/i  .Mij u;xj /;xi D 0 analysis of Galerkin projections. Let V denote a suitable function space for (9) and B.v; w/ the associated weighted-residual semi-linear form for a reduces for smooth solutions to the entropy balance spatial domain  written using the symmetrization equation variables as dependent variables. For v 2 V,   qi Z   .s/;t C .ui s/;xi  w  u;t .v/ C f .i;x/i .v/ dx; 8 w 2 V: B.v; w/ D Cv T ;xi 

(17)

D v;xi  .Mij u;v /v;xj  0:

(22)

Then by choosing the particular test function w D v, an energy associated with the semi-linear form is given by Setting qi D 0 motivates the  0 sign in (13) as a viscosity limit. Substituting  D Cv s and rearranging Z   terms yields the well-known second law of thermodyB.v; v/ D v  u;t .v/ C f .i;x/i .v/ dx namics, also called the Clausius-Duhem inequality [25]  Z after Rudolf Clausius and Pierre Duhem:   D U;t C F;x.ii/ dx: (18) q   i ./;t C.ui /;xi C D Cv v;xi .Mij u;v /v;xj  0: T ;xi Symmetrization of the Compressible (23) Navier-Stokes Equations The compressible Navier-Stokes equations (1) may be A consequence of these inequalities is that for a fixed rewritten in the following form for Mij 2 R.d C2/.d C2/ : domain  with zero heat flux addition and zero flux on @, the total entropy in the system is a nondecreasing .i / u;t C f ;xi D .Mij u;xj /;xi : (19) quantity Z d  dx  0: (24) dt  When written in this form, the matrices M have no ij

particular structure, i.e., Mij are neither symmetric nor positive semi-definite. Unlike the Euler equations, Hughes et al. [12] show that the only suitable entropy Boltzmann Moment Structure of pairs for the compressible Navier-Stokes with Fourier Compressible Flow heat conductivity are given by The kinetic theory of gases describes a gas as a large ensemble of particles (atoms or molecules) in random .i / fU.u/; F .u/g D fc0 .s0  s/; c0 ui .s0  s/g: motion [4]. Rather than tracking the individual motion with position x 2 Rd and particle velocity Under the change of variable u 7! v, the compressible of particles d v 2 R , the Boltzmann transport equation [2], Navier-Stokes equations are symmetrized: u;v v;t C f .i;v/ v;xi D .Mij u;v v;xj /;xi „ƒ‚… „ ƒ‚ … „ƒ‚… SPD

Symm

(20)

@;t f .x; v; t/ C vi @;xi f .x; v; t/ D

1 C.f /; 

(25)

SPSD

describes the evolution of a kinetic distribution funcwith u;v symmetric positive definite (SPD), f .i;v/ tion f .x; v; t/ that carries information about the numsymmetric, and Mij u;v symmetric positive semi- ber of particles at time t in a differential element

C

248

Compressible Flows

dx1 : : : dxd d v1 : : : d vd . The parameter  is the Knudsen number, a ratio of the mean free path R to a characteristic macroscopic length. Let hi Rd ./ d v1 : : : d vd denote an integration over velocity space; C.f / is a collision operator that is assumed to have mass, linear momenta, and energy as collision invariants: hjvj2 C.f /i D 0; (26) and satisfy local entropy dissipation: hC.f /i D 0;

hvi C.f /i D 0;

hlog.f / C.f /i  0 for every f:

(27)

The compressible Euler equations specialized to a monatomic gas are obtained by assuming a gas in local thermodynamics equilibrium (i.e., Maxwellian distribution function) and retaining d C 2 velocity moments m.v/ D .1; vi ; jvj2 =2/T corresponding to mass, linear momenta, and energy 1  A; uj u D hm.v/ fmi D @ 3 1 2 . 2 T C 2 juj / 1 0 ui f .i / D hvi m.v/ fmi D @ ui uj C ıij p A ui . 52 T C 12 juj2 / 0

(32)

Multiplying Boltzmann’s transport equation by the term log.f / and simplifying terms reveals that solutions of Boltzmann’s transport equation satisfy the with zero right-hand-side collision terms by virtue of (26). entropy balance law The compressible Euler equations for a general .i /  -law (polytropic) gas are obtained as moment ap(28) H;t .f / C J;xi .f / D S.f / proximations after the following generalizations: (1) to include internal energy, with H.f / hf log.f /  f i the kinetic entropy, modify the energy moment 2 ı T / D .1; v; jvj =2 C eint / ; (2) increase the dim.v; e int J .i / .f / hvi .f log.f /  f /i the kinetic entropy mensionality of the phase space fluxes, and S.f / hlog.f /C.f /i the kinetic entropy R R integration to include dissipation. The celebrated Boltzmann H -theorem internal energy, hi D RC Rd ./ d v1 : : : d vd deint ; and (3) utilize the generalized Maxwellian for a  -law states that S.f /  0; (29) gas with equality if and only if C.f / D 0 which occurs if and only if the gas is in local thermodynamic equilibrium with Maxwellian distribution

fm .;u;T I v;eint /D

 ı 2 e .juvj =2Ceint /=T ˛.; d / T d=2C1=ı (33)

with ı D .1=.  1/R d=2/1 and ˛.; d / D R ı jvj2 =2 (30) d v1 : : : d vd  RC e eint deint . The nonobviRd e ı has been used [19] ous energy moment jvj2 =2 C eint rather than the more standard moment jvj2 =2Ceint (see for given macroscopic quantities ; u; T . for example [8]) in order that the classical Boltzmann entropy is obtained. Boltzmann Moment Structure of the In the study of moment closures for M  d C 2, Compressible Euler Equations Levermore [14] has shown that the constrained miniThe task of solving the Boltzmann transport equation mization of kinetic entropy may be simplified by retaining only a finite number of velocity moment averages. The resulting moment equations are obtained by introducing a moment vector arg minfH Œf j hm.v/f i D ug; f m.v/ 2 RM with polynomial components that span a velocity subspace and possess translational and roH Œf D hf log.f /  f i; (34) tational invariance. Multiplying (25) by the moment vector m.v/ and integrating over velocities yields the is sufficient to deduce that the distribution function f moment system is of exponential form 2   juvj 2T ; fm .; u; T I x; v; t/ D e .2 T /d=2

hm.v/ f i;t C hvi m.v/ f i;xi D

1 hm.v/ C.f /i: (31) 

f D exp.v.u/  m.v//

(35)

Compressible Flows

249

with v.u/ the symmetrization variables (15). In the and the thermal diffusivity is calculated from the special case of d C 2 moments, f is Maxwellian integral and it can be readily verified that the Maxwellian Z 1 distribution (33) can be rewritten in the form (35) with 1 T 2 ˛.; T; r/.r 2  5/2 r 4 e r =2 dr

.; T / D p T the symmetrization variables v.u/ D .U;u / calcu6 2 0 (39) lated using the macroscopic entropy function U.u/ D c0 .s  s0 /. where ˛.; T; r/ and ˇ.; T; r/ are positive functions that depend on the particular choice of collision model Boltzmann Moment Structure of the C.f /. When the collision model is homogeneous of Compressible Navier-Stokes Equations via degree two, ˛ and ˇ become proportional to 1 so that Chapman-Enskog Expansion  and only depend on temperature in agreement with The compressible Navier-Stokes equations may be classical expressions for these transport coefficients. derived as kinetic moments of the Boltzmann equaExistence of the Chapman-Enskog expansion functions tion corresponding to mass, momentum, and energy .1/ .2/ f and f is formalized in the following theorem: with distribution function f .x; t; v/ formulated as an expansion in the Knudsen number parameter  about Theorem 2 (Bardos et al. [1]) Assume that . ; u ; T / the local thermodynamic equilibrium Maxwellian dis- solve the compressible Navier-Stokes equations with tribution. A distribution function f is an approximate viscosity .; T / given by (38) and thermal diffusivity .1/ .2/ solution of the Boltzmann transport equation of order .; T / given by (39). Then there exist f and f such that f given by (37) is a solution of (36) of order p if p D 2. 1 @;t f .x; v; t/ C vi @;xi f .x; v; t/ D C.f / C O. p /:  (36) Wave Structure of Compressible Flow Using a finite expansion of the specific form  Understanding the wave structure of compressible flow f . ;u ;T Iv/ D fm . ; u ;T I v/ 1Cf.1/ . ;u ;T Iv/ has played an important role in the design and con (37) struction of numerical methods that properly reflect C  2 f.2/ . ; u ; T I v/ ; the finite propagation speed of waves and the resulting the compressible Navier-Stokes equations may be de- finite domain of influence of information signals in rived from solutions of order p D 2 using a successive the flow field. To insure that these finite domains approximation procedure developed by [5] and inde- of influence are accurately modeled, Godunov [10] pendently by [9], now referred to as the Chapman- pioneered the use of the Riemann problem of gas Enskog expansion. In the Chapman-Enskog procedure, dynamics as a fundamental component in the finite.i / the distribution functions f are successively deter- volume discretization of the compressible Euler equamined for increasing i by equating coefficients of equal tions. The Riemann problem of gas dynamics considers powers of  in the (37) expansion of the Boltzmann the time evolution of piecewise constant initial data transport equation (see Cercignani [3]). Note that in ul and ur centered at the origin x D 0 with solution the limit of incompressible flow, the O. 2 / term is not for later time t denoted by uRiemann .ul ; ur I x; t/. The needed in (37) to derive the incompressible Navier- work of Godunov was later extended to high-order acStokes equations. The Chapman-Enskog expansion not curacy by van Leer [13]. For a one-dimensional domain only provides a derivation of the compressible Navier- L tessellated with nonoverlapping control volumes, Stokes equations but also provides explicit expressions xi D xi C1=2  xi 1=2 ; i D 1; : : : ; N , such that for the transport coefficient viscosity  and thermal L D [1i N xi , the Godunov finite-volume method diffusivity for a given collision model C.f /. Specif- in semi-discrete form is given by ically, the viscosity is calculated from the integral F i C1=2  F i 1=2 d Z 1 ui C D 0; for i D 1; : : : ; N 2 T 2 xi ˇ.; T; r/r 6 e r =2 dr (38) dt .; T / D p (40) 15 2 0

C

250

Compressible Flows

R 1 Compressible Flows, Table 1 Eigenvalues and Riemann inwith ui D x xi u dx the cell-averaged solution i and F i C1=2 a numerical flux function obtained from a variants for the Riemann problem of gas dynamics k Riemann invariants solution of the Riemann problem, i.e., C

F i C1=2 D F .ui ; ui C1 / D f .uRiemann .ui ; ui C1 I 0; 0 // (41) that is a consistent and conservative approximation of the true flux f .u/. The development of the Godunov finite-volume method has subsequently motivated a large effort to understand and later approximate [18, 22] solutions of the Riemann problem.

kD1

uc

fu C

kD2

u

fu; pg

kD3

uCc

fu 

u;t C f ;x D 0;

2 c; sg  1

t (1) ρL uL

Riemann Problem of Gas Dynamics The compressible Euler equations in one space dimension simplify from (1) to the following divergence form:

2 c; sg  1

pL

(2) ρ*

ρ*

L u*

R u*

p*

p*

(3) ρR uR p

R

x

(42)

Compressible Flows, Fig. 3 Riemann problem evolution in the .x; t /-plane

with 0

1  u D @uA ; E

0

1 u f D @ u2 C p A : u.E C p/

(43)

The Jacobian matrix f ;u has m D 3 real and distinct eigenvalues 1 .u/ <    < m .u/. Corresponding to each eigenvalue k .u/ is a right eigenvector r k .u/, k D 1; : : : ; m. For each k, there exist m  1 Riemann invariants wj satisfying r k .u/  rwj .u/ D 0;

j D 1; : : : ; m  1:

(44)

Eigenvalues and Riemann invariants are tabulated in Table 1 for the Euler equation system (42). Solutions of the Riemann problem are composed of three wave families: 1. Classical solution rarefaction waves for which the m  1 Riemann invariants are constant throughout the wave. 2. Genuinely nonlinear shock waves that satisfy the Rankine-Hugoniot relations,

3. Linearly degenerate contact waves for which the m  1 Riemann invariants are constant and the fluid density is discontinuous. Contact waves propagate at the fluid velocity so that no material crosses the contact interface. A unique global solution of the Riemann problem 2 .cl C cr /; otherwise, exists if and only if ur  ul <  1 a vacuum is present in the solution [24]. The global solution of the Riemann problem is self-similar in the single parameter x=t (see Fig. 3) with a wave structure of the following composition form: ur D Tx3 Tx2 Tx1 ul ;

(45)

containing three scalar parameters fx1 ; x2 ; x3 g. The transition operator Tx1 consists of either a rarefaction wave or shock wave, Tx2 consists of a contact discontinuity, and Tx3 consists of either a rarefaction wave or a shock wave. Solving the Riemann problem is tantamount to finding these three parameters fx1 ; x2 ; x3 g  Œu.xC ; t/  u.x ; t/ D Œf .xC ; t/  f .x ; t/ ; given the two solution states ful ; ur g; see Smoller [24]. Demanding uniqueness of this solution is sufficient to for a moving shock wave at location x with speed select whether Tx1 and Tx3 are rarefaction waves or  and satisfy the entropy inequality (22) in the limit else viscosity limit shock waves satisfying an entropy of zero viscosity and heat conduction. inequality. Once these parameters are calculated, the

Compressible Flows 15

rho u p T

10 rho, u, p, T

251

7.

8. 5

9. 0

10. −0.5

−1

rarefaction wave

0 x

0.5 contact wave

1 shock wave

Compressible Flows, Fig. 4 Riemann problem solution profiles at the time t > 0

transition states fL ; R ; u ; p  g depicted in Fig. 3 are directly calculated from (45). The self-similar structure of the Riemann problem solution together with the wave family properties outlined above are sufficient to construct a global solution in .x; t/ from the transition states and initial data. A representative Riemann problem solution is given in Fig. 4.

11.

12.

13.

14.

15. 16. 17.

Cross-References

18.

 Lattice Boltzmann Methods  Riemann Problem

19.

20.

References 21. 1. Bardos, C., Golse, F., Levermore, C.: Fluid dynamical limits of kinetic equations. J. Stat. Phys. 63(1–2), 323–344 (1991) 2. Boltzmann, L.: Weitere studien u¨ ber das w¨armegleichgewicht unter gasmolek¨ulen. Wiener Berichte 66, 275–370 (1872) 3. Cercignani, C.: The Boltzmann Equation and Its Application. Springer, New York (1988) 4. Chapman, S., Cowling, T.: The Mathematical Theory of Non-uniform Gases. Cambridge University Press, Cambridge (1939) 5. Chapman, S.: The kinetic theory of simple and composite gases: viscosity, thermal conduction and diffusion. Proc. R. Soc. A93 (1916–1917) 6. Cockburn, B., Lin, S., Shu, C.: TVB Runge-Kutta local projection discontinuous Galerkin finite element method for

22.

23. 24. 25.

26.

conservation laws III: one dimensional systems. J. Comput. Phys. 84, 90–113 (1989) Courant, R., Friedrichs, K.: Supersonic Flow and Shock Waves. Interscience Publishers, New York (1948) Deshpande, S.M.: On the Maxwellian distribution, symmetric form, and entropy conservation for the Euler equations. Tech. Rep. TP-2583, NASA Langley, Hampton (1986) Enskog, D.: Kinetische theorie der vorg¨ange in m¨assig verd¨unnten gasen. Thesis, Upsalla University (1917) Godunov, S.K.: A finite difference method for the numerical computation of discontinuous solutions of the equations of fluid dynamics. Mat. Sb. 47, 271–290 (1959) Harten, A.: On the symmetric form of systems of conservation laws with entropy. J. Comput. Phys. 49, 151–164 (1983) Hughes, T.J.R., Franca, L.P., Mallet, M.: A new finite element formulation for CFD: I. Symmetric forms of the compressible Euler and Navier-Stokes equations and the second law of thermodynamics. Comput. Methods Appl. Mech. Eng. 54, 223–234 (1986) van Leer, B.: Towards the ultimate conservative difference schemes V. A second order sequel to Godunov’s method. J. Comput. Phys. 32, 101–136 (1979) Levermore, C.D.: Moment closure hierarchies for kinetic theories. J. Stat. Phys. 83(5–6), 1021–1065 (1996) Liepmann, H., Puckett, A.: Introduction to Aerodynamics of a Compressible Fluid. Wiley, New York (1947) Liepmann, H., Roshko, A.: Elements of Gasdynamics. Wiley, New York (1957) Mock, M.S.: Systems of conservation laws of mixed type. J. Differ. Equ. 37, 70–88 (1980) Osher, S., Solomon, F.: Upwind difference schemes for hyperbolic systems of conservation laws. Math. Comput. 38(158), 339–374 (1982) Perthame, B.: Boltzmann type schemes for gas dynamics and the entropy property. SIAM J. Numer. Anal. 27(6), 1405–1421 (1990) Reed, W.H., Hill, T.R.: Triangular mesh methods for the neutron transport equation. Tech. Rep. LA-UR-73-479, Los Alamos National Laboratory, Los Alamos (1973) Ringleb, F.: Exakte l¨osungen der differentialgleichungen einer adiabatishen gasstr¨omung. ZAMM 20(4), 185–198 (1940) Roe, P.L.: Approximate Riemann solvers, parameter vectors, and difference schemes. J. Comput. Phys. 43, 357–372 (1981) Shapiro, A.: The Dynamics and Thermodynamics of Compressible Fluid Flow. Wiley, New York (1953) Smoller, J.: Shock Waves and Reaction-Diffusion Equations. Springer, New York (1982) Truesdell, C.: The mechanical foundations of elasticity and fluid dynamics. J. Ration. Mech. Anal. 1, 125–300 (1952) Tschaplygin, S.: On gas jets. Tech. Rep. 1063, NACA, Washington, DC (1944)

C

252

Compressive Sensing Massimo Fornasier Department of Mathematics, Technische Universit¨at M¨unchen, Garching bei M¨unchen, Germany

Synonyms Compressed sensing; Compressive sampling; Sparse sampling

Definition Compressive sensing is a mathematical signal processing theory which exhaustively addresses the efficient practical recovery of nearly sparse vectors from the minimal amount of nonadaptive linear measurements. Usually such evaluations are provided by the application of a random matrix which is guaranteed to possess with highprobability certain spectral properties (e.g., the null space property or the restricted isometry property) for optimal recovery via convex optimization, e.g., `1 -norm minimization over the set of admissible competitors, or by greedy algorithms.

Overview The theory of compressive sensing has been formalized within the seminal works [5] and [6]. Although some of the relevant theoretical [8, 14, 15, 17] and practical [19, 20, 23, 26, 27, 31] aspects of compressed sensing appeared separately in previous studies, the main contribution of the former papers was to realize that one can combine the efficiency of `1 minimization and the spectral properties of random matrices in order to obtain optimal and practical recovering of (approximately) sparse vectors from the fewest linear measurements. In the work [4, 5] of Cand`es, Romberg, and Tao, the restricted isometry property (which was initially called the uniform uncertainty principle) was found playing a crucial role in stable recovery. In particular it was shown that Gaussian, Bernoulli, and partial random Fourier

Compressive Sensing

matrices [4, 22] possess this important property. Later several other types of random matrices, even with structures favorable to efficient computation and to diverse applicability, have been studied [13]. These results require tools from probability theory and finite dimensional Banach space geometry; see, e.g., [16, 18]. Donoho [7] approached the problem of characterizing sparse recovery by `1 minimization via polytope geometry, more precisely, via the notion of k neighborliness. In several papers sharp phase transition curves were shown for Gaussian random matrices separating regions where recovery fails or succeeds with high probability; [7, 9, 10], see Fig. 1 below, for an example of such a graphics. These results build on previous work in pure mathematics by Affentranger and Schneider [1] on randomly projected polytopes. Besides `1 minimization, also several greedy strategies such as orthogonal matching pursuit [28], CoSaMP [29], and iterative hard thresholding [2], which may offer better complexity than standard interior point methods, found a relevant role as practical and efficient recovery methods in compressed sensing. Compressive sensing can be potentially used in all applications where the task is the reconstruction of a signal from devices performing linear measurements, while taking many of those measurements – in particular, a complete set of measurements is a costly, lengthy, difficult, dangerous, impossible, or otherwise undesired procedure. Additionally, there should be reasons to believe that the signal is sparse in a suitable domain. In computerized tomography, for instance, one would like to obtain an image of the inside of a human body by taking X-ray images from different angles. This is the typical situation where one wants to minimize the exposure of the patient to a large amount of measurements, both for limiting the dose of radiation and the discomfort of the procedure. Also radar imaging seems to be a very promising application of compressive sensing techniques [12, 24]. One is usually monitoring only a small number of targets, so that sparsity is a very realistic assumption. Further potential applications include wireless communication [25], astronomical signal and image processing [3], analog to digital conversion [30], camera design [11], and imaging [21], to name a few.

Compressive Sensing

253

Compressive Sensing, Fig. 1 Empirical success probability of recovery of k-sparse vectors x 2 RN from measurements y D Ax, where A 2 RmN is a real random Fourier matrix. The dimension N D 300 of the vectors is fixed. Each point of this diagram with coordinates .m=N; k=m/ 2 Œ0; 1 2 indicates the empirical success probability of exact recovery, which is computed by running 100 experiments with randomly generated k-sparse vectors x and randomly generated matrix

1 0.9 0.8 0.7

C

0.6 0.5 0.4 0.3 0.2 0.1

0.1

0.2

Main Principles and Results The support of a vector x is denoted supp.x/ D fj W xj ¤ 0g, and kxk0 WD j supp.x/j denotes its cardinality. A vector x is called k sparse if kxk0  k. For k 2 f1; 2; : : : ; N g, †k WD fx 2 CN W kxk0  kg denotes the set of k-sparse vectors. Furthermore, the best k-term approximation error of a vector x 2 CN in `p is defined as

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

m N . Without further information, it is, of course, impossible to recover x from y since the linear system (1) is strongly underdetermined and has therefore infinitely many solutions. However, if the additional assumption that the vector x is k sparse or compressible is imposed, then the situation dramatically changes. The typical result in compressed sensing reads as follows. Assume that A 2 CmN be a random matrix drawn from a suitable distribution with concentration properties, suitably designed according to practical uses. For x 2 CN , let y D Ax and x  be the solution of the `1 -minimization problem

k .x/p D inf kx  zkp ; z2†k

P

1=p

min kzk1

subject to Az D y:

N p is the `p norm for where kzkp D j D1 jzj j Then 1  p  2. If k .x/p decays quickly in k, then x k .x/1 is called compressible. Note that if x is k sparse, then kx  x  k2  C1 p k k .x/p D 0. N Taking m linear measurements of a signal x 2 C for m corresponds to applying a matrix A 2 CmN – the k  C2 ; log.N /˛ measurement matrix – with high probability, for suitable constants C1 , C2 > 0 y D Ax: (1) and ˛  1, independent of x and of the dimensions m; N . The vector y 2 Cm is called the measurement vector. To illustrate the result, we show in Fig. 1 a The main interest is in the vastly undersampled case typical phase transition diagram, which describes

254

Computation of Free Energy Differences

´ the empirical success probability of exact recovery a` diff´erentes temp´eratures. J. Ecole Polytech. 1, 24–76 N (1795) of k-sparse vectors x 2 R from measurements 21. Romberg, J.: Imaging via compressive sampling. IEEE y D Ax 2 Rm .

References 1. Affentranger, F., Schneider, R.: Random projections of regular simplices. Discret. Comput. Geom. 7(3), 219–226 (1992) 2. Blumensath, T., Davies, M.: Iterative hard thresholding for compressed sensing. Appl. Comput. Harmon. Anal. 27(3), 265–274 (2009) 3. Bobin, J., Starck, J.L., Ottensamer, R.: Compressed sensing in astronomy. IEEE J. Sel. Top. Signal Process. 2(5), 718– 726 (2008) 4. Cand`es, E.J., Tao, T.: Near optimal signal recovery from random projections: universal encoding strategies? IEEE Trans. Inf. Theory 52(12), 5406–5425 (2006) 5. Cand`es, E.J., Romberg, J., Tao, T.: Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math. 59(8), 1207–1223 (2006) 6. Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006) 7. Donoho, D.L.: High-dimensional centrally symmetric polytopes with neighborliness proportional to dimension. Discret. Comput. Geom. 35(4), 617–652 (2006) 8. Donoho, D., Logan, B.: Signal recovery and the large sieve. SIAM J. Appl. Math. 52(2), 577–591 (1992) 9. Donoho, D.L., Tanner, J.: Neighborliness of randomly projected simplices in high dimensions. Proc. Natl. Acad. Sci. USA 102(27), 9452–9457 (2005) 10. Donoho, D.L., Tanner, J.: Counting faces of randomlyprojected polytopes when the projection radically lowers dimension. J. Am. Math. Soc. 22(1), 1–53 (2009) 11. Duarte, M., Davenport, M., Takhar, D., Laska, J., Ting, S., Kelly, K., Baraniuk, R.: Single-pixel imaging via compressive sampling. IEEE Signal Process. Mag. 25(2), 83–91 (2008) 12. Fannjiang, A., Yan, P., Strohmer, T.: Compressed remote sensing of sparse objects. SIAM J. Imaging Sci. 3(3), 595– 618 (2010) 13. Foucart, S., Rauhut, H.: A Mathematical Introduction to Compressed Sensing. Applied and Numerical Harmonic Analysis. Birkh¨auser, Basel (2013) 14. Garnaev, A., Gluskin, E.: On widths of the Euclidean ball. Sov. Math. Dokl. 30, 200–204 (1984) 15. Gluskin, E.: Norms of random matrices and widths of finitedimensional sets. Math. USSR-Sb. 48, 173–182 (1984) 16. Johnson, W.B., Lindenstrauss, J. (eds.): Handbook of the Geometry of Banach Spaces, vol. I. North-Holland, Amsterdam (2001) 17. Kashin, B.: Diameters of some finite-dimensional sets and classes of smooth functions. Math. USSR Izv. 11, 317–333 (1977) 18. Ledoux, M., Talagrand, M.: Probability in Banach Spaces. Springer, Berlin/New York (1991) 19. Logan, B.: Properties of high-pass signals. PhD thesis, Columbia University (1965) 20. Prony, R.: Essai exp´erimental et analytique sur les lois de la Dilatabilit´e des fluides e´ lastique et sur celles de la Force expansive de la vapeur de l’eau et de la vapeur de l’alkool,

Signal Process. Mag. 25(2), 14–20 (2008) 22. Rudelson, M., Vershynin, R.: On sparse reconstruction from Fourier and Gaussian measurements. Commun. Pure Appl. Math. 61, 1025–1045 (2008) 23. Santosa, F., Symes, W.: Linear inversion of band-limited reflection seismograms. SIAM J. Sci. Stat. Comput. 7(4), 1307–1330 (1986) 24. Strohmer, T., Hermann, M.: Compressed sensing radar. In: IEEE Proceedings of the International Conference on Acoustic, Speech, and Signal Processing, Las Vegas, pp. 1509–1512 (2008) 25. Taub¨ock, G., Hlawatsch, F., Eiwen, D., Rauhut, H.: Compressive estimation of doubly selective channels: exploiting channel sparsity to improve spectral efficiency in multicarrier transmissions. IEEE J. Sel. Top. Signal Process 4(2), 255–271 (2010) 26. Taylor, H., Banks, S., McCoy, J.: Deconvolution with the `1 -norm. Geophys. J. Int. 44(1), 39–52 (1979) 27. Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B 58(1), 267–288 (1996) 28. Tropp, J.A.: Greed is good: algorithmic results for sparse approximation. IEEE Trans. Inf. Theory 50(10), 2231–2242 (2004) 29. Tropp, J., Needell, D.: CoSaMP: iterative signal recovery from incomplete and inaccurate samples. Appl. Comput. Harmon. Anal 26(3), 301–321 (2009) 30. Tropp, J.A., Laska, J.N., Duarte, M.F., Romberg, J.K., Baraniuk, R.G.: Beyond nyquist: efficient sampling of sparse bandlimited signals. IEEE Trans. Inf. Theory 56(1), 520– 544 (2010) 31. Wagner, G., Schmieder, P., Stern, A., Hoch, J.: Application of non-linear sampling schemes to cosy-type spectra. J. Biomol. NMR 3(5), 569 (1993)

Computation of Free Energy Differences Gabriel Stoltz Universit´e Paris Est, CERMICS, Projet MICMAC Ecole des Ponts, ParisTech – INRIA, Marne-la-Vall´ee, France

Mathematical Classification 82B05, 82-08, 82B30

Short Definition Free-energy differences are an important physical quantity since they determine the relative stability of different states. A state is characterized either through the level sets of some function (the reaction coordinate)

Computation of Free Energy Differences

or by some parameter (alchemical case). Although the free energy cannot be obtained from the average of some observable in the thermodynamic ensemble at hand, free-energy differences can be rewritten as ensemble averages (upon derivation or other manipulations), and are therefore amenable to numerical computations. In a mathematical classification, there are four main classes of techniques to compute free-energy differences: free-energy perturbation and histogram methods (which rely on standard ensemble averages); thermodynamic integration (projected equilibrium dynamics); exponential nonequilibrium averages (projected time-inhomogeneous dynamics); and adaptive techniques (nonlinear dynamics).

255

energy is the average energy as given by the laws of statistical physics: Z 1 H.q; p/ eˇH.q;p/ dq dp; (4) E .H / D Z E

while the microscopic counterpart of the thermodynamic entropy is the statistical entropy (see [3])   Z d ˙ D kB ln d: (5) dq dp E Replacing U and S in (3) by (4) and (5), respectively, we obtain the definition (2).

Relative Free Energies In many applications, the important quantity is Description actually the free-energy difference between various macroscopic states of the system, rather than the free Absolute and Relative Free Energies energy itself. Free-energy differences allow to quantify Free energy is a central concept in thermodynamics the relative likelihood of different states  Transition and in modern studies on biochemical and physical Pathways, Rare Events and Related Questions. A state systems. In statistical physics, it is related to the log- should be understood here as either: arithm of the partition function of the thermodynamic 1. The collection of all possible microscopic conensemble at hand. figurations, distributed according to the canonical measure (1), and satisfying a given macroscopic Absolute Free Energies constraint .q/ D z, where  W D ! Rm with m We consider for simplicity the case of systems at consmall. Such macroscopic constraints are for instance stant temperature and volume, in which case the therthe values of a few dihedral angles in the carbon modynamic state is described by the canonical measure backbone of a protein, or the end-to-end distance of on the phase space E D D  RdN :  Calculation of a long molecule. Ensemble Averages In this case, the configurations are restricted to the set ˙.z/ D f.q; p/ 2 E j .q/ D zg where z is .dq dp/ D Z1 eˇH.q;p/ dq dp; the index of the state, and the free-energy difference Z to compute reads eˇH.q;p/ dq dp; (1) Z D E F .1/  F .0/ 1 0Z where H.q; p/ is the Hamiltonian of the system, and ˇH.q;p/ e ı .dq/ dp  .q/1 C B ˙.1/R3N ˇ 1 D kB T . The free energy is then C: D ˇ 1 ln B A @ Z ˇH.q;p/ Z e ı .dq/ dp  .q/ 1 ˙.0/R3N (2) F D  ln eˇH.q;p/ dq dp: ˇ E (6)

This definition is motivated by an analogy with macroscopic thermodynamics, where

A rigorous definition of the measures ı .q/z.dq/ can be given using the co-area formula (see [1, 11] as well as [20], Chap. 3). F D U  T S; (3) 2. The collection of all possible microscopic configuU being the internal energy of the system, and S rations distributed according to the canonical meaits entropy. The microscopic definition of the internal sure associated with a Hamiltonian H depending

C

256

Computation of Free Energy Differences

on some parameter . The parameter  is then the Computational Techniques for Free-Energy index of the state, and the free-energy difference Differences reads We present in this section the key ideas behind the methods currently available to compute free-energy 0Z 1 differences, focusing for simplicity on the alchemical ˇH1 .q;p/ e dq dp case (when possible), and refer to [6,20] for more comB E C C : plete expositions. Some of the techniques are suited Z F .1/  F .0/ D ˇ 1 ln B @ A eˇH0 .q;p/ dq dp both for alchemical transitions and transitions indexed E (7) by a reaction coordinate, but not all of them. Most of the currently available strategies fall within the followTypically,  is a parameter of the potential energy ing four classes, in order of increasing mathematical function, or the intensity of an external perturbation technicality: (such as a magnetic field for Ising systems). 1. Methods of the first class are based on straightforAlchemical transitions can be considered as a special ward sampling methods. In the alchemical case, case of transitions indexed by a reaction coordinate, the free-energy perturbation method, introduced upon introducing the extended variable Q D .; q/ in [28], recasts free-energy differences as usual and the reaction coordinate .Q/ D . Besides, the canonical averages. In the reaction coordinate case, reaction coordinate case is sometimes considered as a usual sampling methods can also be employed, limiting case of the alchemical case, using the family relying on histogram methods. of Hamiltonians 2. The second technique, dating back to [16], is thermodynamic integration, which mimics the quasi 2 1 1 static evolution of a system as a succession of  H .q/ D V .q/ C .q/   C p T M 1 p; equilibrium samplings (this amounts to an infinitely 2 2 slow switching between the initial and final states). In this case, constrained equilibrium dynamics have and letting  ! 0. to be considered. 3. A more recent class of methods relies on dynamics with an imposed schedule for the reaction coordiFree Energy and Metastability nate or the alchemical parameter. These techniques Besides physical or biochemical applications, free entherefore use nonequilibrium dynamics. Equilibergies can also be useful for numerical purposes to rium properties can however be recovered from the devise algorithms which overcome sampling barriers nonequilibrium trajectories with a suitable exponenand enhance the sampling efficiency. Chemical and tial reweighting, see [14, 15]. physical intuitions may guide the practitioners of the field toward the identification of some slowly evolv- 4. Finally, adaptive biasing dynamics may be used in the reaction coordinate case. The switching scheding degree of freedom responsible for the metastable ule is not imposed a priori, but a biasing term in behavior of the system. This quantity is a function the dynamics forces the transition by penalizing .q/ of the configuration of the system, where  W m the regions which have already been visited. This D ! R with m small. The framework to consider is biasing term can be a biasing force as for the therefore the case of transitions indexed by a reaction adaptive biasing force technique of [9], or a biasing coordinate. If the function  is well chosen (i.e., if potential as for the Wang-Landau method [26, 27], the dynamics in the direction orthogonal to  is not nonequilibrium metadynamics [13] or self-healing too metastable), the free energy can be used as a umbrella sampling [22]. biasing potential to accelerate the sampling, relying on importance sampling strategies. This viewpoint allows We refer to Fig. 1 for a schematic comparison to use free-energy techniques in other fields than the of the computational methods in the reaction one traditionally covered by statistical physics, such as coordinate case. All these strategies are based on Bayesian statistics for instance [7,12] – with the caveat appropriate methods to sample canonical measures however that finding a relevant reaction coordinate may  Sampling Techniques for Computational Statistical Physics. be nontrivial.

Computation of Free Energy Differences

257

a

b

C

c

d

Computation of Free Energy Differences, Fig. 1 Cartoon comparison of the different techniques to compute-free energy differences in the reaction coordinate case. (a) Histogram method: sample points around the level sets are generated. (b) Thermodynamic integration: a projected dynamics is used to

sample each “slice” of the phase space. (c) Nonequilibrium dynamics: the switching is imposed a priori and is the same for all trajectories. (d) Adaptive dynamics: the system is forced to leave regions where the sampling is sufficient

Methods Based on Straightforward Sampling

n D 1, the associated free-energy differences Fi D F .i C1 /  F .i / are computed using an estimator similar to (8), and the total free-energy difference is recovered as F D F0 C    C Fn1 . 2. Umbrella sampling [25]: Configurations distributed according to some probability distribution “in between” 0 and 1 are sampled, and the free energy is estimated through some importance sampling technique from the ratio of partition functions. Of course, both strategies can be combined. Finally, let us mention that it is also possible to resort to bridge sampling, where the free-energy difference F is estimated using sample points from both 0 and 1 . In computational chemistry, the method is known as the Bennett acceptance ratio (BAR) method [4].

Free-Energy Perturbation Free-energy perturbation is a technique which is restricted to the computation of free-energy differences in the alchemical case. It consists in rewriting the free-energy difference as Z F D ˇ 1 ln eˇ.H1 H0 / d0 ; E

where the probability measures  are defined as  .dq dp/ D Z1 eˇH .q;p/ dq dp: An approximation of F is then obtained by generating configurations .q n ; p n / distributed according to 0 and computing the empirical average N 1 X ˇ.H1 H0 /.q n ;pn / e : N nD1

(8)

Histogram Methods The idea of histogram methods is to sample configurations centered on some level However, the initial and final distributions 0 set ˙.z/, typically by sampling canonical measures and 1 often hardly overlap, in which case the estimate associated with modified potentials based on (8) is polluted by large statistical errors. There are two ways to improve the situation: 2 1 1. Staging: The free-energy change is decomposed .q/  z ; V .q/ C using n1 intermediate steps 0 D 0 < 1 < : : : < 2

258

Computation of Free Energy Differences

8 where  > 0 is a small parameter, and to construct a dq ˆ < .t/ D rp H.t / .q.t/; p.t//; global sample for the canonical measure .dq dp/ by dt (10) ˆ concatenating the sample points with some appropriate : dp .t/ D rq H.t / .q.t/; p.t//: dt weighting factor. This method was recently put on firm grounds using advances in statistics, and is known as Defining by   the associated flow, the work perMBAR (“multistate BAR”) [23]. Once this global sample is obtained, an approxima- formed on the system starting from some initial contion of the free energy is obtained by estimating the ditions .q; p/ is probability that the value of the reaction coordinate lies Z T @H.t /  in the interval Œz; z C z . This is done by computing .t .q; p// 0 .t/ dt W.q; p/ D @ 0 canonical averages of approximations of ı .q/z (such as bin indicator functions proportional to 1 .q/2Œz;zCz ). D H1 .T .q; p//  H0 .q; p/: Thermodynamic Integration Thermodynamic integration consists in remarking that

Z



F ./  F .0/ D

F 0 .s/ ds;

The last equality is obtained by noticing that  @H.t /  d  .t .q; p// 0 .t/ H.t / .t .q; p// D dt @   rq H.t / .t .q; p//  @t t .q; p//; C rp H.t / .t .q; p//

(9)

0

and that the derivative Z F 0 ./ D

E

@H .q; p/ eˇH .q;p/ dq dp @ Z eˇH .q;p/ dq dp E

is the canonical average of @ H with respect to the canonical measure  . In practice, F 0 .i / is computed using classical sampling techniques for a sequence of values i 2 Œ0; 1 . The integral on the righthand side of (9) is then integrated numerically to obtain the free-energy difference profile. The extension to transitions indexed by a reaction coordinate relies on projected deterministic or stochastic dynamics (see [5, 8, 10, 21, 24]). Nonequilibrium Dynamics Free-energy differences can be expressed as a nonlinear average over nonequilibrium trajectories, using the so-called Jarzynski equality, see (11) below. This equality can easily be obtained for a system governed by Hamiltonian dynamics, with initial conditions at equilibrium, canonically distributed according to 0 , and subjected to a switching schedule  W Œ0; T ! R with .0/ D 0 and .T / D 1. More precisely, we consider initial conditions .q.0/; p.0// 0 , which are evolved according to the following nonautonomous ordinary differential equation for 0  t  T :

and the second term on the right-hand side vanishes in view of (10). Then, Z E

eˇW.q;p/ d0 .q; p/ D Z01

Z



eˇH1 .T .q;p// dq dp: E

Since T defines a change of variables of Jacobian 1, the above equality can be restated as E0 .eˇW / D

Z1 D eˇ.F .1/F .0//; Z0

(11)

where the expectation is taken with respect to initial conditions distributed according to 0 . For stochastic dynamics, results similar to (11) can be obtained, for transitions indexed by a reaction coordinate or an alchemical parameter, using appropriately constrained dynamics (see [17,21]). Expectations have to be understood as over initial conditions and realizations of the Brownian motion in these cases. In view of the equality (11), it is already clear that the lowest values of the work dominate the nonlinear average (11), and the distribution of weights eˇW.q;p/ is often degenerate in practice. This prevents in general accurate numerical computations, and raises issues very similar to the ones encountered with free-energy perturbation (see the discussion in [20], Chap. 4). Refinements are therefore needed to use nonequilibrium methods in practice, and equilibrium

Computation of Free Energy Differences

or adaptive methods generically outperform them. Their interest is therefore rather theoretical except in situations when the underlying physical system is itself genuinely out of equilibrium (as in DNA pulling experiments). Adaptive Dynamics Adaptive dynamics may be seen as some adaptive importance sampling strategy, with a biasing potential at time t which is a function of the reaction coordinate. In essence, the instantaneous reference measure for the system is the canonical measure associated with H.q; p/  Ft ..q//, where Ft is some approximation of the free energy. The biasing potential converges in the longtime limit to the free energy by construction of the dynamics. The main issue is to decide how to adapt the biasing potential. There are two strategies to this end: Update the potential Ft itself (adaptive biasing potential (ABP) strategies, in the classification of [18]), or the gradient of the potential (adaptive biasing force (ABF) strategies). In both cases, the update is done depending on the observed current distribution of configurations. For ABF, this is achieved by adjusting the biasing force in the direction of the gradient of the reaction coordinate in such a way that the average force experienced by the system at a given value of the reaction coordinate vanishes. For ABP, the bias is increased in undervisited regions and decreased in overvisited parts of the phase space, until the distribution of the values of the reaction coordinate is uniform. The resulting dynamics are highly nonlinear, and the mathematical study of their properties is very difficult in general. At the moment, mathematical convergence results exist only for the ABF method [19] and the Wang-Landau algorithm [2].

References 1. Ambrosio, L., Fusco, N., Pallara, D.: Functions of Bounded Variation and Free Discontinuity Problems. Oxford Science Publications, Oxford (2000) 2. Atchade, Y.F., Liu, J.S.: The Wang-Landau algorithm for Monte-Carlo computation in general state spaces. Stat. Sinica 20(1), 209–233 (2010) 3. Balian, R.: From Microphysics to Macrophysics. Methods and Applications of Statistical Physics, vol. I–II. Springer, New York/Berlin (2007)

259 4. Bennett, C.H.: Efficient estimation of free energy differences from Monte-Carlo data. J. Comput. Phys. 22, 245–268 (1976) 5. Carter, E.A., Ciccotti, G., Hynes, J.T., Kapral, R.: Constrained reaction coordinate dynamics for the simulation of rare events. Chem. Phys. Lett. 156(5), 472–477 (1989) 6. Chipot, C., Pohorille, A. (eds.): Free Energy Calculations. Springer Series in Chemical Physics, vol. 86. Springer, New York/Berlin/Heidelberg (2007) 7. Chopin, N., Leli`evre, T., Stoltz, G.: Free energy methods for bayesian inference: efficient exploration of univariate gaussian mixture posteriors. Stat. Comput. (2011, in press) 8. Ciccotti, G., Leli`evre, T., Vanden-Eijnden, E.: Projection of diffusions on submanifolds: application to mean force computation. Commun. Pure Appl. Math. 61(3), 371–408 (2008) 9. Darve, E., Porohille, A.: Calculating free energy using average forces. J. Chem. Phys. 115, 9169–9183 (2001) 10. den Otter, W., Briels, W.J.: The calculation of free-energy differences by constrained molecular-dynamics simulations. J. Chem. Phys. 109(11), 4139–4146 (1998) 11. Evans, L.C., Gariepy, R.F.: Measure Theory and Fine Properties of Functions. Studies in Advanced Mathematics. CRC, Boca Raton (1992) 12. Gelman, A., Meng, X.L.: Simulating normalizing constants: from importance sampling to bridge sampling to path sampling. Stat. Sci. 13(2), 163–185 (1998) 13. Iannuzzi, M., Laio, A., Parrinello, M.: Efficient exploration of reactive potential energy surfaces using Car-Parrinello molecular dynamics. Phys. Rev. Lett. 90(23), 238302 (2003) 14. Jarzynski, C.: Equilibrium free-energy differences from nonequilibrium measurements: a master-equation approach. Phys. Rev. E 56(5), 5018–5035 (1997) 15. Jarzynski, C.: Nonequilibrium equality for free energy differences. Phys. Rev. Lett. 78(14):2690–2693 (1997) 16. Kirkwood, J.G.: Statistical mechanics of fluid mixtures. J. Chem. Phys. 3(5):300–313 (1935) 17. Leli`evre, T., Rousset, M., Stoltz, G.: Computation of free energy differences through nonequilibrium stochastic dynamics: the reaction coordinate case. J. Comput. Phys. 222(2), 624–643 (2007) 18. Leli`evre, T., Rousset, M., Stoltz, G.: Computation of free energy profiles with adaptive parallel dynamics. J. Chem. Phys. 126:134111 (2007) 19. Leli`evre, T., Rousset, M., Stoltz, G.: Long-time convergence of an Adaptive Biasing Force method. Nonlinearity 21, 1155–1181 (2008) 20. Leli`evre, T., Rousset, M., Stoltz, G.: Free Energy Computations. A Mathematical Perspective. Imperial College Press, London/Hackensack (2010) 21. Leli`evre, T., Rousset, M., Stoltz, G.: Langevin dynamics with constraints and computation of free energy differences. Math. Comput. (2011, in press) 22. Marsili, S., Barducci, A., Chelli, R., Procacci, P., Schettino, V.: Self-healing Umbrella Sampling: a non-equilibrium approach for quantitative free energy calculations. J. Phys. Chem. B 110(29):14011–14013 (2006) 23. Shirts, M.R., Chodera, J.D.: Statistically optimal analysis of samples from multiple equilibrium states. J. Chem. Phys. 124(12), 124105 (2008)

C

260 24. Sprik, M., Ciccoti, G.: Free energy from constrained molecular dynamics. J. Chem. Phys. 109(18), 7737–7744 (1998) 25. Torrie, G.M., Valleau, J.P. Non-physical sampling distributions in Monte-Carlo free-energy estimation – Umbrella sampling. J. Comput. Phys. 23(2), 187–199 (1977) 26. Wang, F., Landau, D.: Determining the density of states for classical statistical models: a random walk algorithm to produce a flat histogram. Phys. Rev. E 64, 056101 (2001) 27. Wang, F.G., Landau, D.P.: Efficient, multiple-range random walk algorithm to calculate the density of states. Phys. Rev. Lett. 86(10), 2050–2053 (2001) 28. Zwanzig, R.W.: High-temperature equation of state by a perturbation method I. Nonpolar gases. J. Chem. Phys. 22(8), 1420–1426 (1954)

Computational Complexity Felipe Cucker Department of Mathematics, City University of Hong Kong, Kowloon Tong, Hong Kong

Mathematics Subject Classification 03D15; 65Y20; 68Q15; 68Q25

Short Definition Computational complexity is the study of the resources (mainly computing time) necessary to solve a problem.

Description One of the most immediate experiences when solving problems with a computer is the differences, say in computing time, between different executions. This occurs between different data for the same problem (e.g., two different linear systems, both to be solved for a solution) but also between different problems, since one feels that one of them is more “difficult” to solve than the other (e.g., multiplying matrices is easier than inverting them). The subject of computational complexity gives a formal framework to study this phenomenon.

Partially supported by GRF grant CityU 100810.

Computational Complexity

The Basic Ingredients A problem is a function ' W I ! O from the input space I to the output space O. Elements in I are inputs or instances to the problem. For instance, in the problem of (real) matrix multiplication, the input space is the set of all pairs A; B with A 2 IRmn and B 2 IRnp (for some m; n; p 2 IN). The output for such an input is the matrix AB 2 IRmp . An algorithm A computing the function ' W I ! O is said to solve this problem. This means that for each input a 2 I , the algorithm performs a computation at the end of which it returns '.a/. This computation is no more than a sequence of some elementary operations (more on this soon enough), and the length of this sequence, that is, the number of such operations performed, is the cost of the computation for input a, which we will denote by costA .a/. Two Main Frameworks During the last decades, two scenarios for computing have grown apart: On the one hand, discrete computations, which deal with (basically) combinatorial problems such as searching for a word in a text, compiling a program, or querying a database; and, on the other hand, numerical computations, which (basically again) are related to problems in algebra, geometry, or analysis. A clear distinction between the two can be made by looking at the nature of the occurring input spaces. In discrete computations, inputs (and any other data present during the computation) can be represented using bits. That is, they are taken from the disjoint union f0; 1g1 of vectors of n bits over all possible n  1. Using bits we encode letters of the English alphabet, common punctuation symbols, digits, etc., and, hence, words, texts made with those words, integer, and rational numbers. The size of any such object is the number of bits used in its encoding. That is, if the encoding is a 2 f0; 1g1 , the only n such that a 2 f0; 1gn. The elementary operations in this context are the reading or recording of a bit as well as the replacement of a 0 by a 1 or vice versa. In numerical computations, inputs are vectors of real numbers. That is, they are taken from the disjoint union IR1 of vectors of n reals over all possible n  1, and again, the size of any a 2 IR1 is the only n such that a 2 IRn . In a digital computer, however, real numbers cannot be properly encoded. They are instead approximated by floating-point numbers. (This feature

Computational Complexity

introduces an issue of roundoff errors and their propagation which is not crucial in our account; for more on this see the entrance on conditioning.) Nonetheless, these numbers are treated as indivisible entities in the measure that the elementary operations in this setting are the reading or recording of a real number and the execution of an arithmetic operation (i.e., one in fC; ; ; =g) or a comparison (either < or ) between two reals. The fact that we have identified a set of elementary operations for both the discrete and the numerical settings makes clear our definition of costA .a/ above.

Three Kinds of Analysis The accurate determination of costA .a/ for a specific input a is rarely of interest. Rather, we want to be able to compare the performance of two different algorithms or, simply, to get an idea of how good is a given algorithm. That is, we are interested in the whole of the function costA rather than in a few values of it. To gauge this function, three types of analysis are currently used, all of them relying on a single principle. The principle is the following. For n 2 IN we consider the set IKn  IK1 (here IK is either f0; 1g or IR) and the restriction of costA to IKn . Each form of analysis associates a quantity g A .n/ (depending on n) to this restriction, and the goodness, or lack of it, of an algorithm is given by the asymptotic behavior of g A .n/ for large n. The difference between the three types of analysis is in how g A .n/ is defined. A .n/ WD In worst-case analysis one takes gworst A supa2IKn cost .a/. When IK D f0; 1g, this quantity is finite. In contrast, for some problems and some A .n/ D algorithms, in case IK D IR, we may have gworst 1 for some (or even for all) n 2 IN. An often cited example is the sorting of an array of real numbers, a problem for which there is a vast number of algorithms. The cost of an execution, for most of these algorithms, is the number of comparisons and recordings performed. For instance, for the very popular Quicksort, Quicksort .n/ D O.n2 /. Another example is the we have gworst solution of linear systems of equations Ax D b, where A 2 IRnn and b 2 IRn , using Gaussian elimination. GE .n/ D O.n3 /. For this situation we have gworst A .n/ WD In average-case analysis one takes gavg A IEaDn cost .a/ where Dn is a probability measure on IKn and IE denotes expectation with respect to this

261

measure. The rationale of this analysis is to focus on the behavior of an algorithm over an “average” input instead of a “worst-possible” input. And indeed, in general, this form of analysis appears to be a more accurate description of the behavior of the algorithm in practice. A typical choice for Dn when IK D IR is the standard Gaussian on IRn . With this choice, for Quicksort .n/ D O.n log n/. It is instance, we have gavg worth noting that for an algorithm A and a size n, A A we may have gworst .n/ D 1 but gavg .n/ < 1. When IK D f0; 1g, the typical choice of Dn is the uniform distribution on f0; 1gn. A criticism often done to average-case analysis is the fact that the measure Dn may be too optimistic. This measure is doubtless chosen because of technical considerations (ease of computation) more than because of its accuracy to describe frequencies in real life (an elusive notion in any case). The third form of analysis, recently introduced by D. Spielman and S.-H. Teng with the goal of escaping this criticism, is smoothed analysis (see [7] for a recent survey). The idea is to replace the desiderata “the probability that costA .a/ is large, for a random input a, is small” by “for all input a the probability that costA .a/ is large, for a small random perturbation a of a, is small.” In the case IK D IR, which is the one where smoothed analysis most commonly occurs, we are interested, for  > 0, in the function A gsmoothed .n; / WD sup

IE

a2IRn aN.a; 2 kak2 Id/

costA .a/:

Here N.a;  2 kak2 Id/ is the Gaussian distribution on IRn centered at a with covariance matrix  2 kak2 Id. Smoothed analysis is meant to interpolate between worst-case analysis (obtained when  D 0) and average-case analysis (which is approximated when  is large). Moreover, experience shows that it is quite robust in the sense that a different choice of measure for the random perturbation yields similar results (see [4] for examples of this feature). Upper and Lower Bounds The analyses above provide yardsticks for the performance of an algorithm A which solves a problem '. Using the same yardstick for different algorithms allows one to compare these algorithms (with respect to their computational cost). A related, but different, concern would consider not a few algorithms at hand

C

262

but the set of all possible algorithms solving '. The relevant question is now, which is the smallest cost (with respect to one of our yardsticks) necessary to solve '? The analysis of a particular algorithm for ' provides an upper bound on this cost. A milder form of our question is to provide lower bounds as well. The study of these lower bounds relies on various types of techniques (and is mostly done for the worst-case setting). This study is often a frustrating experience because the gap between provable upper and lower bounds is exponential. The main reference for lower bounds .IK D IR/ is [3].

Computational Dynamics

NP-completeness in discrete computations. A recent textbook in (discrete) complexity is [1]. In the case IK D IR, the standard NPIR -complete problem consists of deciding whether a polynomial of degree 4 (in several variables) with real coefficients has a real zero. A reference for complexity over the reals is [2]. Deciding whether PIK D NPIK is a major open problem. It is widely believed that this equality is false both for IK D f0; 1g and IK D IR (but should equality hold, the consequences would be enormous). For IK D f0; 1g, the P vs. NP question is one of the seven Millennium Prize Problems stated by the Clay Institute in year 2000. This question is also in the list of problems proposed by Steve Smale for the Complexity Classes An idea that gathered strength since the early 1970s mathematicians of the twenty-first century [6] where was to cluster problems with similar cost so that the extension to a more general IK is also mentioned. advances in the study of one of them could lead to advances in the study of others. We next briefly References describe the best known example of this idea. We restrict attention to decisional problems, that is, to problems of the form ' W IK1 ! f0; 1g. We say 1. Arora, S., Barak, B.: Computational Complexity. Cambridge University Press, Cambridge (2009). A modern approach that such a problem is decidable in polynomial time 2. Blum, L., Cucker, F., Shub, M., Smale, S.: Complexity and – or that it is in the class PIK – when there exists an Real Computation. Springer, New York (1998) A .n/ D nO.1/ . We 3. B¨urgisser, P., Clausen, M., Shokrollahi, A.: Algebraic Comalgorithm A solving ' such that gworst plexity Theory. Springer, New York (1996) say that it is decidable in nondeterministic polynomial 4. B¨urgisser, P., Cucker, F., Lotz, M.: The probability that a time – or that it is in the class NPIK – when there exists slightly perturbed numerical analysis problem is difficult. a problem W IK1  IK1 ! f0; 1g and an algorithm Math. Comput. 77, 1559–1583 (2008) 5. Garey, M., Johnson, D.S.: Computers and Intractability: A A solving such that: Guide to the Theory of NP-Completeness. Freeman, San 1. For all a 2 IK1 , '.a/ D 1 iff there exists b 2 IK1 Francisco (1979) such that .a; b/ D 1. 6. Smale, S.: Mathematical problems for the next century. In: A .n/ D nO.1/ . 2. gworst Arnold, V., Atiyah, M., Lax, P., Mazur, B. (eds.) Mathematics: Frontiers and Perspectives, pp. 271–294. AMS, ProviThe class PIK is seen as the class of tractable (in dence (2000) the sense of efficiently solvable) problems. Obviously, 7. Spielman, D.A., Teng, S.-H.: Smoothed analysis: an attempt one has PIK  NPIK , but the converse is unknown. to explain the behavior of algorithms in practice. Commun. It is known that problems in NPIK can be solved ACM 52(10), 77–84 (2009) in exponential time (i.e., that for some algorithm A A .n/ D 2O.n/). In order solving them, one has gworst to decide whether this upper bound can be lowered to polynomial (i.e., whether PIK D NPIK ), a strategy Computational Dynamics was to identify a subclass of NPIK , the class of NPIK complete problems, that has the property that any such Bj¨orn Sandstede problem is in PIK iff PIK D NPIK . This allows to Division of Applied Mathematics, Brown University, focus the efforts for deciding the truth or falsity of Providence, RI, USA this equality in the study of lower bounds for a single problem (any NPIK -complete). For IK D f0; 1g, the number of problems that Mathematics Subject Classification have been established to be NP-complete is very large. The book [5], despite its age, is an excellent text on 65P; 37M

Computational Dynamics

263

Synonyms Computational dynamics; Numerical analysis of dynamical systems

Short Definition Computational dynamics is concerned with numerical techniques that are designed to compute dynamically relevant objects, their stability, and their bifurcations.

Description In this entry, some of the numerical approaches for studying the dynamics of deterministic nonlinear dynamical systems will be reviewed. The focus will be on systems with continuous time and not on discrete dynamical systems generated by iterating nonlinear maps. Thus, let uP D f .u; /;

.u; / 2 X  Rm ;

t >0

(1)

represent a system of ordinary, partial, or delay differential equations posed on a finite- or infinitedimensional space X , where  denotes additional system parameters. We assume that (1) has a solution u.t/ for given initial conditions u.0/ D u0 and are interested in identifying and computing dynamical objects and structures of (1) by numerical means. Common tasks are to find equilibria (stationary, time-independent solutions) or periodic solutions of (1). Once an equilibrium or a periodic orbit has been found for a specific value of the parameter , it is often of interest to continue the solution to other parameter values, assess its stability, and identify the bifurcations it might undergo as  is changed. Other typical tasks include the computations of more general invariant sets such as invariant tori, connecting orbits between equilibria or periodic orbits, invariant manifolds, and attractors. For instance, near an equilibrium, one might be interested in computing center, stable, and unstable manifolds, either globally or as Taylor expansions, or in determining the normal form of the vector field numerically or symbolically. Often it may also be of interest to probe for chaotic dynamics,

for instance, through the computation of Lyapunov exponents. To accomplish these tasks, robust and reliable, yet fast, algorithms are desired. Ideally, these algorithms should have a theoretical foundation, including theoretical error estimates for the difference between the actual and computed objects. It is also often desirable that these algorithms, or appropriate variants of them, respect or exploit additional structure present in (1), for example, symmetries and time reversibility, conserved quantities, symplectic structures such as those afforded by Hamiltonian systems, or multiple time scales. In certain circumstances, verified numerical computations might be feasible that provide a proof that the computed objects indeed exist. In the remainder of this entry, different computational approaches will be reviewed. The statements below should not be interpreted as theorems but rather as results valid under additional assumptions that can be found in the listed references. Direct Numerical Simulations (DNS) The traditional tool for exploring the dynamics of a differential equation comprises numerical initial value problem (IVP) solvers. In their simplest form, namely, as one-step methods, an IVP solver depends on a chosen small time step h and associates to each initial condition u0 an approximation ‰h .u0 / of the solution u.h/ at time h. We say that an IVP solver has order p  1 if there is a constant C0 > 0 such that ju.h/  ‰h .u0 /j  C0 hpC1

8 0 < h 1:

The explicit Euler method, given by ‰h .u/ WD u C hf .u; /, is an example of an IVP solver of order 1. Given a one-step method ‰h of order p, there is, for each fixed T > 0, a constant C.T / such that ju.nh/  ‰hn .u0 /j  C.T /hp

80  n 

T ; h

80 < h 1 for integers n, so that iterates of the nonlinear map ‰h approximate the solution well over each finite time interval Œ0; T . However, to explore the dynamics of (1), we are interested in letting T go to infinity, and the error estimate above is then no longer useful as C.T / may increase without bound. Under appropriate

C

264

Computational Dynamics

assumptions on ‰h , it can be shown that ‰h is the time- the underlying structure to get meaningful results over h map of the modified system longer time intervals [10]. The main advantages of using direct numerical uP D f .u; / C hp g.u; ; h; t= h/ (2) simulations to explore the dynamics are that accurate, reliable, and fast solvers are readily available for a wide for an appropriate function g D g.u; ; h; / that range of problems and that all stable structures can, is 1-periodic in , so that (2) corresponds to adding in principle, be found in this fashion. On the other a small highly oscillatory perturbation to (1); hence, hand, a systematic study of parameter space can be solving (1) numerically with the IVP solver means expensive as the underlying system has to be integrated following the exact solution of the nonautonomous for a long time for each parameter value to ensure system (2) for the same initial data. In particular, that the limiting solution has been reached; another asymptotically stable equilibria and periodic orbits disadvantage is that this approach finds only stable of (1) persist as slightly perturbed asymptotically stable solutions and cannot be used to trace out complete fixed points and invariant circles, respectively, of the bifurcation diagrams. numerical solver ‰h [8, Sect. 6]. In contrast, a nondegenerate homoclinic orbit of (1) will typically be- Continuation Methods come a transverse homoclinic orbit of (2), with the Finding equilibria u of (1) means solving f .u; / D 0 associated complicated chaotic dynamics, although the for u. If a sufficiently accurate guess u0 for an equilibchaotic region is exponentially small in the step size h rium u at  D  is known, then we can efficiently calculate u with Newton’s method by computing the [8, Sect. 6]. Direct numerical simulations are also useful when iterates probing for chaotic dynamics, despite the associated n  0; unC1 WD un  fu .un ;  /1 f .un ;  /; rapid separation of nearby trajectories. If (1) has a chaotic invariant set that possesses a hyperbolic structure, then it also has the shadowing property: for since un will converge to u quadratically in n as n sufficiently small step sizes h, any numerical trajectory approaches infinity. More generally, if  2 R, then the near the hyperbolic chaotic set will lie within order hp set f.u; / 2 X  R W f .u; / D 0g of equilibria of (1) of a genuine trajectory of (1) but for a possibly different will typically consist of curves .u ;  /.s/, which can initial condition, and long-term computations faithfully be computed efficiently by using continuation or pathrepresent the underlying chaotic dynamics [8, Sect. 7]. following methods that are based on Newton’s method. To quantify exponential separation of trajectories and It is useful to write U D .u; / and seek solutions of the dimensionality of the underlying dynamics, Lya- F .U / D 0 for a smooth function F W RN C1 ! RN . punov exponents are often computed simultaneously The solution set fU W F .U / D 0g will typically consist of curves, which can be traced out as follows with the trajectory [4]. To apply these ideas to partial differential equations (see [8, Sect. 4] or [13, Sect. 1]). Starting with a given (PDEs), one would first discretize the PDE in space; solution U0 of F .U0 / D 0, find a vector V0 with jV0 j D if the underlying domain is unbounded, it would need 1 such that FU .U0 /V0 D 0, which exists because to be replaced by a bounded domain together with FU .U0 / maps RN C1 into RN . Next, choose a small appropriate boundary conditions, which could affect step size h > 0 and apply Newton’s method to the map the dynamics, for instance, of traveling waves (see [13, U 7! .F .U /; hU  U0 ; V0 i  h/ with initial guess U0 C Sect. 10] and [8, Sect. 18]). The resulting large system hV0 ; see Fig. 1. Applied to the vector field f .u; /, the of ordinary differential equations (ODEs) is usually continuation method will yield a curve .u ;  /.s/ of stiff and requires IVP solvers that can handle multiple equilibria of (1), regardless of whether these equilibria time scales [2]. If the underlying ODE or PDE has are dynamically stable or not. This algorithm will also conserved quantities or respects a symplectic structure, continue effortlessly around saddle-node bifurcations then the results mentioned above do not apply because where the solution branch folds back. If the eigenvalues none of the objects of interest can be asymptotically of the linearization fu .u .s/;  .s// along the curve of stable; for such systems, care has to be taken to use equilibria are monitored, then bifurcation points can solvers such as geometric integrators, which respect be detected at which two curves of equilibria collide

Computational Dynamics

265

Computational Dynamics, Fig. 1 A schematic illustration of continuation and path-following methods

U1

U∗(s)

or small-amplitude periodic orbits may emerge. At each bifurcation, one can then attempt to switch onto another solution branch [8, Sect. 4]. Periodic orbits can be computed similarly: finding a nontrivial periodic solution u .t/ of uP D f .u/ with period T is equivalent to seeking a zero .v; T / of the function F W C 1 .Œ0; 1 ; Rn /  R ! C 0 .Œ0; 1 ; Rn /  Rn  R .v; T / 7! (3)   Z 1 vP  Tf .v/; v.1/  v.0/; hvP 0 .s/; v0 .s/v.s/i ds ; 0

where v0 .s/ WD u0 .sT0 / is computed from an initial guess .u0 ; T0 / for .u ; T /. The first two components of F ensure that u.t/ D v.t=T / is a T -periodic solution of uP D f .u/. Since any time-shift of a periodic solution is again a periodic solution, the integral condition in the third component selects a specific time-shift and makes the solution unique; see [8, Sect. 4] or [13, Sect. 11]. Discretizing the boundary-value problem (3) and applying Newton’s method allows for the accurate location of periodic orbits, whether stable or not. Similarly, if f depends on a one-dimensional parameter , then periodic orbits and their periods can be continued in  and bifurcations identified by simultaneously computing their Floquet exponents. Similar algorithms exist for locating and continuing connecting orbits between equilibria or periodic orbits [8, Sect. 4]. Continuation methods can also be used to trace out the locations of saddle-node, Hopf, and other bifurcations of equilibria or periodic orbits in systems with two or more parameters; this is achieved by adding defining equations that characterize these bifurcations to the system that describes equilibria or periodic orbits; see [8, Sect. 4] or [9]. Algorithms have also been developed for multiparameter continuation, that is, for tracing out

U0 {U : U − U0, V0 = h}

higher-dimensional surfaces of zeros of functions [13, Sect. 3].

Computing Invariant Manifolds and Sets Direct numerical simulations and continuation methods focus on single trajectories. Often, one is interested in computing larger invariant sets such as stable or unstable manifolds or the global attractor of a dynamical system. Arguably, the most versatile algorithms for computing such objects are based on set-oriented methods. Suppose that ˆ is the time-T map of the differential equation (1) for a fixed parameter value. Given an open bounded set Q  X , we wish to compute the maximal attractor A contained in Q, which is T k defined as the intersection A D k0 ˆ .Q/ of all forward iterates of Q under ˆ. Subdivision algorithms can then be used to approximate A numerically. Starting with a collection B0 of sets whose union is Q, we proceed recursively: given a collection Bk1 of subsets of Q, subdivide each element of Bk1 into smaller sets to obtain a new collection BQk ; next, define a new collection Bk by picking those subsets B of BQk for which there is a BQ in BQk such that ˆ.B/ \ BQ ¤ ;. If the diameter of the elements in Bk converges to zero, then the union of the elements in Bk converges to the attractor A in Q [8, Sect. 5]. Numerically, the condition ˆ.B/ \ BQ ¤ ; is checked on a finite set of test points in B; several algorithms and theoretical error analyses are available for guidance on how to pick these test points. Subdivision algorithms can also be used to compute unstable manifolds and invariant measures [8, Sect. 5]. Various other methods for computing unstable or stable manifolds exist that are based on computing geodesic circles, continuing a set of orbits as solutions to boundary-value problems, or continuing and refining triangulations or meshes [12].

C

266

Rigorous or Verified Computations Starting with Lanford’s investigation of universality in period-doubling cascades and Tucker’s proof of chaos in the Lorenz equations, various methods have emerged for rigorous or verified dynamics computations [14]. Some of these approaches rely on interval arithmetic, which guarantees error bounds on floating-point operations; others use topological methods that can be computed robustly, such as Conley indices, and yet others use a combination of rigorous estimates and numerical computations.

Computational Mechanics 3. Dellnitz, M., Froyland, G., Junge, O.: The algorithms behind GAIO-set oriented numerical methods for dynamical systems. In: Fiedler, B. (ed.) Ergodic Theory, Analysis, and Efficient Simulation of Dynamical Systems, pp. 145–174. Springer, Berlin (2001) 4. Dieci, L., Elia, C.: SVD algorithms to approximate spectra of dynamical systems. Math. Comput. Simul. 79(4), 1235– 1254 (2008) 5. Doedel, E.J., Oldeman, B.: AUTO 07P: Continuation and Bifurcation Software for Ordinary Differential Equations. Tech. Rep., Concordia University (2009) 6. Engelborghs, K., Luzyanina, T., Roose, D.: Numerical bifurcation analysis of delay differential equations using DDEBIFTOOL. ACM Trans. Math. Softw. 28(1), 1–21 (2002) 7. Ermentrout, B.: Simulating, Analyzing, and Animating Dynamical Systems, Software, Environments, and Tools, vol. 14. SIAM, Philadelphia (2002) 8. Fiedler, B. (ed.): Handbook of Dynamical Systems, vol. 2. North-Holland, Amsterdam (2002) 9. Govaerts, W.J.F.: Numerical Methods for Bifurcations of Dynamical Equilibria. SIAM, Philadelphia (2000) 10. Hairer, E., Lubich, C., Wanner, G.: Geometric Numerical Integration. Springer, Heidelberg (2010) 11. Heroux, M.A., Bartlett, R.A., et al.: An overview of the Trilinos project. ACM Trans. Math. Softw. 31(3), 397–423 (2005) 12. Krauskopf, B., Osinga, H.M., Doedel, E.J., Henderson, M.E., Guckenheimer, J., Vladimirsky, A., Dellnitz, M., Junge, O.: A survey of methods for computing (un)stable manifolds of vector fields. Int. J. Bifurc. Chaos Appl. Sci. Eng. 15(3), 763–791 (2005) 13. Krauskopf, B., Osinga, H.M., Gal´an-Vioque, J. (eds.): Numerical Continuation Methods for Dynamical Systems. Springer, Dordrecht (2007) 14. Mischaikow, K.: Topological techniques for efficient rigorous computation in dynamics. Acta Numer. 11, 435–477 (2002)

Software There is a wealth of initial value problem solvers available in various depositories or as part of commercial packages such as MATLAB. The focus here will be on toolboxes that are designed especially for computational dynamics. AUTO07P [5] is a package that implements continuation methods for algebraic and boundary-value problems. Among other features, AUTO 07 P accurately locates and continues equilibria, periodic orbits, and connecting orbits; determines their stability; locates and continues their bifurcations; and implements branch-switching routines. XPPAUT [7] provides a graphical interface to a suite of IVP solvers that can be used to solve ODEs and delay differential equations; it also provides a user interface to some of AUTO 07 P ’s capabilities. DDE - BIFTOOL [6] is a continuation code for delay differential equations. TRILINOS [11] provides a suite of continuation methods for large-scale systems through its packages LOCA and PARACONT . Set-oriented subdivision algorithms have been implemented in GAIO [3]. Other computationaldynamics toolboxes are reviewed in [13, Sect. 2], Computational Mechanics and additional continuation codes are listed in [9]. Symbolic software packages such as MAPLE or J. Tinsley Oden MATHEMATICA can be used to implement algorithms Institute for Computational Engineering and Sciences, The University of Texas at Austin, Austin, TX, USA for calculating normal forms near equilibria [1].

References 1. Algaba, A., Freire, E., Gamero, E.: Characterizing and computing normal forms using Lie transforms: a survey. Dyn. Contin. Discret. Impuls. Syst. Ser. A Math. Anal. 8(4), 449–475 (2001) 2. Ascher, U.M., Petzold, L.R.: Computer methods for ordinary differential equations and differential-algebraic equations. SIAM, Philadelphia (1998)

Theoretical mechanics is the study of the motion of bodies under the action of forces. Thus, it embodies a vast area of mathematical physics and chemistry, as well as all of classical mechanics, including elements of quantum mechanics, molecular dynamics, celestial mechanics, and the theories underlying solid mechanics, fluid mechanics, and much of materials science. Computational mechanics is the discipline concerned

Computational Mechanics

with the use of computational methods and devices to study theoretical mechanics and to apply it to problems of interest in engineering and applied sciences. It is one of the most successful branches of computational science and has been instrumental in enriching our understanding of countless physical phenomena and in the study and design of untold thousands of engineering systems. It has affected virtually every aspect of human existence in the industrialized world and stands as one of the most important areas of both engineering science and science itself. Although it is not always the case, the term computational mechanics generally brings to mind the use of mathematical models involving partial differential equations. There are many exceptions to this rule, such as modern computational models of molecular and electronic systems, chemical reactions, and models of types of biological systems. Because of the more common use of the term, we shall limit this account of computational mechanics to the study of models primarily characterized by partial differential equations. Other types of models within this broad discipline are dealt with elsewhere in this volume. In particular, most of the subjects we target fall within the general area of computational fluid mechanics, computational solid mechanics, and computational electromagnetics characterized by continuum-phenomenological models. It is appropriate to comment on the strong connection between modern and classical mathematics and mathematical physics and the subject of theoretical mechanics and computational mechanics. The structure of the theory of partial differential equations is basically partitioned into areas motivated by longaccepted models of physical phenomena. The notion of the propagation of waves and signals cannot be understood without knowledge of the properties of hyperbolic systems and hyperbolic partial differential equations. The concept of diffusion, while prominent in the area of mechanics, thermodynamics, and molecular science, is intrinsically related to properties of parabolic equations, which dominate the literature in heat transfer and heat conduction. The study of the equilibrium of deformable bodies is deeply rooted in the theory of elliptic partial differential equations. Even the notion of signals, frequencies, and equilibrium states corresponding to energy levels cannot be appreciated without an understanding of the properties of eigenvalue problems for Hermitian

267

operators. Thus, computational mechanics and the mathematical foundations of partial differential equations are intrinsically interwoven into a fabric that has made the subject not only challenging but intellectually rich and enormously useful and important. The complete history of computational mechanics is difficult to trace. Certainly, the numerical algorithms developed by Newton himself which were applied to study dynamical events qualify as one of the earliest examples of computational mechanics, and some also give credit to Leibniz for discretizing the domain of solutions to ordinary differential equations in his study of the brachistochrone problem. In the 1930s and 1940s, the work of Sir Richard Southwell on the use of manually operated calculators to obtain solutions of finite-difference approximations of the equations of elasticity or potential flow certainty qualifies as a landmark in computational mechanics. But what we think of as computational mechanics today is widely associated with the use of digital computers to solve problems in mechanics. Here the work of John von Neumann and his colleagues in the 1940s on numerical solutions of problems in fluid mechanics was perhaps the beginning of computational fluid dynamics and was launched on versions of the earlier digital computers, which he also helped design. Certainly, one of the greatest events in the history of computational mechanics was the development of the finite element method. The first complete formulation of the method, together with significant applications solved on digital computers, appeared in a paper by John Turner, Ray Clough, Harold Martin, and L. J. Topp in 1956, although several of the underlying ideas were mentioned in the appendix of a 1940 paper by Richard Courant and many of the algorithms of implementation were described in work of John Argyris in the early 1950s. The explosion of literature on computational mechanics began in the mid-1960s, as the speed and capacity of computers reached a level at which important applications in science and engineering could be addressed. The mathematical foundations of the subject followed the development of numerical mathematics, involving mainly work on algorithms, until the mid-1960s and early 1970s, when the subject was broadened to encompass areas of approximation theory and the theory of partial differential equations. The era of the development of the mathematical foundations of computational mechanics began in the late 1960s

C

268

Computational Mechanics

and early 1970s and continues today. Its boundaries have expanded to new and challenging areas, including stochastic systems, optimization, inverse analysis, and applications in chemistry, quantum physics, biology, and materials science. As the speed and capacity of high-performance computers continue to increase and as new algorithms and methods emerge, computational mechanics continues to be an indispensible discipline within applied science and engineering and an area rich in challenges in computational and applied mathematics and computational science and engineering. Most of the subfields of computational mechanics are based on various discrete approximations of the conservation and balance laws governing thermomechanical and electromagnetic phenomena in material bodies. For instance, if the motion of a body B carries it from a reference configuration identified with a fixed region ˝0  0, then the dynamics appears as if it is generated by a continuous parameterized family of maps f ./ WD ' .T; /W X ! X . From a computational perspective, this latter approach is often more useful and thus we use this framework for most of our presentation. If the dynamics is generated by a partial or functional differential equation, then in general one cannot expect f W X ! X to be invertible. In practice this has limited conceptual consequences but can significantly increase the technical challenges; thus, we assume that f is a homeomorphism. A set S  X is invariant under f if f .S / D S . These are the fundamental objects of study. While general invariant sets are too complicated to be classified, there are well-understood invariant sets which can be used to describe many aspects of the dynamics. A point x 2 X is a fixed point if f .x/ D x. It is a periodic point if there exits N > 0 such that f N .x/ D x. The associated invariant set ff n .x/ j n D 1; : : : ; N g is called a periodic orbit. Similarly, x 2 X is a heteroclinic point if limn!˙1 f n .x/ D y ˙ where y ˙ are distinct fixed points. If y C D y  , then x is a homoclinic point. Again, the complete set ff n .x/ j n 2 Zg is a heteroclinic or homoclinic orbit. Because they are both mathematically tractable and arise naturally, invariant manifolds play an important role. For example, periodic orbits for flows form invariant circles and integrable Hamiltonian

289

systems give rise to invariant tori. If xN is a hyperbolic fixed point, that is the spectrum of Df .x/ N does not intersect the unit circle in the the complex plane, then N WD fx 2 X j limn!1 f n .x/ D xg N the sets W s .x/ u and W .x/ N WD fx 2 X j limn!1 f n .x/ D xg N are immersed manifolds called the stable and unstable manifolds, respectively. The concept of stable and unstable manifolds extends to hyperbolic invariant sets [39]. Cantor sets also play an important role. Subshifts on finite symbols arise as explicit examples of invariant sets with complicated dynamics. For a positive integer K, let † D fk j k D 0; : : : ; K  1gZ with the product topology, and consider the dynamical system generated by W † ! † given by .a/j D aj C1 . Observe that if ˚A is ˚a K  K matrix with 0;  1 entries and †A WD a D aj 2 † j Aaj ;aj C1 ¤ 0 , then †A is an invariant set for . If the spectral radius .A/ of the matrix A is greater than one, then the invariant set †A is said to be chaotic. In particular, it can be shown that †A contains infinitely many periodic, heteroclinic, and homoclinic orbits, and furthermore, one can impose a metric d on † compatible with the product topology such that given distinct elements a; b 2 †A there exists n 2 Z such that d. n .a/;  n .b//  1. Topological entropy provides a measure of how chaotic an invariant set is. In the case of subshift dynamics, the entropy is given by ln..A//. Given that the focus of dynamics is on invariant sets and their structure, the appropriate comparison of different dynamical systems is as follows. Two maps f W X ! X and gW Y ! Y generate topologically conjugate dynamical systems if there exists a homeomorphism hW X ! Y such that h ı f D g ı h. Returning to the context of a parameterized family of dynamical systems, 0 2 ƒ is a bifurcation point if for any neighborhood U of 0 there exists 1 2 U such that f1 is not conjugate to f0 , i.e., the set of invariant sets of f1 differs from that of f0 . Our understanding of bifurcations arises from normal forms, polynomial approximations of the dynamics from which one can extract the conjugacy classes of dynamics in a neighborhood of the bifurcation point. The presence of chaotic invariant sets has profound implications for computations. In particular, arbitrarily small perturbations, e.g., numerical errors, lead to globally distinct trajectories. Nevertheless for some chaotic systems, one can show that numerical trajectories are shadowed by true trajectories, that is, there are

C

290

true trajectories that lie within a given bound of the numerical trajectory. From the perspective of applications and computations, an even more profound realization is the fact that there exist parameterized families of dynamical systems for which the set of bifurcation points form a Cantor set of positive measure. This implies that invariant sets associated to the dynamics of the numerical scheme used for computations cannot be expected to converge to the invariant sets of the true dynamics.

A Posteriori Functional Analytic Methods Newton’s method is a classical tool of numerical analysis for finding approximate solutions to F .x/ D 0. The Newton-Kantorovich theorem provides sufficient a posteriori conditions to rigorously conclude the existence of a true solution within an explicit bound of the approximate solution. INTLAB is a Matlab toolbox that using interval arithmetic can rigorously carry out these types of computations for x 2 Rn [60, 61]. Since fixed points and periodic points of a map f W X ! X can be viewed as zeros of an appropriate function, this provides an archetypical approach to computational proofs in dynamics: establish the equivalence between an invariant set and a solution to an operator equation, develop an efficient numerical method for identifying an approximate solution, and prove a theorem – that can be verified by establishing explicit bounds – that guarantees a rigorous solution in a neighborhood of the approximate solution. Even in rather general settings, this philosophy is not new. As an example, observe that x.t/ is a -periodic solution of the differential equation xP D V .x/ if and only if x is a solution of R the operator equation ˆŒx D 0 where ˆŒx .t/ D 0 V Œx.t/ dt. Representing x in Fourier space and using theoretical and computer-assisted arguments to verify functional analytic bounds allows one to rigorously conclude the existence of the desired zero. This was done as early as 1963 [14]. By now there are a significant number of results of this nature, especially with regard to fixed points for PDEs [53, 57]. The field of computer-assisted proof in dynamics arguably came into its own through the study of the Feigenbaum conjecture; a large class of unimodal differentiable mappings W Œ1; 1 ! R exhibit an infinite sequence of period doubling bifurcations, and

Computational Proofs in Dynamics

the values of the bifurcation points are governed by a universal constant ı [34]. This conjecture is equivalent to the statement that the doubling operator T Œ .x/ D  a1  ı .ax/ has a hyperbolic fixed point N and that N has a the Frechet derivative at the fixed point DT Œ single unstable eigenvalue with value ı [18,19]. A good approximation of the fixed point and the unstable eigenvalue was determined using standard numerical methods. Newton-Kantorovich was then used to conclude the existence and bounds of a true fixed point and unstable eigenvalue, where the latter computations are done using upper and lower bounds to control for the errors arising from the finite dimensional truncation and the finite precision of the computer [45, 46]. Examples of invariant sets that can be formulated as the zero of a typically infinite-dimensional operator include stable and unstable manifolds of fixed points and equilibria [10, 11], invariant tori in Hamiltonian systems [28], hyperbolic invariant tori and their stable and unstable manifolds [38], existence of heteroclinic and homoclinic orbits [8, 9], and shadowing orbits for systems with exponential dichotomies [56]. Thus, for all these problems, there are numerical methods that can be used to find approximate solutions. Furthermore, for parameterized families continuation methods can be used to identify smooth branches of approximate zeros [43]. In tandem with nontrivial analytic estimates, this has been successfully exploited to obtain computational proofs in a variety of settings: universal properties of area-preserving maps [32], KAM semiconjugacies for elliptic fixed points [27], relativistic stability of matter against collapse of a many-body system in the Born-Oppenheimer approximation [33], computation of stable and unstable manifolds for differential equations [40, 70], existence of connecting orbits for differential equations and maps [21, 22, 70], existence of chaotic dynamics for maps and differential equations [6, 65, 68], equilibria and periodic solutions of PDEs [4, 5, 25], and efficient computation of one parameter branches of equilibria and periodic orbits for families of PDEs and FDEs [25, 35, 47, 68, 69].

A Posteriori Topological Methods An alternative approach to extracting the existence and structure of invariant sets is to localize them in phase space and then deduce their existence using a topological argument. The common element of this approach is

Computational Proofs in Dynamics

to replace the study of f W X ! X by that of an outer ! approximation, a multivalued map F W X  ! X whose images are compact sets that satisfy the property that for each x 2 X , f .x/ 2 int F .x/ where int denotes interior. This implies that precise information about the nonlinear dynamics is lost – at best one has information about neighborhoods of orbits – however, this is the maximal direct information one can expect using a numerical approximation. The central concept in this approach is the following. A compact set N  X is an isolating neighborhood under f if Inv.N; f /, the maximal invariant set in N under f , is contained in the interior of N . The theoretical underpinnings for these methods go back to [20, 31, 50], where, for example, the existence of the stable and unstable manifolds of a hyperbolic fixed point is proven by studying iterates of an isolating neighborhood using the contractive and expansive properties guaranteed by the hyperbolicity. The first rigorous computational proof using these types of ideas was the demonstration of the existence of chaotic dynamics in Hamiltonian systems [16]. To indicate the breadth of this approach, we provide a few examples of computational implementations. By definition a homoclinic point to a hyperbolic fixed point xN corresponds to an intersection point of the N and unstable W s .x/ N manifolds of x. N stable W s .x/ If this intersection is transverse, then there exists an invariant set which is conjugate to subshift dynamics with positive entropy [64]. This suggests the following computational strategies: find a hyperbolic fixed N and point x; N compute geometric enclosures of W s .x/ s W .x/; N verify the transverse intersection; and if one wants a lower bound on the associated entropy, use the identified homoclinic points to construct the appropriate subshift dynamics. Beginning with the work of [54], this approach has been applied repeatedly. The accuracy of the bound on entropy is limited by the N and W s .x/. N An efficient impleenclosure of W s .x/ mentation of higher-order Taylor methods [7] that leads to high precision outer approximations was used to attain the best current lower bounds on the entropy of the Henon map at the classical parameter values [55]. A constraint for this strategy is the rapid growth in cost of approximating invariant manifolds as a function of dimension. This can be avoided by isolating only the invariant set of interest using covering relations, parallelograms which are properly aligned under the differential of the map. While applications of this idea to planar maps appear as early as [63],

291

a general theory along with efficient numerical implementation has been developed by Capi´nski and Zgliczy´nski [13], Zgliczy´nski and Gidea [75], and Gidea and Zgliczy´nski [37]. In conjunction with rigorous tools for integrating differential equations [12], this method has found wide application including proofs of the existence of heteroclinic and homoclinic connecting orbits and chaotic dynamics in celestial mechanics [72,73], uniformly hyperbolic invariant sets for differential equations [71], and particular orbits in PDEs [74]. These techniques can also be applied to detect complicated bifurcations. A family of ODEs in R3 exhibits a cocooning cascade of heteroclinic tangencies (CCHT) centered at  , if there is a closed solid torus T , equilibria x ˙ 62 T , and a monotone infinite sequence of parameters n converging to  for which Wu .x C / and Ws .x  / intersect tangentially in T and the length of the corresponding heteroclinic orbit within T becomes unbounded as n tends to infinity. For systems with appropriate symmetry, a CCHT can be characterized in terms of the topologically transverse intersection of stable and unstable manifolds between the fixed points and a periodic orbit [30] and therefore can be detected using the abovementioned techniques. This was used in [44] to obtain tight bounds on parameter values at which the Michelson equation exhibits a CCHT. Identifying structurally stable parameter values is important. This is associated with hyperbolic invariant sets. Let X be a manifold. If N  X is an isolating neighborhood and Inv.N; f / is chain recurrent, then to prove that Inv.N; f / is hyperbolic, it is sufficient to show that there is an isolating neighborhood NN  TX , the tangent bundle, under Tf W TX ! TX such that Inv.NN ; Tf / is the zero section over N [17, 62]. This was used by Arai [1] to determine lower bounds on the set of parameter values for which the Henon map is hyperbolic (see Fig. 1). To efficiently identify isolating neighborhoods, it helps to work with a special class of outer approximations. For f W Rn ! Rn let X denote a cubical grid which forms a cover for a compact set X  Rn . For each  cube  2 X , let F ./  X such that f ./  int [ 0 2F ./  0 . Observe that F can be viewed both as an outer approximation and as a directed graph. The latter perspective is useful since it suggests the use of efficient algorithms from computer science. The construction X and the search for isolating neighborhoods

C

292

Computational Proofs in Dynamics 1

0.8 0.6 0.4

b

0.2 0

−0.2 −0.4 −0.6 −0.8 −1

−1

0

1

2

3

4

5

6

Conceptually, partitioning a posteriori methods of computational proofs in dynamics into functional analytic and topological methods is useful, but for applications a combination of these tools is often desirable. For example, the proof of the existence of the Lorenz attractor at the classic parameter values [67] is based on a posteriori topological arguments. However, the construction of the rigorous numerical outer approximation exploits a high-order normal form computed at the origin, and a-posteriori functional analytic tools are used in order to obtain rigorous bounds on truncation errors for the normal form and its derivative.

a

Computational Proofs in Dynamics, Fig. 1 Black region indicates parameter values at which the Henon map is hyperbolic

Global Topological Methods

can be done in an adaptive manner which in some settings implies that the computational cost is determined by the dimension of the invariant set as opposed to the ambient space Rn [29, 41]. A representative of any isolating neighborhood can be identified by this process if the grid and the outer approximation are computed with sufficiently fine resolution [42]. Given an isolating neighborhood N , the Conley index can be used to characterize the structure of Inv.N; f /. To compute this one needs to construct a pair of compact sets P D .P1 ; P0 /, called an index pair, on which f induces a continuous function fP  W .P1 =P0 ; ŒP0 / ! .P1 =P0 ; ŒP0 / on the quotient space [58]. The induced map on homology fP  W H .P1 =P0 ; ŒP0 / ! H .P1 =P0 ; ŒP0 / is a representative for the Conley index. Given an outer ! approximation F W X  ! X there are efficient directed graph algorithms to construct index pairs. Furthermore, fP  can be computed using F [15, 41, 52]. The first nontrivial computational use of these ideas was a proof that the Lorenz equations exhibit chaotic subshift dynamics [51]. Since then Conley index techniques have been applied in the context of rigorous computations to a variety of problems concerning the existence and structure of invariant sets including chaotic dynamics in the Henon map [26] and the infinite dimensional Kot-Shaffer map [23], homoclinic tangencies in the H´enon map [2], global dynamics of variational PDEs [24, 49], and chaotic dynamics in fast-slow systems [36].

The underlying strategy for a posteriori analytic and topological techniques is to identify a priori a class of invariant sets, numerically approximate and then rigorously verify the existence. Since it is impossible to enumerate all invariant sets, an algorithmic analysis of arbitrary dynamical systems using a classification based on structural stability is impossible. An alternative approach based on using isolating neighborhoods to characterize the objects of interest in dynamical systems [20] appears to be well suited for rigorous systematic computational exploration of global dynamics. To provide partial justification for this method, we return to the setting of an outer approximation F of f defined on a cubical grid X covering a compact set X 2 Rn . Let S WD Inv.X; f /. Viewing F as a directed graph, there exist efficient algorithms for identifying the strongly connected path components fM.p/  X j p 2 .P; 0 and set F  ./ WD f 0 2 X  j f ./ \  0 ¤ ;g. As indicated above this defines fM.p  /  X  j p  g f2 .P ; N , so that the solution  can be then expressed as  D 0 ˚   , that is,   is an orthogonal correction to 0 . CC is formulated in terms of excitation operators

r:s $ $ Vp;q ap aq as ar ;

;:::;ar D Xia11    Xiar r D aa$1 ai1    aa$r air ; X WD Xia11;:::;i r

p;q;r;s p

(2)

p;q

with the one- and two-electron integrals hr , Vr;s . For more detailed information on second quantization formulation of electronic structure problems, compare, for example, [4]. In practice, the basis B (and thus B) is substituted by a finite basis set Bd , inducing a Galerkin basis Bd for a trial space contained in H1 . A Galerkin method for (1) with Bd as basis for the ansatz space is termed Full-CI in quantum chemistry. Because Bd usually contains far too many functions (their number scaling exponentially with the size of Bd ), a subset BD of Bd is chosen for discretization. Unfortunately, traditional restricted CI-methods, like the CISD method described in the entry  Post-Hartree-Fock Methods and Excited States Modeling, thereby lose size-consistency, meaning that for a system AB consisting of two independent subsystems A and B, the energy of AB as computed by the truncated CI model is no longer the sum of the energies of A and B. In practice, this leads to inaccurate computations with a relative error increasing with the size of the system. Therefore, size-consistency and the related property of size-extensivity are essential properties of quantum chemical methods (see, e.g.,

where r  N , i1 < : : : < ir  N , N C 1  a1 < : : : < ar . These X can also be characterized by their action on the basis functions  Œp1 ; : : : ; pN 2 B: If fp1 ; : : : ; pN g contains all indices i1 ; : : : ; ir , the operator replaces them (up to a sign factor ˙1) by the or;:::;ar  Œp1 ; : : : ; pN D bitals a1 ; : : : ; ar ; otherwise, Xia11;:::;i r 0: Indexing the set of all excitation operators by a set M, we have in particular that B D f0 g [ f j D X 0 ;  2 Mg. The convention that 'i ?'a implies two essential properties, namely, excitation operators commute (X X  X X D 0), and are nilpotent, that is, X2 D 0. Note that these only hold within the singlereference ansatz described here. Exponential ansatz. The cluster operator of a coefficient vector t 2 `2 .M/, t D .t /2M is defined P replaces as T .t/ D 2M t X . The CC method P the linear parametrization  D 0 ˚ 2M t X 0 (of functions normalized by h0 ;  i D 1) by an exponential (or multiplicative) parametrization  D e T .t/ 0 D e .

P

2M t X /

0 D ˘2M .1Ct X /0 :

Coupled-Cluster Methods

317

Choosing a suitable coefficient space V  `2 .M/ reflecting the H1 -regularity of the solution, it can be shown that there is a one-to-one correspondence between the sets f0 C   j 0 ?  2 H1 g; f0 C T .t/0 j t 2 Vg, and fe T .t/ 0 j t 2 Vg: Coupled-Cluster equations. The latter exponential representation of all possible solutions 0 C   is used to reformulate (1) as the set of unlinked Full-CC equations for a coefficient vector t 2 V, h˚ ; .H  E/e

T .t/

0 i D 0; for all ˚ 2 B:

Inserting e T .t/ yields the equivalent linked Full-CC equations

the number r of one-electron functions in which  differs from the reference 0 , see, e.g., [4]). For example, including at most twofold excitations (i.e., r  2 in (2)) gives the common CCSD (CC Singles/Doubles) method.

C Numerical Treatment of the CC Equations The numerical treatment of the CC ansatz consists mainly in the computation of a solution of the nonlinear equation f.tD / D 0. This is usually performed by quasi-Newton methods, .nC1/

h˚ ; e

T

tD

He 0 i D 0 for all  2 M; T

E  D h0 ; He T 0 i D h0 ; e T He T 0 i: For finite resp. infinite underlying one-particle basis B resp. Bd , both of these two sets of equations are equivalent to the Schr¨odinger equation (1) resp. the linear Full-CI ansatz. For two subsystems A and B and corresponding excitation operators TA and TB , the exponential ansatz admits for the simple factorization e TA CTB D e TA e TB . Therefore, aside from other advantages, the CC ansatz maintains the property of sizeconsistency. The restriction to a feasible basis set BD  B corresponds in the linked formulation to a Galerkin procedure for the nonlinear function   f W V ! V0 ; f .t/ WD h˛ ; e T .t/ He T .t/ 0 i ˛2M ;

the roots t of which correspond to solutions e 0 of the original Schr¨odinger equation. This gives the projected CC equations hf .tD /; vD i D 0 for all vD 2 VD , where VD D l2 .MD / is the chosen coefficient Galerkin space, indexed by a subset MD of M. This is a nonlinear equation for the Galerkin discretization f of the function f :

.n/

.n/

.0/

tD D 0

(5)

with an approximate Jacobian F given by the Fock matrix, see below. On top of this method, it is standard to use the DIIS method (“direct inversion in the iterative subspace”), for acceleration of convergence. Convergence of these iteration techniques is backed by the theoretical results detailed later. We note that the widely used Møller–Plesset second-order perturbation computation (MP2), being the simplest post-Hartree– Fock or wave function method, is obtained by terminating (5) after the first iteration. For application of the iteration (5), the discrete CC function (3) has to be evaluated. Using the properties of the algebra of annihilation and creation operators, it can be shown that for the linked CC equation, the Baker–Campbell–Hausdorff expansion terminates, that is,

(3) T .t /

D tD  F1 f.tD /;

e T He T D

1 4 X X 1 1 ŒH; T .n/ D ŒH; T .n/ nŠ nŠ nD0 nD0

with the n-fold commutators ŒA; T .0/ WD A; ŒA; T .1/ WD AT  TA, ŒA; T .n/ WD ŒŒA; T .n1/ ; T . It is common use to decompose the Hamiltonian into one- and two-body operators H D F C U , where F normally is the Fock operator from the   preliminary self-consistent Hartree–Fock (or Kohn– f.tD / WD h˛ ; e T .tD / He T .tD / 0 i ˛2MD D 0: Sham) calculation, and where the one-particle basis (4) set 'p consists of the eigenfunctions of the discrete canonical Hartree–Fock (or Kohn–Sham) equations Usually, the Galerkin space VD is chosen based on the with corresponding eigenvalues p . The CC equations so-called excitation level r of the basis functions (i.e., (4) then read

318

Coupled-Cluster Methods

for all  2 M, while the derivatives with respect to  yield exactly the CC equations f .t/ D 0 providing the exact CC wave function  D e T .t/ 0 . Afterward, (6) the Lagrange multiplier  can be computed from (8). for all  2 MD ; Introducing the functions with Pr the Fock matrix F D diag. / D diag X   e WD  e.t; / D 0 C   e T .t/  D e T .t/ lD1 .al il / . The commutators are then evaluated within the framework of second quantization by  X using Wick’s theorem and diagrammatic techniques, .1 C  X /0 ;  .t/ D e T .t/ 0 ; resulting in an explicit representation of f as a fourth order polynomial in the coefficients t , see [3] for a comprehensible derivation. where T  is the adjoint of the operator T , there holds Complexity. The most common variants of CC e.t; /; H  .t/i together with the duality L.t; / D h methods are the CCSD (see above) and for even e h ;  i D 1. As an important consequence, one can higher accuracy the CCSD(T) method, see [1]. In compute derivatives of energy with respect to certhe latter, often termed the “golden standard of tain parameters, for example, forces, by the Hellman– quantum chemistry,” a CCSD method is converged Feynman theorem. If the Hamiltonian depends on a at first, scaling with the number N of electrons as e; .@! H / i parameter !, H D H.!/, then @! E D h N 6 ; then, the result is enhanced by treating triple holds for the respective derivatives with respect to excitations perturbatively by one step of N 7 cost in !. Accordingly discrete equations are obtained by rethe CCSDT basis set. While the computational cost for placing V; M by their discrete counterparts VD ; MD : calculating small- to medium-sized molecules stays The above Lagrangian has been introduced in quantum reasonable, it is thereby possible to obtain results that chemistry by Monkhorst; the formalism has been exlie within the error bars of corresponding practical tended in [7] for a linear, size-consistent CC response experiments. theory. 4 X 1 F; t  h ; ŒU; T .n/ 0 i D 0; nŠ nD0

Lagrange Formulation and Gradients

Theoretical Results: Convergence and The CC method is not variational, which is a certain Error Estimates disadvantage of the method. For instance, the computed CC energy is no longer an upper bound for the It has been shown recently in [8] that if the reference exact energy. The following duality concept is helpful 0 is sufficiently close to an exact wave function  belonging to a non-degenerate ground state and in this context: Introducing a formal Lagrangian if VD is sufficiently large, the discrete CC equation X (4) locally permits a unique solution tD . If the basis L.t; / W D h0 ; He T .t/ 0 i C  h ; e T .t/ set size is increased, the solutions tD converge quasi2M optimally in the Sobolev H 1 -norm toward a vector He T .t/ 0 i; (7) t 2 V parameterizing the exact wave function  D e T .t/ 0 . The involved constant (and therefore the qualthe CC ground state is E D inft2V sup2V L.t; /: The ity of approximation) depends on the gap between corresponding stationary condition with respect to t lowest and second lowest eigenvalues and on k0  reads  kH1 . The above assumptions and restrictions mean that CC works well in the regime of dynamical or X @L weak correlation, which is in agreement with practical .t; / D h0 ; HX e T .t/ 0 i C  h ; e T .t/ @t experience. 2M The error jE.t/  E.tD /j of a discrete groundT .t/ 0 0 ŒH; X e 0 i D E .t/ C h; f .t/i D 0 state energy E.tD / computed on Vd can be bounded (8) using the dual weighted residual approach of Ran-

Coupled-Cluster Methods

319

nacher: Denoting by .t; / the stationary points of the Lagrangian (7) belonging to the full energy E  , and by tD the solution of the corresponding discretized equation f.tD / D 0; the error of the energy can be bounded by jE.t/  E.tD /j .



d.t; Vd / C d.; Vd /

2

and thus depends quadratically on the distance of the approximation subspace to the primal and dual solutions t;  in V. Note that these estimates are a generalization of error bounds for variational methods, which allow for error bounds depending solely on d.t; Vd /2 , and an improvement of the error estimates given in [6]. Roughly speaking, this shows that CC shares the favorable convergence behavior of the CI methods, while being superior due to the size-consistency of the CC approximation.

approximation (RPA) or electron pair methods like CEPA methods may be derived this way, and these approaches may serve as starting point for developing efficient numerical methods providing almost CCSD accuracy within a much lower computational expense. Multi-reference CC. Multi-reference methods [1] aim at situations where the reference determinant is not close to the true wave function, so that classical CC methods fail. Unfortunately, multi-reference CC in its present stage is much more complicated, less developed, and computationally often prohibitively more expensive than usual Coupled Cluster.

Cross-References  Density Functional Theory  Hartree–Fock Type Methods  Post-Hartree-Fock Methods and Excited States Mod-

eling  Schr¨odinger Equation for Chemistry

Outlook: Enhancements and Simplifications of the Canonical CC Method

References

To reduce the complexity or to remedy other weak- 1. Bartlett, R.J., Musial, M.: Coupled-cluster theory in quantum chemistry. Rev. Mod. Phys. 79, 291 (2007) nesses of the method, various variants of the above 2. C´ ˇ ızˇ ek, J.: Origins of coupled cluster technique for atoms and standard CC method have been proposed. We only give molecules. Theor. Chim. Acta 80, 91 (1991) 3. Crawford, T.D., Schaeffer, H.F., III.: An introduction to a short, incomplete overview. coupled cluster theory for computational chemists. Rev. Local CC methods. These techniques allow to accelComput. Chem. 14, 33 (2000) erate the CCSD computations drastically by utilizing 4. Helgaker, T., Jørgensen, P., Olsen, J.: Molecular localized basis functions, for which the two-electron Electronic-Structure Theory. Wiley, Chichester/New York RR p;q (2000) .'p .xi /'q .xj /'r .xi /'s .xj //=jri  integrals Vr;s D rj jdxi dxj decay with the third power of the distance 5. Klopper, W., Manby, F.R., Ten-no, S., Vallev, E.F.: R12 methods in explicitly correlated molecular structure theory. of the support of 'p 'q and 'r 's . Integrals over distant Int. Rev. Phys. Chem. 25, 427 (2006) pairs can thus be neglected. 6. Kutzelnigg, W.: Error analysis and improvement of coupled cluster theory. Theor. Chim. Acta 80, 349 (1991) Explicitly correlated CC methods. Fast convergence 7. Pedersen, T.B., Koch, H., H¨attig, C.: Gauge invariant of CC to the full basis set limit is hampered by the coupled cluster response theory. J. Chem. Phys. 100(17), electron–electron cusp, caused by discontinuous higher 8318–8327 (1999) derivatives of the wave function where the coordinates 8. Rohwedder, T., Schneider, R.: Error estimates for the Coupled Cluster Method, ESAIM Math. Model. Numer. Anal. of two particles coincide. Explicit incorporation of the 47(6), 1553–1582 (2013) electron–electron cusp by an r12  or f12  ansatz [5] 9. Sch¨utz, M., Werner, H.-J.: Low-order scaling local correlacan improve convergence significantly. Recent density tion methods. IV. Linear scaling coupled cluster (LCCSD). fitting techniques (see [10]) herein allow the efficient J. Chem. Phys. 114, 661 (2000) 10. Sherrill, C.D.: Frontiers in electronic structure theory. treatment of the arising three-body integrals. J. Chem. Phys. 132, 110902 (2010) Simplified approaches. The CCSD equation can 11. Yserentant, H.: Regularity and Approximability of Elecbe simplified by linearization and/or by leaving out tronic Wave Functions. Lecture Notes in Mathematics Secertain terms in the CC equations. The random phase ries, vol. 2000. Springer, Heidelberg/New York (2010)

C

320

Curvelets

contain complete information about f (supposed that fO./ vanishes for jj < 2), i.e., there is a reproducing formula of the form

Curvelets Gerlind Plonka1 and Jianwei Ma2 1 Institute for Numerical and Applied Mathematics, University of G¨ottingen, G¨ottingen, Germany 2 Department of Mathematics, Harbin Institute of Technology, Harbin, China

f .x/ D

1 .ln 2/

R 2 R 0

R1

R2

0

hf; 'a;b;! i 'a;b;! .x/

da db d! : a3=2 a1=2 a

Discretization and Algorithm Short Definition 2

2

Curvelets are highly anisotropic functions in L .R / with compact support in angular wedges in frequency domain and with effective support shaped according to the parabolic scaling principle length2 width in spatial domain. Generalizing the idea of wavelets, curvelets are ideally adapted to sparsely represent twodimensional functions (images) that are piecewise smooth with discontinuities along smooth curves with bounded curvature [2]. The curvelet transform is a nonadaptive multiscale transform with strong directional selectivity [3].

Curvelet Transform The curvelet elements are obtained by rotation, translation, and parabolic dilation of a suitable basic curvelet function ' with compact support in frequency domain bounded away from zero; see Fig. 1 (left). For the scale a 2 .0; 1 , the location b 2 R2 , and the orientation ! 2 Œ0; 2 /, they have the form 'a;b;! .x/ D a3=4 '.Da R! .x  b//;

Restricting to dilations aj D 2j , j D 0; 1; : : :, , l D 0; 1; : : : ; Nj 1; Nj D rotation angles !j;l D 2 l Nj 4  2bj=2c , and translations bk D R!1j;l .Daj k/, k 2 Z, the collection .'j;l;k / with 'j;l;k WD 'a ;b j;l ;! forms j k j;l (together with a suitable low-pass function) a Parseval frame of L2 .R2 /, and each function f 2 L2 .R2 / can be represented as j;l

f D

X

hf; 'j;l;k i'j;l;k

j;l;k

P 2 with kf k2L2 D j;l;k jhf; 'j;l;k ij . In particular, the curvelets 'j;l;k form a tiling of the frequency domain, where 'Oj;l;k has its essential support in an angular wedge; see Fig.1 (right). In order to derive a fast curvelet transform, one usually transfers the polar tiling of the frequency domain into a pseudo-polar tiling based on concentric squares and shears. The fast curvelet transform is based on the fast Fourier transform [3]. It is freely available under http://www. curvelet.org. The curvelet transform has been also generalized to three dimensions [3].

x 2 R2 (1)

Applications

with the dilation matrix Da D diag . a1 ; p1a / and with the rotation matrix R! defined by the angle !. The Curvelets have been used in various applications in imcontinuous curvelet transform f of f 2 L2 .R2 / is age processing as image denoising, sparsity-promoting given as regularization in image reconstruction, contrast enhancement, and morphological component separation Z [4, 6, 7]. Further applications include seismic explof .a; b; !/ WD hf; 'a;b;! i D f .x/'a;b;! .x/ dx: R2 ration, fluid mechanics, solving of PDEs, and compressed sensing [5,6]. Moreover, the curvelet transform Observe that since the scale a is bounded from above is also of theoretical interest in physics; it accurately by a D 1, low-frequency functions are not contained in models the geometry of wave propagation and sparsely the system in (1). The curvelet coefficients hf; 'a;b;! i represents Fourier integral operators [1].

Curvelets

321

1 0.8

C

0.6 0.4 0.2 0 2 1

2 1.5

0

1

−1 −2

0.5 0

Curvelets, Fig. 1 Example of a basic curvelet ' D '0;0;0 in frequency domain (left) and tiling of the frequency domain into wedges that determine the support of 'j;l;k (right) (The figures are taken from [6])

References 1. Cand`es, E., Demanet, L.: Curvelets and Fourier integral operators. C R Math. 336(5), 395–398 (2003) 2. Cand`es, E., Donoho, D.: New tight frames of curvelets and optimal representations of objects with piecewise C 2 singularities. Comm. Pure Appl. Math. 57, 219–266 (2004) 3. Cand`es, E., Demanet, L., Donoho, D., Ying, L.: Fast discrete curvelet transforms. Multiscale Model. Simul. 5(3), 861–899 (2006)

4. Fadili, M.J., Starck, J.L.: Curvelets and ridgelets. In: Meyers, R. (ed.) Encyclopedia of Complexity and Systems Science, vol. 3, pp. 1718–1738. Springer, New York (2009) 5. Ma, J., Plonka, G.: Computing with curvelets: from image processing to turbulent flows. Comput. Sci. Eng. 11(2), 72– 80 (2009) 6. Ma, J., Plonka, G.: The curvelet transform. IEEE Signal Process. Mag. 27(2), 118–133 (2010) 7. Starck, J.L., Cand`es, E., Donoho, D.: The curvelet transform for image denoising. IEEE Trans. Image Process. 11(6), 670– 684 (2002)

D

Defect Correction Methods Winfried Auzinger Institute for Analysis und Scientific Computing, Technische Universit¨at Wien, Wien, Austria

Synonyms Boundary value problem (BVP); Correction scheme (CS); Defect correction, DeC, deferred correction; Defect, residual; Finite element method (FEM); Full approximation scheme (FAS); Initial value problem (IVP); Neighboring problem (NP); Ordinary differential equation (ODE); Original problem (OP); Partial differential equation (PDE); Truncation error (TE); [Implicit] Runge–Kutta ([I]RK)

algorithmic components is not always straightforward, and we discuss some of the relevant issues. There are many versions and application areas with various pros and cons, for which we give an overview in the final sections. Applications to partial differential equations (PDEs) in a variational context are briefly discussed. In this entry, we are not specifying all algorithmic components in detail, e.g., concerning the required interpolation and quadrature processes. These are numerical standard procedures which are easy to understand and to realize. Also, an exhaustive survey of the available literature on the topic is not provided here. A word on notation: We use upper indices for iteration counts and lower indices for numbering along discrete grids.

Introduction Underlying Concepts and General Defect correction (DeC) methods (also: “deferred cor- Algorithmic Templates rection methods”) are based on a particular way to estimate local or global errors, especially for differential and integral equations. The use of simple and stable integration schemes in combination with defect (residual) evaluation leads to computable error estimates and, in an iterative fashion, yields improved numerical solutions. In the first part of this entry, the underlying principle is motivated and described in a general setting, with focus on the main ideas and algorithmic templates. In the sequel, we consider its application to ordinary differential equations in more detail. The proper choice of

Many iterative numerical algorithms are based on the following principle: • Compute the residual, or “defect,” d i of a current iterate ui . • Backsolve for a correction "i using an approximate solver. • Apply the correction to obtain the next iterate ui C1 WD ui  "i . Stationary iterative methods for linear systems of equations are the classical examples (cf., e.g., [14]). For starting our considerations, we think of a given,

© Springer-Verlag Berlin Heidelberg 2015 B. Engquist (ed.), Encyclopedia of Applied and Computational Mathematics, DOI 10.1007/978-3-540-70529-1

324

Defect Correction Methods

original problem [OP] in the form of a system of reasonable linear or nonlinear approximation FQ i  F nonlinear equations: to be used for iteratively solving [OP]. In view of typical applications of DeC methods, we assume that FQ  F is kept fixed but is admitted to be [OP] F .u/ D 0; with exact solution u D u . (1) nonlinear. Let us first consider a single step of such a procedure for the purpose of estimating the error of a More generally, the mapping F may also represent a given approximation u0 to u . Consider the defect (2), (sufficiently well-behaved) operator between infinite and associate u0 ; d 0 with the so-called neighboring dimensional spaces; the general considerations below problem related to (1), equally apply in such a more general setting. [NP] F .u/ D d 0 ; with exact solution u D u0 . Example: Local Linearization and Newton (4) Iteration principles, (A) and (B), for Let an initial approximation u0 to u be given, with the We invoke two heuristic estimating the error of u0 . Originally introduced in [17] (see also [7]), these are based on the idea that [NP] may (2) defect d 0 WD F .u0 / of u0 , be considered to be closely related to [OP], provided d 0 is small enough. the amount by which F .u0 / fails approximate (A) Let uQ and uQ 0 be the solutions of FQ .u/ D 0 and 0 D F .u /. Replace F .u/ by its local linearization FQ .u/ D d 0 , respectively; we assume that these FQ 0 .u/ WD F .u0 / C DF .u0 / .u  u0 /  F .u/, and solve can be formed at low computational cost. Confor the new approximation u1 , i.e., sidering the original and neighboring problem together with their approximations, FQ 0 .u1 / D 0 , DF .u0 /.u0  u1 / D d 0 : [OP]: F .u / D 0 [NP]: F .u0 / D d 0 Iteration leads to the classical Newton method, FQ .Qu/ D 0 FQ .Qu0 / D d 0 FQ i .ui C1 / D 0

,

i DF .ui /.u ui C1 „ ƒ‚ …/

suggests the approximate identity

DW "i

D F .ui / ; „ƒ‚…

i D 0; 1; 2; : : :

(3)

DW d i

based on the local linearizations FQ i .u/ D F .ui / C DF .ui / .u  ui /. Due to the affine nature of the FQ i .u/, each step (3) takes the form of a linear system for the Newton correction "i WD ui  ui C1 with defect d i D F .ui / on the right-hand side, such that ui C1 D ui  "i . The correction "i is an approximation for the “exact correction,” the error e i D ui  u . Simplified, “quasi-Newton” schemes work in a similar way, but with approximations J i  DF .ui / which may also be kept (partially) fixed, e.g., J i  DF .u0 /. Templates for Error Estimation Based on Nonlinear Approximation [Quasi-] Newton iteration is rather special concerning the choice of the FQ i in form of local affine approximations to F . More generally, we may consider any

uQ 0  uQ  u0  u :

(5)

This leads to the defect-based error estimator "0 WD uQ 0  uQ (6) as a computable estimate for the error e 0 WD u0  u . We can use it to obtain an updated approximation u1 in the form u1 WD u0  "0 D u0  .Qu0  uQ /: (B) Consider the truncation error (TE) t  WD FQ .u /, the amount by which u fails to satisfy the approximate equation FQ .u/ D 0. With dQ 0 WD FQ .u0 /, considering the approximate identity FQ .u /  FQ .u0 /  F .u /  F .u0 /; i.e., t   dQ 0  d 0

Defect Correction Methods

325

suggests to choose the defect-based TE estimator  0 WD dQ 0  d 0 D .FQ  F /.u0 / (7) as a computable estimate for the TE. Note that d 0 D F .u /  d 0 is the TE of u with respect to [NP]. In the case u0 D uQ , i.e., FQ .u0 / D 0, we have  0 D d 0  t  . We can use  0 to obtain an updated approximation u1 as the solution of FQ .u1 / D  0 ;

(8)

which also provides an estimate for the error: "0 WD u0  u1  u0  u D e 0 . Equation (8) can also be written in terms of the error estimate as (9) FQ .u0  "0 / D  0 ;

For i D 0; 1; 2; : : :: • Compute d i WD F .ui / • Solve FQ .Qui / D d i • Update ui C1 WD ui  .Qui  uQ / The solution uQ of FQ .Qu/ D 0 is required in the initialization step, and it is usually natural to choose u0 D uQ . The corrections uQ  ui play the role of successive estimates for the errors e i D ui  u . IDeC (B) : Choose an initial iterate u0 , and let D 1 WD FQ .u0 /. For i D 0; 1; 2; : : :: • Compute d i WD F .ui / • Update D i WD D i 1 C d i • Solve FQ .ui C1 / D D i Again it is natural to choose u0 D uQ , such that dQ 0 D D 1 D 0. Then the D i are simply accumulated defects, D i D d 0 C: : :Cd i , and the D i play the role of successive approximations for the TE t  D FQ .u /.

Q 0  e 0 / D t  . Remarks: approximating the error equation F.u Q If F .u/ D P u  c is an affine mapping, it is easy to • Nonlinear IDeC is sometimes called a full approximation scheme (FAS), where we directly solve check that (A) and (B) result in the same error estimate 0 for an approximation in each step. If FQ is affine, " , which can be directly obtained as the solution of the IDeC (A) and IDeC (B) are again equivalent and correction scheme (CS) can be reformulated as a CS in terms of linear (10) P "0 D d 0 ; backsolving steps for the correction "i D uQ i  uQ , as in (10). similar to (3), and the corresponding TE estimate is • IDeC (B) can also be rewritten in the spirit of (9).  0 D .P u0  c/  d 0 . • Note that u is a fixed point of an IDeC iteration In general, (A) and (B) are not equivalent. The since d  WD F .u / D 0. computational effort amounts to: For systems of algebraic equations, choosing FQ (A): Forming the defect d 0 , and solving two approxi- to be nonlinear is usually not very relevant from a mate problems to construct the error estimate "0 practical point of view. Rather, such a procedure turns (B): Forming the defect d 0 , and also dQ 0 , to obtain out to be useful in a more general context, where F the TE estimate  0 , and solving one approximate represents an operator between function spaces (typiproblem to obtain the error estimate "0 cally a differential or integral operator), and where FQ is a discretization of F . This leads us to the class of Iterated Defect Correction (IDeC) DeC methods for differential or integral equations. Both approaches (A) and (B) are designed for a posteriori error estimation, and they can also be used to design iterative solution algorithms, involving updated Application to Ordinary Differential versions of [NP] in the course of the iteration. This Equations (ODEs) leads in a straightforward way to two alternative versions the method of Iterated Defect Correction We mainly focus on IDeC (A), the “classical” IDeC (IDeC). method originally due to [18]. IDeC (B) can be realized IDeC (A) : Solve FQ .Qu/ D 0 and choose an initial in a similar way, and we will remark on this where appropriate. iterate u0 .

D

326

Defect Correction Methods

A Basic Version: IDeC (A) Based on Forward [NP] u0 .x/ D f .x; u.x// C d 0 .x/; u.a/ D ˛: Euler (16) Let us identify the “original problem” F .u/ D 0 with We now consider a correction step u0 7! u1 of type (A), an initial value problem (IVP) for a system of n ODEs, u0 .x/ D f .x; u.x//;

u.a/ D ˛;

(11)

Solve FQ .Qu0 / D d 0 ; followed by

u1 .x/ WD u0 .x/  .Qu0  u0 /.x/:

with exact solution u .x/ 2 Rn . This means that our original problem is given by This means that uQ 0 2 Pp is to be understood as the interpolant of the discrete values UQ j0 obtained by the [OP] F .u/.x/ WD u0 .x/  f .x; u.x// D 0: (12) solution of UQ j0C1  UQ j0 More precisely, the underlying function spaces and the D f .xj ; UQ j0 / C d 0 .xj /; initial condition u.a/ D ˛ are part of the complete h problem specification. j D 0; 1; 2; : : : ; UQ 0 D a; Furthermore, we identify the problem FQ .u/ D 0 with a discretization scheme for (11); at the moment we assume that a constant stepsize h is used, with which is the forward Euler approximation of (16), discrete grid points xj D a C j h; j D 0; 1; 2; : : :. involving pointwise defect evaluation at the grid Consider, for instance, the first-order accurate forward points xj . According to our general characterization of Euler scheme IDeC (A), this process is to be continued to obtain further iterates ui .x/. If we use m IDeC steps in the first Uj0C1  Uj0 D f .xj ; Uj0 /; subinterval I1 D Œa; aCp h, we can restart the process h at the starting point a C p h of the second subinterval j D 0; 1; 2; : : : ; U00 D ˛; (13) I , with the new initial value u.aCp h/ D um .aCp h/. 2 This is called local, or active mode. Alternatively, one and associate it with the operator FQ acting on may integrate with forward Euler over a longer interval continuous functions u satisfying the initial condition I encompassing several of the Ik and perform IDeC u.a/ D ˛, on I , where each individual ui .x/ is forwarded over the complete interval. This is called global or passive mode. u.x /  u.x / j C1 j  f .xj ; u.xj // D 0: FQ .u/.xj / WD h (14) Remarks: • In general, the exact solution u is not in the scope Choose a continuous function u0 .x/ interpolating the of the iteration, since the ui live in the space Pp . Uj0 at the grid points xj . The standard choice is a But there is a fixed point uO 2 Pp related to u : It continuous piecewise polynomial interpolant of degree is characterized by the property dO WD F .Ou/ D 0, p over p C 1 successive grid points, i.e., piecewise i.e., uO 0 .xj / D f .xj ; uO .xj // for all j . (Technical interpolation over subintervals Ik of length p h. In the detail: Since we are using the forward Euler scheme corresponding piecewise-polynomial space Pp , u0 .x/ Uj 7! Uj C1 evaluating f and the d i at x D xj , is the solution of FQ .u/ D 0. The defect d 0 WD F .u0 / is for u 2 P and x an initial point of a subinterval well defined, Ik the derivative u0 .x/ is to be understood as the right-hand limit.) This means that uO is a so-called d 0 .x/ D F .u0 /.x/ D .u0 /0 .x/  f .x; u0 .x//; (15) collocation polynomial, and IDeC can be regarded as an iterative method to approximate collocation solutions. In fact, this means that, instead of (12), and u0 .x/ is the exact solution of the neighboring IVP the system of collocation equations

Defect Correction Methods

327

F .Ou/.xj / D uO 0 .xj /  f .x; uO .xj // D 0

interpolation interval, which is not a very restrictive requirement. On the other hand, it is indeed necessary, at collocation nodes xj as has been demonstrated in [2]. Otherwise the error uQ u usually lacks the required smoothness properties, is rather to be considered as the effective original despite its asymptotic order. problem. Naturally, IDeC can also be applied to boundary • IDeC can be combined in a natural way with grid value problems (BVPs). For second-order two-point adaptation strategies, because the requisite local or boundary value problems, the necessary algorithmic global error estimates are built-in to the procedure. modifications have first been described in [10]. Here, special care has to be taken at the endpoints of the interpolation intervals Ik , where an additional defect IDeC Based on Higher Order Schemes FQ : term arises due to jumps in the derivatives of the local A Bit of Theory For IDeC applied to IVPs, any basic scheme FQ may interpolants. be used instead of forward Euler. For example, in the pioneering paper [18] a classical Runge–Kutta (RK) Reformulation in Terms of Integral Equations: scheme of order 4 was used. Using RK in the correction IQDeC (A) and “Spectral IDeC” (= IQDeC (B)) steps means that in each individual evaluation of the An ODE can be transformed into an integral equation. right-hand side the pointwise value of the current Taking the integral means of (11) over the interval defect is to be added (RK applied to [NP]). Many spanned by two successive grid points gives other authors have also considered and analyzed IDeC Z xj C1 versions based on RK schemes. u.xj C1 /  u.xj / Despite the natural idea behind IDeC, the converD f .x; u.x// dx: (17) h xj gence analysis is not straightforward. Obtaining a full higher order of convergence asymptotically for h ! 0 Observe that the left-hand side is of the same type as requires: in the Euler approximation (13). Therefore it appears • A sufficiently well-behaved, smooth problem • A sufficiently high degree p for the local inter- natural to consider (17) instead of (11) as the original problem. In addition, for numerical evaluation the polants ui .x/ • Sufficient smoothness of these interpolants, in the integral on the right-hand side has to be approximated, sense of boundedness of a certain number of deriva- typically by polynomial quadratures using the p C 1 nodes available in the current working interval Ik 3 xj . tives of the ui .x/, uniformly for h ! 0 A theoretical tool to assure the latter smoothness prop- The coefficients depend on the location of xj within Ik ; erty are asymptotic expansions of the global discretiza- ignoring this aspect and using Q as a generic symbol tion error uQ u for the underlying scheme, which have for these quadratures leads to the “computationally been proved to exist for RK methods over constant tractable,” modified original problem [OP] replacing stepsize sequences. A standard convergence result for [OP] from (12), defined over the grid fxj g as IDeC derived in this way is given in [11]; see also [16]. u.xj C1 /  u.xj / A typical convergence result reads as follows: [OP] F .u/.xj / WD h If the sequence of grids is equidistant and the under.Qf /.x; u.x//j D 0; lying scheme has order q, then m IDeC steps result (18) in an error as uk .x/  u .x/ D O.hminfp; m qg / for h ! 0, where p is the degree of interpolation. more precisely, or its effective version restricted to The achievable order p is usually identical to the u 2 Pp . This is to be compared to FQ from (14), which approximation order of the fixed point uO 2 Pp , a poly- is used in the same way as before. The treatment of nomial of collocation or generalized collocation type. the leading derivative term u0 is the same in (18) and The assumption on the grids appears quite restrictive. in (14), which turns out to be advantageous. EquaIn fact, this can be relaxed in a natural way in the tion 18 leads to an alternative definition of the defect sense that the stepsize h has to be kept fixed over each at the evaluation points xj , namely [OP]

D

328

Defect Correction Methods

ui .xj C1 /  ui .xj / dN i .xj / WD F .ui /.xj / D h .Qf /.x; ui .x//j :

(19)

method, the convergence properties strongly depend on the problem at hand. The main difficulty for DeC is that the convergence rate may be rather poor for error components associated with stiff eigendirections. An overview and further material on this topic can be found in [3] or [9]; see also the numerical example below. Actually, “stiffness” is paraphrased for quite a large variety of linear and nonlinear phenomena and still an active area of research, not only in the context of DeC. Similar remarks apply to problems with singularities.

This may be interpreted in the sense that the original, pointwise defect d i .x/ is “preconditioned” by applying local quadrature. All other algorithmic components of IDeC remain unchanged, with correspondingly defined neighboring problems [NP]. In [2] this version is introduced and denoted as IQDeC (type (A)). Variants in the spirit of IQDeC of type (B) have also received attention in the literature; this is usually Boundary Value Problems (BVPs) and “Deferred called “spectral defect correction” and has first been Correction” described in [9]. For a convergence proof, see [12]. Historically, one of the first applications of a type (B) truncation error estimator (7) appears in the context Remarks: finite-difference approximations to a BVP • With an appropriate choice of defect quadrature, the fixed point of IQDeC is the same as for IDeC. (20) u0 .x/ D f .x; u.x//; B.u.a/; u.b// D 0; In fact, equation dO D F .Ou/ D 0 turns out to be closely related to a reformulation of the associated posed on an interval Œa; b (with boundary conditions collocation equations uO 0 .xj / D f .xj ; uO .xj // in represented by the function B), or higher order probthe form of an equivalent implicit Runge–Kutta lems. (A classical text on the topic is [13].) For a finite(IRK) scheme. In other words: The “pointwise,” difference approximation of u0 .xj /, e.g., as in (13), an or “collocation defect” has been replaced with a asymptotic expansion of the TE t  is straightforward related defect of IRK type. using Taylor series and using (20): • There are several motivations for considering IQDeC. One major point is that, as demonstrated t  .xj / D FQ .u /.xj / in [2], its convergence properties are much less u .xj C1 /  u .xj /   0 affected by irregular distribution of the xj . In the  u .xj / D forward Euler case, the normal order sequence h 1; 2; 3; : : : shows up, in contrast to IDeC. For higher 1  00 1  000 D u .xj / C u .xj / C : : : (21) order FQ the precise construction of IQDeC or 2 6 spectral IDeC and its convergence behavior is more  00 involved and subject to recent investigations. The idea is to approximate the leading term 12 u .xj / • For a related approach in the context of second- by a second-order difference quotient involving three order two-point boundary value problems, also per- successive nodes. This defines an “approximate TE” mitting variable mesh spacing, see [8]. associated with an “approximate [OP],” which corre• Another modification can be used to construct su- sponds to a higher order discretization of (20). The perconvergent IDeC methods: In [2] (“IPDeC”) and corresponding TE estimator  0 is obtained by evaluin [15], the use of an equidistant basic grid is ating the approximate TE at a given u D u0 . This is combined with defect evaluation at Gaussian nodes, used in the first step of an IDeC (B) procedure (see (7)– in a way that the resulting iterates converge to the (9)). In this context, updating the (approximate) [OP] corresponding superconvergent fixed point (collo- in course of the iteration is natural, involving difference cation at Gaussian nodes). approximations of the higher order terms in (21), to be successively evaluated at the iterates ui. IDeC (B) versions of this type are usually addressed Stiff and Singular Problems For stiff systems of ODEs, DeC methods have been as deferred correction techniques, and they have been used with some success. However, as for any other extensively used, especially in the context of boundary

Defect Correction Methods

329

value problems. The analysis heavily relies on the describes the deflection of a driven linear, damped ossmoothness properties of the error. Piecewise equidis- cillator. We consider the equivalent first-order system tant meshes are usually required. A difficulty to be and consider: coped with is the fact that the difference quotients in- (i) ! D  D 1 (low frequency, critical damping); volved increase in complexity and have to be modified collocation (degree p D 4) on piecewise equidisnear the boundary and at points where the stepsize is tant interior nodes; estimator based on the box changed. scheme and the modified defect dN 0 . (ii) ! D  D 100 (high frequency, critical damping); collocation (degree p D 4) over Gauss–Legendre Defect-Based Error Estimation and Adaptivity nodes; estimator based on the box scheme and the In practice, the DeC principle is also applied – in the pointwise defect d 0 . This is a rather stiff situation, spirit of our original motivation – for estimating the but both integration methods used are A-stable. error of a given numerical solution with the purpose The initial data are chosen such that the resulting u.t/ of adapting the mesh. A typical case is described and 0 is a smooth, resonant solution (a linear combination of analyzed in [4]: Assume that u is a piecewise polynomial collocation solution to the BVP (20). Collocation sin t and cos t). Figure 1 shows the global error e.t/ methods are very popular and have favorable conver- and its QDeC estimate ".t/ on a mesh of 32 collogence properties; they can be implemented directly or cation subintervals over Œ0; 2. For (i), the deviation via some version of IDeC. By definition of u0 , its point- ".t/  e.t/ amounts to approximately 1 % of the error. 4 wise defect d 0 .x/ D .u0 /0 .x/  f .x; u0 .x// vanishes On refining the mesh, one observes e.t/ D O.h / 6 at the collocation nodes which are, e.g., chosen in the and ".t/  e.t/ D O.h / asymptotically for h ! interior of the collocation subintervals Ik . Therefore, 0, which is in accordance with the theory from [4]. information about the quality of u0 is to be obtained For (ii), the size of the collocation error is similar as evaluating d 0 .x/ at other nodes, e.g., the endpoints of for (i); the estimator is approximately following but underestimating it. Here, using dN 0 instead of d 0 turns the Ik . 0 0  For estimating the global error e .x/ D .u  u /.x/, out to be less favorable, demonstrating that the correct one can use the type (A) error estimator (6) based handling of stiff problems is not a priori obvious. on a low-order auxiliary scheme FQ , e.g., an Euler or box scheme, over the collocation grid. Replacing the pointwise defect d 0 by the modified defect dN 0 , Partial Differential Equations (PDEs) and analogously as in (19), is significantly advantageous, Variational Formulation because this version is robust with respect to the lack of smoothness of u0 which is only a C 1 function. In the context of PDEs, in particular elliptic boundary In [4] it has been proved that such a procedure value problem, a weak formulation of DeC techniques leads to a reliable and asymptotically correct error is of interest. We will be brief and restrict ourselves estimator of QDeC type. This method of error to an abstract, linear variational formulation. Let a.; / estimation is implemented in the software package be a bounded coercive bilinear form on a Hilbert space sbvp described in [5]. sbvp is an adaptive collocation V , and f a continuous linear functional defined on V (e.g., V D H 1 ./ with respect to a domain  2 Rn ). solver especially tuned for singular BVPs. 0 N With an appropriately modified version of d , Given the original problem in variational form, closely related to the defect definition from [8], the QDeC estimator can also be extended to second (or [OP] Find u 2 V with a.u; v/ D f .v/ 8 v 2 V; higher order) problems. (23) with solution u D u , we consider its discrete analog (e.g., arising from a Galerkin Finite Element (FEM) discretization) defined on finite-dimensional subspace Vh  V ,

Example: Damped Oscillator The second-order IVP:

u00 .t/ C 2 u0 .t/ C ! 2 u.t/ D 2 cos t; u.0/ D ˛; u0 .0/ D ˇ

(22)

[OP]h

Find uh 2 Vh with

D

330

Defect Correction Methods 6e−07

3e−06

4e−07

2e−06

2e−07

1e−06

0

0

−2e−07

−1e−06

−4e−07

−2e−06

−6e−07 0

1

2

3

4

5

6

0

1

2

3

4

5

6

Defect Correction Methods, Fig. 1 Error (red) and its DeC estimate (green), e.g., (22). Left: case (i). Right: case (ii)

a.uh ; vh / D f .vh / 8 vh 2 Vh ;

(24) is concerned with a posteriori estimation of the discretization error uh u (or some related functional).  with solution uh D uh . Here, h symbolizes the • Multigrid (multilevel) methods: These are based on recursive CS or FAS type DeC steps over hiunderlying mesh spacing. (In practice, a.; / and f ./ erarchical grids, in combination with smoothing are approximated numerically, but we are neglecting procedures on each level to make the coarse grid this aspect.) 0  corrections work. Assume that uh is a given approximation to uh , and Identify (24) with the problem on the finest diswe wish to estimate the error eh0 WD u0h  uh . The cretization level and consider the analogous prob(weak) defect dh0 of u0h with respect to the discrete lem on a coarser level associated with a subspace problem (24) is a linear functional on Vh , VH  Vh . For a current iterate u0h and with an 0 0 appropriately chosen prolongation operator P W dh .vh / WD a.uh ; vh /  f .vh /; vh 2 Vh ; VH ! Vh , the solution of 0 and uh is the exact solution of the neighboring problem Find "0H 2 VH with [NP]h Find uh 2 Vh with a."0 ; v / D P T d 0 .v / 8 v 2 V ; H

a.uh ; vh / D f .vh / C dh0 .vh / 8 vh 2 Vh :

H

h

H

H

H

playing the role of (26), gives rise to the coarse-grid correction "0h WD P "0H of Galerkin type. Multilevel methods have become very popular, With an approximate form a.; Q /, consider the soluin particular as global or local preconditioners for tions uQ h and uQ 0h of (24) and (25), respectively, with Krylov-based solvers, cf., e.g., [14]. 0 0 a.; / replaced by a.; Q /. Then, as in (5), "h WD uQ h  • Local defect correction methods: These are related  0 uQ h  uQ h  uh D eh serves as an error estimate. For to domain decomposition (Schwarz type) methu0h D uQ h , we can also determine "0h from the solution of ods; a globally defined approximate solution u0h is the CS improved by adding up defect corrections acting locally on subdomains; this implicitly defines the Find "0h 2 Vh with a." Q 0h ; vh / D dh0 .vh / 8 vh 2 Vh : approximate form a.; Q /. The idea appears in [7, (26) p. 89]; see also [1]. This can also be combined with We list some specific techniques. The first and multigrid solvers for the local problems. second are used for handling the large systems of • In the context of PDEs, a posteriori error estimates equations arising after discretization. The third one for the purpose of mesh adaptation are frequently (25)

Defect Correction Methods

331

based on a norm kdh k of an appropriately defined defect dh , e.g., of a FEM approximation uh . Solving back for a direct error estimate as in the ODE case (cf. the QDeC approach from [4]) is not so obvious here and is not an established technique. Rather, local contributions to kdh k are frequently used as a measure for the local quality of uh over the computational cells. This methodology is refined in goal-oriented, dual-weighted residual (DWR) methods. Consider a discrete variational problem (24) and assume that uh  u has been computed. Let the functional J.u/ represent a quantity of interest which is to be controlled, e.g., a weighted average of u. Here we assume that J.u/ is linear, and we wish to estimate the deviation J.uh /  J.u / D J.eh /. The idea is to consider the dual problem [DP]

Find w 2 V with

a.v; w/ D J.v/ 8 v 2 V;

(27)

If wQ h is obtained from a Galerkin approximation of (29), in the spirit of (24), we may invoke the WD wQ h  wh  wh  type (A) DeC estimator "dual h  dual dual w D eh , and evaluate f ."h /. However, in order to make sense, the defect functional ıh .v/ has to be evaluated with higher accuracy, e.g., using a higher order interpolant of wh . Such an interpolation plays the same role as the interpolation of a given approximation in the context of conventional IDeC schemes.

Cross-References  Collocation Methods  Euler Methods, Explicit, Implicit, Symplectic  Finite Difference Methods  Finite Element Methods  Interpolation  Runge–Kutta Methods, Explicit, Implicit  Variational Integrators

with solution w D w . The solutions u and w of (23) and (27) are adjoint to each other via the reReferences lation J.u / D a.u ; w / D f .w /. The analogous relation holds between the discrete versions uh and 1. Anthonissen, M.J.H., Mattheij, R.M.M., ten Thije wh , which leads to Boonkkamp, J.H.M.: Convergence analysis of the local J.eh /

D

J.uh /

D

f .wh /



 J.u / D 

a.uh ; wh /

 f .w / D





 a.u ; w /

f .ehdual /;

2.

(28)

where ehdual WD wh  w denotes the discretization error in approximating (27). Thus, provided the solution w of the dual problem (27) is available, f .ehdual / yields an exact representation for the deviation J.eh /. Practical realization of such a DWR estimate relies on the availability of an efficient and sufficiently accurate approximation of w , of a better quality than wh . Several techniques of this type, and the extension to nonlinear problems, are discussed [6]. In the context of DeC, an option is to consider a dual neighboring problem [DNP] to [DP], defined via the defect functional ıh .v/ WD a.v; wh /  J.v/. Then, wh is the exact solution of

3.

4.

5.

6.

7.

8.

[DNP]

Find wh 2 Vh with

a.v; wh / D J.v/ C ıh .v/ 8 v 2 V:

(29)

defect correction method for diffusion equations. Numer. Math. 95(3), 401–425 (2003) Auzinger, W., Hofst¨atter, H., Kreuzer, W., Weinm¨uller, E.: Modified defect correction algorithms for ODEs. Part I: general theory. Numer. Algorithm 36, 135–156 (2004) Auzinger, W., Hofst¨atter, H., Kreuzer, W., Weinm¨uller, E.: Modified defect correction algorithms for ODEs. Part II: stiff initial value problems. Numer. Algorithm 40, 285–303 (2005) Auzinger, W., Koch, O., Weinm¨uller, E.: Efficient collocation schemes for singular boundary value problems. Numer. Algorithm 85, 5–25 (2002) Auzinger, W., Kneisl, G., Koch, O., Weinm¨uller, E.: SBVP 1.0 – A MATLAB solver for singular boundary value problems. Technical Report/ANUM Preprint No. 2/02, Vienna University of Technology. See also: http://www. mathworks.com/matlabcentral/fileexchange/1464-sbvp-1-0package (2002) Becker, R., Rannacher, R.: An optimal control approach to a posteriori error estimation in finite element methods. Acta Numer. 10, 1–102 (2001) B¨ohmer, W., Stetter, H.J. (eds.): Defect Correction Methods – Theory and Applications. Computing Suppl. 5, Springer, Wien/New York (1984) Butcher, J.C., Cash, J.R., Moore, G., Russell, R.D.: Defect correction for two-point boundary value problems on nonequidistant meshes. Math. Comput. 64(210), 629–648 (1995)

D

332 9. Dutt, A., Greengard, L., Rokhlin, V.: Spectral deferred correction methods for ordinary differential equations. BIT 40(2), 241–266 (2000) 10. Frank, R.: The method of iterated defect-correction and its application to two-point boundary value problems. Numer. Math 25(4), 409–419 (1976) 11. Frank, R., Ueberhuber, C.W.: Iterated defect correction for differential equations. Part I: theoretical results. Computing 20, 207–228 (1978) 12. Hansen, A.C., Strain, J.: On the order of deferred correction. Appl. Numer. Math. 61(8), 961–973 (2011) 13. Pereyra, V.: Iterated deferred correction for nonlinear boundary value problems. Numer. Math. 11(2), 111–125 (1968) 14. Saad, Y.: Iterative Methods for Sparse Linear Systems, p. 495. SIAM, Philadelphia (2003) 15. Schild, K.H.: Gaussian collocation via defect correction. Numer. Math. 58(4), 369–386 (1990) 16. Skeel, R.D.: A theoretical framework for proving accuracy results for deferred corrections. SIAM J. Numer. Anal. 19(1), 171–196 (1981) 17. Stetter, H.J.: The defect correction principle and discretization methods. Numer. Math. 29(4), 425–443 (1978) 18. Zadunaisky, P.E.: On the estimation of errors propagated in the numerical integration of ODEs. Numer. Math. 27(1), 21–39 (1976)

Delaunay Triangulation Yasushi Ito Aviation Program Group, Japan Aerospace Exploration Agency, Mitaka, Tokyo, Japan

Mathematics Subject Classification 32B25

Synonyms Delaunay Tessellation

Short Definition For a set P of points in the n-dimensional Euclidean space, the Delaunay triangulation is the triangulation D(P) of P such that no point in P is inside the circumscribed n-sphere (e.g., circumcircle in two dimensions

Delaunay Triangulation

(2D) and circumsphere in three dimensions (3D)) of any simplex (triangle in 2D and tetrahedron in 3D) in D(P). This fundamental property of the Delaunay triangulation is known as the empty circle property. D(P) is also the dual of the Voronoi tessellation of P.

Description The Delaunay triangulation introduced by Boris Delaunay in 1934 [1] has been useful in many applications, such as scattered data fitting and unstructured mesh generation [2–4]. The Delaunay triangulation has been used for mesh generation because it provides connectivity information for a given set of points (Points are sometimes referred to as nodes in mesh generation). In addition, it has a good property in 2D: the minimum angle of all angles of resulting triangles is minimized [2]. For better understanding, Fig. 1 shows simple examples of triangulation of four points. Unfortunately, this property cannot be extended to 3D (and higher dimensions) because of the existence of almost-zerovolume tetrahedra known as slivers, and special care must be taken to remove such unwanted elements [5]. The Delaunay triangulation does not exist in degenerated cases, such as a given set of points on the same line. In n-dimensions, the Delaunay triangulation is not unique if nC2 or more points are on the same n-sphere. For example, a rectangle in 2D can be subdivided into two triangles by a diagonal, and either of the two diagonals creates two Delaunay triangles because the four points of the rectangle are cocircular (e.g., Fig. 1c, d). The Delaunay triangulation, however, does not address by itself other two important aspects of mesh generation [6]: how to generate the points for creating well-shaped elements and for achieving a smooth element size transition and how to ensure boundary conformity of non-convex hulls. The latter problem is known as constrained Delaunay triangulation [7]. Moreover, the actual implementation of the Delaunay triangulation needs a method that is not sensitive to round-off and truncation errors when locating a point inside or outside of the circumscribed n-sphere of an existing simplex. Therefore, typical Delaunay triangulation methods require the following steps in 3D [6, 8–10]:

Delaunay Triangulation

333

Delaunay Triangulation, Fig. 1 (a) Non-Delaunay and (b–d) Delaunay triangles with their circumcircles for points of (a, b) a quadrilateral and (c, d) a rectangle: the circumcircles of the non-Delaunay triangles in (a) contain other nodes

Delaunay Triangulation, Fig. 2 Delaunay triangulation in 2D for letter i: (a) boundary points and edges; (b) two triangles covering the entire meshing domain and boundary points (white) for reference; (c) Delaunay triangulation with 20 boundary points;

(d) Delaunay triangulation with all boundary points; (e) triangles after those on the outside removed; (f) Delaunay triangulation with additional interior points

1. Discretize the boundaries of the domain to be meshed as a surface mesh, which consists of points P, edges E, and faces F (Fig. 2a). 2. Create a box as a set of tetrahedra that covers the entire surface mesh (Fig. 2b). 3. Perform the Delaunay triangulation of P to generate tetrahedra. This is done by adding P one by one to the existing tetrahedra and then updating their connectivity (Fig. 2c). 4. Recover E and then F to obtain the original boundary surface (Fig. 2d). Note that additional points may have to be added on the boundaries to create valid tetrahedra. 5. Remove tetrahedra outside of the meshing domain (Fig. 2e). 6. Generate interior points and add them to the existing tetrahedral mesh to update it using the Delaunay triangulation (Fig. 2f).

Those steps should be easily modified for 2D triangulation problems shown in Fig. 2. The points added in steps 4 and 6 are sometimes called as Steiner points. The Delaunay triangulation can be combined with an advancing front method to better control point distribution [11–13], especially for anisotropic or hybrid mesh generation for high Reynolds number viscous flow simulations.

References 1. Delaunay, B.: Sur la sph`ere vide. A la m´emoire de Georges Voronoi. Izv. Akad. Nauk SSSR, Otdelenie Matematicheskih i Estestvennyh Nauk 7, 793800 (1934) 2. Lawson, C.L.: Software for C1 surface interpolation. In: Rice, J.R. (ed.) Mathematical Software III, pp. 161–194. Academic, New York (1977) 3. Bowyer, A.: Computing dirichlet tessellations. Comput. J. 24(2), 162–166 (1981). doi:10.1093/comjnl/24.2.162

D

334 4. Watson, D.F.: Computing the n-dimensional delaunay tessellation with application to voronoi polytopes. Comput. J. 24(2), 167172 (1981). doi:10.1093/comjnl/24.2.162 5. Dey, T.K., Bajaj, C.L., Sugihara, K.: On good triangulations in three dimensions. In: Proceedings of the 1st ACM Symposium on Solid Modeling Foundations and CAD/CAM Applications, TX, Austin, pp. 431–441 (1991). doi:10.1145/112515.112578 6. Weatherill, N.P., Hassan, O.: Efficient three-dimensional delaunay triangulation with automatic point creation and imposed boundary constrains. Int. J. Numer. Methods Eng. 37, 2005–2039 (1994). doi:10.1002/nme.1620371203 7. Chew, L.P.: Constrained delaunay triangulations. Algorithmica 4, 97–108 (1989). doi:10.1007/BF01553881 8. Baker, T.: Three-dimensional mesh generation by triangulation of arbitrary point sets. In: Proceedings of the AIAA 8th CFD Conference, Honolulu, HI, AIAA Paper 87-1124-CP, pp. 255–269 (1987) 9. George, P., Hecht, F., Saltel, E.: Automatic 3D mesh generation with prescribed meshed boundaries. IEEE Trans. Magn. 26(2), 771–774 (1990). doi:10.1109/20.106431 10. Weatherill, N.P.: Delaunay triangulation in computational fluid dynamics. Comput. Math. Appl. 24(5–6), 129–150 (1992). doi:10.1016/0898-1221(92)90045-J 11. M¨uller, J.-D., Roe, P.L., Deconinck, H.: A frontal approach for internal node generation in delaunay triangulations. Int. J. Numer. Methods Fluids 13, 1–31 (1991). doi:10.1002/fld.1650170305 12. Marcum, D.L., Weatherill, N.P.: Unstructured grid generation using iterative point insertion and local reconnection. AIAA J. 33(9), 1619–1625 (1995). doi:10.2514/3.12701 13. Mavriplis, D.J.: An advancing front delaunay triangulation algorithm designed for robustness. J. Comput. Phys. 117, 90–101 (1995). doi:10.1006/jcph.1995.1047

Delay Differential Equations Nicola Guglielmi Dipartimento di Matematica Pura e Applicata, Universit`a dell’Aquila, L’Aquila, Italy

Delay Differential Equations

is not possible in this short entry. Nevertheless for a deep knowledge of the subject, we remand the reader to the recent monographs by Bellen and Zennaro [1] and by Brunner [3] (the last one is more focused on integral functional differential equations) and to the wide bibliographies therein. Delay differential equations (in short DDEs) provide a powerful means of modeling many phenomena in applied sciences. Recent studies in several fields as physics, biology, economy, electrodynamics (see e.g., [5, 9, 12] and their references) have shown that DDEs play an important role in explaining many different behaviors. In particular, they become very important when ODE-based models are not able to describe the considered phenomena due to the presence of time lags which determine a memory in the system (as an example relevant to Maxwell equations see [11]). In this entry, we give an essential description of the numerical methods for approximating solutions of a class of DDEs. For a comprehensive introduction to the treated subject we refer the reader to the book by Bellen and Zennaro [1] and to the extensive bibliography contained therein. For software issues, we refer the reader to the last section and for recent results, which include a new class of so-called functional continuous numerical schemes, we refer to [2].

A Simple Illustrative Example Consider the initial value problem for the scalar DDE with a constant delay  D 1: (

y 0 .t/ D  y.t  1/ y.0/ D 1;

y.t/ D 0

for t > 0 for  1  t < 0

(1)

Subject Classifications 65L06, 65Q20, 34K28

Introduction The aim of this entry is to provide a concise introduction to the numerical integration of a class of delay differential equations. The literature on this subject is very broad and we apologize for not quoting many interesting papers, as an exhaustive list of references

whose solution is shown in Fig 1. The solution can be easily computed by the so-called method of steps which consistsin solving a sequence of ODEs y 0 .t/ D gk .t/, being gk .t/ D y.t  1/, for t 2 Œk; k C 1, k  0. Some important features are evident. 1. In order to give a meaning to the Cauchy problem, it is necessary to provide it by an initial function in the interval Œ1; 0. 2. The solution is not globally smooth but only piecewise even if the right-hand side and the initial function are C 1 .

Delay Differential Equations y disc.

335

high derivative of the solution are not significant in terms of the numerical method.

y disc. .5

−1

y disc. 1

−.5

2

3

4 y

5

6

disc.

Numerical Integration

Consider a mesh  D ft0 ; t1 ; : : : ; tN D T g which is usually determined adaptively. A general approach should take into account that in general, the solution Explicit Delay Differential Equations in the past required at mesh points is unknown since We start by considering systems of delay differential ˛.tn ; yn / is not a mesh point. This suggests the followequations in the autonomous (which is not restrictive) ing strategy: explicit form: 1. Choose a discrete method for solving ODEs. 2. Choose a continuous extension of the method (see  (   e.g., [13]) which provides a uniform approximation t 2 .t0 ; T  y 0 .t/ D f y.t/; y ˛.t; y.t// ; .t/ of the solution y.t/. t < t0 y.t/ D '.t/; y.t0 / D y0 3. Compute the relevant breaking points fj gj 1 (or(2) dered in ascending way) and apply subsequently the method to the problems where T > t0 is a given constant, y 2 Rd .d  1/,  (   '.t/ 2 Rd is a given initial function, f is a real valt 2 Œk1 ; k  x 0 .t/ Df x.t/; ˛.t; x.t// ued vector function, ˛.t; y.t// represents the deviating x.k1 / D .k1 / argument (often written as ˛.t; y.t// D t  .t; y.t//  in terms of the delay .t; y.t//  0), y ˛.t; y.t// de(3) notes the solution computed at ˛.t; y.t//  t. The case of several delays is straightforward. The value '.t0 / Note that this is possible only if the delay does not may be different from y0 , allowing a discontinuity at t0 . vanish; otherwise, breaking points would not be well Delay Differential Equations, Fig. 1 Solution of (1)

Breaking Points If y0 ¤ '.t0 /, the solution is obviously discontinuous at t0 . However, even when y0 D '.t0 / the righthand derivative y 0 .t0 /, which is given by the delay differential equation, is not equal in general to the lefthand derivative ' 0 .t0 /. This lack of smoothness at t0 typically propagates along the integration interval. In fact, as soon as ˛.; y.// D t0 for some  > t0 , the function f .y.t/; y .˛.t; y.t//// is not smooth at . In general, this creates a further jump discontinuity in some derivatives of the solution y.t/ at , which will be propagated in turn. In the literature, these points are called breaking points (see e.g., [1, 10]). In the case of a constant delay ˛.t; y/ D t  , the breaking points are k D t0 C k for k  1. A jump discontinuity at t0 leads in general to a jump discontinuity at 1 in the first derivative of y.t/; this determines in turn a jump discontinuity at 2 in the second derivative of y.t/ and similarly for further points. Observe that only some of these breaking points are important for the numerical integration, because discontinuities in a sufficiently

separated. If the chosen method is a continuous method (like a collocation method) step (2) is obtained directly. In this article, for the sake of brevity, we confine the discussion to Runge-Kutta (RK) methods. Continuous Runge-Kutta Methods For a classical ODE:



  y 0 .t/ D g t; y.t/ ; y.t0 / D y0 ;

t0  t  T;

an s-stage Runge-Kutta method with coefficients faij g, abscissæ fci g, and weights fbi g (i; j D 1; : : : ; s) has the form (where hn is the step size): .n/ Yi

D yn C hn

s X

  .n/ ; aij g tn C cj hn ; Yj

j D1

i D 1; : : : ; s ynC1 D yn C hn

s X i D1

  .n/ ; bi g tn C ci hn ; Yi

D

336

Delay Differential Equations

which provides an approximate solution ynC1 of the Implicit, Stiff, and Neutral Problems solution y.tnC1 / starting from the knowledge of an approximate solution yn of the solution y.tn /. The We consider next IVPs for implicit delay differential continuous extension of the method .t/ is defined equations of the form: in each subinterval Œtn ; tnC1  of the mesh  by a 8   continuous quadrature rule of the form: 0) we just have ˛j D tn C cj hn  .

Collocation Methods for Implicit Problems Consider an s-stage implicit Runge-Kutta collocation method (see [8]) and assume that the method is stiffly accurate so that bi D asi for i D 1; : : : ; s. One step is given by: s     X .n/ .n/ .n/ aij f Yj ; Zj ; M Yi  yn D hn j D1

i D 1; : : : ; s .n/

(6)

where ynC1 D Ys approximates the solution at time .n/ tn Chn . In order to determine Zj which has to provide an approximation to y .˛.t; y.t/// at t D tn Ccj hn , we .n/ .n/ recall that ˛j WD ˛.tn C cj hn ; Yj /. Here the dense

Delay Differential Equations

337

output .t/ is given step-by-step by the collocation Runge-Kutta methods). Assume a breaking point  has polynomial um .t/ of degree s, which is available for been detected, that is, ˇ.tn I / ˇ.tn C hn I / < 0: In tm  t  tmC1 : order to obtain an accurate approximation of  (that is, with the same accuracy of the solution), one can consider hn as a variable in (6) and impose that  is s X .m/ approximated by tn C hn . This leads to the augmented um .tm C #hm / D Li .#/Yi # 2 Œ0; 1; system: i D0 s     X .n/ .n/ .n/ aij f Yj ; Zj ; M Yi  yn D hn

.m/ Y0

where WD ym and Li .#/ is the Lagrange polynomial satisfying Li .cj / D ıij with ıij the Kronecker delta symbol (we add c0 D 0 to the abscissas of the method). In intervals succeeding to breaking points, the dense output polynomial can be better replaced by: s X

j D1

i D 1; : : : ; s ˛ .tn C hn ; un .tn C hn // D  .n/

(8) (9)

.n/

for the unknowns Y1 ; : : : ; Ys , and hn . Note that u n .t/ is the dense output of the current step and therei D1 .n/ fore depends on the stage values fYi gsiD1 . For given hn , the system (8) is solved by a simplified Newton which interpolates the internal stage values but not ym iteration which exploits the structure of the system (see (see Sect. 2 of [6]). The use of this option is important [6]). In order to preserve such structure, the system (8) in order to improve the accuracy in the presence of a and (9) can be efficiently solved by means of a splitting jump discontinuity in the solution since it allows to also method. If this converges and the error is small enough, have a discontinuity in the dense output approximation t C h is labeled as a breaking point. Otherwise the n n of the solution. In fact, one has in general vm .tm / ¤ step size is reduced. .n/ ym D um1 .tm /. Whenever ˛j 2 Œtn ; tnC1 , that is, in case the delay is smaller than the current step size, .n/ .n/ um .˛j / (and also vm .˛j /) depend on the unknown Solving the Runge-Kutta Equations .n/ .n/ stage values Y1 ;    ; Ys . This determines a stronger Finally, we discuss the solution of the RK equations coupling of the system of Runge-Kutta equations (6). focusing our attention to the particular case when The use of the three-stage Radau IIA method as basic overlapping occurs (that is, the deviating argument falls integrator, whose good stability properties have been into the current interval Œtn ; tn C hn ). The RK system widely investigated (see [1]), has led to the code (6) has the form (for i D 1; : : : ; s): vm .tm C #hm / D

.m/

Li .#/Yi

# 2 Œ0; 1;

RADAR5 (see [6] and [7]). .n/

Fi Accurate Computation of Breaking Points A fundamental issue in order to preserve the expected precision of a numerical method is that of accurately computing breaking points. The problem is to find the zeros of:

    .n/ .n/ Y1 ;    ; Ys.n/ D M Yi  yn

hn

s X

   .n/ .n/ .n/ aij f Yj ; Zj Y1 ;    ; Ys.n/ D 0

j D1

.n/

.n/

for the unknowns Y1 ;    ; Ys . For convenience, in ˇ.tI / D ˛ .t; u.t//  ; (7) the sequel, we omit the dependence on n and denote by f .y; z/ the right-hand side of (5). We are interested in where  is a previous breaking point and u.t/ is a solving previous system by means of a Newton-based suitable approximation to the solution. A natural way process. We consider the approximation: to detect the presence of a breaking point in the current interval Œtn ; tn C hn  is to look for a sign change in @Fi  M ıi k  hn .ai k Dk C Ei k /; (7) when the step is rejected (see also [4] for explicit @Yk

D

338

Delay Differential Equations

where ıi k is the Kronecker delta, ˛0 D ˛.tn ; yn / and Dk WD

Ei k WD

@f @f @˛ .yn ; zn / C .yn ; zn / 0 .˛0 / .yn /; @y @z @y s X j D1

aij

@Zj @f .yn ; zn / : @z @Yk

  We get (with j D ˛.tn C cj hn ; Yj /  tn = hn ): @Zj D Uj k I; @Yk

 Uj k D

Lk . j / if j > 0 0 otherwise:

Shampine and Thompson), and RETARD (by Hairer and Wanner) are based on explicit RungeKutta methods. • SNDDELM (by Jackiewicz and Lo) is based on an explicit multistep method. • DDE-STRIDE (by Baker, Butcher and Paul) and RADAR5 (by Guglielmi and Hairer) are based on implicit Runge-Kutta methods. • DIFSUB-DDE (by Bocharov, Marchuk and Romanyukha) is based on BDF formulas.

References (10)

Note that U D O if no overlapping occurs. We compute the Jacobian of F as:  J D I ˝ M  hn A ˝ hn A  U ˝

@f ; @z

@f 0 @˛ @f C .˛0 / @y @z @y



(11)

where A D faij g is the matrix of the coefficients of the Runge-Kutta method, @f =@y and @f =@z denote the matrices of the partial derivatives of f with respect to the variables y and z respectively, @˛=@y denotes the row vector of the partial derivatives of ˛ with respect to y, and the matrix U is given by (10). Based on the approximation (see [2]) U  Is ; where ! min O 2R kU  O Is k2F , and Is is the s s identity matrix, it is possible to take advantage of the tensor structure of J and transform it (if A is invertible) into blockdiagonal form, in analogy to what is done in the ODE case [8]. For the three-stage Radau-IIa collocation method, the LU decomposition of the transformed Jacobian is about five times less expensive than the pretransformed matrix (11).

Software Here is a list of codes for the time integration of systems of delay differential equations: • ARCHI (by Paul), DDE23 (by Shampine and Thompson), DDVERK (by Hayashi and Enright), DKLAG6 (by Thompson), DDE-SOLVER (by

1. Bellen, A., Zennaro, M.: Numerical Methods for Delay Differential Equations. Oxford University Press, Oxford (2003) 2. Bellen, A., Guglielmi, N., Maset, S., Zennaro, M.: Recent trends in the numerical solution of retarded functional differential equations. Acta Numer. 18, 1–110 (2009) 3. Brunner, H.: Collocation Methods for Volterra Integral and Related Functional Differential Equations. Cambridge Monographs on Applied and Computational Mathematics, vol. 15. Cambridge University Press, Cambridge (2004) 4. Enright, W.H., Hayashi, H.: A delay differential equation solver based on a continuous Runge-Kutta method with defect control. Numer. Algorithm. 16, 349–364 (1997) 5. Erneux, T.: Applied Delay Differential Equations. Surveys and Tutorials in the Applied Mathematical Sciences, vol. 3. Springer, New York (2009) 6. Guglielmi, N., Hairer, E.: Implementing Radau IIA methods for stiff delay differential equations. Computing 67(1), 1–12 (2001) 7. Guglielmi, N., Hairer, E.: Computing breaking points in implicit delay differential equations. Adv. Comput. Math. 29(3), 229–247 (2008) 8. Hairer, E., Wanner, G.: Solving Ordinary Differential Equations II. Stiff and Differential-Algebraic Problems. Springer Series in Computational Mathematics, vol. 14, second edn. Springer, Berlin (1996) 9. Kuang, Y.: Delay Differential Equations with Applications in Population Dynamics. Mathematics in Science and Engineering, vol. 191. Academic Press, Boston (1993) 10. Neves, K.W., Feldstein, A.: Characterization of jump discontinuities for state dependent delay differential equations. J. Math. Anal. Appl. 56, 689–707 (1976) 11. Ruehli, A.E., Heeb, H.: Circuit models for three dimensional geometries including dielectrics. IEEE Trans. Microw. Theory Tech. 40, 1507–1516 (1992) 12. Smith, H.: An Introduction to Delay Differential Equations with Applications to the Life Sciences. Texts in Applied Mathematics, vol. 57. Springer, New York (2011) 13. Zennaro, M.: Natural continuous extensions of Runge-Kutta methods. Math. Comput. 46(173), 119–133 (1986)

Dense Output

339

Dense Output Lawrence F. Shampine1 and Laurent O. Jay2 1 Department of Mathematics, Southern Methodist University, Dallas, TX, USA 2 Department of Mathematics, The University of Iowa, Iowa City, IA, USA

Introduction We solve numerically an initial value problem, IVP, for a first-order system of ordinary differential equations, ODEs. That is, we approximate the solution y.t/ of y 0 .t/ D f .t; y.t//;

t0  t  tF

that has given initial value y.t0 /. In the early days this was done with pencil and paper or mechanical calculator. A numerical solution then was a table of values, yj  y.tj /, for mesh points tj that were generally at an equal spacing or step size of h. On reaching tn where we have an approximation yn , we take a step of size h to form an approximation at tnC1 D tn C h. This was commonly done with previously computed approximations and an Adams-Bashforth formula like ynC1 D yn C h

16 5 23 fn  fn1 C fn2 : (1) 12 12 12

Here fj D f .tj ; yj /  f .tj ; y.tj // D y 0 .tj /. The number of times the function f .t; y/ is evaluated is an important measure of the cost of the computation. This kind of formula requires only one function evaluation per step. The approach outlined is an example of a discrete variable method [9]. However, even in the earliest computations, there was a need for an approximation to y.t/ between mesh points, what is now called a continuous extension. A continuous extension on Œtn ; tnC1  is a polynomial Pn .t/ that approximates y.t/ accurately not just at end of the step where Pn .tnC1 / D ynC1 , but throughout the step. Solving IVPs by hand is (very) tedious, so if the approximations were found to be more accurate than required, a bigger step size would be used for efficiency. This was generally done by doubling h so as to reuse previously computed values. Much more troublesome was a step size that was not

small enough to resolve the behavior of the solution past tn . Reducing h to h0 amounts to forming a new table of approximations at times tn  h0 ; tn  2h0 ; : : : and continuing the integration with this new table and step size. This was generally done by halving h so as to reuse some of the previously computed values, but values at tn  h=2; tn  3h=2; : : : had to be obtained with special formulas. Continuous extensions make this easy because the values are obtained by evaluating polynomials. Indeed, with this tool, there is no real advantage to halving the step size. To solve hard problems, it is necessary to vary the step size, possibly often and possibly by large amounts. In addition, it is necessary to control the size of the step so as to keep the computation stable. Computers made this practical. One important use of continuous extensions is to facilitate variation of step size. Some applications require approximate solutions at specific points. Before continuous extensions were developed, this was done by adjusting the step size so that these points were mesh points. If the natural step size has to be reduced many times for this reason, we speak of dense output. This expense can be avoided with a continuous extension because the step size can be chosen to provide an accurate result efficiently and a polynomial evaluated to obtain as many approximations in the course of a step as needed. This is especially important now that problem-solving environments like MATLAB and graphics calculators are in wide use. In these computing environments, the solutions of IVPs are generally interpreted graphically and correspondingly, we require approximate solutions at enough points to get a smooth graph. The numerical solution of ODEs underlies continuous simulation. In this context, it is common that a model is valid until an event occurs, at which time the differential equations change. An event is said to occur at time t  if g.t  ; y.t  // D 0 for a given event function g.t; y/. There may be many event functions associated with an IVP. Event location presents many difficulties, but a fundamental one is that in solving the algebraic equations, we must have approximations to y.t/ at times t that are not known in advance. With a continuous extension, this can be done effectively by testing g.tn ; yn / and g.tnC1 ; ynC1 / for a change of sign. If this test shows an event in Œtn ; tnC1 , it is located accurately by solving g.t  ; Pn .t  // D 0. In the following sections, we discuss briefly continuous extensions for the most important methods for

D

340

Dense Output

solving IVPs numerically. In the course of this dis- tnC1 , i.e., Pn0 .tnC1 / D f .tnC1 ; Pn .tnC1 //. For instance, cussion, we encounter other applications of continuous BDF3 is extensions. Providing an event location capability for 11 3 1 a wide range of methods was the principal reason ynC1  3yn C yn1  yn2 : hf .tnC1 ; ynC1 / D for developing the MATLAB ODE suite [13]. A few 6 2 3 details about this will make concrete our discussion of These formulas are implicit, and evaluating them efsome approaches to continuous extensions. ficiently is the principal challenge when solving stiff IVPs. Here, however, the point is that these methods are defined in terms of polynomials Pn .t/ which are Linear Multistep Methods natural continuous extensions. The formulas exhibited are linear combinations of Adams-Bashforth formulas are derived by approximatpreviously computed values and slopes and in the ing the integrated form of the differential equation case of implicit formulas, the value and slope at the Z t next step. They are representative of linear multistep y.t/ D y.tn / C f .x; y.x// dx; methods, LMMs [9]. By using more data, it is possible tn to obtain formulas of higher order, but they have seriwith an interpolating polynomial. Previously computed ous defects. The Adams methods and closely related slopes fn ; fn1 ; : : : ; fnkC1 are interpolated with a methods called predictor-corrector methods are very popular for the solution of non-stiff IVPs, and the polynomial Q.x/ and then BDFs are very popular for stiff IVPs. All these methods Z t have natural continuous extensions, which contributes Pn .t/ D yn C Q.x/ dx: (2) to their popularity. And, from the derivation outlined, tn it is clear that the methods are defined for mesh points The Adams-Bashforth formula of order k, ABk, is that are not equally spaced. Some popular programs ynC1 D Pn .tnC1 /. The example (1) is AB3. A very work with constant step size until a change appears convenient aspect of this family of formulas is that the worth the cost. This is standard for BDFs, including polynomial Pn .t/ is a natural continuous extension. the ode15s program of Shampine and Reichelt [13]. The Adams-Moulton formulas are constructed in the Other programs vary the step size, perhaps at every same way except that Q.x/ also interpolates the un- step. This is less common, but is the case for the known value f .tnC1 ; ynC1 /. This results in implicitly Adams-Bashforth-Moulton predictor-corrector method of ode113 [13]. A continuous extension for other defined formulas such as AM3 LMMs can be constructed by interpolation to all the

values and slopes used by the formula (Hermite in 8 1 5 f .tnC1 ; ynC1 / C fn  fn1 : terpolation). With some care in the selection of step ynC1 D yn Ch 12 12 12 size, this is a satisfactory continuous extension. Still, The new value ynC1 of an implicit Adams-Moulton only a very few other LMMs are seen in practice. formula is computed by iteration. In practice, this costs One, the midpoint rule, underlies a popular approach a little less than twice as many function evaluations to solving IVPs discussed in the section “Extrapolation as an explicit Adams-Bashforth formula. However, the Methods.” Adams-Moulton formulas are more accurate and more stable, so this is a bargain. The point here, however, is that a natural continuous extension (2) is available for these formulas too. Another important family of formulas is based on interpolation of previously computed values. The backward differentiation formulas, BDFs, are defined by a polynomial Pn .t/ that interpolates solution values ynC1 ; yn ; : : : and satisfies the differential equation at

Runge-Kutta Methods Using previously computed values causes some difficulties for LMMs. For instance, where do these values come from at the start of the integration? RungeKutta, RK, methods use only information gathered in the current step, so are called one-step methods.

Dense Output

341

An explicit RK formula of three function evaluations, rate of increase of step size. This approach can be or stages, has the form used at low orders, but a more fundamental difficulty was recognized as higher-order formulas came into use. In the case of explicit Adams methods, the step yn;1 D yn ; size is chosen so that an interpolating polynomial yn;2 D yn C hˇ2;1 fn;1 ; provides an accurate approximation throughout the step. Runge-Kutta methods of even moderate order use yn;3 D yn C h Œˇ3;1 fn;1 C ˇ3;2 fn;2  ; much larger step sizes that are chosen independent ynC1 D yn C h Œ 1 fn;1 C 2 fn;2 C 3 fn;3  : of any polynomial interpolating at previous steps. In practice, it was found that the interpolating polynomial Here tn;j D tn C ˛j h and fn;j D f .tn;j ; yn;j /. The does not achieve anything like the accuracy of the coefficients ˛j ; ˇj;k ; and j are chosen primarily to approximations at mesh points. The resolution of an important difference between make ynC1 approximate y.tnC1 / to high order. It is RK methods and LMMs is crucial to the construction easy to find coefficients that make a LMM as high an order as possible because they appear in a linear of satisfactory continuous extensions. This difference way. This is very much more complicated and difficult is in the estimation of the error of a step. LMMs can with RK methods because the coefficients appear in a use previously computed values for this purpose. There nonlinear way. The higher the order, the more algebraic are several approaches taken to error estimates for RK equations, the equations of condition, and the number methods, but they are equivalent to taking each step increases rapidly with the order. It is actually easy to with two formulas of different orders and estimating find formulas of any given order – the trick is to find the error of the lower-order formula by comparison. formulas of few stages. It is known that it takes at least RK methods involve a good many stages per step, so k stages to get a formula of order k. In this case, it is to make this practical, the two formulas are constructed possible to get order 3 with just three stages. Typically so that they share many function evaluations. Generally RK formulas involve families of parameters, and that this is done by starting with a family of formulas and looking for a good formula that uses the same stages is the case for this example. Explicit RK methods are much more expensive in and is of one order lower. Fehlberg [5] was the first to terms of function evaluations than an explicit Adams introduce these embedded formulas and produce useful method, but they are competitive because they are more pairs. For example, it takes at least six stages to obtain accurate. However, for this argument to be valid, a an explicit RK formula of order 5. He found a pair of program must be allowed to use the largest step sizes orders 4 and 5 that requires only the minimum of six that provide the specified accuracy. As a consequence, stages to evaluate both formulas. Later he developed it is especially inefficient with RK methods to obtain pairs of higher order [4], including a very efficient (7, output at specific points by reducing the step size so 8) pair of 13 stages. Another matter requires discussion at this point. as to produce a result at those points. Event location is scarcely practical for RK methods without a con- If each step is taken with two formulas, it is only tinuous extension. Unfortunately, it is much harder to natural to advance the integration with the higher-order construct continuous extensions for RK methods than formula provided, of course, that other properties like stability are acceptable. After all, the reliability of the for LMMs. An obvious approach to constructing a continuous error estimate depends on the higher-order formula extension is to use Hermite polynomial interpolation being more accurate. In this approach, called local exto yn ; yn1 ; : : : and fn ; fn1 ; : : :, much as with LMMs. trapolation, the step size is chosen to make the lowerGladwell [6] discusses the difficulties that arise when order result pass an error test, but a value believed to interpolating over just two steps. An important ad- be more accurate is used to advance the integration. All vantage of RK methods is that they do not require a the popular programs based on explicit RK methods do starting procedure like the methods of section “Linear local extrapolation. There is a related question about Multistep Methods,” but this approach to continuous the order of a continuous extension. If the formula extension does require starting values. Further, con- used to advance the integration has a local error that vergence of the approximation requires control of the is O.hpC1 /, the true, or global, error y.tn /  yn is

D

342

O.hp /, which is to say that the formula is of order p. Roughly speaking, for a stable problem and formula, errors at each step that are O.hpC1 / accumulate after O.1= h/ steps to yield a uniform error that is O.hp /. This leads us to the question as to the appropriate order of a continuous extension. It would be natural to ask that it has the order of the formula used to advance the integration, but it is not used to propagate the solution, so it can be one lower order and still achieve the global order of accuracy. Because it can be expensive to obtain a continuous extension at high orders, this is an important practical matter. Horn [10] was the first to present a modern approach to continuous extensions for RK methods. In her approach, a family of formulas is created, one for each point in the span of a step. Each member of the family is a linear combination of the stages used in the basic formula plus other stages as necessary. By virtue of reusing stages, it is possible to approximate the solution anywhere in the span of the step with a small number of extra stages. In more detail, suppose that a total of s stages are formed in evaluating the pair of formulas. For some 0 < < 1, we approximate the solution at tnC D tn C h with an explicit RK formula that uses these stages: ynC Dyn C h Œ 1 . /fn;1 C 2 . /fn;2 C : : : s . /fn;s  : This is a conventional explicit RK formula of s stages with specified coefficients ˛j ; ˇj;k for approximating y.tn C h/. We look for coefficients j . / which provide an accurate approximation ynC . This is comparatively easy because these coefficients appear in a linear way. Although we have described this as finding a family of RK formulas with parameter , the coefficients j . / turn out to be polynomials in

, so we have a continuous extension Pn . /. It can happen that there is enough information available to obtain approximations that have an order uniform in

that corresponds to the global order of the method. For instance, the (4, 5) pair due to Dormand and Prince [2] that is implemented in the ode45 program of MATLAB is used with a continuous extension that is of order 4. We digress to discuss some practical aspects of continuous extensions of RK formulas with this pair as example. Solutions of IVPs are customarily studied in graphical form in MATLAB, so the output of the solvers is tailored to this. For nearly all the solvers, which

Dense Output

implement a wide range of methods, the default output is the set ftn ; yn g chosen by the solver to obtain accurate results efficiently. Generally this provides a smooth graph, but there is an option that computes additional results at a fixed number of equally spaced points in each step using a continuous extension. The (4, 5) pair implemented in ode45 must take relatively large steps if it is to compete with Adams methods, and correspondingly, a solution component can change significantly in the span of a step. For this reason, results at mesh points alone often do not provide a smooth graph. The default output of this program is not just results at mesh points but results at four equally spaced points in the span of each step. This usually provides a smooth graph. In this context, a continuous extension is formed and evaluated at every step. The pair does not involve many stages, so any additional function evaluations would be a significant expense. This is why a “free” continuous extension of order 4 was chosen for implementation. Some of the continuous extensions can be derived in a more direct way by interpolation [12] that we use to raise another matter. The yn;j approximate y.tn;j /, but these approximations are generally of low order. Some information of high order of accuracy is to hand. After forming the result ynC1 that will be used to advance the integration, we can form f .tnC1 ; ynC1 / for use in a continuous extension. Certainly we would prefer continuous extensions that are not only continuous but also have a continuous derivative from one step to the next. To construct such an extension, we must have fnC1 . Fortunately, the first stage of an explicit RK formula is always fn D f .tn ; yn /, so the value fnC1 is “free” in this step because it will be used in the next step. We can then use the cubic Hermite interpolant to value and slope at both ends of the step as continuous extension. Interpolation theory can be used to show that it is an excellent continuous extension for any formula of order no higher than 3. It is used in the MATLAB program ode23 [13]. Some of the higher-order formulas that have been implemented have one or more intermediate values yn;j that are sufficiently accurate that Hermite interpolation at these values, and the two ends of the step provides satisfactory continuous extensions. If the stages that are readily available do not lead to a continuous extension that has a sufficiently high order uniformly in 0   1, we must somehow obtain additional information that will allow us to achieve our goal. A tactic [3] that has proved useful is to observe

Dense Output

that in addition to the ends of the step, the extension Pn;s . / based on s stages may be of higher order at one or more points in .0; 1/. If  is such a point, we define tn;sC1 D tn C  h and yn;sC1 D ynC  and evaluate fn;sC1 D f .tn;sC1 ; yn;sC1 /. If there is more than one such point, we do this for each of the points. We now try to find a continuous extension that uses these new stages in addition to the ones previously formed. If this new continuous extension has a uniform order that is acceptable, we are done and otherwise we repeat. This tactic has resulted in continuous extensions for some popular formulas of relatively high order. After Fehlberg showed the effectiveness of a (7, 8) pair of 13 stages, several authors produced pairs that are better in some respects and more to the point have continuous extensions. Current information about quality RK pairs is found at Verner [15]. Included there are (7, 8) pairs with a continuous extension of order 7 that requires three additional stages and order 8 that requires four. Implicit Runge-Kutta, IRK, formulas are exemplified by the two-stage formula yn;1 D yn C h Œˇ1;1 fn;1 C ˇ1;2 fn;2  ; yn;2 D yn C h Œˇ2;1 fn;1 C ˇ2;2 fn;2  ; ynC1 D yn C h Œ 1 fn;1 C 2 fn;2  : This is a pair of simultaneous algebraic equations for yn;1 and yn;2 , and as a consequence, it is much more trouble to evaluate an IRK than an explicit RK formula. On the other hand, they can be much more accurate. Indeed, if tn;1 and tn;2 are the nodes of the two-point Gauss-Legendre quadrature formula shifted to the interval Œtn ; tnC1 , the other coefficients can be chosen to achieve order 4. For non-stiff IVPs, this high order is not worth the cost. However, IRKs can also be very much more stable. Indeed, the two-stage Gaussian formula is A-stable. This makes them attractive for stiff problems despite the high costs of evaluating the formulas for such problems. IRKs are also commonly used to solve boundary value problems for ODEs. IVPs specify a solution of a set of ODEs by the value y.t0 / at the initial point of the interval t0  t  tF . Two-point boundary value problems, BVPs, specify a solution by means of values of components of the solution at the two ends of the interval. More specifically, the vector solution y.t/ is to satisfy a set of equations, g.y.t0 /; y.tF // D 0. In this context, the formula must be evaluated on all subintervals Œtn ; tnC1  simultane-

343

ously. This is typically a large system of nonlinear equations that is solved by an iterative procedure. If an approximation to y.t/ is not satisfactory, the mesh is refined and a larger system of algebraic equations is solved. A continuous extension is fundamental to this computation because it is used to generate starting guesses for the iterative procedure. The IRKs commonly implemented are based on Gaussian quadrature methods or equivalently collocation. There is a sense of direction with IVPs, so the formulas for stiff IVPs in wide use are based on Radau formulas. The lowest-order case is the implicit backward Euler method ynC1 D yn Chf .tnC1 ; ynC1 /, a formula that happens to be AM1 and BDF1. There is no preferred direction when solving BVPs with implicit RK methods, so the symmetric Gauss-Legendre or Gauss-Lobatto formulas are used. The nodes of the former do not include an endpoint of the step, and the nodes of the latter include both. As mentioned above, the two-point Gauss-Legendre formula is of order 4. It can be derived by collocation rather like the BDFs. This particular formula is equivalent to collocation with a quadratic polynomial Pn .t/ that interpolates Pn .tn / D yn and also P .tn;j / D yn;j for j D 1; 2. The yn;j are determined by the collocation conditions Pn0 .tn;j / D f .tn;j ; P .tn;j // for j D 1; 2. Although the formula is of order 4 at mesh points, this quadratic approximation has a uniform order of 2. This is typical. Popular codes like COLSYS [1] use Gauss-Legendre formulas of quite high order for which the uniform order of approximation by the collocation polynomial is roughly half the order of approximation at mesh points. This is not all that one might hope for, but a convenient continuous extension is very important and formulas of a wide range of orders are available. The three-point Gauss-Lobatto formula collocates at both endpoints of the step and the midpoint. The underlying cubic polynomial is uniformly of order 4, which is adequate for solving BVPs in MATLAB. The collocation conditions imply that the approximation is C 1 Œt0 ; tF , which is useful in a computing environment where results are often studied graphically.

Extrapolation Methods Extrapolation methods are built upon relatively simple methods of order 1 or 2 such as the explicit/forward Euler method and the implicit midpoint rule.

D

344

Dense Output

The construction of extrapolation methods relies on the theoretical existence of an asymptotic expansion of the global error of the low-order underlying method in terms of a constant step size h. For example, let us consider the explicit/forward Euler method ykC1 D yk C hf .tk ; yk /: We denote yh .tk C nh/ D ykCn for n D 0; 1; 2; : : : With initial condition y.tk / D yk , it can be shown that for sufficiently smooth f .t; y/, the global error at t D tk C nh of the explicit Euler method has an asymptotic expansion of the form yh .t/  y.t/ D e1 .t/h C e2 .t/h2 C : : : C eN .t/hN C EN .t; h/hN C1 ; where e1 .t/; : : : ; eN .t/ are smooth functions and EN .t; h/ is bounded for jhj sufficiently small. For a symmetric method such as the implicit midpoint rule, all the odd terms e2kC1 .t/ vanish. Given a finite sequence of increasing natural numbers ni for i D 1; : : : ; I such as ni D i , for a given macro step size H , we define the micro step sizes hi D H=ni . By independent applications of ni steps of the explicit Euler method with constant step size hi , we obtain a finite sequence Yi1 D yhi .tk C H / of approximations to the solution y.tk C H / of the ODE passing through y.tk / D yk . Defining the table of values Yi;j C1

Yij  Yi 1;j D Yij C .ni =ni j /  1

Due to their possible high-order, extrapolation methods may take large step sizes H . Hence, the use of a sufficiently high order continuous extension is really required if an accurate approximation at intermediate points is needed. A continuous extension can be obtained by building a polynomial approximation to the solution. First finite-difference approximations .m/ D1i .t/ to the derivatives y .m/ .t/ at the left endpoint t D tk , at the midpoint t D tk C H=2, or/and at the right endpoint t D tk C H are built for each index i when possible based on the intermediate values of f .t; y/ or y. In the presence of stiffness, it is not recommended to use the intermediate values based on f .t; y/ since f may amplify errors catastrophically, and approximations to the derivatives should be based only on the intermediate values of y in this .m/ situation. The values D1i .t/ are extrapolated to obtain higher-order approximations. We denote the most extrapolated value by D .m/ .t/. A polynomial P . / approximating f .tk C H / is then defined through Hermite interpolation conditions. For example, for the GBS algorithm, we consider a sequence of increasing even natural numbers ni satisfying ni C1  ni mod 4. We define a polynomial Pd . / of degree d C 4 with 1  d  2I satisfying the Hermite interpolation conditions Pd .0/ D yk ;

Pd .1/ D YII ;

Pd0 .0/ D Hf .tk ; yk /;

Pd0 .1/ D Hf .tk C H; YII /; .m/

Pd .1=2/ D H m D .m/ .tk C H=2/ for m D 0; : : : ; d:

For n1 D 4 and d  2I 4, it can be shown that Pd . / (3) is an approximation of order 2I in H to y.t C H /, k i.e., Pd . /  y.tk C H / D O.H 2I C1 / for 2 Œ0; 1. the extrapolated values If one wants to have a continuous extension with Yij are of order j , i.e., Yij   y.tk C H / D O H j C1 . For symmetric methods, we a certain required accuracy, one also needs to control 2 replace the term ni =ni j in (3)by .ni =n  i j / , and we its error and not just the error at the endpoint. This 2j C1 . If there is no can be done by using an upper bound on the norm obtain Yij  y.tk C H / D O H stiffness, an efficient symmetric extrapolation method of the difference between the continuous extension is given by the Gragg-Bulirsch-Stoer (GBS) algorithm and another continuous extension of lower order. For where yh .tk C nh/ for n  2 with n even is obtained example, for the GBS algorithm for d  0, one can starting from z0 D yk as follows: consider the difference for i D 2; : : : ; I; j D 1; : : : ; i  1;

z1 D z0 C hf .tk ; z0 /; zlC1 D zl1 C 2hf .tk C lh; zl / for l D 1; : : : ; n; yh .tk C nh/ D

1 .zn1 C 2zn C znC1 /: 4

Pd . /  Pd 1 . / D 2 .1  /2 .  1=2/d cd C4 ; where cd C4 is the coefficient of d C4 in Pd . /. The function j 2 .1  /2 .  1=2/d j is maximum on Œ0; 1

Density Functional Theory

at d D 12 .1 ˙ estimate

345

p d=.d C 4//, and we obtain the error 14. Shampine, L.F., Baca, L.S., Bauer, H.J.: Output in extrapo-

lation codes. Comput. Math. Appl. 9, 245–255 (1983) 15. Verner, J.: Jim Verner’s refuge for Runge-Kutta pairs. http:// people.math.sfu.ca/jverner/ (2011)

max kPd . /  Pd 1 . /k  j d2 .1  d /2

2Œ0;1

. d  1=2/d j  kcd C4 k; which can be used in a step size controller. For more information on continuous extensions for extrapolation methods, we refer the reader to Hairer and Ostermann [7] for the extrapolated Euler method and the linearly implicit Euler method; to Hairer and Ostermann [7], Hairer et al. [8], and Shampine et al. [14] for the GBS algorithm; and to Jay [11] for the semi-implicit midpoint rule.

Density Functional Theory Rafael D. Benguria Departamento de F´ısica, Pontificia Universidad Cat´olica de Chile, Santiago de Chile, Chile

Synonyms Exchange corrections; Generalized gradient corrections; Kohn–Sham equations; Local density approximation; Statistical model of atoms; Thomas–Fermi

References 1. Ascher, U.M., Christiansen, J., Russell, R.D.: Collocation software for boundary value ODEs. ACM Trans. Math. Softw. 7, 209–222 (1981) 2. Dormand, J.R., Prince, P.J.: A family of embedded RungeKutta formulae. J. Comput. Appl. Math. 6, 19–26 (1980) 3. Enright, W.H., Jackson, K.R., Nørsett, S.P., Thomson, P.G.: Interpolants for Runge-Kutta formulas. ACM Trans. Math. Softw. 12, 193–218 (1986) 4. Fehlberg, E.: Classical fifth-, sixth-, seventh-, and eighth order Runge-Kutta formulas with step size control. Technical report, 287, NASA (1968) 5. Fehlberg, E.: Klassische Runge-Kutta-Formeln vierter und niedrigerer Ordnung mit Schrittweiten-Kontrolle und ihre Anwendung auf W¨armeleitungsprobleme. Computing 6, 61–71 (1970) 6. Gladwell, I.: Initial value routines in the NAG library. ACM Trans. Math. Softw. 5, 386–400 (1979) 7. Hairer, E., Ostermann, A.: Dense output for extrapolation methods. Numer. Math. 58, 419–439 (1990) 8. Hairer, E., Nørsett, S.P., Wanner, G.: Solving Ordinary Differential Equations I. Nonstiff Problems. Springer Series in Computational Mathematics, vol. 18, 2nd edn. Springer, Berlin (1993) 9. Henrici, P.: Discrete Variable Methods in Ordinary Differential Equations. Wiley, New York (1962) 10. Horn, M.K.: Fourth and fifth-order scaled Runge-Kutta algorithms for treating dense output. SIAM J. Numer. Anal. 20, 558–568 (1983) 11. Jay, L.O.: Dense output for extrapolation based on the semiimplicit midpoint rule. Z. Angew. Math. Mech. 73, 325–329 (1993) 12. Shampine, L.F.: Interpolation for Runge-Kutta methods. SIAM J. Numer. Anal. 22, 1014–1027 (1985) 13. Shampine, L.F., Reichelt, M.W.: The MATLAB ODE suite. SIAM J. Sci. Comput. 18, 1–22 (1997)

Definition Density functional theory (DFT for short) is a powerful, widely used method for computing approximations of ground state electronic energies and densities in chemistry, material science, and biology. The purpose of DFT is to express the ground state energy (as well as many other quantities of physical and chemical interest) of a multiparticle system as a functional of the single-particle density  .

Overview Since the advent of quantum mechanics [20], the impossibility of solving exactly problems involving many particles has been clear. These problems are of interest in such areas as atomic and molecular physics, condensed matter physics, and nuclear physics. It was, therefore, necessary from the early beginnings to introduce approximative methods such as the Thomas–Fermi model [4, 21], (see J. P. Solovej  Thomas–Fermi Type Theories (and Their Relation to Exact Models) in this encyclopedia) and the Hartree–Fock approximation [5, 6] (see I. Catto  Hartree–Fock Type Methods in this encyclopedia), to compute quantities of physical interest in these areas. In quantum mechanics of many particle systems,

D

346

Density Functional Theory

N N the main object of interest is the wavefunction X „2 X VN 2 3

i  e V .xi / H D L .R /, (the antisymmetric tensor 2 2m i D1 i D1 product of L2 .R3 /). More explicitly, for a system X of N fermions, .x1 ; : : : ; xi ; : : : ; : : : xj ; : : : xN / D 1 Ce 2 C U; (2)  .x1 ; : : : ; xj ; : : : ; : : : xi ;R: : : xN /, in view of Pauli jxi  xj j 1i 1:234. The first to estimate the difference between the expectation value of I and the direct term was Dirac [3], who estimated this difference for an ideal gas of N electrons in a box of volume V and obtained the expression cD e 2=3 .N=V /4=3 , with cD D .3=4/.3=/1=3  0:74. The Gradient Correction The TF model is attractive because of its simplicity, is not satisfactory for atomic problems because it yields Thomas–Fermi Model and Corrections an electron density with incorrect behavior very close and very far away from the nucleus. Moreover, acThe statistical model of atoms and molecules (hence- cording to Teller’s Lemma (see, e.g., [14], Sect. III.C), forth TF) [4, 14, 21] was the first DFT model to be it does not allow the existence of molecules. As we introduced. In units in which „2 =.2m/ D 1 and e D 1, mention above, also there are no negative ions in TF. it is defined via the functional In 1935, Weizs¨acker [22] suggested the addition of the inhomogeneity correction (gradient correction) Z Z 3 E./ D dTF 5=3 dx  V .x/.x/ dx Z 5 R3 R3 .r/2 dx; (12) c X Zk Z` W  R3 ; (10) CD.; / C jRk  R` j 1k 0. are quickly smoothed out. The diffusion equation requires boundary conditions. As an example, we consider diffusion in the unit square  D f.x; y/ W 0  x  1; 0  y  1g Numerical Methods with prescribed values for u on the boundary @. The When computing the numerical solution, we use a complete initial–boundary value problem is different formulation of (1). The differential equation     is multiplied by a function .x; y/ and integrated. @ @u @ @u @u In other words, we take the scalar product of the D d C d ; @t @x @x @y @y differential equation with . We apply the integration by parts procedure precisely as above and obtain .x; y/ 2 ; t  0 ; (1) u.x; t/ D e d! t sin.!x/ : 2

u.x; y; t/ D g.x; y; t/ ; u.x; y; 0/ D f .x; y/ :

.x; y/ 2 @ ;



@u ; @t





@ @u D ;d @x @x



 

@ @u ;d @y @y

 : (2)

Diffusion Equation: Computation

355



We now define the function space

@uN @uN ;d D 2 @x @x



@v 2 @v 2



S D v.x; y/ W kvk2 C C < 1; @x @y  v.x; y/ D 0 for .x; y/ 2 @˝ ;

 2d0





@uN @uN ;d 2 @y @y





!

@uN 2 @uN 2



@x C @y :

Here we used (2) with  D uN , which is possible since uN 2 SN S . If this requirement is violated (nonconforming elements), we may still have a good approximation, but the theory becomes more complicated. For nonzero data g, we require that the boundary condition in (1) is satisfied for the FEM solution uN . However, the -functions still belongs to the space SN as defined above with homogeneous conditions. With Neumann boundary conditions @u=@nC ˛u D g, an extra term must be added to (2), but the format of this article does not allow for further discussion of this case. The most elementary finite elements are piecewise linear functions. In two space dimensions, they are defined on a mesh composed of triangles with nodes at the corners. Each basis function is one at a given node and zero at all other nodes. On each triangle, every function has the form ax C by C c, which is uniquely determined by its values at the corner nodes. The piecewise polynomials are continuous across all edges. N On most of the triangles, they are identically zero. If h X cj .t/j .x; y/ ; uN .x; y; t/ D is the typical side length of the triangles, one can prove j D1 that the error jjuN  ujj is of the order h2 . For higher order accuracy, higher order polynomials are used on such that (2) is satisfied with u D uN and .uN ; / D each triangle. That requires more nodes associated with .f; / for t D 0 and all functions  2 SN . each triangle. Figure 1 shows the location of these This is the Galerkin finite element method. A very nodes for the quadratic and cubic case. nice consequence of the Galerkin formulation is that The introduction of uN .x; y; t/ into the Galerkin stability follows automatically. We have formulation results in a system of ordinary differential equations   d @uN dc 2 jjuN jj D 2 uN ; D Qc ; M dt @t dt

and formulate the problem • Find the function u.x; y; t/ with u 2 S for any t such that (2) is satisfied and .u; / D .f; / for t D 0 and all functions  2 S . (Here the initial condition is also formulated in its integrated form.) This is called the weak form of the problem. It is obviously satisfied by the “classical” solution of (1), but it also allows for relaxed regularity requirements on the solution. The integrals exist for piecewise differentiable functions, and we do not have to worry about the second derivatives. For the numerical solution v.x; y; t/, we now define an approximation space SN S . The subscript N indicates that the subspace has N degrees of freedom, and there are N basis functions j .x; y/. If all functions in SN are piecewise polynomials, we have a finite element space. The numerical solution is formally defined by • Find the function

a

b

linear

quadratic

c

cubic

Diffusion Equation: Computation, Fig. 1 Nodes for the specification of polynomials on triangles. (a) Linear. (b) Quadratic. (c) Cubic

D

356

Diffusion Equation: Computation Temperature [°C]

1

0.5

100

0.6

90

0.4

80

0.2

70

0

0

60

−0.2

50

−0.4

−0.5

40

−0.6 −1

Max:100.768

0.8

30

−0.8 −1

−0.5

0 Grid

0.5

1

20 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 Min:19.232 Temperature after 10 minutes

Diffusion Equation: Computation, Fig. 2 FEM computation of temperature distribution. (a) Grid. (b) Temperature after 10 min

where the vector c contains the coefficients cj .t/. The elements of the matrices M and Q contain integrals of i j and its derivatives. Since each i is zero in most of the domain, there is a nonzero overlap with only a few of the neighboring functions. As a consequence, the matrices are sparse. For time discretization, standard difference methods are used in most cases. The trapezoidal rule and the backward Euler method are both unconditionally stable, that is, the choice of time step is governed only by accuracy considerations. At each time level tn , a large system of equations must be solved. For realistic problems in two and three space dimensions, an iterative solution method is required. There is an abundance of such methods; the conjugate gradient methods are very common. Multigrid methods are also very effective. Starting from the given fine grid, they solve a number of different systems corresponding to a sequence of coarser grids. As a computational example, we choose a heat conduction problem for a plate, which has the temperature 20ı C initially. At t D 0, two edges are suddenly given the temperature 100ı C. Figure 2 shows the grid and the computed solution at t D 10 minutes. Note the smooth temperature distribution despite the severe discontinuity at the start. The computation was done using the COMSOL system.

Further Reading The book [3] contains a basic discussion of scientific computing and numerical methods.The books [6,7] are recommended for those who want to learn more about finite element methods. Solution methods for linear systems of algebraic equations are thoroughly treated in the books [1, 5]. Finite element methods are dominating when it comes to numerical solution of diffusion problems. However, there are also effective difference methods, in particular if it is possible to construct a structured curvilinear grid. Such methods are discussed in [2, 4].

References 1. Golub, G., van Loan, C.: Matrix Computations, 3rd edn. John Hopkins University Press, Baltimore (1996) 2. Gustafsson, B.: High Order Difference Methods for Time Dependent PDE. Springer, Berlin/Heidelberg (2008) 3. Gustafsson, B.: Fundamentals of Scientific Computing. Texts in Computational Science and Engineering. Springer, Berlin/Heidelberg/New York (2011) 4. Gustafsson, B., Kreiss, H.-O., Oliger, J.: Time Dependent Problems and Difference Methods. Wiley, New York (1995) 5. Saad, Y.: Iterative Methods for Sparse Linear Systems. SIAM, Philadelphia (2003) 6. Strang, G., Fix, G.: An Analysis of the Finite Element Method, 2nd edn. Wellesley–Cambridge Press, Wellesley (2008) 7. Zienkiewics, O., Taylor, R.: The Finite Element Method, Vol 1: The Basis. Butterworth/Heineman, Oxford (2008)

Direct Methods for Linear Algebraic Systems

Direct Methods for Linear Algebraic Systems Iain Duff Scientific Computing Department, STFC – Rutherford Appleton Laboratory, Oxfordshire, UK CERFACS, Toulouse, France

Synonyms Matrix factorization

Definition Direct methods for solving linear algebraic equations differ from iterative methods in that they provide a solution to a system of equations in a finite and predetermined number of steps. They involve expressing the matrix as a product of simpler matrices (called factors) where the solution of equations involving these matrices is very easy.

Overview We discuss matrix factorization and consider in particular the factorization of a matrix A into the product of a lower triangular matrix L and an upper triangular matrix U . We examine the complexity, stability, and efficiency of LU factorization in the case where the matrix A is dense. We then consider the case when the matrix is sparse. Finally, we discuss the limitations of direct methods and discuss an approach by which they may be overcome.

357

the entries in the matrix and vectors are real numbers but mutatis mutandis our remarks apply equally to the case when the entries are complex numbers. The class of methods that we will discuss for solving equation (1) are called direct methods and involve expressing A as the product of simpler matrices through a technique called matrix factorization. The essential feature is that equations involving these simpler matrices are easy to solve. The two main factorizations that are used are the LU factorization, where L is lower triangular and U is upper triangular, and the QR factorization, where Q is orthogonal and R is upper triangular. The solution of equations using the LU factorization first solves the system Ly D b. When using the QR factorization, the vector y is obtained just by multiplying the right-hand side by QT . In both cases, the solution of the system is then obtained through solving a triangular system (with matrix R or U ) using backward substitution. LU factorization is by far the most common approach so we will consider it in the remainder of this entry. An excellent and standard text for matrix factorization algorithms is the book by Golub and Van Loan [6]. We first consider the case when the matrix A is dense, that is, when there are insufficiently zero entries for us to take advantage of this fact. We will later treat the case when A is sparse.

LU Factorization

The LU factorization of a matrix A of order n proceeds in n  1 major steps. At step 1, multiples of the first equation are subtracted from equations 2 to n where the multiples are chosen so that variable x1 is removed from the succeeding equations. At step k (k D 2; n  1), the reduced equation k is used to remove variable k from all the later equations. After all major steps are completed, the remaining system will be upper Matrix Factorizations triangular with coefficient matrix U . Each major step corresponds to multiplying the matrix prior to that step We consider the solution of the linear equations by an elementary lower triangular matrix, Lk , which Ax D b (1) is the identity matrix except in column k where the lower triangular entries correspond to the multiples where A is a matrix of order n and x and b are used to eliminate variable k from each equation. Thus, column vectors of length n. The matrix A and vector the factorization can be expressed as b (the right-hand side) are known and we wish to find the solution x. In our discussion we will assume that (2) Ln1 Ln2 : : : L2 L1 A D U:

D

358

Direct Methods for Linear Algebraic Systems

As the inverse of these elementary matrices are the We note that the abovementioned complexity is only same as the matrix except that the signs of the off- applicable for dense matrices. We will see later that the diagonals are reversed, we have situation is quite different for sparse matrices. 1 1 1 A D L1 1 L2 : : : Ln2 Ln1 U D LU;

(3) Stability of LU Factorization The stability of the factorization can be measured using because the product of the n  1 lower triangular the backward error analysis of [9] where bounds can be N matrices is lower triangular. computed for the matrix E where ACE D LN UN with L If the matrix is symmetric (A D AT ), the factoriza- and UN the computed result for L and U , respectively. tion can be written as Clearly, if the simple algorithm described earlier is used, the factorization can fail, for example, if variable (4) 1 does not appear in (1) (entry a11 is zero). However, A D LLT even if it were nonzero, it might be very small in which is called a Cholesky factorization. Note that this magnitude with respect to other entries in the matrix, involves taking square roots for the diagonal entries. and its use as a pivot would cause growth in the This can cause problems if the matrix is not positive entries of the reduced matrix and consequent large entries in E. definite so that an alternative factorization To avoid the worst excesses of this, the normal T (5) recourse is to use what is called partial pivoting. At the A D LDL beginning of major step k, the rows are interchanged so that the coefficient of variable k in equation k is larger is often used where D is block diagonal with blocks of or equal in modulus to the coefficient of variable k in order 1 or 2. all the remaining equations. In terms of the matrix, we will choose akk so that Complexity of LU Factorization The complexity of the factorization is easily obtained by examining the algorithm described previously. At major step k, there are n  k minor steps involving the subtraction of one vector of length n  k from another so the complexity of step k in terms of number of arithmetic operations is 2 .nk/2 plus nk divisions to compute the multipliers. When this is summed from k D 1 to n  1, the overall complexity becomes 2=3n3 C O.n2 /:

n

jakk j  max jai k j: i Dk

(6)

Although this algorithm is not backward stable (in the sense that jjEjj is guaranteed small), it has been proven to work well in practice and is the strategy employed by most computer codes for direct solution. A good and standard text for error analysis is the book by Higham [7].

Implementation of LU Factorization There are several variants of algorithms for implementing LU factorization (also known as Gaussian elimination) but all have the same complexity. However, the efficiency (in terms of computer time) can be greatly influenced by the details of the implementation. The best current implementations of LU factorization make use of the Level 3 BLAS kernels, the most important of which is GEMM for matrix-matrix multiplication. This kernel is often tuned by vendors for their A major benefit of direct methods is that, once the computer architectures and commonly will perform at factorization has been performed, the factors can be close to the peak performance of the machine. There used repeatedly to solve any system with the same co- are versions of the BLAS ( e.g., the multithreaded efficient matrix at the same lower order of complexity. BLAS) that are tuned for parallel architectures. These

In the symmetric case, the complexity is approximately halved and is 1=3n3 C O.n2 /. It is important to note that this complexity only refers to the initial factorization. Once this has been done, the solution is obtained through a single pass on the triangular factors so that the complexity for doing this is only twice the number of entries in the factors and so is 2 n2 C O.n/:

Direct Methods for Linear Algebraic Systems

kernels can be used in the LU factorization by performing the factorization by blocks. The effect of this is to replace computations of the form aij D aij  li k ukj by Aij D Aij  Li k Ukj where the matrix-matrix multiplies are performed using GEMM. The effect of using this approach is that when n is large the speed of the LU factorization approaches that for GEMM so that the factorization can usually get quite close to peak machine performance. Indeed although the actual implementation is often more complicated, this is the flavor of the algorithms used by vendors in the LINPACK benchmark [3]. We also note that since the kernel of such a factorization technique uses matrix–matrix multiplications, it is possible to use techniques based on Strassen’s algorithm to effect a dense factorization with complexity less than O.n3 /.

Solution of Sparse Equations

359 × × × × × × × × × ×

× × ×

and then eliminate without interchanges, the resulting factors would be dense. This example is extreme but in general permutations can greatly affect the increase in number of entries between the original matrix and the factors (the fill-in). There are many strategies for reducing fill-in in sparse factorization. One of the earliest was proposed by Markowitz [8] which is to choose, at stage k, an entry from the reduced matrix that minimizes the product of the number of entries in the row and the number of entries in the column and then to permute its row and column to position k. The symmetric analogue of this selects the diagonal entry in the row/column with the least number of entries. This is called the minimum degree algorithm and would give an ordering in the small example above that avoids any fill-in. Of course, it is necessary to still guard against instability. Partial pivoting is not an option as it would restrict too much the ability to preserve sparsity, but a weaker form of this called threshold pivoting is used, viz,

When the matrix is sparse, that is to say when many entries are zero, the situation is quite different from that described in previously. This is a very common occurrence as matrices coming from discretizations of partial differential equations, structural analysis, largescale optimization, linear programming, chemical engineering, and in fact nearly all large-scale problems are sparse. In such cases, although the underlying aln gorithm remains the same, the complexity, implemen(7) jakk j  u max jai k j; i Dk tation, and performance characteristics differ markedly from the dense case. We indicate this difference in the remainder of this entry. where u is a threshold parameter such that 0 < u  1. The benefit of this strategy is that the choice of u can determine the balance between sparsity preservation Matrix Factorizations for Sparse Systems When the matrix is sparse, the main concern is to (u near to 0) and stability (u near to 1:0). There are many variations and modifications to preserve this sparsity as much as possible. For example, if no interchanges are performed and pivoting is these simple strategies. Indeed although simple, their performed down the diagonal on the arrowhead matrix implementation can be anything but. In addition, it is also possible to perform a priori permutations to × × restrict fill-in. These can include orderings like [1] for × × narrow bandwidth, orderings to block triangular form, × × and nested dissection. Nested dissection can be shown × × to be optimal on matrices arising from simple finite× × × × × difference discretizations of Laplace’s equation on a square domain. there would be same number of entries in the factors as Information on these orderings can be found in the in the original matrix. However, if we were to permute books [2, 4, 5] which have a full discussion of direct the matrix to the form methods for solving sparse systems.

D

360

Direct Methods for Linear Algebraic Systems

almost proportional to the number of entries in the matrix. The subsequent numerical factorization can use Grid dimensions Matrix order Work to factorize Factor storage dense BLAS kernels so that the efficiency of a sparse kk k2 O .k 3 / O .k 2 log k/ factorization can often reach half of the peak perfor3 6 kkk k O .k / O .k 4 / mance on a wide range of computer architectures. There are many codes for sparse factorization and some are tuned for parallel computers both for shared Complexity and Stability for Sparse Systems memory (including multicore) and for distributed It is not possible to give a simple formula for the memory. complexity of a sparse factorization. From the previous discussion this will clearly depend on the ordering but the structure of the matrix will also influence this. Limitations of Direct Methods and One simple example for which results exist is for the Alternatives factorization of a matrix resulting from a simple finitedifference discretization of a Laplacian matrix in 2 or In spite of the abovementioned recent advances in 3 dimensions. If the grid has k grid points in each sparse factorization, the approach can become infeasidirection, we show the complexity when the matrix is ble for very large three-dimensional problems normally factorized using a nested dissection ordering in Table 1. because of the number of entries in the factors. In such This has been proved to be the best ordering in an cases, direct methods can still be used but either as an approximate factorization for use as a preconditioner asymptotic sense for such matrices. We note the difference between the complexity of for iterative methods or on a subproblem of the original the factorization and the subsequent solution and the problem. A good example of this would be in a dofact that, in the two-dimensional case, this solution main decomposition approach where a hybrid solution is almost linear in the matrix order. Although these scheme is used with a direct solver being used on the figures are often used to suggest that direct methods are local subdomains and an iterative technique on the only feasible for two-dimensional problems, the shape boundary. Another example of a hybrid method would of a cube is almost optimally bad, and there are many be a block iterative method. 3D problems for which direct methods work well, for example, a narrow pipe geometry. Cross-References Some very recent work that effectively makes use of the structure of the Green’s function has led to direct  Classical Iterative Methods factorization algorithms with linear complexity.  Domain Decomposition As might be expected, the stability of threshold  Preconditioning pivoting is even worse than that of partial pivoting although in practice it has been found to work well. It is important, however, to build safeguards in sparse References codes, for example, to perform iterative refinement if 1. Cuthill, E., McKee, J.: Reducing the bandwidth of sparse the residual (b  Ax) is large. Direct Methods for Linear Algebraic Systems, Table 1 Complexity of sparse Gaussian elimination on 2D and 3D grids

Implementation of Sparse Direct Solution Even more than in the dense case, the implementation of sparse LU factorization is crucial for the viability of direct methods. There have been significant advances in the last few years that have enabled direct solution methods to be used on problems with over a million degrees of freedom. For a large class of methods, we can obtain an ordering based only on the pattern of the matrix and then, if necessary, perform interchanges in a subsequent factorization. The original symbolic analysis can be performed very efficiently, in time

2. 3.

4. 5.

6.

symmetric matrices. In: Proceedings of the 24th National Conference of the Association for Computing Machinery, New York, pp. 157–172. Brandon, Princeton (1969) Davis, T.A.: Direct Methods for Sparse Linear Systems. SIAM, Philadelphia (2006) Dongarra, J.J., Luszczek, P., Petitet, A.: The LINPACK benchmark: past, present and future. Concurr. Comput. Pract. Exp. 15, 803–820 (2003) Duff, I.S., Erisman, A.M., Reid, J.K.: Direct Methods for Sparse Matrices. Oxford University Press, Oxford (1986) George, A., Liu, J.W.H.: Computer Solution of Large Sparse Positive Definite Systems. Prentice-Hall, Englewood Cliffs (1981) Golub, G.H., Van Loan, C.F.: Matrix Computations, 4th edn. The Johns Hopkins University Press, Baltimore (2013)

Discontinous Galerkin Methods: Basic Algorithms

361

• The functions satisfy the boundary conditions specified on the solution to the PDE, strongly in the case of Dirichlet boundary conditions and weakly in the case of Neumann boundary conditions. Thus, in the traditional finite element method, the numerical solution to the PDE is a continuous, piecewise polynomial defined on a collection of elements. The discontinuous Galerkin finite element method, or DG method for short, breaks the continuity Discontinous Galerkin Methods: Basic requirement on the numerical solution, allowing Algorithms the solution to be discontinuous at inter-element boundaries, and allows for a weak approximation of Clint N. Dawson boundary conditions, including Dirichlet boundary Institute for Computational Engineering and Sciences, conditions. The perceived advantages of the DG University of Texas, Austin, TX, USA method are: • The ability to preserve local conservation properties, such as conservation of mass, Synonyms • The ability to easily refine the mesh locally within an element without the difficulty of dealing with DG methods; Interior penalty methods; Nonconformhanging nodes (h adaptivity), ing finite element methods • The ability to use different polynomials on each element (p adaptivity) depending on the smoothness of the problem, Definition • The ability to treat boundary and other external conditions weakly, and Discontinuous Galerkin finite element methods are • The method is highly parallelizable, as most of the a class of numerical methods used to approximate work is done at the element level and the stencil solutions to partial differential equations. usually involves only neighboring elements. There has been recent speculation that, because of the latter property, DG methods may work well on new computer architectures which use a mixture of CPUs Overview and GPUs (Graphical Processing Units).

7. Higham, N.J.: Accuracy and Stability of Numerical Algorithms, 2nd edn. SIAM, Philadelphia (2002) 8. Markowitz, H.M.: The elimination form of the inverse and its application to linear programming. Manag. Sci. 3, 255–269 (1957) 9. Wilkinson, J.H.: Error analysis of direct methods of matrix inversion. J. ACM 8, 281–330 (1961)

The Galerkin finite element method has long been used in the numerical solution of partial differential equations (PDEs). In these methods, the domain over which the PDE is defined is discretized into elements; that is, the domain is covered by a finite number of geometrical objects, such as intervals in a onedimensional domain, triangles or rectangles in a twodimensional domain, and tetrahedra, prisms, or hexahedra in a three-dimensional domain. On this set of elements, the solution to the PDE is approximated in a space of functions which typically satisfy the following properties: • The functions are polynomials of degree at most k on each element, where k  1. • The functions are globally continuous; that is, continuity between the polynomials is enforced at element boundaries.

Historical Background The basic idea behind DG methods can be traced to a paper by Lions in 1968 [1] for a second-order elliptic Dirichlet boundary-value problem. The idea was to approximate the Dirichlet boundary conditions weakly through adding a penalty term. Nitsche in 1971 [2] formalized this method and proved its convergence. Based on these works, the notion of enforcing inter-element continuity through penalties was investigated, leading to the so-called interior penalty (IP) Galerkin methods for elliptic and parabolic equations. Despite extensive analysis in the 1970s, IP methods were never adopted by the computational science community because they were viewed as being computationally inefficient on the computers available at that time. With the advent of more powerful, parallel

D

362

Discontinous Galerkin Methods: Basic Algorithms

computers, IP methods were revived in the 1990s for elliptic boundary-value problems and a substantial literature now exists; an excellent summary is given in [3] with a unified presentation of many wellknown DG and IP methods for elliptic boundary-value problems. The DG method was introduced for first-order hyperbolic partial differential equations by Reed and Hill in 1973 [4]. This initial work led to extensive investigation and application of DG methods for hyperbolic conservation laws which continues today. For hyperbolic PDEs, DG methods can be viewed as an extension of low-order finite volume methods to higherorder approximations and very general geometries. The application of DG methods for problems which are nearly hyperbolic, for example, advectiondominated advection-diffusion equations, combines the ideas of the IP methods for elliptic PDEs with the DG method for hyperbolic PDEs. We will discuss the mechanics of the DG method for a simple advection-diffusion model problem below. We note that DG methods have been investigated now for a wide variety of problems, including compressible and incompressible flows, multiphase flow and transport in porous media, shallow water and ocean hydrodynamics, electromagnetics, and semiconductors, just to name a few. We conclude this section by noting that there are several excellent reviews and textbooks on DG methods in the literature; see, for example, [5–7].

fairly general boundary conditions ˛0 f .c/  ˇ0 cx D 0 ; x D 0; t > 0 ˛1 f .c/  ˇ1 cx D 1 ; x D L; t > 0 where the ˛’s and ˇ’s are coefficients which could be zero or one, although both coefficients can’t be zero simultaneously. The interval Œ0; L is divided into elements Bj D Œxj 1=2 ; xj C1=2  of length hj , with xj denoting the midpoint of the element, j D 1; : : : ; J . We consider a test space Vh of functions which are in H 2 inside each element, but are not continuous at the interior interface points xj C1=2 . Notationally, let   v  .xj C1=2 / D lim v xj C1=2 C ı (2) ı!0

    v xj C1=2 D lim v xj C1=2 C ı (3) C

ı!0C

  ŒŒv.xj C1=2 / D v .xj C1=2 /  v C xj C1=2 (4) 

fv.xj C1=2 /g D

 1  v .xj C1=2 / C v C .xj C1=2 / : (5) 2

A Weak Formulation Multiplying (1) by v 2 Vh , integrating over a single element Bj , and integrating by parts, we arrive at the weak form of (1): Z Œct v  f .c/vx C cx vx  dx Bj

Basic Methodology

x

C Œf .c/  cx  vjxjj C1=2 1=2 D

Z gvdx:

(6)

Bj

In order to explain the basic methodology behind the DG and related IP methods, we will focus on Summing over all Bj , a simple one-space-dimensional advection-diffusion equation with solution c.x; t/ satisfying J Z X Œct v  f .c/vx C cx vx dx ct C f .c/x  cxx D g;

0 < x < L; t > 0

(1)

j D1 Bj J 1 X

Z

C Œf .c/  cx ŒŒvjxj C1=2 D gvdx where the flux function f could be linear in c, f .c/ D Bj j D1 uc for some given coefficient u, or f .c/ could be nonCŒf .c/  cx vjxD0  Œf .c/  cx vjxDL : (7) linear in c, for example, in Burger’s equation f .c/ D 2 c =2. The diffusion coefficient   0. We will focus only on DG discretization in space and comment on the Notice that we have not applied any of the boundary time discretization in the next section. We assume an conditions to the test space or to the weak formulation initial condition c.x; 0/ D c0 .x/ and we will assume at this point.

Discontinous Galerkin Methods: Basic Algorithms

363

numerical fluxes have been proposed and can be found in the literature. Let

Approximating Spaces and Numerical Fluxes Next define an approximating space

b.C  ; C C /  fCx g C ŒŒC  F .C  ; C C / D f

Wh D fv W v 2 P k .Bj /; j D 1; : : : ; J g which is a finite dimensional subspace of Vh , where P k denotes the set of all polynomials of degree less than or equal to k, k  1. The dimension of Wh is J .k C 1/, and given a set of basis functions Pj;l , one can write any function v 2 Wh as

at any interior interface xj C1=2 . The boundary conditions may be enforced as follows. We consider the left boundary x D 0 and the right boundary x D L analogous: • If ˛0 D ˇ0 D 1, then .f .c/  cx /jxD0 D 0  F0 :

v.x/ D

k J X X

vj;l Pj;l .x/

• If ˛0 D 1 and ˇ0 D 0, then

j D1 lD0

f .c/jxD0 D 0 for some coefficients vj;l . Typical basis functions are the set of Legendre polynomials defined on Bj up and we define F0 D 0  Cx jxD0 . through order k. We will approximate c by a function • If ˛0 D 0 and ˇ0 D 1, then C in Wh ; however, we must modify (7) in several ways. First, note that since C will be discontinuous at inter-element boundaries, the “flux” terms f .c/ and cx cx jxD0 D 0 are not well defined. We will approximate these terms by “numerical fluxes”: and we define F0 D f .C /jxD0 C 0 . Here C and Cx are taken from inside the domain. b.C  .xj C1=2 /; C C .xj C1=2 //; (8) We define a boundary flux FL analogously to F0 . f .c.xj C1=2 //  f Finally, in order to complete the definition of the cx .xj C1=2 /  fCx .xj C1=2 /g C ŒŒC.xj C1=2 /: (9) DG method, we may add another stabilization term. This terms is “zero” because it involve jumps in the In (9), the second term is the so-called interior penalty true solution, which we assume to be smooth. The final term, since it penalizes the jump in the approximate DG weak form is written as follows: solution, where the penalty parameter  > 0 must be J Z b.C  ; C C / is the X chosen. A simple numerical flux f ŒCt v  f .C /vx C Cx vx  dx local Lax-Friedrichs flux given by B j D1

b.C  ; C C / D ff .C /g C  ŒŒf .C / f 2 0

(10) C

where  D sup jf .c/j. Note that if f D uc for some coefficient u, then

J 1 X

j

F .C  ; C C /ŒŒvjxj C1=2  s

j D1

J 1 X

ŒŒC fvx gjxj C1=2

j D1

Z D

gvdx C F0 vjxD0  FL vjxDL v 2 Wh : Bj

(12) b.C  ; C C / D f



C ; C C;

u>0 u 0 is a given function. The computational domain Œ0; 1 is discretized into

Discontinuous Galerkin method (DG); Local discontinuous Galerkin method (LDG); Runge–Kutta disconwith tinuous Galerkin method (RKDG)

0 D x 1 < x 3 <    < xN C 1 D 1 2

2

Ii D .xi  1 ; xi C 1 /I

Short Definition

u.0/ D g (1)

2

2

2

xi D xi C 1  xi  1 ; 2

2

h D max xi : 1i N

The discontinuous Galerkin method is a class of finite element methods using completely discontinuous basis The finite element space is given by functions to approximate partial differential equations. ˚ Vh WD v W vjIi 2 P k .Ii /I 1  i  N ; where P k .Ii / denotes the set of polynomials of degree up to k defined on the cell Ii . The DG method for solving (1) is defined as follows: find the unique function The discontinuous Galerkin (DG) method is a class uh 2 Vh such that, for all test functions vh 2 Vh and all of finite element methods using completely discon- 1  i  N , we have tinuous basis functions to approximate partial dif    R ferential equations (PDEs). It was first designed for  Ii a.x/uh .x/@x vh .x/dx C a xi C 1 uO h xi C 1 2 2 steady-state scalar linear hyperbolic equations [15] in         1973. Early analysis of the method was performed vh xiC 1  a xi  1 uO h xi  1 vh xiC 1 2 2 2 2 in [11, 13]. Runge–Kutta DG (RKDG) method for R solving time-dependent nonlinear hyperbolic conser(2) D Ii f .x/vh .x/dx: vation laws was designed [3] in 1989. Local DG (LDG) method for solving time-dependent convection- Here, uO h is the so-called numerical flux, which is a dominated PDEs with higher-order spatial derivatives single-valued function defined at the cell interfaces

Description

D

366

Discontinous Galerkin Methods: Time-dependent Problems

and in general depends on the values of the numerical solution uh from both sides of the interface. The choice of numerical flux is one of the crucial ingredients of the design of stable and accurate DG methods. For the simple PDE (1), the choice is simply upwinding     uO h xi C 1 D uh xiC 1 ; 2

2

1  i  NI   uO h x 1 D g:

The RKDG method [3] avoids this difficulty nicely, by using the DG formulation only in the spatial variables and using an explicit, nonlinearly stable Runge– Kutta time discretization in time. The resulting scheme is explicit, just like a finite difference or a finite volume scheme, without the need to solve any large linear or nonlinear systems. It has nice stability properties [3, 9, 10] and performs well for multidimensional systems with discontinuous solutions [5].

2

With this choice, the scheme (2) in cell Ii depends only  on the  solution in the left neighbor at the interface uh xi 1 . Therefore, we can use (2) to explicitly 2

obtain the solution (which is a polynomial of degree at most k) inthe first cell I1 with the given boundary condition uO h x 1 D g. Once this is done, we can 2 use (2) again to explicitly obtain  the solution   in the second cell I2 with the flux uO h x 3 D uh x 3 which 2

Time-Dependent Nonlinear Convection-Diffusion Equations: LDG Method In order to compute problems with physical viscosities, such as high Reynolds number Navier–Stokes equations, the DG method has been generalized to handle higher (than first)-order spatial derivatives. One of the successful methods is the LDG method [4], which, for the simple heat equation @t u D @2x u;

(3)

2

has already been computed. Proceeding in this way, we can compute the solution in the whole computational domain cell by cell without solving any large systems. This method can be easily constructed along the same lines for multidimensional scalar linear hyperbolic equations with given boundary conditions at the inflow boundary. The finite element space is again the set of piecewise polynomials on any structured or unstructured triangulation. The polynomial degree k does not need to be the same in different cells; hence this first DG method already has the full flexibility in h-p adaptivity. Early analysis of the method was performed in [11, 13], indicating that the method is at least (k C 12 )th order accurate in L2 for arbitrary triangulations for smooth solutions.

proceeds to first defining an auxiliary variable p D @x u and rewriting the Eq. (3) into the following first-order system @t u  @x p D 0;

then discretizing this first-order system by the usual DG procedure. Of course, the choice of the numerical fluxes uO h and pOh can no longer be guided by the upwinding principle, which applies only to wave equations. For the heat Eq. (3), it turns out [4] that the following choice of alternating flux     uO h xi C 1 D uh xiC 1 ; 2

Time-Dependent Nonlinear Hyperbolic Equations: RKDG Method The first DG method described in the previous section can also be applied to initial-boundary value problems of linear time-dependent scalar hyperbolic equations, simply by treating the time variable as one of the spatial variables and use the DG method in space and time. However, this DG method is difficult to design and implement for linear hyperbolic systems (for which characteristics flow in different directions) and for nonlinear hyperbolic equations (for which the flow direction actually depends on the solution itself).

p  @x u D 0;

2

    pOh xi C 1 D ph xiCC 1 2

2

would lead to stability and optimal L2 order of accuracy (the other alternating pair is also fine). The method and the analysis can be easily generalized to very general nonlinear convection-diffusion equations. Time-Dependent Nonlinear Equations of Higher Order: LDG Method The LDG method, described in the previous section for convection-diffusion equations, can also be designed and analyzed for many higher-order nonlinear wave or diffusion equations. Examples include the KdV equations, the Kadomtsev–Petviashvili (KP) equations, the

Discrete and Continuous Dispersion Relations

367

Zakharov–Kuznetsov (ZK) equations, the Kuramoto– 11. Johnson, C., Pitk¨aranta, J.: An analysis of the discontinuous Galerkin method for a scalar hyperbolic equation. Math. Sivashinsky-type equations, the Cahn–Hilliard equaComput. 46, 1–26 (1986) tion, and the equations for surface diffusion and Will- 12. Kanschat, G.: Discontinuous Galerkin methods for viscous more flow of graphs. We refer to the review paper [18] flow. Deutscher Universit¨atsverlag, Wiesbaden (2007) 13. Lesaint, P., Raviart, P.A.: On a Finite Element Method for more details. More Equations and Conclusions The DG method has various versions, for example, the hybridizable DG (HDG) methods [7], and has been designed to solve many more types of PDEs, for example, elliptic equations [1] and Hamilton–Jacobitype equations. We will not list all the details here. There are several recent reviews, books, and lecture notes [2, 6, 8, 12, 14, 16, 17] in which more details can be found.

14. 15.

16.

17.

18.

References 1. Arnold, D., Brezzi, F., Cockburn, B., Marini, L.: Unified analysis of discontinuous Galerkin methods for elliptic problems. SIAM J. Numer. Anal. 39, 1749–1779 (2002) 2. Cockburn, B.: Discontinuous Galerkin methods for convection-dominated problems. In: Barth, T.J., Deconinck, H. (eds.) High-Order Methods for Computational Physics. Lecture Notes in Computational Science and Engineering, vol. 9, pp. 69–224. Springer, Berlin/New York (1999) 3. Cockburn, B., Shu, C.-W.: TVB Runge–Kutta local projection discontinuous Galerkin finite element method for conservation laws II: general framework. Math. Comput. 52, 411–435 (1989) 4. Cockburn, B., Shu, C.-W.: The local discontinuous Galerkin method for time-dependent convection diffusion systems. SIAM J. Numer. Anal. 35, 2440–2463 (1998) 5. Cockburn, B., Shu, C.-W.: The Runge–Kutta discontinuous Galerkin method for conservation laws V: multidimensional systems. J. Comput. Phys. 141, 199–224 (1998) 6. Cockburn, B., Shu, C.-W.: Runge–Kutta discontinuous Galerkin methods for convection-dominated problems. J. Sci. Comput. 16, 173–261 (2001) 7. Cockburn, B., Dong, B., Guzman, J., Restelli, M., Sacco, R.: A hybridizable discontinuous Galerkin method for steadystate convection-diffusion-reaction problems. SIAM J. Sci. Comput. 31, 3827–3846 (2009) 8. Hesthaven, J., Warburton, T.: Nodal Discontinuous Galerkin Methods. Springer, New York (2008) 9. Hou, S., Liu, X.-D.: Solutions of multi-dimensional hyperbolic systems of conservation laws by square entropy condition satisfying discontinuous Galerkin method. J. Sci. Comput. 31, 127–151 (2007) 10. Jiang, G., Shu, C.-W.: On a cell entropy inequality for discontinuous Galerkin methods. Math. Comput. 62, 531– 538 (1994)

for Solving the Neutron Transport Equation, Mathematical Aspects of Finite Elements in Partial Differential Equations (C. de Boor, ed.), pp. 89–145. Academic, New York (1974) Li, B.: Discontinuous Finite Elements in Fluid Dynamics and Heat Transfer. Birkhauser, Basel (2006) Reed, W.H., Hill, T.R.: Triangular Mesh Methods for the Neutron Transport Equation, Tech. Rep. LA-UR-73-479, Los Alamos Scientific Laboratory (1973) Rivi`ere, B.: Discontinuous Galerkin Methods for Solving Elliptic and Parabolic Equations. Theory and Implementation, SIAM, Philadelphia (2008) Shu, C.-W.: Discontinuous Galerkin methods: general approach and stability. In: Bertoluzza, S., Falletta, S., Russo, G., Shu, C.-W. (eds.) Numerical Solutions of Partial Differential Equations. Advanced Courses in Mathematics CRM Barcelona, pp. 149–201. Birkh¨auser, Basel (2009) Xu, Y., Shu, C.-W.: Local discontinuous Galerkin methods for high-order time-dependent partial differential equations. Commun. Comput. Phys. 7, 1–46 (2010)

Discrete and Continuous Dispersion Relations Geir K. Pedersen Department of Mathematics, University of Oslo, Oslo, Norway

In a uniform medium, small-amplitude waves may exist as periodic modes. The frequency and wavelength then fulfill a relation generally denoted as the dispersion relation. A more general solution can be obtained through Fourier synthesis of such modes. Among other things, the dispersion relation will determine the dispersion of a pulse in the direction of wave advance due to wavelength dependence of the group velocity. Wave phenomena are generally described by partial differential equations, or sometimes integral equations, derived from the fundamental physical laws. When the medium for wave propagation is uniform and the amplitudes are small, harmonic solutions may exist as separable solutions on the form D A sin.‚/;

‚ D k  r  !t;

(1)

D

368

Discrete and Continuous Dispersion Relations

where is some field variable (such as particle excursion, velocity, pressure, temperature, strength of electromagnetic field) describing the state of the medium, r is the position vector, t is time, k is the wave-number vector, and ! is the frequency. The quantity A may be a constant amplitude or may vary in the direction normal to k for waves guided by an interface, such as surface waves in the ocean. The phase function ‚ is constant in the direction normal to the wave number. However, (1) is a solution of the underlying equations only if a dispersion relation ! D !i .k/

(2)

is fulfilled. As indicated by the subscript, i , different families of modes may exist, owing to different physical mechanisms. For instance, elastic waves in the crust of the earth may be associated with compression (P-waves) or shearing (S-waves) of the medium, each kind with a different dispersion relation. In the remainder of this entry we assume the mode number as implicit. Wave modes of the form (1), including ranges of values for k and A, may be combined in a spectrum to provide solutions to, for instance, initial value problems through Fourier transforms. The relation (1) then contains all information of the medium needed to construct the solution, and the dispersion relation may, in this sense, be regarded as equivalent to the underlying governing equations. In a strict sense the harmonic mode is a solution only for a uniform

c √∗ gH c/

√

medium. However, (2) and (1) may be applied locally also for nonuniform media, provided the wavelength is short compared to the characteristic length of variation. This is the basis of ray theory, or geometrical optics [5, 6]. From (2) phase and group velocities are defined as cD

! k

 and cg D fcg i g D

@! @ki

 ;

respectively, where k D jkj. An observer moving with speed c in the direction of k will experience a phase (‚) remaining at a constant value. The group velocity, cg , defines the propagation speed and direction of wave properties such as energy. For theories on nonuniform wave patterns, the group velocity plays an essential role [6]. Particularly, the energy associated with each spectral component of an initially confined pulse will be propagated with the group velocity of that component. When cg is dependent on the wave number, the pulse will then undergo dispersion in the direction of wave advance. An example is displayed in Fig. 1 (right panel). Otherwise we have cg D ck=jkj and the waves will be nondispersive. An isotropic medium is characterized by the frequency depending on the wavelength only, and not the direction of wave advance (! D !.k/). As a consequence cg is parallel to k. On the other hand, for anisotropic waves the group velocity may be at an angle with the direction of wave advance (gravity waves in a density stratification, crystal optics). Isotropic dispersion is denoted by normal if the

h A0

gH √ cg / gH

λ/H

Discrete and Continuous Dispersion Relations, Fig. 1 Left panel: dispersion relation for surface gravity waves, with asymptotes for long and short waves depicted with dashes. Right panel: the mild effects of dispersion on a tsunami-type signal in deep ocean (H D 5 km). The dashes show the initial elevation (divided by two), while the solid line shows the surface elevation after t D 45 min of propagation towards decreasing. The latter

x−c 0 t (km)

p gH t to allow for comparison of data are shifted c0 t D the shapes. Dispersion effects are manifest as a stretching and reduction of the leading pulse and the evolution of short, trailing waves. These effects may not be important for all tsunamis, and particularly not for those associated with the largest earthquakes, such as the mega-disasters of 2004 and 2011

Discrete and Continuous Dispersion Relations

369

longer waves (smaller k) are the faster ones, which dc leads to cg D c C k dk < c. Correspondingly, for dc > 0 and cg > c. abnormal dispersion we have dk Plane surface, gravity, waves on water of equilibrium depth H obey the dispersion relation !2 D

g tanh .kH / ; k

(3)

where g  9:81 m=s2 is the constant of gravity. These waves are sometimes referred to as Airy waves in honor of the scientist who first presented a comprehensive theory [1]. Equation (3) is depicted in Fig. 1 (left panel). For waves much longer than the depth (kH 1), the phase, and group, velocities become, p approximately, cg D c D gH , which yields c D 797 km=h for H D 5;000 m, which is characteristic for a tsunami propagating in deep ocean. For wavelength similar to, or shorter than, the depth, the phase speed p becomes c D g=k, while cg D 12 c. According to this, an ocean swell of period 20 s inherits a phase speed of c D 112 km=h and a wavelength  D 624 m. Other examples on dispersive waves are elastic surface waves and electromagnetic waves in liquids and solids, yielding color-specific refraction indices (see, for instance [2]). When a set of equations is approximated by a finite difference, volume or element technique, also the dispersion properties become approximate and depend upon on the resolution (grid increments) and the method. A further discussion of numerical dispersion effects is best aided by a simple, yet fundamental, example. The standard wave equation for plane waves reads 2 @2 2@  c D 0; 0 @t 2 @x 2

(4)

where c0 is a constant, owing to medium, x is a spatial coordinate in the direction of wave propagation, and t is time. The general solution of (4) is derived and discussed in any elementary textbook on partial differential equations and is composed of two systems of permanent shaped waves, moving in the positive and negative x direction, respectively, with speed c0 . This implies the dispersion relation ! D c0 k;

c D c0 :

(5)

Hence, there is no dispersion and (4) serves as model equation for nondispersive waves of all kinds of physical origin, among them accoustic waves. In particular it describes the asymptotic long wave limit of dispersive wave classes such as the Airy waves. If we approximate the solution of (4) by a discrete solution, defined on a uniform grid ! will depend on the spacings of the grid,

x and t. Then ! D c0 k f .k x; c0 k t/, where the function f depends on the discretization method. Unlike (5) such a relation generally yields waves with dispersion, denoted as numerical dispersion. One consequence is that any finite pulse eventually will be dispersed into a wave train in the discrete approximation, while it translates with unaltered form according to the partial differential equation. Just as physical dispersion, the numerical counterpart is accumulative and may cause artificial disintegration of wave systems even for fine grids, provided the propagation distance is long. The simplest way to solve the wave equation (4) numerically is to employ the common symmetric finite difference for the second-order derivatives to employ a “+” shaped stencil in the x, t plane. Insertion of an harmonic mode, corresponding to (1), then yields 

! t sin 2



  c0 t k x sin D˙ :

x 2

(6)

Important information is offered by the numerical dispersion relation (6). First, a right-hand side value larger than unity implies a non-real frequency and thereby instability. To avoid this, for the whole range of possible wave numbers (k), we must require c0 t= x  1. This is the Courant-Friedrichs-Levy condition and may be related to the maximum, discrete signal speed in the grid. Moreover, save for c0 t= x D 1 when the numerical method becomes exact; (6) displays normal dispersion, with properties for long waves (small k) akin to those of surface gravity waves. Even though this has inspired attempts to model real dispersion for long ocean waves by means of numerical dispersion of the standard wave equations, the solution is generally degraded by numerical dispersion. Moreover, in multiple spatial dimensions numerical dispersion will be anisotropic. All results cited so far are linear in the sense that they are valid for asymptotically small amplitudes. If finite amplitude effects are taken into account, the propagation speed may be altered, such as for the

D

370

Distributions and the Fourier Transform

periodic Stokes’s wave where (3) is extended by terms involving the wave steepness (see, for instance [3]). Nonlinear effects may also counteract dispersion effects and inhibit the evolution of a wave train. An example to this is the solitary wave (see [4] for overview).

The theory in its modern form arose from the work of Laurent Schwartz in the late 1940s, although it certainly had important precursors such as Heaviside’s operational calculus in the 1890s and Sobolev’s generalized functions in the 1930s. The approach of Schwartz had the important feature of being completely mathematically rigorous while retaining the ease of calculation of the operational methods. DisReferences tributions have played a prominent role in the modern 1. Airy, G.B.: Tides and Waves, Encyclopaedia Metropolitana, theory of partial differential equations. The idea behind distribution theory is easily ilvol 3, publ. j.j. Griffin, London (1841) 2. Born, M., Wolf, E.: Principles of Optics: Electromagnetic lustrated by the standard example, the Dirac delta Theory of Propagation, Interference and Diffraction of Light. function. On the real line, the Dirac delta is a “function Cambridge University Press, Cambridge with contributions from A.B. Bhatia, P.C. Clemmow, D. Gabor, A.R. Stokes, ı.x/ which is zero for x ¤ 0 with an infinitely high peak at x D 0, with area equal to one.” Thus, if A.M. Taylor, P.A. Wayman, W.L. Wilcock (1999) 3. Lamb, H.: Hydrodynamics, 6th edn. Cambridge University f .x/ is a smooth function, then integrating ı.x/f .x/ Press, Cambridge (1994) is supposed to give

4. Miles, J.W.: Solitary waves. Ann. Rev. Fluid Mech. 12, 11–43 (1980) 5. Peregrine, D.H.: Interaction of water waves and currents. Adv. Appl. Mech. 16, 10–117 (1976) 6. Whitham, G.B.: Linear and nonlinear waves. Pure & Applied Mathematics. Wiley, New York (1974)

Distributions and the Fourier Transform Mikko Salo Department of Mathematics and Statistics, University of Jyv¨askyl¨a, Jyv¨askyl¨a, Finland

Z

1

ı.x/f .x/ D f .0/: 1

The Dirac delta is not a well-defined function (in fact it is a measure), but integration against ı.x/ may be thought of as a linear operator defined on some class of test functions which for any test function f gives out the number f .0/. After suitable choices of test function spaces, distributions are introduced as continuous linear functionals on these spaces. The following will be a quick introduction to distributions and the Fourier transform, mostly avoiding proofs. Further details can be found in [1–3].

Introduction Test Functions and Distributions The theory of distributions, or generalized functions, provides a unified framework for performing standard calculus operations on nonsmooth functions, measures (such as the Dirac delta function), and even more general measure-like objects in the same way as they are done for smooth functions. In this theory, any distribution can be differentiated arbitrarily many times, a large class of distributions have well-defined Fourier transforms, and general linear operators can be expressed as integral operators with distributional kernel. The distributional point of view is very useful since it easily allows to perform such operations in a certain weak sense. However, often additional work is required if stronger statements are needed.

Let ˝ Rn be an open set. We recall that if f is a continuous function on ˝, the support of f is the set supp.f / WD˝ n V;

V is the largest open subset in ˝

with f jV D 0: Some notation: any n-tuple ˛ D .˛1 ; : : : ; ˛n / 2 Nn where N D f0; 1; 2; : : :g is called a multi-index and its norm is j˛j D ˛1 C    C ˛n . We write  @ f .x/ D ˛

@ @x1

˛1



@  @xn

˛n f .x/:

Distributions and the Fourier Transform

A function f on ˝ is called C 1 or, infinitely differentiable, if @˛ f is a continuous function on ˝ for all ˛ 2 Nn . The following test function space will be used to define distributions.

371

L1 .˝/ function is locally integrable.) We define uf W Cc1 .˝/ ! C; uf .'/ D

Z f .x/'.x/ dx: ˝

Definition 1 The space of infinitely differentiable functions with compact support in ˝ is defined as

By the definition of convergence of sequences, uf is a well-defined distribution. If f1 ; f2 are two locally integrable functions and uf1 D uf2 , then f1 D Cc1 .˝/ WDff W ˝ ! C I f is C 1 and supp.f / f2 almost everywhere by the du Bois-Reymond lemma. Thus a locally integrable function f can be is compact in ˝g: (1) identified with the corresponding distribution uf . 2. (Dirac mass) Fix x0 2 ˝ and define If ˝ is a domain with smooth boundary, then supp.f / is compact in ˝ if and only if f vanishes ıx0 W Cc1 .˝/ ! C; ıx0 .'/ D '.x0 /: near @˝. The space Cc1 .˝/ contains many functions, for instance, it is not hard to see that This is a well-defined distribution, called the Dirac mass at x0 .  1=.1jxj/2 ; jxj < 1; e 3. (Measures) If  is a positive or complex .x/ WD R Borel mea0; jxj  1 sure in ˝ such that the total variation K d jj < 1 for any compact K ˝, then the operator is in Cc1 .Rn /. Generally, if K V V ˝ where Z K is compact and V is open, there exists ' 2 Cc1 .˝/ W ' ! 7 '.x/ d.x/ u  such that ' D 1 on K and supp.'/ V . ˝ 1 To define continuous linear functionals on Cc .˝/, we need a notion of convergence: is a distribution that can be identified with . 4. (Derivative of Dirac mass) On the real line, the Definition 2 We say that a sequence .'j /1 j D1 con1 operator verges to ' in Cc .˝/ if there is a compact set K ˝ ı00 W ' 7! ' 0 .0/ such that supp.'j / K for all j and if k@˛ .'j  '/kL1 .K/ ! 0 as j ! 1; for all ˛ 2 Nn :

is a distribution which is not a measure. We now wish to extend some operations, defined for smooth functions, to the case of distributions. This is possible via the duality of test functions and distributions. To emphasize this point, we employ the notation

More precisely, one can define a topology on Cc1 .˝/ such that this space becomes a complete locally convex topological vector space, and a linear functional u W Cc1 .˝/ ! C is continuous if and only if u.'j / ! 0 for any sequence .'j / such that hu; 'i WD u.'/; u 2 D0 .˝/; ' 2 Cc1 .˝/: 'j ! 0 in Cc1 .˝/. We will not go further on this since the convergence of sequences is sufficient for most practical purposes. Note that if u is a smooth function, then the duality is We can now give a precise definition of distribu- given by Z tions. hu; 'i D u.x/'.x/ dx: ˝ Definition 3 The set of distributions on ˝, denoted by D0 .˝/, is the set of all continuous linear functionals u W Cc1 .˝/ ! C.

Multiplication by Functions

Examples 1. (Locally integrable functions) Let f be a locally integrable function inR˝, that is, f W ˝ ! C Let a be a C 1 function in ˝. If u; ' 2 Cc1 .˝/, we is Lebesgue measurable and K jf j dx < 1 for any clearly have compact K ˝. (In particular, any continuous or hau; 'i D hu; a'i:

D

372

Distributions and the Fourier Transform

Since ' 7! a' is continuous on Cc1 .˝/, we may for Homogeneous Distributions any u 2 D0 .˝/ define the product au as the distribution given by We wish to discuss homogeneous distributions, which are useful in representing fundamental solutions of hau; 'i WD hu; a'i; ' 2 Cc1 .˝/: differential operators, for instance. We concentrate on a particular example following [2, Section 3.2]. If a > 1, define Distributional Derivatives  a x ; x > 0; fa .x/ WD Similarly, motivated by the corresponding property for 0; x < 0: smooth functions, if u 2 D0 .˝/ and ˇ 2 Nn is a multi-index, then the (distributional) derivative @ˇ u is This is a locally integrable function and positively homogeneous of degree a in the sense that fa .tx/ D the distribution given by t a f .x/ for t > 0. For a > 1 we can define the a distribution xC WD fa . If a > 0 it has the properties h@ˇ u; 'i WD .1/jˇj hu; @ˇ 'i; ' 2 Cc1 .˝/: a1 a D xC ; xxC

(If u is a smooth function, this is true by the integration by parts formula Z Z u.x/@xj '.x/ dx D  @xj u.x/'.x/ dx:/

a 0 a1 .xC / D axC :

a We would like to define xC for any real number a as 0 an element of D .R/ so that some of these properties The last fact is one of the most striking features remain valid. of distributions: in this theory, any distribution (no First note that if a > 1, then for k 2 N by repeated matter how irregular) has infinitely many well-defined differentiation derivatives! 1 a Example 1 1. In Example 4 above, the distribution ı00 hx aC1 ; ' 0 i D    ; 'i D  hxC aC1 C is in fact the distributional derivative of the Dirac 1 mass ı0 . D .1/k .a C 1/.a C 2/ : : : .a C k/ 2. Let u.x/ WD jxj; x 2 R. Since u is continuous, 0 aCk we have u 2 D .R/. We claim this one has the hxC ; ' .k/ i: distributional derivatives ˝

˝

a If a … f1; 2; : : :g we can define xC 2 D0 .R/ by the last formula. 00 u D 2ı0 : If a is a negative integer, we need to regularize a the expression xC to obtain a valid distribution. For a 1 In fact, if ' 2 Cc1 .R/, one has .R/, the quantity F .a/ D hxC ; 'i D fixed ' 2 C c R .x/'.x/ dx can be extended as an analytic funcf a Z 0 Z 1 0 0 0 0 tion for complex a with Re.a/ > 1. The previous x' .x/ dx  x' .x/ dx hu ; 'i D hu; ' i D a with negative a then shows that F is formula for xC 1 0 Z analytic is C n f1; 2; : : :g, and it has simple poles at sign.x/'.x/ dx D hsign.x/; 'i; D the negative integers with residues R

u0 D sign.x/;

using integration by parts and the compact support of '. Similarly, 00

0

0

Z

0

hu ; 'i D hu ; ' i D

0

Z

1

' .x/ dx  1

D 2'.0/ D h2ı0 ; 'i:

0

' .x/ dx 0

lim .a C k/F .a/ D

a!k

D

0 ; ' .k/ i .1/k hxC .k C 1/.k C 2/ : : : .1/

' .k1/ .0/ : .k  1/Š

k We define xC 2 D0 .R/, after a computation, by

Distributions and the Fourier Transform

k hxC ; 'i

373

WD lima!k .F .a/  D

1 .k1/Š

' .k1/ .0/ / .k1/Š.aCk/

 R  P  1 k1 1 .k1/  0 .log x/' .k/ .x/ dx C .0/ : j D1 j '

a1 a a 0 Then xxC D xC is valid for all a 2 R, and .xC / D Definition 4 The Schwartz space S .Rn / is the set of a1 axC holds true except for nonpositive integers a. all infinitely differentiable functions f W Rn ! C such that the seminorms

Schwartz Kernel Theorem

kf k˛;ˇ WD kx ˛ @ˇ f .x/kL1 .Rn /

One of the important features of distribution theory is that it allows us to write almost any linear operator as an integral operator, at least in a weak sense. If ˝; ˝ 0 Rn are open sets and if K 2 L2 .˝ ˝ 0 /, by Cauchy-Schwarz, one has a bounded linear operator T W L2 .˝ 0 / !L2 .˝/; T '.x/ WD Z K.x; y/'.y/ dy:

are finite for all multi-indices ˛; ˇ 2 Nn . If .fj /1 j D1 is a sequence in S , we say that fj ! f in S if kfj  f k˛;ˇ ! 0 for all ˛; ˇ. It follows from the definition that a smooth function f is in S .Rn / iff for all ˇ and N there exists C > 0 such that j@ˇ f .x/j  C hxiN ;

x 2 Rn :

˝0

Here and below, hxi WD .1 C jxj2 /1=2 . Based on The function K is called the integral kernel of the this, Schwartz space is sometimes called the space of operator T . There is a general one-to-one correspon- rapidly decreasing test functions. dence between continuous linear operators and integral Example 2 Any function in Cc1 .Rn / is in Schwartz kernels. 2 space, and functions like e  jxj , > 0, also belong 1 0 0 Theorem 1 If T W Cc .˝ / ! D .˝/ is a continuous to S . The function e  jxj is not in Schwartz space linear map, then there is K 2 D0 .˝ ˝ 0 / such that because it is not smooth at the origin, and also hxiN is not in S because it does not decrease sufficiently rapidly at infinity. hT .'/; i D hK; ˝ 'i; There is a topology on S such that S becomes a complete metric space. The operations f 7! Pf and f 7! @ˇ f are continuous maps S ! S , if P is any Here . ˝'/.x; y/ D .x/'.y/. Conversely, any K 2 polynomial and ˇ any multi-index. More generally, let D0 .˝ ˝ 0 / gives rise to a continuous linear map T by the above formula. ' 2 Cc1 .˝ 0 /;

2 Cc1 .˝/:

OM .Rn / WDff 2 C 1 .Rn / I for all ˇ there exist C;

Tempered Distributions

N >0 such that j@ˇ f .x/j  C hxiN g:

In the following, we will give a brief review of the Fourier transform in the general setting of tempered distributions. We introduce a test function space de- It is easy to see that the map f 7! af is continuous S ! S if a 2 OM . signed for the purposes of Fourier analysis.

D

374

Distributions and the Fourier Transform

Definition 5 If f 2 S .Rn /, then the Fourier trans- 3. Let  be a positive Borel measure in Rn such that form of f is the function F f D fO W Rn ! C defined Z by Z hxiN d.x/ < 1 fO./ WD e ix f .x/ dx;  2 Rn : n R

Rn

The importance of Schwartz space is based on the fact that it is invariant under the Fourier transform.

for some N > 0. Then the corresponding distribution u is tempered. In particular, the Dirac mass ıx0 Theorem 2 (Fourier inversion formula) The at x0 2 Rn is in S 0 . n Fourier transform is an isomorphism from S .R / 4. The function e x is in D0 .R/ but not in S 0 .R/ onto S .Rn /. A Schwartz function f may be if R x ¤ 0, since it is not possible to define recovered from its Fourier transform by the inversion R e '.x/ dx for all Schwartz functions '. formula However, e x cos.e x / belongs to S 0 since it is the distributional derivative (see below) of the bounded Z function 1 sin.e x / 2 S 0 . e ix fO./ d ; f .x/ DF 1 fO.x/ D .2/n Rn

x2R : n

Operations on Tempered Distributions After introducing the Fourier transform on nicely The operations defined above for D0 .Rn / have natural behaving functions, it is possible to define it in a very analogues for tempered distributions. For instance, if general setting by duality. a 2 OM .Rn / and u 2 S 0 .Rn /, then au is a tempered 0 n Definition 6 Let S .R / be the set of continuous distribution where linear functionals S .Rn / ! C. The elements of S 0 are called tempered distributions, and their action on hau; 'i D hu; a'i: test functions is written as Similarly, if u 2 S 0 .Rn / then the distributional derivative @ˇ u is also a tempered distribution. Finally, we can define the Fourier transform of Since the embedding Cc1 .Rn / S .Rn / is con- any u 2 S 0 as the tempered distribution F u D uO tinuous, it follows that S 0 .Rn / D0 .Rn /, that is, with tempered distributions are distributions. The elements hOu; 'i WD hu; 'i; O ' 2 S: in S 0 are somewhat loosely also called distributions of polynomial growth. The following examples are In fact, this identity is true if u; ' 2 S , and it then similar to the case of D0 .Rn / above. extends the Fourier transform on Schwartz space to the Example 3 1. Let f W Rn ! C be a measurable case of tempered distributions. function, such that for some C; N > 0, one has Example 4 The Fourier transform of ı0 is the constant 1, since jf .x/j  C hxiN ; for a.e. x 2 Rn : Z 1 O hı0 ; 'i D hı0 ; 'i O D '.0/ O D '.x/ dx Then the corresponding distribution uf is in 1 S 0 .Rn /. The function f is usually identified with D h1; 'i: the tempered distribution uf . 2. In a similar way, any function f 2 Lp .Rn / with If u 2 L2 .Rn / then u is a tempered distribu1  p  1 is a tempered distribution (with the tion, and the Fourier transform uO is another element identification f D uf /. hu; 'i WD u.'/;

u 2 S 0; ' 2 S :

Domain Decomposition

375

of S 0 . The Plancherel theorem (which is the exact Domain Decomposition analogue of Parseval’s theorem for Fourier series) 2 states that in fact uO 2 L and that the Fourier transform Barry Smith1 and Xuemin Tu2 is an isometry on L2 up to a constant. We state 1 Mathematics and Computer Science Division, the basic properties of the Fourier transform as a Argonne National Laboratory, Argonne, IL, USA theorem. 2 Department of Mathematics, University of Kansas, Theorem 3 The Fourier transform u 7! uO is a bijec- Lawrence, KS, USA tive map from S 0 onto S 0 , and one has the inversion formula

Synonyms hu; 'i D .2/

n

hOu; '. O  /i;

' 2 S:

The Fourier transform is also an isomorphism from L2 .Rn / onto L2 .Rn /, and kOukL2 D .2/n=2 kukL2 : It is remarkable that any tempered distribution has a Fourier transform (thus, also any Lp function or measurable polynomially bounded function), and there is a Fourier inversion formula for recovering the original distribution from its Fourier transform. We end this section by noting the identities .@˛ u/O D .i /˛ uO ; .x ˛ u/O D .i @ /˛ uO ; where x ˛ D x1˛1 : : : xn˛n . These hold for Schwartz functions u by a direct computation and remain true for tempered distributions u by duality. Thus the Fourier transform converts derivatives into multiplication by polynomials and vice versa. This explains why the Fourier transform is useful in the study of partial differential equations, since it can be used to convert constant coefficient differential equations into algebraic equations.

DDM

Definition Domain decomposition methods are solvers for partial differential equations that compute the solution by solving a series of problems on subdomains of the original domain. Domain decomposition methods are efficient (generally iterative) methods for the implicit solution of linear and nonlinear partial differential equations (PDEs). Such methods are motivated by four considerations: complex geometry, coupling of multiple physical regimes (e.g., fluid and structure), the divide-and-conquer paradigm, and parallel computing. Domain decomposition methods may be applied to all three classes of PDEs – elliptic, parabolic, and hyperbolic – as well as coupled and mixed PDEs. A large number of domain decomposition methods have been developed, including those based on overlapping and nonoverlapping subdomains. The convergence analysis of domain decomposition methods is a rich mathematical topic with both a general, broad abstract theory and detailed technical estimates. Domain decomposition methods are closely related to block Jacobi methods, multigrid methods, as well as other iterative and direct solution schemes.

References

Motivating Aspects

1. Friedlander, F.G., Joshi, M.: Introduction to the Theory of Distributions, 2nd edn. Cambridge University Press, Cambridge/New York (1998) 2. H¨ormander, L.: The Analysis of Linear Partial Differential Operators, vol. I, 2nd edn. Springer, Berlin/New York (1990) 3. Schwartz, L.: Th´eorie des Distributions. Hermann, Paris (1950)

Complex Geometry and Multiphysics The first domain decomposition method is often attributed to H. A. Schwarz in 1870. He used it to demonstrate the existence of a solution to a Poisson problem (4u D f ) with a Dirichlet boundary

D

376

Domain Decomposition

Ω1

Ω1 Γ2

Γ1

Ω2 Γ3

Ω2

Fig. 1 Overlapping Subdomains of Schwarz

Fig. 2 Nonoverlapping subdomains of capacitance matrix methods

condition on a domain that is the overlapping union of a circle (1 ) and rectangle (2 ), where no closed-form solution exists (both circles and rectangles have closedform solutions); see Fig. 1. He proposed the following abstract iteration. Select an initial guess on the interior boundary 1 , and solve the Dirichlet problem on 1 . Then, using the trace of that solution on 2 , solve the Dirichlet problem on 2 . Repeat the process with the latest values on 1 . This is the Schwarz alternating process. The process can be interpreted physically as having two interleaved tanks with two gates 1 and 2 and letting the residual 4u  f represent water in the tanks. Solving on 1 can be interpreted as closing the gate on 1 and pumping all the water out of tank 1. Opening the gate equalizes the water height across the dual tanks; one then closes the second gate and removes the water from the second tank. Intuitively, this iterative process will eventually remove all the water. To express the overlapping alternating Schwarz method algebraically, it is useful to introduce the restriction operators R1 and R2 that select all the degrees of freedom interior to each subdomain. The discrete alternating Schwarz method can then be written as follows:

In the 1970s, an awareness of fast solvers on simple geometries (e.g., based on fast Fourier transforms and cyclic reduction) led to the development of capacitance matrix methods. Consider the L-shaped region in Fig. 2 that is the union of two nonoverlapping rectangles, 1 and 2 . If the values on 3 are known, then the solutions in 1 and 2 can be computed independently in parallel. Thus, it is useful to derive an equation for the values only on 3 . This formulation can be done either at the continuum level or after the PDE has been discretized into an algebraic equation. For simplicity, consider the L-shaped region discretized by using centered finite differences. The resulting algebraic equations can be written as

unC1=2 D un C R1T .R1 AR1T /1 R1 .f  Aun /;

.2/ where S .1/ D A33  A31 A1 D A33  11 A13 and S 1 A32 A22 A23 are the Schur complements from 1 and 2 , respectively. Note that an application of S requires the solution of the PDE on each subdomain. This Schur complement system can be solved by using a preconditioned Krylov subspace method such as the conjugate gradient method or the generalized minimum residual (GMRES) method. The preconditioner of S can be the square root of the negative of a discretization of the Laplacian on 3 , sometimes called the J operator (or K 1=2 ), which can be applied by

unC1 D unC1=2 C R2T .R2 AR2T /1 R2 .f  AunC1=2 /: All the details in the transformations to this form from the continuum algorithm may be found in [4, Chapter 2]. Thus, we see the first motivation for domain decomposition: it provides a method for computing a PDE solution on a complicated geometric region by applying PDE solvers on simpler geometric regions.

0 @

A11 A31

A22 A32

10 1 0 1 A13 u1 f1 A @ u2 A D @ f2 A : A23 .1/ .2/ u3 f3 A33 C A33

Eliminating the first two sets of variables (e.g., with a block LU factorization) results in the reduced Schur complement system, 1 S u3 D .S .1/ C S .2/ /u3 D f3  A31 A1 11 f1  A32 A22 f2 ; .1/

.2/

Domain Decomposition

377

using a fast Fourier transform. Alternatively, one can 1 1 choose S .1/ or S .2/ or the summation of both as a preconditioner, which is related to the NeumannNeumann preconditioners. An efficient application of 1 S .1/ can be computed by using the relationship

S

.1/1



vD 0 I



A11 A13 .1/ A31 A33

!1   0 v; I

where the solution of the right-hand-side system represents solving the subproblem with Neumann boundary conditions along 3 . Full details may be found in [4, Section 4.2.1]. Both the J operator and the Neumann-Neumann approach result in optimal preconditioners, in the sense that the condition number (and hence the number of iterations needed for convergence) does not increase with the size of the algebraic problem. The algorithms as presented are not optimal in work estimates unless an optimal solver is used for each of the subdomains. Both overlapping and nonoverlapping domain decomposition methods may be extended for use when different PDE models are used for different parts of the domain. For example, in fluid-structure interaction, the nonoverlapping methods are often used with information being passed between the domains as Dirichlet and Neumann boundary conditions along the interface. For coupling of Navier-Stokes and Boltzmann regimes, overlapping regions are often used, where the Boltzmann representation is converted to the Navier-Stokes representation in the overlapped region and vice versa at each iteration. Divide and Conquer and Parallelism The other motivator for domain decomposition methods is to decompose a large problem into a collection of small problems that may be solved in parallel. In general, the smaller subproblems are all coupled, requiring an iterative solution process. The process is as follows: decompose the entire computational domain  into several small subdomains i ; i D 1; : : : ; N . For simplicity, consider the discretized linear problem Ax D b, and denote the space of degrees of freedom by V . This decomposition process can also be carried out at the continuum level and for nonlinear problems with changes only in notation and more technical details.

Consider a family of spaces fVi ; i D 0; : : : ; N g. Here, Vi usually are related to the degrees of freedom on subdomains i , for i D 1;    N , called local spaces. V0 usually is related to a coarse problem, called the coarse space. The coarse space provides global communication across the domain in each iteration and contains the coarser grid eigenmodes to satisfy the local null space property. Specifically, the coarse space includes the null spaces of the local problems defined on the local spaces Vi , for i D 1;    N . The dimensions of these null spaces are much smaller than the dimension of V . In addition, assume there exist rectangular (restriction) matrices Ri that return the degrees of freedom in spaces Vi : Ri W V ! Vi . Then P T V can be written as V D R0T V0 C N i D1 Ri Vi . This decomposition is not a direct sum in most cases. Define the subproblem (simply the original problem restricted to one domain) Ai D Ri ARiT and the projection 1 operator Pi D RiT A1 i Ri A. When Ai is solved only approximately, the Pi are no longer projections. Using the decomposition of the space V and the projection-like operators Pi , the following Schwarz preconditioned operators are defined: P • Additive operator – Pad D P0 C N i D1 Pi D PN R /A . i D0 RiT A1 i i • Multiplicative operator – Pmu D I  …N i D0 .I  Pi / and symmetric version Pmu D I  …N i D0 .I  Pi /  …0iDN 1 .I  Pi / • Hybrid operator – Phy D I  .I  P0 /.I  PN i D1 Pi /.I  P0 / For the additive operator Pad , one can solve subproblems Ai in parallel. For the multiplicative operator Pmu , these subproblems must be solved sequentially. However, if the subdomains are colored so that no two subspaces of a color share degrees of freedom, then all the subproblems of the same color may be solved in parallel.

Standard Approaches Two standard approaches to domain decomposition methods exist based on whether the subdomains are overlapped. Overlapping Domains Overlapping domains are usually obtained by decomposing the computational domain  into nonoverlapping subdomains i with diameters Hi . These

D

378

Domain Decomposition

domains are then each extended to a larger region main local matrices and applying the Schur comple0i by repeatedly adding a layer of elements into i . ment implicitly as needed. The distance parameter ıi measures the width of the Depending on how one eliminates the coupling in regions n0i on i . If there is a constant c such that the Schur complement and forms the local spaces Vi , Hi  1c , the overlap is called generous. The two primal nonoverlapping domain decomposition maxN i D1 ıi convergence of the overlapping domain decomposition approaches can be distinguished. One approach eliminates the coupling between all pairs of faces, method depends on the size of the overlap. The local spaces Vi are defined as the degrees of edges, and vertices; methods using this approach are freedom on the extended subdomains 0i with zeros on called primal iterative substructuring methods. The @0i , for i D 1; : : : ; N . The restriction matrices Ri second approach eliminates the coupling between are 0-1 rectangle matrices that extract the degrees of subdomains, leading to Neumann-Neumann methods. Finite-element tearing and interconnecting (FETI) freedom in Vi from V . The coarse space V0 is usually not necessary for methods are another type of nonoverlapping domain parabolic problems but is crucial in order to make the decomposition methods. These methods iterate on the algorithms converge independently of the number of dual variable Lagrange multipliers that are introduced the subdomains for elliptic PDEs. The standard coarse to enforce the continuity of the solutions across the space V0 can be formed by the degrees of freedom subdomain interface. For the primal iterative substructuring methods, defined on a shape-regular coarse mesh on the domain the local spaces Vi are the degrees of freedom related . The original mesh need not be a refinement of this to individual faces, edges, and vertices. In order to T coarse mesh. The matrix R0 is the interpolation from avoid computing the elements of the global Schur the coarse to a fine mesh. For unstructured meshes, complement, the local solvers (Si ) can be replaced by especially in three dimensions, the construction of R0 inexpensive inexact solvers. The coarse space can be might not be straightforward. Several nonstandard coarse grid spaces have been vertex based, which gives optimal convergence bounds introduced for specific circumstances. For example, for two dimensions but not for three dimensions. Wiresome coarse spaces, from nonoverlapping methods basket-based and face-based coarse spaces can give such as face-based and Neumann-Neumann coarse optimal convergence bounds for three-dimensional space, are used for nonconforming finite-element dis- problems. For the Neumann-Neumann methods, the local cretization to make the algorithms independent of the spaces Vi contain the boundary degrees of freedom jump coefficients. Similar coarse spaces are also used for almost-incompressible elasticity. With such coarse on each subdomain, and the 1local solvers are the space, a condition number bound for saddle point local Schur complement Si . For the floating problems was established, which is also independent subdomains, whose boundary have no intersection of the jump coefficients. Several other coarse spaces with the boundary of the original domain, the exist, for example, partition of unity and aggregation; local Schur complements are singular. The coarse space is spanned by the pseudoinverses of the for more details, see [5, Section 3.10]. counting functions from each subdomain. These pseudoinverses form a partition of unity and ensure that the local problems are balancing for floating domains when the hybrid Schwarz framework is Nonoverlapping Domains In the first step of nonoverlapping domain decompo- used. For the additive Schwarz framework, a lowsition methods, the original problem A is reduced to order term can be added into the local solvers to make the Schur complement S by eliminating the degrees them nonsingular. Balancing domain decomposition of freedom interior to each subdomain. The Schur methods by constraints (BDDC) is an advanced version complement has a smaller size and a better condition of the balancing methods. With BDDC algorithms, number compared with that of the original matrix A. the subdomain interface degrees of freedom are However, it is denser and expensive to form and store. divided into primal and dual parts. The local solvers In practice, therefore, one tries to avoid forming the have a Neumann condition on the dual variables global Schur complement, instead storing the subdo- but a Dirichlet condition for the primal variables.

Domain Decomposition

The singularity of the local problems from floating subdomains is thus removed. The coarse space of BDDC consists of the functions given by their values for the primal variables and energy minimal extension to each subdomain independently. An additive Schwarz framework is used for BDDC methods. FETI-DP methods are an advanced version of the FETI family. They share coarse and local spaces similar to those of BDDC algorithms, but they do iterations on the dual variable Lagrange multipliers. It has been established that the preconditioned BDDC and FETIDP algorithms, with the same coarse spaces, have the same nontrivial nonzero eigenvalues.

379

organized around the following three assumptions. The hybrid method can be interpreted as an additive method on the range space of I  P0 . Assumption 1 (Stable Decomposition) There exists a constant C0 such that for all u 2 V , there exists a P T decomposition u D N i D0 Ri ui , ui 2 Vi such that N X

D uTi Ai ui



C02 uT Au:

i D0

The constant C0 is related to the minimal eigenvalue of Pad , and this assumption ensures that the spaces Nonlinear Problems: ASPIN Vi provide a stable splitting of the space V . Note that The previous material was premised on solving non- the more overlap between the subspaces, the better this linear PDEs by using Newton’s method, thus reducing bound will be. a nonlinear problem to a series of linear problems. One can also apply domain decomposition principles Assumption 2 (Strengthened Cauchy-Schwarz) Inequality There exist constants 0  ij  1, 1  i; j  directly to the nonlinear problem. Discretization of nonlinear PDEs results in nonlin- N such that 8ui 2 Vi ; uj 2 Vj ; i; j D 1; : : : ; N , ear algebraic equations F .u/ D 0: Additive Schwarz  1=2 preconditioned inexact Newton (ASPIN) methods conjuTi Ri ARjT uj j  ij uTi Ri ARiT ui vert this system to a different, better conditioned, non 1=2 linear system F .u/ D 0, which has the same solution uTj Rj ARjT uj : as the original system. The new nonlinear system is then solved by inexact Newton methods. The construction of F is based on the decompo- The spectral radius of , ./, measures the orthogosition of the computational domain into overlapped nality of the spaces Vi for i D 1; : : : ; N , and it will subdomains, as for linear problems. There is no coarse appear in the upper bound of the maximal eigenvalues space V0 . The subdomain local nonlinear functions of Pad . The more orthogonal the subspaces, the better are defined as Fi D Ri F , where Ri are the square the constant will be. restriction matrices that take the degrees of freedom Assumption 3 (Local Stability) There exists a connot in 0i zero. For any u 2 V , Pi .u/ 2 Vi is defined stant ! > 0 such that for all u 2 V , i D 0; : : : ; N , i i as the solution of the subdomain nonlinear problems: Fi .uPi .u// D 0. The nonlinear function F is defined PN uTi Ri ARiT ui  !uTi Ai ui : as F .u/ D Pi .u/. Thus, an evaluation of F i D1

involves a nonlinear solve on each subdomain. The Jacobian matrix of F can be approximated by Here, ! gives a one-sided measure of the approxPN 1 J D i D1 Ji J , where J is the Jacobian of the imation properties of the local problem, especially original function F and Ji D Ri JRiT . Thus, one sees when Ai is an approximation of Ri ARiT . that the ASPIN nonlinear system can be interpreted as The fundamental theorems of Schwarz analysis are a nonlinear preconditioning of the original system. given by the following two results. Theorem 1

Analysis: The Schwarz Framework Proofs for the convergence of additive and multiplicative domain decomposition preconditioners are

C02 uT Au  uT APad u  ! ../ C 1/ uT Au; 8u 2 V:

380

Domain Decomposition

Theorem 2 For symmetric multiplicative operator, 2! uT Au  uT APmu u  uT Au; .1 C 2! 2 2 .//C02

for the lower bound can be obtained easily. However, the estimate for the upper bound is difficult; it can be obtained by estimating the so-called jump or average operators.

8u 2 V: Convergence proofs for any domain decomposition method then consist of verifying the three assumptions and determining the dependency of the constants C02 , ./, and ! on the subdomain size H , the element size h, and the PDE-specific parameters. Each domain decomposition method has its own bounds and proof of convergence. The following two theorems are typical of the results that may be obtained. For two-level overlapping Schwarz methods with standard coarse space V0 , let N c represent the number of colors needed to color the overlapping regions. One can easily obtain the estimate ./  N c for the upper bound. However, it is much more technically   difficult to obtain the estimate C02  C 1 C Hı for the lower bound. If exact solvers are used for all subspace problems, the following theorem holds. Theorem 3   H ; .Pad /  C 1 C ı where .Pad / is the condition number of the twolevel overlapping Schwarz method and the constant C depends on N c but not on the mesh size h, the subdomain size H , or the overlapping constant ı. It has been proved that this bound is sharp. For well-designed nonoverlapping domain decomposition methods, one typically obtains the following condition number bound. Theorem 4   H 2 .Pad /  C 1 C log ; h where the constant C is independent of h, H , and the specific coefficients of the PDE. This bound is sharp as well. For the analysis of the balancing NeumannNeumann methods with hybrid Schwarz framework, BDDC, and FETI-DP algorithms, the estimate of C02

Relationship to Other Algebraic Solver Methods For the linearized problem, one-level overlapping additive Schwarz methods may be considered generalizations of the block Jacobi method. The key differences are that the matrix rows and columns are preordered (at least conceptually, though not necessarily in practice) so that each diagonal block is associated with geometric subdomain and the blocks are “extended” to include degrees of freedom that are coupled to degrees of freedom associated with the block. This extension can be done at the geometric level or at the algebraic level by using information from the sparse matrix data structure. Schur complement methods are closely related to sparse direct solver methods. Substructuring is an approach to organizing the computations in a sparse direct solver by using the geometric/mesh information to determine an efficient ordering used during the factorization; this is also closely related to nested dissection orderings. These orderings result in small separators that are ordered last and couple the subdomains resulting from the substructuring or nested dissection ordering. A Schur complement domain decomposition method can then be thought of as a hybrid direct-iterative method where the factorization is stopped at the separators and an iterative method is used to complete the solution process along the separators. Multigrid methods are closely related to two-level and multilevel domain decomposition methods. In fact, much of the mathematics used in multigrid analysis is common to Schwarz analysis, and many mathematicians work in both domain decomposition and multigrid methods. Domain decomposition methods are generally designed with more focus on concurrency, whereas multigrid emphasizes optimality in terms of work that needs to be performed. Both approaches can suffer from the telescoping problem in which a coarse grid solver requires far fewer compute resources than do the finer grid solves, resulting in possibly idle processes during the coarse grid solve.

Dry Particulate Flows

Domain Decomposition Research Community Domain decomposition has an active research community that includes mathematicians, computer scientists, and engineers. A domain decomposition conference series has been held since its initiation in Paris in 1987. The proceedings for these conferences are a rich source of material for both the practical and the mathematical aspects of domain decomposition. Information on the conference series may be found at the domain decomposition website [1]. Currently, four books have been devoted to domain decomposition methods [2–5].

381

as sand and gravel, associated with coastal erosion, landslides, and avalanches. For reviews, see the texts of Rietma [6], P¨oschel and Schwager [5], Duran [1], and Zohdi [8]. In the manufacturing of particulate composite materials, small-scale particles, which are transported and introduced into a molten matrix, play a central role. For example, see Hashin [2], Mura [3], Nemat-Nasser and Hori [4], Torquato [7], and Zohdi and Wriggers [9]. In this brief introduction, the following topics are covered: • The dynamics of a single, isolated, particle • The dynamics of loosely flowing groups of particles • The dynamics of a cluster of rigidly bonded particles.

References 1. Gander, M: Domain Decomposition website. http://www. ddm.org (2012) 2. Mathew, T.: Domain Decomposition Methods for the Numerical Solution of Partial Differential Equations, vol. 61. Springer, Berlin (2008) 3. Quarteroni, A., Valli, A.: Domain Decomposition Methods for Partial Differential Equations, vol. 10. Clarendon Press, Oxford/New York (1999) 4. Smith, B.F., Bjørstad, P., Gropp, W.D.: Domain Decomposition: Parallel Multilevel Methods for Elliptic Partial Differential Equations. Cambridge University Press, Cambridge (1996) 5. Toselli, A., Widlund, O.: Domain Decomposition Methods– Algorithms and Theory. Springer, Berlin (2005)

Dry Particulate Flows Tarek I. Zohdi Department of Mechanical Engineering, University of California, Berkeley, CA, USA

Dynamics of an Individual Particle We start with an introduction to the dynamics of a single particle. A fixed Cartesian coordinate system will be used throughout this chapter. The unit vectors for such a system are given by the mutually orthogonal triad .e1 ; e2 ; e3 /. The temporal differentiation of a vector u is given by (Boldface symbols imply vectors or tensors.) d u1 d d u2 d u3 uD e1 C e2 C e3 D uP 1 e1 C uP 2 e2 C uP 3 e3 : dt dt dt dt (1) Kinetics of a Single Particle The fundamental relation between force and acceleration of a mass point is given by Newton’s second law of motion, in vector form: d .mv/ D dt

Introduction

;

(2)

where is the sum (resultant) of all the applied forces Flowing, small-scale, particles (“particulates”) are instantaneously acting on mass m. Newton’s second ubiquitous in industrial processes and in the natural law can be rewritten as sciences. Applications include electrostatic copiers, Z t2 d.mv/ inkjet printers, powder coating machines and a variety D ) G.t1 / C dt D G.t2 /; (3) dt of manufacturing processes. The study of uncharged t1 “granular” or “particulate” media, in the absence of electromagnetic effects, is wide-ranging. Classical where G.t/ D .mv/jt is the linear momentum. Clearly examples include the study of natural materials, such if D 0, then G.t1 / D G.t2 / and linear momentum is

D

382

Dry Particulate Flows

said to be conserved. A related quantity is the angular Subtracting the two expressions yields momentum, for example, relative to the origin def

Ho D r mv:

(4)

vi .t C t/  vi .t/ d vi O jt C t D C O. t/; dt

t

(8)

Clearly, the resultant moment M implies MDr

d Ho ) Ho .t1 / C D dt

Z

t2 t1

r „ƒ‚…

O where O. t/ D O.. t/2 /, when  D 12 . Thus, inserting this into the equation of motion yields

M

dt D Ho .t2 /:

(5) vi .t C t/ D vi .t/ C t mi

t ot

2 O .t C  t/ C O.. t/ /:

(9) Thus, if M D 0, then Ho .t1 / D Ho .t2 /, and angular momentum is said to be conserved. Note that adding a weighted sum of (7) and (8) yields

Multiple Particle Flow

vi .t C t/ D vi .t C t/C.1/vi .t/CO.. t/2 /; (10)

In the simplest particulate flow models, the objects in the flow are assumed to be small enough to be which will be useful shortly. Now, expanding the poconsidered (idealized) as spherical particles and that sition of the center of mass in a Taylor series about the effects of their rotation with respect to their center t C  t, we obtain of mass are unimportant to their overall motion. We consider a group of nonintersecting particles (Np in d ri jt C t .1  / t ri .t C t/ D ri .t C  t/ C total). The equation of motion for the i th particle in dt a flow is 1 d 2 ri C jt C t .1  /2 . t/2 2 dt 2 (6) mi rR i D ti ot .r1 ; r2 ; : : : ; rNp /; CO.. t/3 / (11) where ri is the position vector of the i th particle and t ot where i represents all forces acting on particle i , for example, contact forces from other particles and and near-field interactions. In order to simulate such systems, one employs numerical time-stepping schemes. d ri ri .t/ D ri .t C  t/  jt C t  t (For more details, in particular on the forces that dt t ot , namely contact, friction and near-field comprise 1 d 2 ri C jt C t  2 . t/2 C O.. t/3 /: interaction, see Zohdi [8].) For example, expanding the 2 dt 2 velocity in a Taylor series about t C  t, we obtain (12) d vi jt C t .1  / t vi .t C t/ D vi .t C  t/ C dt Subtracting the two expressions yields 2 1 d vi 2 2 jt C t .1  / . t/ C 2 dt 2 ri .t C t/  ri .t/ O D vi .t C  t/ C O. t/: (13) CO.. t/3 / (7)

t and Inserting (10) yields d vi jt C t  t vi .t/ D vi .t C  t/  dt 1 d 2 vi C jt C t  2 . t/2 C O.. t/3 /: 2 dt 2

ri .t C t/ D ri .t/ C .vi .t C t/ C .1  /vi .t// 2 O

t C O.. t/ /;

(14)

Dry Particulate Flows

383

and thus using (9) yields ri .t C t/ D ri .t/ C vi .t/ t C

. t/2 mi

2 O .t C  t/ C O.. t/ /:

where M is the total system mass and Nc is the number of particles in the cluster. A decomposition of the position vector for particle i , of the form ri D rcm Crcm!i , t ot where rcm!i is the position vector from the center of i mass to the particle i , allows the linear momentum of (15) the system of particles (G) to be written as

Nc Nc Nc X X X The term ti ot .t C  t/ can be handled in two main mi rPi D mi .Prcm C rP cm!i / D mi rP cm „ƒ‚… ways: i D1 i D1 i D1 t ot t ot Gi • i .t C  t/  i .ri .t C t/ C .1  /ri .t// Nc or X def t ot t ot D rP cm mi D MPrcm D Gcm ; (18) • i .t C  t/   i .ri .t C t// C .1  i D1 / ti ot .ri .t//. The differences are quite minute between either of PNc P cm D P cm!i D 0. Furthermore, G the above; thus, for brevity, we choose the latter. In since i D1 mi r MRrcm ; thus summary, we have the following:

. t/2   ri .t C t/ D ri .t/ C vi .t/ t C mi .ri .t C t// C .1  /

t ot

P cm D MRrcm D G

t ot i

.ri .t//

Nc X

ext def D i

 EX T ;

(19)

i D1



where ext represents the total external force acting on i EX T O represents the total external CO.. t/ /; (16) the particle i and  force acting on the cluster. The angular momentum of the system relative to the center of mass can be written where as (utilizing rP i D vi D vcm C vcm!i ) • When  D 1, then this is the (implicit) Backward Euler scheme, which is very stable (very dissipaNc Nc X X 2 O / D O.. t/2 / locally in time. tive) and O.. t/ Hcm!i D .rcm!i mi vcm!i / • When  D 0, then this is the (explicit) Forward i D1 i D1 Euler scheme, which is conditionally stable and Nc X 2 O / D O.. t/2 / locally in time. O.. t/ D .rcm!i mi .vi  vcm // • When  D 0:5, then this is the (implicit) “Midi D1 (20) 2 O point” scheme, which is stable and O.. t/ / D O.. t/3 / locally in time. 0 1 For more on time-stepping schemes for these types of systems, see, for example, P¨oschel and Schwager [5] BX C Nc X B Nc C or Zohdi [8]. C D .mi rcm!i vi /  B m r i cm!i C B @ i D1 A i D1 „ ƒ‚ … 2

D0

Clusters of Particles

vcm D Hcm :

(21)

In many cases, particles will agglomerate into rigid Since v cm!i D ! rcm!i , then clusters. When we consider a collection of particles Nc Nc that are rigidly bound together, the position vector of X X the center of mass of the system is given by Hcm!i D mi .rcm!i vcm!i / Hcm D i D1

PNc

rcm D Pi D1 Nc def

m i ri

i D1

mi

c 1 X m i ri ; M i D1

N

D

(17)

D

Nc X i D1

i D1

mi .rcm!i .! rcm!i //:

(22)

D

384

Dry Particulate Flows

Decomposing the relative position vector into its components,

Nc Nc X X D xO i 2 xO i 3 mi ; I 13 D I 31 D  xO i1 xO i 3 mi ; i D1

rcm!i D ri  rcm D xO i1 e1 C xO i 2 e2 C xO i 3 e3 ;

(29)

(23)

where xO i1 , xO i 2 , and xO i 3 are the coordinates of the mass or explicitly point measured relative to the center of mass, and expanding the angular momentum expression yields H1 D !1

Nc X

Nc X

i D1

i D1

.xO i22 C xO i23 / mi  !2

!3

Nc X

xO i1 xO i 2 mi

xO i1 xO i 3 mi

i D1

i D1

2

3 I 11 I 12 I 13 I D 4 I 21 I 22 I 23 5 : I 31 I 32 I 33

(30)

The particles’ own inertia contribution about their respective mass-centers to the overall moment of inertia (24) of the agglomerated body can be described by the Huygens-Steiner (generalized “parallel axis” theorem) formula (p; s D 1; 2; 3)

and H2 D !1

Nc X

xO i1 xO i 2 mi C !2

i D1

Nc X

2 .xO i1 C xO i23 / mi

IN ps D

Nc   X i C mi .jjri  rcm jj2 ıps  xO ip xO i s / : IN ps i D1

(31)

i D1

i For a spherical particle, IN pp D 25 mi Ri2 , and for p ¤ s, (25) IN i D 0 (no products of inertia), R being the particle i ps i D1 radius. (If the particles are sufficiently small, each particle’s own moment of inertia (about its own center) PNc and is insignificant, leading to IN ps D i D1 mi .jjri  2 jj ı  x O x O /.) Finally, for the derivative of the r cm ps ip i s N N c c X X angular momentum, utilizing rR i D ai D acm C acm!i , xO i1 xO i 3 mi  !2 xO i 2 xO i 3 mi H3 D !1 we obtain i D1 i D1 Nc X 2 Nc C!3 .xO i1 C xO i22 / mi ; (26) X rel P H D .rcm!i mi acm!i / i D1 cm

!3

Nc X

xO i 2 xO i 3 mi

i D1

which can be concisely written as D Hcm D I  !;

(27)

where we define the moments of inertia with respect to the center of mass

Nc X .rcm!i mi .ai  acm //

Nc Nc X X D .mi rcm!i ai /  mi rcm!i i D1

I 11 D

Nc X

.xO i22 C xO i23 / mi ;

I 22

i D1

I 33 D

Nc X

Nc X 2 D .xO i1 C xO i23 / mi ; i D1

2 .xO i1 C xO i22 / mi ;

(28)

i D1

I 12 D I 21 D 

(32)

i D1

„

i D1

ƒ‚ D0

P cm ; acm D H

! … (33)

and consequently c X P cm D d.I  !/ D H rcm!i dt i D1

N

Nc X i D1

xO i1 xO i 2 mi ;

I 23 D I 32

ext def D i

T MEX cm ;

(34)

Dynamic Programming T where MEX is the total external moment about the cm center of mass. Equations such as (34) are typically integrated numerically using techniques similar to the ones described earlier for individual particles (P¨oschel and Schwager [5] or Zohdi [8]).

References 1. Duran, J.: Sands, Powders and Grains. An Introduction to the Physics of Granular Matter. Springer, Heidelberg (1997) 2. Hashin, Z.: Analysis of composite materials: a survey. ASME J. Appl. Mech. 50, 481–505 (1983) 3. Mura, T.: Micromechanics of Defects in Solids, 2nd edn. Kluwer Academic, Dordrecht (1993) 4. Nemat-Nasser, S., Hori, M.: Micromechanics: Overall Properties of Heterogeneous Solids, 2nd edn. Elsevier, Amsterdam (1999) 5. P¨oschel, T., Schwager, T.: Computational Granular Dynamics. Springer, Heidelberg (2004) 6. Rietema, K.: Dynamics of Fine Powders. Springer, Heidelberg (1991) 7. Torquato, S.: Random Heterogeneous Materials: Microstructure and Macroscopic Properties. Springer, New York (2002) 8. Zohdi, T.I.: Dynamics of Charged Particulate Systems. Modeling, Theory and Computation. Springer, Berlin/ New York (2012) 9. Zohdi, T.I., Wriggers, P.: Introduction to Computational Micromechanics, Second Reprinting. Springer, Berlin/Heidelberg (2008)

385

Example To illustrate this idea, consider the network depicted in Fig. 1, where the numbers on the arcs denote their lengths. Suppose that the problem of interest is to find the shortest path from node 1 to node 7, where the length of a path is equal to the sum of the arcs’ lengths on that path. Clearly, the solution to this problem can be worked out from the solutions to the following three (modified) problems: • Determine the shortest path from node 2 to node 7. • Determine the shortest path from node 3 to node 7. • Determine the shortest path from node 4 to node 7. To state this idea formally, let f .n/ denote the length of the shortest path from node n to node 7 and d.i; j / denote the length of arc .i; j /. Then clearly f .1/ is equal to either d.1; 2/ C f .2/, or d.1; 3/ C f .3/, or d.1; 4/ C f .4/, depending on which is smaller. We can therefore write f .1/ D min fd.1; x/ C f .x/g

(1)

x2f2;3;4g

Applying the same analysis to other nodes on the network yields the following typical dynamic programming functional equation: f .n/ D min fd.n; x/ C f .x/g ; n D 1; : : : ; 6 x2Suc.n/

(2)

Dynamic Programming Moshe Sniedovich Department of Mathematics and Statistics, The University of Melbourne, Melbourne, VIC, Australia

where Suc.n/ denotes the set of immediate successors of node n. It can be easily solved for n D 6; 5; : : : ; 1 (in this order), observing that by definition f .7/ D 0. The results are shown in Fig. 2, where the f .n/ values are displayed (in square brackets) above the nodes and

2

Description

1

9 2

5 4

3

Dynamic programming (DP) is a general-purpose 1 4 7 8 9 problem-solving paradigm. It is based on the proposition that in many situations a problem can be 4 5 1 1 decomposed into a family of related problems so that 10 3 6 the solution to the problem of interest (target problem) is expressed in terms of the solutions to these related problems (modified problems). Dynamic Programming, Fig. 1 A shortest path problem

D

386

Dynamic Programming

[4] 9

2 [5]

1

1

2 8

4

[7] 3

[2]

5

Based on this idea, the plan of attack that dynamic programming puts forward to tackle problems such as the above is summed up in the following metarecipe: • Step 1: Embed the target problem in a family of related problems. • Step 2: Derive the relationship between the solutions to these problems. • Step 3: Solve this relationship. • Step 4: Recover a solution to the target problem from the solution to this relationship. One might argue that this approach to problemsolving had been employed by humans, even if informally, ever since rational planning began. Still, it was the mathematician Richard E. Bellman (1920– 1984) who develop this approach into a full-blown systematic theory, which he called dynamic programming. Bellman formulated this approach in terms of a (generic) sequential decision process. Meaning that in this setting, the network depicted in Fig. 1 represents a sequential decision process whose states are represented by the nodes and the decisions are represented by the arcs. To describe this symbolically, let S denote the state space, i.e., the set of all admissible states, and let sT 2 S denote the final (target) state. The task is then to determine the best (optimal) way of reaching a given target state  from a given initial state  by implementing a sequence of decisions. The dynamics of this process is governed by a transition law T meaning that s 0 D T .s; x/ is the state resulting from applying decision x to state s. In this process, all decisions are constrained by the requirement that the decision applied to state s must be an element of some given set D.s/  D, where D denotes the decision space. It is assumed that applying decision x to state s generates the return r.s; x/. For simplicity assume that the total return is equal to the sum of returns generated by the process as it proceeds from the initial state  to the final state . To illustrate the working of Step 1 in the metarecipe, let f .s/ denote the optimal total return given that the initial state is s, namely, define

3 9

1 10

Methodology

4

4 5

[3]

[1]

[0] 7

1

6

Dynamic Programming, Fig. 2 Optimal solution of the DP functional equation associated with the shortest path problem

the bold arcs represent the solution to the functional equation. The optimal (shortest) path is (1,2,4,6,7) yielding a total length f .1/ D 5. Principle of Optimality The rationale behind the proposition to relate the modified problems to one another, with the view to derive the dynamic programming functional equation from this relation, is based on the following argument. Suppose that the objective is to get to node 7 along the shortest path and that (a) we are at node n and (b) the next transition is to node m 2 Suc.n/. Then the best way to reach node 7 from node m is along the shortest path from node m to node 7. The point is here that it is immaterial how node m was reached. Once the process is at node m, then the best way to reach node 7 is the shortest path from node m to node 7. This argument puts into action what is known in dynamic programming as the Principle of Optimality. This principle was formulated by Richard Bellman, the father of dynamic programming, as follows: PRINCIPLE OF OPTIMALITY . An optimal policy has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision. Bellman [1, p. 83]

Thus, a dynamic programming model is formulated in a way ensuring that this principle holds, whereupon the dynamic programming functional equation can be viewed as a mathematical transliteration of the principle.

f .s/ WD max fr.s1 ; x1 / C r.s2 ; x2 / C    k x1 ;:::;xk

C r.sk ; xk /g ; s 2 S n fg

(3)

Dynamic Programming

387

s1 D s

(4) gramming functional equations, consider a case where the state space S and the decision space D consist (5) of a finite number of elements. Assume also that the skC1 D  functional equation (8) can be solved iteratively by snC1 D T .sn ; xn / ; n D 1; 2; : : : ; k (6) enumerating the value of s 2 S n fg in an order such that the value of f .T .s; x// is known for all x 2 D.s/ (7) when the equation is solved for state s. In short, assume xn 2 D.sn / ; n D 1; 2; : : : ; k that for each s 2 S , solving the right-hand side of (8) is a simple matter. with f ./ WD 0. The factor that determines whether or not this It should be noted that the max operation is subequation will lend itself to solution is then the number scripted by k to indicate that the value of k is not of admissible states in S , namely, the size of S . necessarily fixed in advance as it can be contingent The point to note here is that in various dynamic on the initial state s and the decisions made. Also, the programming applications, even a modest increase target problem, associated with s D , is embedded in a family of modified problems associated with the in the problem’s size causes a blowout in the size of S which results in a blowout in the amount of other states in S . computation required to solve the functional equation. As a consequence, exact solutions cannot be recovered Functional Equation It is straightforward to show that if optimal solutions for the functional equation in such cases. A good exist for all s 2 S , then the following typical dynamic illustration of this difficulty would be the genetic job scheduling problem, where n jobs are to be scheduled programming functional equation holds: so as to meet certain goals and constraints. Suffice it to say that in many dynamic programming formulations f .s/ D max fr.s; x/ C f .T .s; x//g ; s 2 S n fg of this problem, the size of S is equal to c2n where c is s2D.s/ (8) some constant. Unsurprisingly, considerable effort has gone into As for the solution of an equation of this type, finding methods and techniques to deal with the comthis will depend on the case considered, for there is putational requirements of dynamic programming alno general-purpose solution method for this task. In gorithms, in response to the huge challenge presented some cases the solution procedure is straightforward, by the Curse of Dimensionality. This challenge has in others it is considerably complicated; in some it is also stimulated research into methods aimed at yielding numeric, in others it is analytic. For this, and other approximate solutions to dynamic programming funcreasons, user-friendly software support for dynamic tional equations, including heuristic methods (see [5] and [6]). programming is rather limited. s:t:

The Curse of Dimensionality Applications The most serious impediment obstructing the solution of dynamic programming functional equations was dubbed by Bellman: the Curse of Dimensionality. Bellman [1, p. ix] coined this colorful phrase to describe the crippling effect that an increase in a problem’s “size” can have on the ability to solve the problem. It is important to appreciate that this ill does not afflict only dynamic programming functional equations. Still, to see how it affects the solution of dynamic pro-

Dynamic programming’s basic character as a “general-purpose” methodology to problem solving is manifested, among other things, in its wide spectrum of application areas: operations research, economics, sport, engineering, business, computer science, computational biology, optimal control, agriculture, medicine, health, ecology, military, typesetting, recreation, artificial intelligence, and more (see [2–6]).

D

388

Dynamical Models for Climate Change

References

Description

1. Bellman, R.: Dynamic Programming. Princeton University Press, Princeton (1957) 2. Denardo, E.V.: Dynamic Programming: Models and Applications. Dover, Mineola (2003) 3. Dreyfus, S.E., Law, A.M.: The Art and Theory of Dynamic Programming. Academic, New York (1977) 4. Nemhauser, G.L.: Introduction to Dynamic Programming. Wiley, New York (1966) 5. Powell, W.B.: Approximate Dynamic Programming: Solving the Curses of Dimensionality. Wiley-Interscience, Hoboken (2007) 6. Sniedovich, M.: Dynamic Programming: Foundations and Principles, 2nd edn. CRC, Boca Raton/London/New York (2011)

Climate Change Human societies have dramatically changed the composition of the atmosphere, raising carbon dioxide levels from a preindustrial value of 280 parts per million to over 400 parts per million, primarily from fossil fuel burning and deforestation. Since carbon dioxide is a greenhouse gas, one expects an increase in global temperature due to the modified atmospheric composition, and temperatures have warmed about 0.8 ı C since the early twentieth century. Independent evidence from satellite data, ground stations, ship measurements, and mountain glaciers all show that global warming has occurred [1]. Other aspects of climate that have changed include atmospheric water vapor concentration, Arctic sea ice coverage, precipitation intensity, extent of the Hadley circulation, and height of the tropopause in accordance with predictions.

Dynamical Models for Climate Change Dargan M.W. Frierson Department of Atmospheric Sciences, University of Washington, Seattle, WA, USA

Mathematics Subject Classification 86A10; 76U05

Synonyms General circulation models; Global climate models; Global warming

Short Definition

General Circulation Models Because there is considerable interest in predicting future climate changes over the coming decades, there is a large international effort focused on climate modeling at high spatial resolution, with detailed treatment of complex physics. These models, known as general circulation models or global climate models (GCMs), incorporate physical effects such as the fluid dynamics of the atmosphere and ocean, radiative transfer, shallow and deep moist convection, cloud formation, boundary layer turbulence, and gravity wave drag [2, 3]. More recently, Earth system models have additionally incorporated effects such as atmospheric chemistry and aerosol formation, the carbon cycle, and dynamic vegetation. Due to the complexity of GCMs, output from these models can often be difficult to interpret. To make progress in understanding, there has been a concerted effort among climate scientists and applied mathematicians to develop a hierarchy of dynamical models, designed to better understand climate phenomena. Held [4] has provided an argument for the usefulness of hierarchies within climate science. Four different classes of simplified dynamical models of climate change are discussed in this entry.

Climate change, or global warming, refers to the human-caused increase in temperature that has occurred since industrialization, which is expected to intensify over the rest of the century. The term also refers to the associated changes in climatic features such as precipitation patterns, storm tracks, overturning circulations, and jet streams. Dynamical Radiative-Convective Models models for climate change use basic physical principles The essence of global warming can be elegantly expressed as a one-dimensional system, with temperature to calculate changes in climatic features.

Dynamical Models for Climate Change

and atmospheric composition a function of height alone. The first necessary physical process is radiative transfer, which includes both solar heating and radiation emitted from the Earth, which is partially absorbed and reemitted by greenhouse gases. The second necessary ingredient is convection, which mixes energy vertically within the troposphere, or weather layer, on Earth. The first radiative-convective calculation of carbon dioxide-induced global warming was performed by Manabe and Strickler in 1964 [5]. Many example radiative-convective codes are publicly available. Idealized Dry GCMs GCMs typically use the primitive equations as their dynamical equations, which assume hydrostatic balance and the corresponding small-aspect ratio assumptions that are consistent with this. The primitive equations are @u @u C v  ru C ! D fv @t @p C

u v tan. / 1 @˚   F a a cos @

@v @v C v  rv C ! D f u @t @p 

u2 tan. / 1 @˚   F

a a @

@T T ! @T C v  rT C ! D CQ @t @p p @˚ D Rd Tv @ln p v C

@! D0 @p

where  D longitude, D latitude, p D pressure, u D zonal wind, v D meridional wind, f D 2˝ sin. / D Coriolis parameter, a D Earth radius, ˚ D g z D geopotential with g D gravitational acceleration and z D height, ! D Dp D pressure Dt velocity, T D temperature, Tv D T =.1  .1  Rd =Rv /q/ D virtual temperature (which takes into account the density difference of water vapor), Rd D ideal gas constant for dry air, Rv D ideal gas constant for water vapor,  D cRp , and cp D specific heat of dry air.

389

Much of the complexity of comprehensive GCMs comes from parameterizations of Q D heating and F = momentum sources. A realistic circulation can be produced from remarkably simple parameterizations of Q and F. Held and Suarez [6] used “Newtonian cooling” and “Rayleigh friction” in their dry dynamical core model: T  Teq Q v FD F

QD

The equilibrium temperature Teq is chosen essentially to approximate the temperature structure of Earth if atmospheric motions were not present. Rayleigh friction exists within the near-surface planetary boundary layer. The relaxation times Q and D are chosen to approximate the typical timescales of radiation and planetary boundary layer processes. Hyperdiffusion is typically added as a final ingredient. The dry dynamical core model can be used to calculate dynamical responses to climate change by prescribing heating patterns similar to those experienced with global warming, for instance, warming in the upper tropical troposphere, tropopause height increases, stratospheric cooling, and polar amplification. An example of such a study, which examines responses of the midlatitude jet stream and storm tracks, is Butler et al. [7]. Idealized Moist GCMs One of the most rapidly changing quantities in a warming climate is water vapor. The Clausius-Clapeyron equation states that there is an approximately 7 % per degree increase in the water vapor content of the atmosphere at constant relative humidity. Increases in water vapor are fundamental to many aspects of climate change. Water vapor is a positive feedback to global warming from its radiative impact. Precipitation in the rainiest regions increases. Because there is a release of latent heat when condensation occurs, temperature structure, eddy intensity, and energy transports are also affected by increased water vapor content. In order to study these impacts of climate change in an idealized model with an active moisture budget, Frierson et al. [8] developed the gray-radiation moist (GRaM) GCM. This model has radiation that is only a function of temperature, and simplified surface flux

D

390

Dynamical Models for Climate Change

boundary layer, and moist convection schemes. In ad- References dition to the influences of water vapor listed above, this model has been used to study the effect on overturning 1. IPCC: Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment circulations, precipitation extremes, and movement of Report of the Intergovernmental Panel on Climate Change rain bands. [Stocker, T.F., Qin, D., Plattner, G.-K., Tignor, M., Allen, S.K., Boschung, J., Nauels, A., Xia, Y., Bex, V., Midgley, P.M. (eds.)]. Cambridge University Press, Cambridge/New Models with Simplified Vertical Structure York, pp. 1535 (2013) A final class of simplified dynamical models of climate 2. Donner, L.J., et al.: The dynamical core, physical paramechange are those with simplified vertical structure. terizations, and basic simulation characteristics of the atmoMost famous of these are energy balance models, spheric component AM3 of the GFDL global coupled model CM3. J. Clim. 24, 3484–3519 (2011). doi:http://dx.doi.org/ reviewed in North et al. [9], which represent the ver10.1175/2011JCLI3955.1 tically integrated energy transport divergence in the 3. Gent, P.R., Danabasoglu, G., Donner, L.J., Holland, M.M., atmosphere as a diffusion. The simplest steady-state Hunke, E.C., Jayne, S.R., Lawrence, D.M., Neale, R.B., energy balance model can be written as Rasch, P.J., Vertenstein, M., Worley, P.H., Yang, Z.-L.,

S  L C Dr T D 0 2

where S is the net (downward minus upward) solar radiation; L D A C BT is the outgoing longwave radiation, a linear function of temperature T ; and D is diffusivity. Typically written as a function of latitude alone, the energy balance model can incorporate climate feedbacks such as the ice-albedo feedback and can calculate the temperature response due to changes in outgoing radiation (e.g., by modifying A). More recently, energy balance models have been used to interpret the results from both comprehensive GCMs and idealized GCMs such as those described above. This exemplifies the usefulness of a hierarchy of models for developing understanding about the climate and climate change.

4.

5.

6.

7.

8.

9.

Zhang, M.: The community climate system model version 4. J. Clim. 24(19), 4973–4991 (2011) Held, I.M.: The gap between simulation and understanding in climate modeling. Bull. Am. Meteor. Soc. 86, 1609–1614 (2005). doi:10.1175/BAMS-86-11-1609 Manabe, S., Strickler, R.F.: Thermal equilibrium of the atmosphere with a convective adjustment. J. Atmos. Sci. 21, 361–385 (1964) Held, I.M., Suarez, M.J.: A proposal for the intercomparison of the dynamical cores of atmospheric general circulation models. Bull. Am. Meteor. Soc. 75, 1825–1830 (1994) Butler, A.H., Thompson, D.W.J., Heikes, R.: The steadystate atmospheric circulation response to climate change-like thermal forcings in a simple general circulation model. J. Clim. 23, 3474–3496 (2010) Frierson, D.M.W., Held, I.M., Zurita-Gotor, P.: A grayradiation aquaplanet moist GCM. Part I: static stability and eddy scale. J. Atmos. Sci. 63, 2548–2566 (2006) North, G.R., Cahalan, R.F., Coakley, J.A.: Energy balance climate models. Rev. Geophys. Space Phys. 19, 91–121 (1981)

E

eigenvalues of A and in some cases the corresponding eigenvectors.

Eigenvalues and Eigenvectors: Computation

Description

Franc¸oise Tisseur School of Mathematics, The University of Manchester, Manchester, UK

Mathematics Subject Classification 15A18; 65F15

Synonyms Eigendecompositions; Eigenproblems; Eigensolvers

Short Definition A vector x 2 Cn is called an eigenvector of A 2 Cnn if x is nonzero and Ax is a multiple of x, that is, there is a  2 C such that Ax D x;

x ¤ 0:

(1)

No algorithm can calculate eigenvalues of n  n matrices exactly in a finite number of steps for n > 4, so eigensolvers must be iterative. There are many methods aiming at approximating eigenvalues and optionally eigenvectors of a matrix. The following points should be considered when choosing an appropriate eigensolver for a given eigenproblem [2]: (i) Structure of the matrices defining the problem: Is the matrix real or complex? Is it Hermitian, symmetric, or sparse? (ii) Desired spectral quantities: Do we need to calculate all the eigenvalues or just a few or maybe just the largest one? Do we need the corresponding eigenvectors? (iii) Available operations on the matrix and their cost: Can we perform similarity transformations? Can we solve linear systems directly or with an iterative method? Can we use only matrix-vector products? Eigensolvers do not in general compute eigenpairs exactly. Condition numbers and backward errors are useful to get error bounds on the computed solution. A condition number measures the sensitivity of the solution of a problem to perturbations in the data, whereas a backward error measures how far a problem has to be perturbed for an approximate solution to be an exact solution of the perturbed problem. With consistent definitions, we have the rule of thumb that

The complex scalar  is called the eigenvalue of A associated with the right eigenvector x. The pair .; x/ is called an eigenpair of A. The eigenvalues of A are the roots of the characteristic polynomial det.I A/ D 0, which has degree n, so A has n eigenvalues, some of which may be repeated. An eigensolver is a program or an algorithm that < condition numberbackward error: computes an approximation to some of or all the error in solution  © Springer-Verlag Berlin Heidelberg 2015 B. Engquist (ed.), Encyclopedia of Applied and Computational Mathematics, DOI 10.1007/978-3-540-70529-1

392

Eigenvalues and Eigenvectors: Computation

An eigensolver is backward stable if for any matrix A, it produces a solution with a small backward error. Thus, if an eigensolver is backward stable, then the error in the solution is small unless the condition number is large. There are roughly two groups of eigensolvers: those for small-to medium-size matrices and those for large and usually sparse matrices. Small-to Medium-Size Eigenproblems Most methods for small-to medium-size problems work in two phases: a reduction to a condensed form such as Hessenberg form or tridiagonal form in a finite number of steps followed by the eigendecomposition of the condensed form. These methods use similarity transformations and require O.n2 / storage and O.n3 / operations, so they cannot be used when n is very large. Any matrix A 2 Cnn is unitarily similar to a triangular matrix T : U  AU D T;

U  U D I:

(2)

This is a Schur decomposition of A and it reveals the eigenvalues of A on the diagonal of T . This decomposition cannot exist over R when A 2 Rnn has complex conjugate eigenvalues. However, any real matrix A is orthogonally similar to a real quasi-triangular matrix T with 1  1 and 2  2 blocks on its diagonal, where the 1  1 diagonal blocks display the real eigenvalues and the 22 diagonal blocks contain the complex conjugate eigenpairs. The eigenvectors of A are of the form U v, where v is an eigenvector of T . Note that an eigenvalue of a nonnormal matrix A (i.e., AA ¤ A A) can have a large condition number when it is very close to another eigenvalue. Such eigenvalues can be difficult to compute accurately. The QR algorithm computes the Schur (or real Schur) decomposition of a complex (or real) A. It starts by reducing A to Hessenberg form H in a finite number of steps using unitary (or orthogonal) transformations, and then it applies the QR iteration, whose simplest form is given by H0 D H;

Hk1 D Qk Rk (QR factorization); Hk D Rk Qk ; k D 1; 2; : : : :

Under certain conditions, Hk converges to a (real) Schur form of A. To make the QR iterations effective, multishifts implemented implicitly are used as well as (aggressive) deflation. The QR algorithm is a backward-stable algorithm [3]. Often in applications, A is Hermitian, that is, A D T  A .D A /, or real symmetric, that is, AT D A with A 2 Rnn . Then the Schur decomposition (2) simplifies to X  AX D diag.1 ; 2 ; : : : ; n / 2 Rnn ; X  X D In ; that is, A is unitarily diagonalizable and has real eigenvalues. The columns of X are eigenvectors and they are mutually orthogonal, and X can be taken real when A is real. The eigenvalues of A are always well conditioned in the sense that changing A in norm by at most  changes any eigenvalue by at most . To preserve these nice properties numerically, it is best to use an eigensolver for Hermitian matrices. Such eigensolvers start by reducing A to real tridiagonal form T by unitary similarity transformation, Q AQ D T , and then compute the eigendecomposition of T . There are several algorithms for computing eigenvalues of tridiagonal matrices: (a) The symmetric QR algorithm finds all the eigenvalues of a tridiagonal matrix and computes the eigenvectors optionally. It is backward stable. (b) The divide and conquer method divides the tridiagonal matrix into two smaller tridiagonal matrices, solves the two smaller eigenproblems, and glues the solutions together by solving a secular equation. It can be much faster than the QR algorithm on large problems but requires more storage. It is backward stable. (c) Bisection is usually used when just a small subset of the eigenvalues is needed. The corresponding eigenvectors can be computed by inverse iteration. This approach is faster than the QR algorithm and divide and conquer method when the eigenvalues are not clustered. (d) The relatively robust representation algorithm is based on LDLT factorizations of the shifted tridiagonal matrix T  I for a number of shifts  near clusters of eigenvalues and computes the small eigenvalues of T  I very accurately. It is faster than the other methods on most problems.

Eigenvalues and Eigenvectors: Computation

Large and Sparse Eigenproblems For large problems for which O.n2 / storage and O.n3 / operations are prohibitive, there are algorithms that calculate just one or a few eigenpairs at a much lower cost. Most of these algorithms proceed by generating a sequence of subspaces fKk gk0 that contain increasingly accurate approximations to the desired eigenvectors. A projection method is then used to extract approximate eigenpairs from the largest Kk . The projection method requires the matrix A and a subspace Kk of dimension k  n containing an approximate eigenspace of A. It proceeds as follows: 1. Let the columns of V 2 Cnk be a basis for Kk and let W 2 Cnk be such that W  V D I . (V and W are bi-orthogonal.) 2. Form Ak D W  AV (projection step). 3. Compute the m desired eigenpairs .e j ; j / of Ak , j D 1W m  k. 4. Return .e j ; V j / as approximate eigenpairs of A (Ritz pairs). If the approximate eigenvectors of A are not satisfactory, they can be reused in some way to restart the projection method [4, Chap. 9]. The projection method approximates an eigenvector x of A by a vector e x D V  2 Kk with corresponding approximate eigenvalue e . Usually, the projection method does a better job of estimating exterior eigenvalues of A than interior eigenvalues. If these are not the eigenvalues of interest, then prior to any computation, we can apply a spectral transformation that maps the desired eigenvalues to the periphery of the spectrum, a common example of which is the shift-and-invert transformation, f ./ D 1=.  /. This transformation  yields the matrix .A  I /1 , which has eigenpairs ./1 ; x/ corresponding to the eigenpair .; x/ of A. The eigenvalues of .AI /1 of greatest absolute value correspond to the eigenvalues of A closest to the shift . The numerical methods do not form .A  I /1 explicitly but solve linear systems with A  I instead. Iterations for large sparse problems are usually based on matrix-vector products. When shift-and-invert is used, the matrix vector products are replaced with linear solvers with the matrix A  I . The power method is the simplest method. From a given vector x0 , it computes and iterates

xkC1 D Axk =kxk k2 ;

k D 1; 2; : : : ;

393

until xkC1 becomes parallel to the eigenvector associated with the largest eigenvalue in absolute value, which is then approximated by xk xkC1 =kxk k2 . When shift-and-invert is used, the power method is called inverse iteration. Subspace iteration operates on several vectors simultaneously as opposed to one vector for the power method. It approximates the largest eigenvalues in absolute value together with their corresponding eigenvectors. The Arnoldi method is a projection method. Starting with a vector v, it builds a matrix Qk with orthonormal columns that form a basis for the Krylov subspace Kk .A; v/ D spanfv; Av; : : : ; Ak1 vg: It approximate the eigenvalues of A by the eigenvalues (the Ritz values) of the Hessenberg matrix Hk D Qk AQk , which are computed with the QR algorithm. Restart techniques exist to keep memory requirement and computational overhead low, and a shift-and-invert spectral transform can be used to target eigenvalues close to the shift . Shift-and-invert Arnoldi requires accurate solution of large systems with A  I , which may not be practical when A is too large. The JacobiDavidson method can be used in this case as it requires less accurate solution to linear systems, which are preconditioned and solved iteratively. When A is Hermitian or symmetric, then as for the Arnoldi method, the Lanczos method builds step by step a matrix Qk whose columns are orthonormal and form a basis for the Krylov subspace Kk .A; b/ D spanfb; Ab; : : : ; Ak1 bg, where b is a given vector. It approximates the eigenvalues of A by the eigenvalues of the symmetric tridiagonal matrix Tk D Qk AQk of smaller size. The Lanczos vectors can suffer from loss of orthogonality and reorthogonalization is sometimes necessary. Spectral transformations and restarting techniques can also be used [2]. Generalized Eigenproblems An eigenproblem defined by a single square matrix as in (1) is called a standard eigenvalue problem as opposed to a generalized eigenproblem Ax D Bx;

x¤0

(3)

defined by two matrices A and B. The main differences between the standard and generalized eigenvalue

E

394

problems are that when B is singular, there are infinite eigenvalues and when det.A  B/ is identically zero (i.e., A  B is a singular pencil), any  is an eigenvalue. The QZ algorithm is an extension of the QR algorithm for regular generalized eigenproblems (i.e., det.AB/ 6 0) of small to medium size. It computes the generalized Schur decomposition Q .AB/Z D T  S , where Q; Z are unitary and T; S are upper (quasi-)triangular and whose eigenvalues can be read off the diagonal entries of T and S . The QZ algorithm starts by reducing A  B to Hessenbergtriangular form and then applies the QZ steps iteratively. Implementations of the QZ algorithm return all the eigenvalues and optionally the eigenvectors. It is a backward-stable algorithm. When A and B are Hermitian with B positive definite, (i.e., B’s eigenvalues are positive), (3) can be rewritten as a Hermitian standard eigenvalue problem

Eikonal Equation: Computation

References 1. Anderson, E., Bai, Z., Bischof, C.H., Blackford, S., Demmel, J.W., Dongarra, J.J., Du Croz, J.J., Greenbaum, A., Hammarling, S.J., McKenney, A., Sorensen, D.C.: LAPACK Users’ Guide, 3rd edn. Society for Industrial and Applied Mathematics, Philadelphia (1999) 2. Bai, Z., Demmel, J.W., Dongarra, J.J., Ruhe, A., van der Vorst, H.A. (eds.): Templates for the Solution of Algebraic Eigenvalue Problems: A Practical Guide. Society for Industrial and Applied Mathematics, Philadelphia (2000) 3. Golub, G.H., Van Loan, C.F.: Matrix Computations, 3rd edn. Johns Hopkins University Press, Baltimore (1996) 4. Watkins, D.S.: The Matrix Eigenvalue Problem: GR and Krylov Subspace Methods. Society for Industrial and Applied Mathematics, Philadelphia (2007)

Eikonal Equation: Computation

Vincent Jacquemet Centre de Recherche, Hˆopital du Sacr´e-Coeur de Montr´eal, Montr´eal, QC, Canada Department of Physiology, Universit´e de Montr´eal, x D L y; L1 AL y D y; Institut de G´enie Biom´edical and Groupe de Recherche en Sciences et Technologies Biom´edicales, where B D LL is the Cholesky factorization of B, Montr´eal, QC, Canada which can be solved with eigensolvers for the standard Mathematics Subject Classification Hermitian problem. For large sparse problems, inverse iteration and subspace iteration can be applied to L1 AL 35J60; 65N30 kept in factored form. There is a variant of the Lanczos algorithm, which builds a B-orthogonal Synonyms matrix Qk of Lanczos vectors such that range.Qk / D spanfb; B 1 Ab; : : : ; .B 1 A/k1 bg for a given vector Eikonal-diffusion equation; Eikonal equation b and approximates the eigenvalues of A  B by the eigenvalues of the symmetric matrix tridiagonal Short Definition matrix Tk D Qk AQk . There is also a variant of the Jacobi-Davidson method that uses B orthogonality. The eikonal equation is a nonlinear partial differential There are software repository and good online equation that describes wave propagation in terms of search facilities for mathematical software such as arrival times and wave front velocity. Applications Netlib (http://www.netlib.org) and GAMS (http:// include modeling seismic waves, combustion, comgams.nist.gov, developed by the National Institute putational geometry, image processing, and cardiac of Standards and Technology.) where eigensolvers can electrophysiology. be downloaded. Eigensolvers for small-to mediumsize problems and also some eigensolvers for large problems are available in almost all linear-algebra- Description related software packages such as LAPACK [1] and MATLAB, (MATLAB is a registered trademark of The Problem Statement MathWorks, Inc.) as well as general libraries such as A wave propagation process may be meaningfully represented by its arrival time .x/ at every point the NAG Library. (http://www.nag.co.uk)

Eikonal Equation: Computation

395

2  x y Cy Cx x in space (e.g., shock wave, seismic wave, sound ; Di;j ; 0 Cmax.Di;j ; Di;j ; 0/2 max Di;j propagation). The local propagation velocity, which 2 D 1=ci;j (3) can be computed as krk1 , is often determined by the physical properties of the medium and therefore may be assumed to be a known positive scalar field c.x/. where the finite difference operators are defined as Cx x This relation leads to the so-called eikonal equation Di;j  D .i;j  i 1;j /=x, Di;j  D .i C1;j  y Cy [5, 9]: i;j /=x, and similarly for Di;j and Di;j . The algorithm maintains three lists of nodes: accepted c krk D 1 (1) nodes (for which  has been determined), considered nodes (for which  is being computed, one grid point whose purpose is to compute arrival times from local away from an accepted node), and far nodes (for which propagation velocity. The zero of arrival time is defined  is set to C1). The quadratic equation (3) is used to on a curve 0 as Dirichlet boundary condition  D 0 determine the values of  in increasing order. At each (the wave front originates from the source 0 ). The step,  is computed at considered nodes from known eikonal equation may also be derived from the (hypervalues at accepted nodes and the smallest value of  bolic) wave equation [6]. among those considered becomes accepted. The lists The eikonal-diffusion equation is a generalization are then updated and another step is performed until that involves an additional diffusive term [11]: all nodes are accepted. The efficiency of the method relies on the implementation of list data structures and kcrk D 1 C r  .Dr/: (2) sorting algorithms. The fast marching method can be extended to triangulated surface [8, 10] by adapting (3). A Matlab/C implementation (used here) has To account for possible anisotropic material properties, been made available on Matlab Central (http://www. the propagation velocity c and the diffusion coefficient mathworks.com/matlabcentral/fileexchange/6110) by D are symmetric positive definite tensor fields. Gabriel Peyr´e for both structured and unstructured The boundary condition is  D 0 on 0 and meshes. n  Dr D 0 on other boundaries (n is normal to the boundary). The diffusive term creates wave Newton-Based Method for the front curvature-dependent propagation velocity, Eikonal-Diffusion Equation smoothens the solution, and enforces numerical When physically relevant (e.g., for cardiac propagastability. In the context of wave propagation in tion, see Pernod et al. [7]), the eikonal-diffusion equanonlinear reaction-diffusion systems (e.g., electrical tion may be used to refine the solution provided by impulse propagation in the heart), the eikonal-diffusion the fast marching algorithm. In this case, if  is an equation may also be derived from the reaction- approximate solution to (2) satisfying the boundary diffusion equation using singular perturbation theory conditions, a better approximation  C can be ob[1]. tained by substituting  C into (2) and finding a The objective is to compute the arrival time field solution up to second order in . This is equivalent to () knowing the material properties (c and D) and the Newton iterations for nonlinear system solving. Taylor location of the source (0 ). expansion of (2) leads to the following steady-state convection-diffusion equation for the correction : Fast Marching Method for the Eikonal Equation kcrk  r  .Dr/  1 D r  .Dr /  kcrk1 The fast marching method [9] is an efficient algorithm r c cr (4) to solve (1) in a single pass. Its principle, based on Dijkstra’s shortest path algorithm, exploits the causality of wave front propagation. In a structured grid with boundary condition n  Dr D 0 and D 0 in 0 . This linearized equation can be solved using finite with space steps x and y (the value of a field F at coordinate .ix; jy/ is denoted by Fi;j ), (1) is elements. The procedure is given here for a triangular discretized as [10]: mesh composed of a set of nodes i 2 V and a set of

E

396

Eikonal Equation: Computation

X ˝ij k  triangles .ij k/ 2 T , of area ˝ij k , with i; j; k 2 V. kcij k rij k k  1 fm ./ D Linear shape functions, denoted by Ni for P i 2 V, are 3 .ij k/ 2 T used to reconstruct the scalar fields  D i 2V i Ni P m 2 fij kg  and D i 2V i Ni . These functions are linear in each triangle; the gradient operator evaluated in triangle C3 rij k Nm  Dij k rij k  : .ij k/ is noted rij k . Similarly, the parameters cij k and (6) Dij k denote the values of c and D at the center of gravity of the triangle .ij k/. The application of the Galerkin finite element approach [2] to (4) leads to For vertices m 2 0 , the boundary condition D the linear system A./  D f./, where the matrix and 0 is applied by setting Amn D ımn and fm D 0, the right-hand side are computed as: which ensures that A is not singular. An easy and efficient implementation in Matlab based on sparse matrix manipulation is possible after reformulation [4]. X ˝ij k rij k Nm  Dij k rij k Nn Amn ./ D  Practically, the first estimate  0 is given by the fast .ij k/2T marching method (neglecting diffusion). At iteration X ˝ij k n C 1, the correction nC1 is obtained by solving the     kcij k rij k k1 cij k rij k   linear system A. n /  nC1 D f. n /. Then  nC1 D 3 .ij k/2T  n C nC1 is updated until the norm of the correction   cij k rij k Nn (5) falls below a given tolerance k nC1 k < tol.

Eikonal Equation: Computation, Fig. 1 Propagation of the electrical impulse in an anisotropic surface model of the human atria computed using the eikonal-diffusion equation. (a) Normal propagation from the sinoatrial node region. Activation time is color-coded. Isochrones are displayed every 10 ms. White arrows illustrate propagation pathways. (b) Reentrant propagation sim-

ilar to typical atrial flutter in a model with slower propagation velocity. The line of block is represented as a thick black line. TV tricuspid valve, MV mitral valve, LAA left atrial appendage, RAA right atrial appendage, PVs pulmonary veins, IVC inferior vena cava, SVC superior vena cava, SAN sinoatrial node

Elastodynamics

397

Extension to Reentrant Waves The eikonal-diffusion equation, due to its local nature, is also valid for reentrant wave propagation. In this case, the wave does not originate from a focal source but instead is self-maintained by following a closed circuit. To account for the periodicity of the propagation pattern and avoid phase unwrapping issue, a phase transformation D exp.i / is applied, where  is now normalized between 0 and 2. The transformed eikonal-diffusion equation reads [3]:   kcr k D 1 C Im r   Dr

(7)

The boundary condition n  Dr D 0 still holds and the constraint j j D 1 must be preserved. The star (*) denotes the complex conjugate and “Im” the imaginary part. A Newton-based finite element method can be applied to solve (7) as described in Jacquemet [4]. Examples in Cardiac Electrophysiology The eikonal approach is illustrated here in a triangular mesh (about 5,000 nodes) representing the atrial epicardium derived from magnetic resonance images of a patient. Fiber orientation (anisotropic properties) was obtained from anatomical and histological data. Propagation velocity was set to 100 cm/s (along fiber) and 50 cm/s (across fiber). 0 was placed near the anatomical location of the sinoatrial node. The diffusion coefficient D was set to 10 cm2 . The activation map (arrival times) computed using the eikonal-diffusion solver (iteration from the solution provided by the fast marching algorithm) is displayed in Fig. 1a. With tol D 1010 , 16 iterations were needed. Figure 1b shows a reentrant activation map corresponding to an arrhythmia called typical atrial flutter, simulated using the eikonal-diffusion solver extended for reentrant propagation. The reentrant pathway  was formed by a closed circuit connecting the two vena cava. Propagation velocity was reduced by 40 %. With tol D 1010 , 50 iterations were needed. The resulting period of reentry was 240 ms, a value within physiological range.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

asymptotic analysis and numerical simulations. J. Math. Biol. 28(2), 121–176 (1990) Huebner, K.H.H., Dewhirst, D.L., Smith, D.E., Byrom, T.G.: Finite Element Method. Wiley, New York (2001) Jacquemet, V.: An eikonal approach for the initiation of reentrant cardiac propagation in reaction-diffusion models. IEEE Trans. Biomed. Eng. 57(9), 2090–2098 (2010) Jacquemet, V.: An eikonal-diffusion solver and its application to the interpolation and the simulation of reentrant cardiac activations. Comput. Method Program Biomed. (2011). doi:10.1016/j.cmpb.2011.05.003 Keener, J.P., Sneyd, J.: Mathematical Physiology. Interdisciplinary Applied Mathematics, vol. 8, 2nd edn. Springer, New York (2001) Landau, L.D., Lifshitz, E.M.: The Classical Theory of Fields, 4th edn. Butterworth-Heinemann, Oxford (1975) Pernod, E., Sermesant, M., Konukoglu, E., Relan, J., Delingette, H., Ayache, N.: A multi-front eikonal model of cardiac electrophysiology for interactive simulation of radio-frequency ablation. Comput. Graph. 35(2), 431–440 (2011) Qian, J., Zhang, Y.T., Zhao, H.K.: Fast sweeping methods for eikonal equations on triangular meshes. SIAM J. Numer. Anal. 45(1), 83–107 (2007) Sethian, J.A.: Level Set Methods and Fast Marching Methods. Cambridge University Press, Cambridge (1999) Sethian, J.A., Vladimirsky, A.: Fast methods for the eikonal and related Hamilton–Jacobi equations on unstructured meshes. Proc. Natl. Acad. Sci. 97(11), 5699–5703 (2011) Tomlinson, K.A., Hunter, P.J., Pullan, A.J.: A finite element method for an eikonal equation model of myocardial excitation wavefront propagation. SIAM J. Appl. Math. 63(1), 324–350 (2002)

Elastodynamics Adrian J. Lew Mechanical Engineering, Stanford University, Stanford, CA, USA

Description

Elastodynamics refers to the study of the motion of a continuum made of an elastic material. Elastic behavior References is the dominant component of the response of many 1. Franzone, P.C., Guerri, L., Rovida, S.: Wave-front propaga- solid objects to mechanical loads. A continuum made tion in an activation model of the anisotropic cardiac tissue – of an elastic material is a model for such type of solid

E

398

objects. Common materials that display elastic behavior under some conditions, generally small deformations, include glasses, most metals, many composites, rubber, and many geological materials. When an elastic continuum is locally excited, elastic waves are radiated that travel away from the point of excitation. Intuitively, the basic phenomena in elastodynamics involve small regions of the continuum exerting forces on neighboring ones and accelerating and hence deforming them as a result. Thus, the speed at which elastic waves travel is determined by the mass density of the continuum and by the relation between force (stress) and deformation (strain). Elastodynamics finds its range of applications in problems involving highly transient phenomena. After a mechanical load is imposed on an object, elastic waves are emitted and travel through it, reflecting from its boundaries. After many transit times of the waves through the object, static equilibrium is attained, generally attributed to the presence of dissipative mechanisms. Therefore, wave propagation need only be considered during a short time after the imposed load changes. Early interests in elastodynamics stemmed from applications in geophysics and seismology. Elastic waves traveling through the Earth’s crust find applications in earthquake and nuclear explosion monitoring and analysis, quarrying, and oil and gas exploration. The most popular engineering application of elastodynamics is in ultrasonics, involving low-energy waves, used for imaging (including medical imaging) and for non-destructive evaluation of structures and devices. High-energy waves find applications in high-speed machinery and metal-forming processes, and, of course, diverse military applications related to the response of structures to impacts and blast loads. Elastic waves are substantially more complex than electromagnetic or acoustic waves. Even in the most common and simplest case of an isotropic material, two types of waves with different wave speeds are found: the faster pressure or volumetric waves and the shear waves. When a wave of one type finds a boundary or interface, it generally spawns reflected or refracted waves of the other type. Anisotropy can introduce up to three different wave speeds for each direction of propagation. The coordinated interaction of volumetric or shear waves at boundaries or interfaces adds up to engender

Elastodynamics

other types of waves, which propagate with their own effective wave speeds. Along a free surface, the precise superposition of volumetric and shear waves to satisfy the traction-free condition on the surface generates Rayleigh waves or Love waves. Stoneley waves appear along a material interface as a result of the continuity of tractions and displacement. In the presence of confined geometries, such as plates, beams, or rods, the more complicated boundary conditions give rise to a host of interesting phenomena. Lamb waves, for example, are waves that propagate in the direction parallel to a plate’s surface but are standing waves along its thickness. Elastodynamics is a classical subject. We refer the reader to the books by Achenbach [2] and Graff [5] for an extensive introduction to wave motion in elastic solids as well as for a brief historical account. A mathematical description of elasticity, including some important aspects of elastodynamics, can be found in Marsden and Hughes [7]. For mathematical aspects of elastostatics instead, prime references are either this last reference or the book by Ciarlet [4]. The first chapter of Marsden and Hughes [7] is particularly useful as an overview of the essential features of elasticity. For an engineering perspective, see Holzapfel [6]. The introduction of Ciarlet [4] contains a rather extensive account of classical expositions.

The Elastic Continuum The description of an elastic continuum begins with the selection of a reference configuration, an open set 2 R3 . The reference configuration serves the important functions of: 1. Labeling particles in the continuum: the position of particles after deformation is described through the deformation or configuration 3 X 7! '.X / 2 R3 , or equivalently, through the displacement field u.X / D '.X /  X (see Fig. 1). 2. Indicating neighborhood among particles: strains are computed as F D rX ' or, in Cartesian coordinates, FiJ D @'i =@XJ , a second-order tensor field known as the deformation gradient. A deformation should be admissible, namely, it should be (i) injective to prevent interpenetration, (ii) orientation preserving, and (iii) sufficiently smooth, to have derivatives defined almost everywhere (e.g., W 1;1 . /) and avoid fracture. As a result, detF > 0 almost

Elastodynamics

399

ϕ(Ω,t1) ϕ(X,t1) u(X,t1)

W .F / ! C1 as det F ! 0C

(2)

to prevent the material from reaching arbitrarily large compressions. As an example, consider the stored energy function of a compressible neo-Hookean material

X u(X,t2) Ω

ϕ(Ω,t2)

decomposition theorem that W can only depend on the symmetric part of F or on the right Cauchy-Green deformation tensor C D F T F . An often adopted second condition on W is that

ϕ(X,t2)

Elastodynamics, Fig. 1 Sketch of a reference configuration for an elastic continuum and two deformations at times t1 and t2 of a motion '. The displacement vectors for a generic point X 2 at the two times are also shown

W .F / D f .det F / C

 trace.F T F /; 2

(3)

where the real-valued function f is such that f .J / ! C1 as J ! 0C ; for example, f .J / D  ln.det F /2 =2   ln.det F /, with material constants  > 0 and  > 0. For elastodynamics, the mass density of the material is important. The mass density per unit volume in the reference configuration is denoted with 0 .X /. Therefore, the choice of the reference configuration defines both W and 0 , and both should change accordingly if the reference configuration is changed.

everywhere. A time-dependent family of admissible deformations '.X; t/ is a motion. Evidently, there is freedom in the choice of the reference configuration. When possible, it is customary to choose it so that the elastic continuum is stress-free when u  0. The continuum is made of an elastic material if the first Piola-Kirchhoff stress tensor P at each point The Equations of Elastodynamics X 2 is a function of the deformation gradient F only, P .X; t/ D PO .F .X; t//, where PO W R3 ! R3 The initial value nonlinear elastodynamics problem in is the constitutive relation. The more widely known the time interval Œ0; T is Cauchy stress tensor is computed from P and F as @2 '  D .det F /1 PF T . An elastic material is hyperelastic (4a) 0 2 D divX P C 0 B in  .0; T / @t if there exists a real-valued stored energy function W .F / defined whenever det F > 0 such that PO .F / D P  N D T on @  .0; T /; (4b) @W=@F . The potential energy stored in an elastic continuum due to its deformation follows as ' D ' on @d  .0; T / (4c) Z W .r'/ d : (1) Ielastic Œ' D ' D ' 0 on  f0g; (4d) The dependence of W on F should be such that, upon rigidly rotating the material, the value of W should not change, since the material is not further strained as a result. More precisely, for each value of F such that det F > 0, W .X; F / D W .X; QF / for all Q 2 SO.3/. This is part of the principle of material frame indifference; see 19 in Truesdell and Noll [9] for a discussion. It then follows from the polar

@' D v0 @t

on  f0g:

(4e)

Equation (4a) is the statement of Newton’s second law per unit volume in the reference configuration, or the balance of linear momentum. To avoid possible ambiguities, the divergence of the stress tensor is computed as .divX P /i D PiJ;J for i D 1; 2; 3 (Hereafter, only

E

400

Elastodynamics

components in a Cartesian basis will be used, and the summation convention will be adopted: (i) an index repeated twice in a term indicates sum from 1 to 3 over it (ii) for a function a.X /, a;i D @a=@Xi ). This term is the net force per unit volume exerted on the “particle” at X by neighboring “particles.” The last term contains the body force per unit mass BW  .0; T / ! R3 . For example, gravitational force near the Earth would produce a B.X; t/ D g, where g is the acceleration of gravity, and position-dependent body forces such as those arising from an electrostatic field would have the form B.X; t/ D b.'.X; t/; t/ for some function b. Dirichlet boundary conditions are imposed on @d  @ in (4c), and the imposed values of the deformation are specified with 'W @d  .0; T / ! R3 . Forces are imposed on @  D @ n @d in (4b), and the imposed tractions are specified with T W @˝  .0; T / ! R3 . Both the initial deformation and the initial velocity field, ' 0 ; v 0 W ! R3 , are prescribed in (4d) and (4e), respectively. More generally, it is also possible to have mixed boundary conditions at the same point of the boundary, at which mutually orthogonal components of the deformation and the imposed tractions are prescribed, for example, in smooth sliding contact situations. Hamilton’s Principle For a hyperelastic materials and conservative body forces, the equations of elastodynamics can be obtained from Hamilton’s principle (see, e.g., 5 in Marsden and Hughes [7]). To this end, consider the Lagrangian density over  .0; T /: L.'; v; F / D

1 0 v 2  W .F /  0 V .'/; 2

where V W R ! R is the potential energy per unit mass for the body force, so B D rV . Hamilton’s principle then seeks a motion ' over Œ0; T satisfying (4c) that is the stationary point of the action

  @' .X; t/; rX '.X; t/ d dt L '.X; t/; @t .0;T / Z C T .X; t/  '.X; t/ dS dt (6) @ .0;T /

ˇ d ˇ S Œ' C ı' ˇ D 0; D0 d

(7)

for all variations ı' that satisfy ı' D 0 on  f0; T g [ @ d  .0; T /. If L and ' are smooth enough (e.g., C 2 . /), then (7) implies the Euler-Lagrange equations: @L  divX 0D @'



@L @F

 

@ @L @t @v

in  .0; T / (8a)

0DT C

@L N @F

on @   .0; T /; (8b)

where the arguments of the derivatives of the La grangian density are '.X; t/; @' .X; t/; r '.X; t/ . X @t After replacing with (5), these equations are precisely (4a) and (4b). Conservation properties, such as energy, and linear and angular momentum could be obtained from Noether’s theorem. By accounting for the potential location of discontinuities in the derivatives of ' in  .0; T /, the jump conditions across a shock are obtained. Some of the differential geometry aspects of this formulation can be consulted in 5 of Marsden and Hughes [7].

(5) Linear Elastodynamics

3

S Œ' D Z

among all variations that leave the value of ' fixed on the Dirichlet boundary and at times 0 and T; namely,

The nonlinear elastodynamics problem (4) is very complex, and few exact solutions or even regularity or existence results have been obtained. Insight into the wave propagation phenomena has therefore been obtained by analyzing the linearized problem instead. This problem is also attractive because it is a good model for every elastodynamics problem in which the displacement and displacement gradients are sufficiently small. The equations of linear elastodynamics describe to first order the dynamics resulting from small perturbations of the initial conditions or the forcing of a motion ' s that satisfies (4). Often ' s is simply a static solution, namely, 'R s D 0. For simplicity, we will only

Elastodynamics

401

  discuss the linearization around a stress-free reference Aij kl D ıij ıkl C  ıi k ıj l C ıi l ıj k ; configuration (' s .X; t/ D X ) (see 4 of Marsden Pij D uk;k ıij C .ui;j C uj;i /; (12) and Hughes [7] for a general case). Under these conditions, the linear elastodynamics equations stated in terms of the components of the displacement field which involve only the two Lam´e constants  > 0 u are and  > 0. The latter is the shear modulus, and the two can be used to obtain the Young modulus E D 0 uR i D .Aij kl uk;l /;j C 0 Bi in  .0; T / (9a) .3 C 2/=. C /. Finally, for an isotropic linear elastic material, (9a) is Aij kl uk;l Nj D T i

on @  .0; T /;

E

(9b) 0 uR i D uk;ki C .ui;jj C uj;j i / C 0 Bi :

ui D ' i  Xi WD ui

on @d  .0; T /

(13)

(9c)

It should be mentioned that when linearization is per(9d) formed at a stressed configuration, the elastic moduli generally lose the minor symmetries and depend on @ui the spatial position; it is effectively a continuum with a 0 D vi on  f0g; (9e) different linear elastic material at each point and a body @t force that reflects the stresses at ' s . for each i D 1; 2; 3. (In the following, uP D @u=@t, uR D @2 u=@t 2 for a function u.X; t/.) The fourth-order A Word About Solutions tensor 2 Under some mild assumptions, the solution of the @W .I / (10) homogeneous linear elastodynamics problem (9) (B D Aij kl D @Fij @Fkl 0, T D 0, and u D 0) exists and satisfies u 2 is the elastic moduli at the stress-free reference con- C 0 .Œ0; T ; H 1 . // and uP 2 C 0 .Œ0; T ; L2 . // (see 6 figuration, i.e., when F is the identity I . The linear in Marsden and Hughes [7]). This result holds if is elastodynamics equations follow directly from (4) by compact and has a smooth boundary, u0 2 H 1 . /, adopting the linearized stored energy density v 0 2 L2 . /, 0 > 0, and most importantly, the moduli Aij kl are strongly elliptic, namely, there exists  > 0 1 such that Wlinear .F / D .Fij  ıij /Aij kl .Fkl  ıkl / 2 1 Aij kl i k j l  2 i i j j ; (14) D Aij kl ui;j uk;l ; (11) 2 ui D 'i0  Xi WD u0i

on  f0g;

where ıij are the components of I . The first PiolaKirchhoff stress tensor is then Pij D Aij kl uk;l , and ij D Pij to first order in rX u. If W is twice continuously differentiable at I , the second derivatives in (10) commute and the moduli have the major symmetry Aij kl D Aklij . Minor symmetries Aij kl D Aj i kl follow from the material frame indifference of W. The linearized stored energy density is not material frame indifferent, but due to the minor symmetries, it is invariant under infinitesimal rigid rotations, namely, skew-symmetric displacement gradients. The symmetries of Aij kl leave a maximum of 21 different material constants. For isotropic materials, the elastic moduli and the stress are

for all ;  2 R3 . The elastic moduli in (12) trivially satisfy this condition. Solutions can be arbitrarily smooth, even C 1 , provided the initial and boundary conditions are themselves smooth and satisfy some compatibility conditions. These conditions, as well as explicit results for the nonhomogeneous, stressed case and spatially varying mass density and elastic moduli, follow from the general results for first-order hyperbolic systems in Rauch and Massey [8] and are briefly discussed in 6 of Marsden and Hughes [7]. The general problem of existence, uniqueness, and regularity of solutions for nonlinear elastodynamics equations (4) is still open. A relatively recent discussion of existing results was provided by Ball [3].

402

Elastodynamics

p-wave

d

p

reference configuration

s-wave

d p

Elastodynamics, Fig. 2 Deformation of the reference configuration (middle) by a plane progressive p-wave (left) and s-wave (right). The wave has the form d expŒ.p  x  ct /2 , and the

corresponding vectors p and d are shown. Both waves are moving towards the top of the page. The coloring was used solely to visualize the deformation

Linear Elastic Waves Equation (9a), which states the balance of linear momentum for linear elastodynamics, admits solutions in free space ( D R3 ) in the form of plane progressive waves when B D 0. These solutions illuminate how waves propagate in linear elastic solids. A plane progressive wave is a displacement field that has the form

Elastodynamics, Table 1 Typical representative wave speeds for some common materials (from Achenbach [2]). Liquids do not display elastic shear waves

u.x; t/ D d .p  x  ct/

(15)

where d 2 R3 is the polarization vector, p 2 R3 is a unit vector that defines the direction of propagation of the wave, c > 0 is the speed of propagation, and 2 C 2 .R/ is the shape of the wave. For a plane progressive wave to propagate in a linear elastic material, it should satisfy (9a), which happens if and only if .Aij kl pj pl /dk D ƒi k dk D 0 c 2 di :

(16)

Therefore, the only plane progressive waves that can propagate have d as an eigenvector of the acoustic tensor ƒ and 0 c 2 as an eigenvalue. If Aij kl is strongly elliptic, cf. (14), then ƒ is a symmetric and positive definite matrix. It has then three mutually orthogonal polarization vectors and real and positive eigenvalues and hence real wave speeds. In the particular case of

Material

0 [kg/m3 ]

cL [m/s]

cT [m/s]

 D cL =cT

Air Water Steel Copper Aluminum Glass Rubber

1.2 1,000 7,800 8,900 2,700 2,500 930

340 1,480 5,900 4,600 6,300 5,800 1,040

3,200 2,300 3,100 3,400 27

1.845 2 2.03 1.707 38.5

isotropic materials, c.f. (12), there are two types of plane progressive waves. The first p ones have p D d and propagate at a speed cL D . C 2/=0 . These waves, in which the polarization vector is parallel to the direction of propagation, are called longitudinal or p-waves (p for primary, or pressure). The second type ofp waves has d ? p and propagates at a speed cT D =0 . The polarization vector is any vector in the plane orthogonal to the direction of propagation of the wave, and hence, these are called transverse or shear waves, also known as s-waves (s for secondary, or shear). Snapshots of how a continuum is deformed under the action of each one of these waves are shown in Fig. 2. Typical representative wave speeds for some standard materials are shown in Table 1.

Elastodynamics

403

Helmholtz Decomposition and Free-Space Solutions In isotropic materials, p-waves are particular cases of irrotational, volumetric, dilatational, or pressure waves, and s-waves are particular cases of rotational or isochoric waves. This is because any displacement field u that satisfies (9a) for an isotropic linear elastic material can be decomposed as u D rˆ C r  ‰

(17)

for a scalar potential ˆW ! R and a vector potential ‰W ! R3 with r  ‰ D 0 that satisfy the wave equations 0D

1 R ˆ  ˆ C Bˆ cL2

(18a)

0D

1 R ‰  ‰ C B‰ ; cT2

(18b)

where the same decomposition was adopted for the body force B D cL2 rBˆ C cT2 r  B‰ . The decomposition (17) is known as the Helmholtz decomposition [2] or the Hodge decomposition [1]. The vector potential ‰ represents waves that travel at a speed cT , induce shear deformations, and are isochoric, since rr‰ D 0. On the other hand, the vector potential ˆ represents the waves that travel at speed cL , induce changes in the mass density of the material, and are irrotational, because r  rˆ D 0. Since the pressure is computed from (12) as p D Pi i =3 D . C 2/ˆ, (18a) also

governs the propagation of pressure waves. The two types of waves are completely decoupled in free space. A number of elementary solutions follow directly from this perspective, such as radiation of elastic waves from point or line loads in an infinite medium [2, 5].

The Role of Boundary and Interfaces The elegant and convenient decoupling of the longitudinal and transverse waves in (18) is lost when waves find boundaries or interfaces between materials of different elastic properties. This is why elastic waves are more complex than acoustic or electromagnetic waves. Upon finding an interface, transverse waves may spawn refracted and reflected longitudinal waves and conversely. The appearance of these new waves is needed to satisfy the conditions at the interface. For example, if the interface is a traction-free boundary with normal n, then it should happen that P  n D 0 therein at all times. An incident longitudinal wave at an arbitrary angle will generally not satisfy this condition, so new waves need to appear to do so. The particular case of a p-wave incident on a traction-free boundary is discussed below, to showcase how the boundary conditions play a role in spawning waves of a different type. For the more general case, see Achenbach [2] and Graff [5]. The essential phenomena are typically showcased with a harmonic plane progressive wave

un D d n cosŒkn .p n  x  c n t/

(19)

x2

d0

p0 x1

θ0

θ1 θ2

d1 p1 d2 p2

Elastodynamics, Fig. 3 A longitudinal (transverse) wave incident on a boundary or interface may spawn both a reflected longitudinal (transverse) wave and a transverse (longitudinal) reflected wave. The wave diagram on the left shows the direction (p) and polarization (d ) vectors for the p-wave incident on the

free-surface x2 D 0 and the two reflected waves. A possible deformation induced by three waves on an elastic continuum whose reference configuration is a regularly gridded rectangle is shown on the right

E

404

Elastodynamics

traveling through an isotropic linear elastic half-space x2 < 0 and encountering a flat boundary x2 D 0 (e.g., [2, 5]). Here kn > 0 is the wave number; .x1 ; x2 ; x3 / are the Cartesian coordinates of a point. The index n D 0 corresponds to the incident wave. Without loss of generality, it is assumed that the direction of propagation p 0 lies on the x1 -x2 -plane, so p 0 D sin 0 e1 C cos 0 e2 for 0 2 Œ0; =2 where fe1 ; e2 ; e3 g is the Cartesian basis see Fig. 3. A longitudinal incident wave will have d 0 D A0 p 0 , A0 > 0, and c 0 D cL . It is also convenient to identify two independent families of transverse waves: the SV-waves in which d 0 lies in the x1 -x2 -plane and the SH-waves in which d 0 lies along the x3 direction. Both families of waves have c 0 D cT and d 0 ? p0. The traction at the boundary for the incident p-wave is 8 k1 D k0 ; ˆ ˆ ˆ ˆ < 1 D 0 ; ˆ ˆ ˆ sin 2 0 sin 2 2   2 cos2 2 2 ˆ : A1 D A0 ; sin 2 0 sin 2 2 C  2 cos2 2 2 Notable cases are (a) normal incidence ( 0 D 0) or grazing incidence ( 0 D =2), for which there is no reflected S V -wave, and (b) angle of incidence such that there is no reflected p-wave, or A1 D 0, but there is a reflected S V -wave with A2 D  cot 2 0 . Similar results are found for incident S V -waves or for different boundary conditions on the surface. Incident SH -waves, on the other hand, only spawn reflected SH -waves for typical boundary conditions. For many of these cases, the relation between the incident and reflected waves can be gracefully depicted through slowness diagrams (see [2]). Surface Waves A mechanical excitation in the bulk of the elastic continuum, such as an underground detonation or an earthquake, excites both volumetric and shear waves. These waves decay in amplitude as they travel away from the source, since the energy of the perturbation decays at least with the square of the distance to the source. When these waves encounter an interface, they may excite surface waves. Surface waves can travel along the surface, and their amplitude decays exponentially away from the surface, so they are also

0 D A0 k0  sin.2 0 / sin.k0 x1 sin 0  k0 cL t/ P12   0 P22 D A0 k0 C 2 cos2 0 sin.k0 x1 sin 0  k0 cL t/ 0 P32 D 0: 0 0 D P22 D 0 for It follows from here that if P12 all x1 and t, then necessarily A0 D 0. Considering the superposition of the incident wave with a p-wave reflected from the boundary also leads to trivial solutions. Only by additionally including a reflected S V wave is it possible to obtain nontrivial ones. Therefore, the displacement field induced by the harmonic plane wave incident on the boundary is u0 C u1 C u2 , where n D 1 labels the reflected p-wave and n D 2 labels the reflected S V -wave. These are defined with (see [2]) p n D sin n e1  cos n e2 for n D 1; 2, d1 D A1 p 1 , d 2 D A2 e3 p 2 D A2 .cos 2 e1 Csin 2 e2 /, c 1 D cL , c 2 D cT , and

k2 D k0 cL =cT D ko ; sin 2 D  1 sin 0 ; A2 D

2 sin 2 0 cos 2 2 A0 : sin 2 0 sin 2 2 C  2 cos2 2 2

known as evanescent waves. For example, Rayleigh waves have the form u D .d1 e1 C d2 e2 /e bx2 cosŒk.x1  ct/

(20)

in the coordinate system in Fig. 3, for b > 0. Because these waves are localized to a small region near the surface, their energy decay, only inversely with the distance to the source. Consequently, far away from the source, the disturbance carried by the surface waves will be the dominant one. This is why these waves are of interest in seismology and nondestructive evaluation. The speed of propagation of surface waves is generally different than that of volumetric or shear waves. For example, the speed c and decay rate b of Rayleigh waves are obtained after replacing (20) in (9a) with B D 0 and looking for nontrivial solutions that satisfy the traction-free condition on the surface (see [2]). It follows from there that c < cT < cL . Horizontally polarized or SH -surface waves can also appear under some circumstances and receive the name of Love waves. If instead of a free surface there is a material interface, then disturbances confined to

Elastography, Applications Using MRI Technology

405

a neighborhood of the interface may also propagate; Short Definition these are called Stoneley waves. Elasticity imaging is a rather recent non invasive imagWave Guides ing modality which provides in vivo data about the Thin structures, such as plates or rods, which are viscoelastic properties of tissue. With manual palpation much larger along one direction than the others, give being an integral part of many diagnostic procedures, rise to waves that are the product of a standing wave it is obvious that elasticity imaging has many interacross the thickness, due to the boundary conditions, esting and promising potentials in medical imaging, and traveling waves along the structure. Because the i.e., from lesion/tissue detection and characterization wave, and hence the energy, travels along the structure, to therapy follow-up. The general concept of this these are also called wave guides. In contrast with the method is to displace the material mechanically and waves from previous sections, the speed of these waves infer from displacement measurements the intrinsic depends on the wave number; it is a dispersive medium. local viscoelastic properties. Many different techniThese types of waves in solid plates receive the name cal realizations exist (static, dynamic, transient) utiof Lamb waves. lizing different imaging modalities (MRI, ultrasound) which all probe different frequency domains. Since References viscoelastic properties of tissue change strongly with frequency, care must be taken when interpreting the 1. Abraham, R., Marsden, J., Raiu, T.: Manifolds, Tensor Anal- data in terms of elastic and viscous component. Here, ysis, and Applications, vol. 75. Springer, New York (1988) we will focus on the dynamic 3D approach via MRI, 2. Achenbach, J.: Wave Propagation in Elastic Solids. North Holland Series in Applied Mathematics and Mechanics, i.e., a mono-frequent mechanical excitation and a volvol. 16. North-Holland, Amsterdam (1973) umetric assessment of the displacement field. This 3. Ball, J.: Some open problems in elasticity. In: Geometry, allows overcoming several physical difficulties: Firstly Mechanics, and Dynamics, pp. 3–59. Springer, New York compressional waves can be properly suppressed via (2002) 4. Ciarlet, P.: Mathematical Elasticity: Three-Dimensional Elas- the application of the curl operator, secondly wavegticity, vol. 1. North Holland, Amsterdam (1988) uide effects are eliminated, and finally the calculation 5. Graff, K.: Wave Motion in Elastic Solids. Dover, New York of the complex shear modulus does not necessitate (1991) 6. Holzapfel, G.: Nonlinear Solid Mechanics. Wiley, Chich- any assumption of the underlying rheological model. Clinical results on liver fibrosis and on breast cancer ester/New York (2000) 7. Marsden, J., Hughes, T.: Mathematical Foundations of are discussed. Elasticity. Dover, New York (1994) 8. Rauch, J., Massey, F.: Differentiability of solutions to hyperbolic initial-boundary value problems. Trans. Am. Math. Soc. 189(197), 303–318 (1974) 9. Truesdell, C., Noll, W.: The Non-linear Field Theories of Mechanics, 3rd edn. Springer, Berlin/New York (2004)

Introduction

From the dawn of time, manual palpation has been recognized as an important part in many diagnostic Elastography, Applications Using MRI procedures since pathological changes are often acTechnology companied with changes in the stiffness of tissue. Palpation has already been mentioned by Hippocrates Ralph Sinkus and its importance is unbroken until nowadays. The CRB3, Centre de Recherches Biom´edicales aim of palpation is to deduce information about the Bichat-Beaujon, Hˆopital Beaujon, Clichy, France internal mechanical properties of soft tissue by applying an external force. Thus, when considering the linear regime of small deformations and weak forces, Synonyms we intend to measure the material parameter which links stress to strain. In other words, given a certain Aspartate to Platelets Ratio Index (APRI); Hepatocel- force (i.e., the locally imposed stress), can we predict lularcarcinoma (HCC); Magnetic Resonance Elastog- the resulting deformation (i.e., the local strain)? In raphy (MRE); Magnetic Resonance Imaging (MRI) general, any deformation (ignoring flow effects) can

E

406

be decomposed into a pure compressional component and a pure shear component [4]. Consequently, an isotropic material can mechanically be characterized in the linear regime by a compressional modulus and a shear modulus. Soft tissue consists approximately 70 % of water rendering it mechanically incompressible. However, water is extremely soft in terms of shear. It is therefore obvious that we need to investigate the shear stress-strain relationship of tissue in order to probe its mechanical integrity for disease characterization. The compressional stress-strain relationship reflects mainly the properties of water which are far less sensitive to alterations of the solid matrix. In this context it is important to realize that when a doctor palpates, i.e., when he or she exerts a stress on the tissue, the generated deformation will be pure shear due to the incompressible nature of tissue. Consequently, the common saying “this tumour is less compressible” is misleading, and it should be corrected to “the mechanical shear properties of this tumour are elevated compared to the surrounding tissue.” The qualitative aspect of manual palpation was brought to a quantitative imaging technique via the pioneering work of Ophir [9]. Here, ultrasound speckle tracking methods were used to image the strain field which was created by an externally applied static force. This development triggered a whole new ultrasound research area with many exciting applications. The conceptual drawback of this method is the lack of any force information, i.e., while the strain is locally measured via ultrasound, the local stress is not accessible with this approach. Hence, the calculation of the shear modulus as a local intrinsic property of tissue is only possible under very specific assumptions (like plane strain or constant stress) which are rarely met in reality. Dynamic elastography was developed in order to overcome the lack of missing local stress information using now a sinusoidally changing stress source at a fixed frequency (typically of the order of 10–100 Hz) [5]. Thereby, mono-chromatic mechanical waves propagate through the organ of interest which can be visualized via motion-sensitive imaging methods. Dealing with monochromatic waves has the big advantage of controlling time: hence, acceleration and thereby force, i.e., local stress and strain information, can be calculated from time series of wave images (see details below). Lewa was the first one proposing motion-sensitized MRI sequences for the detection of propagating mechanical waves in tissue [6] cumulating shortly later in a Science publication for the

Elastography, Applications Using MRI Technology

visualization of propagating shear waves via MRI [8]. This triggered the formation of a steadily growing MR elastography community with different approaches and different methods for reconstructing the shear modulus [1, 10, 12, 14]. Many in vivo applications have in the meantime been developed for different organs (breast [7, 12, 13], brain [15], liver [2, 3, 11]) with focus on diffuse diseases (like, fibrosis, demyelination) or focal lesions. The MR-based approach has the advantage of volumetric data acquisition with equal motion sensitivity to all spatial directions. Its disadvantage compared to the ultrasound-based approach is certainly the lack of real-time capability. This significant advantage of ultrasound is balanced by the difficulties of obtaining volumetric data, reduced SNR, and very different motion sensitivity for the three spatial directions. We will recall the basics of shear modulus reconstruction in case of dynamic MR elastography and highlight its fundamental difference to shear wave speed based methods. Clinical results on liver fibrosis and breast cancer are presented. Theory of Mechanical Wave Propagation The propagation of a monochromatic mechanical wave in a linear isotropic viscoelastic material is given by     ! 2 ui D @xk G  @xk ui C @xi  C G  @xk uk ; „ ƒ‚ … „ ƒ‚ … „ ƒ‚ … force-term

shear-term

mixing/compressional-term

(1) with  the density of the material, ! the circular frequency, and ui the i ’s component of the 3D displacement vector uE.  refers to the compressional modulus, G  refers to the shear modulus, and Einstein convention is assumed for identical indices. Since we intend to apply this equation to tissue and use vibration frequencies of the order of 100 Hz, it is important to keep several physical conditions in mind: • Tissue is almost incompressible. Thus,  is of the order of GPa which leads to a speed of sound of approximately 1,550 m/s in tissue almost independent of the frequency. • Shear waves are slow (1–10 m/s) which causes G  being of the order of kPa, i.e., 6 orders of magnitude smaller than the compressional modulus . • The term @xk uk represents the relative volume change and is consequently of very small magnitude in tissue [4]. • Longitudinal waves are not attenuated when operating at frequencies of the order of 100 Hz. Therefore,

Elastography, Applications Using MRI Technology

407

a monochromatic experiment at different frequen is real-valued. Contrarily, shear waves are attencies; uated in this frequency range, and therefore, G  is • The calculation of G  is based upon spatial derivacomplex-valued. tives of the measured displacement fields and NOT The small magnitude of the @xk uk -term often leads on the measurement of the wave speed. Waveguide to the wrong assumption that it might be possible to effects therefore do not affect the correctness of simply ignore the entire second term on the right hand Eq. 4. It is importanthq to keep side of Eq. 1. This is however not correct. As can be i in mind that the shear G seen, when considering the limit of an incompressible is a composite variable wave speed cs D <  material (i.e., Poisson’s ratio  approaches 0.5), the which depends on the real and imaginary part of G  small magnitude of the @xk uk -term is balanced by the as well as on geometrical effects. It is therefore only large magnitude of the compressional modulus (which justified under very specific conditions to assume  2 approaches infinity in case of incompressibility) leadthat Gd D  app  D  cs2 holds (with app the ing to a so-called finite pressure term, i.e., apparent local shear wavelength). From these considerations, it is obvious that an un!1 biased reconstruction of G  necessitates volumetric @xk uk ! 0 acquisition (in order to correctly calculate the D (2) data p  @xk uk D finite r 2 -term) and at least two of the three motion comwhich is nonzero and finite even when assuming the ponents (in order to get at least one component of qr D "rsi @xs ui ). Nevertheless, it is advised to acquire material to be incompressible. all three motion components because Eq. 4 becomes Often, the local spatial derivatives of the material numerically ill-defined once hitting zero-crossings for properties are ignored which leads to the following one of the components of qE . In those cases the other simplified equation: two components, which are unlikely to also have a zero-crossing at the same spatial location, can be used ! 2 ui D G  r 2 ui C @xi p : (3) to properly calculate G  . An obvious drawback of the curl-based reconstruction method is the necessity of In order to eliminate the unknown pressure component, taking 3rd-order spatial derivatives which requires high we apply the curl operator "rsi @xs to both sides of Eq. 3 values of SNR. This requirement is met because MR leading to a simple Helmholtz-type equation data acquisition time is of the order of several minutes providing high-quality data. 2  2 ! qr D G r qr ; qr D "rsi @xs ui : (4) This equation can be solved analytically at each point within the imaging volume because qE and r 2 qE can be obtained from MR elastography data (see below), for the density  a value corresponding to water is assumed, and the vibration frequency ! is known from the experimental conditions. It is important to realize several consequences of Eq. 4: • The involved derivatives are local: thus, boundary conditions of the experiment do not invalidate the correctness of the equation even so they certainly influence qE and r 2 qE , • Equation 4 represents the complex-valued local stress-strain relationship for a monochromatic shear wave. The obtained values for G  D Gd C iGl (with Gd the so-called dynamic modulus and Gl the so-called loss modulus) are therefore independent of any rheological model, and its frequency dependence can be obtained by repeating or multiplexing

MR Elastography Data Acquisition and Reconstruction The details of the MR elastography (MRE) data acquisition have been described elsewhere [12]. In short, a mechanical transducer is placed at the surface of the object under investigation and vibrates sinusoidal at typical frequencies of 100 Hz. A motion-sensitized MRI sequence is applied in synchrony (phase-locked) to the mechanical vibration, thereby providing snapshots of the propagating waves at different time points during the oscillatory cycle. This specific method allows the measurement of all three components of the displacement vector with equal precision within a volume. Contrary to ultrasound, MRI is not capable to acquire the displacement field in real time. The finite sampling provides in return broadband sensitivity which can be used to accelerate or improve data acquisition.

E

408

Elastography, Applications Using MRI Technology

In general, it is obvious from Eq. 4 that finding G  necessitates the assessment of uE.x; E t/ at various times of the oscillatory cycle in order to calculate its real and imaginary part via Fourier transformation. Then, Eq. 4 can be rewritten to

! 2

qi< qi=



D



 r 2 qi< r 2 qi= Gd  ; r 2 qi= r 2 qi< Gl i 2 Œ1; 2; 3 :

(5)

Consequently, a reconstruction algorithm ignoring the complex-valued nature of the displacement field is prone to provide biased values for G  . There are in general two different approaches to obtain local maps for the complex shear modulus: • The direct approach, i.e., solving Eq. 5 locally with more or less assumptions in order to simplify things [12] • The indirect approach [14], i.e., simulating the expected displacement field within a small region of interest based upon an initial guess for the distribution of the viscoelastic parameters and updating the guess iteratively via 2 -minimization between simulated and measured displacement data Both approaches have pros and cons: the direct method is certainly more sensitive to noise while the indirect is sensitive to boundary conditions. An additional challenge in case of the indirect method is the proper calculation of the @xk uk -term in Eq. 1. Its true value in tissue is so minute (due to the quasi incompressible nature) that an estimation of the compressional modulus is easily off by several orders of magnitude (it should be of the order of GPa to yield the correct speed of sound in tissue of 1,550 m/s). MRE Applied to Stage Liver Fibrosis and Characterize Breast Cancer Chronic liver diseases typically lead to liver fibrosis. Recent investigations demonstrate that liver fibrosis is reversible using effective treatment during the early phase of disease progression. In this context, the stage of liver fibrosis plays a major role: it determines firstly the treatment options and secondly also the prognosis. The current gold standard for determining the stage of liver fibrosis is the biopsy. As an invasive procedure, it is, for instance, not well suited for treatment followup studies, which is however mandatory in order to separate in the early phase responders from nonrespon-

ders. Moreover, needle biopsy is probing only a tiny quasi 1D volume of the entire liver and is thus prone to sampling variability and interobserver variation in the interpretation of the semiquantitative scoring systems. Thus, there is a need for noninvasive alternatives to liver biopsy which should at least be capable to reliably differentiate between three stages of fibrosis: none/early, intermediate, and advanced/cirrhotic. An identification of the intermediate stage is necessary since patients with hepatitises B and C and nonalcoholic liver disease should be treated. Late-stage patients require, for instance, follow-up studies regarding potential hepatocellular carcinomas. Various noninvasive methods have been proposed to assess the stage of liver fibrosis. These methods include liver imaging methods via MRI or ultrasound and biochemical scores. The most common score is the so-called aspartate to platelet ratio index (APRI). Although those techniques certainly carry diagnostic value, their accuracy for staging intermediate fibrosis remains debated. From clinical experience it is well known that liver stiffness changes with the grade of fibrosis. Here, MRE as a novel noninvasive method for measuring the viscoelastic properties of the liver may play an important role. Preliminary reports [2, 3, 11] suggest that MRE is a feasible method to stage liver fibrosis. Figure 1 shows a selected example of a patient with already substantially developed fibrosis (stage F3 as confirmed by histology) in combination with a large hepatocellularcarcinoma (HCC). Very good wave penetration can be observed, henceforth allowing an exploration of the mechanical parameters over the entire right liver lobe. The HCC can clearly be differentiated from the surrounding parenchyma as an area with significantly enhanced elasticity values. The parenchyma with an average value of G d D 3:3 kPa appears – as expected – significantly enhanced when compared to normal liver tissue with values around 2 kPa. More recent clinical results clearly demonstrate that MRE can separate those three stages of liver fibrosis. A large comparative study (MRE, FibroScan, and APRI) demonstrated the superiority of MRE over the other two methods (see Fig. 2) and the ability to efficiently separate between low-grade fibrosis (F0–F1) and intermediate- to high-grade fibrosis (F2–F4) [3]. Obviously 3D MRE outperforms the 1D Fibroscan approach which is an expected result after all theoretical considerations discussed before. Figure 2d shows the

Elastography, Applications Using MRI Technology

409

Elastography, Applications Using MRI Technology, Fig. 1 Liver fibrosis & tumor example. This example shows a patient with a large hepatocellularcarcinoma (a, red ROI) in combination with an advanced fibrosis (grade F3). The mechanical transducer is attached from the ide (blue rectangle) Wave penetration Gd [kPa] 7

is very good (b, actually the z-component of the curl is shown) and the resulting image of Gd (c, in units of [kPa]) clearly shows the presence of the lesion. Enhanced elasticity values for the liver parenchyma are recorded .G d D 3:3 kPa/ compared to normal liver tissue (2 kPa)

E=3Gd [kPa] 80

a

APRI index 8

c

b

6

FIBROSCAN Ultrasound based

60

MRE 5 4

APRI

6

40

4

20

2

3 2 0

1 0

1

2

3

METAVIR Score

0

4

0

α 0.2

1

2

3

4

0

METAVIR Score

1

2

3

4

METAVIR Score

d

F0 F1 F2 F3 F4

0.1

∼1/Gd 0.0 120

160

200

Elastography, Applications Using MRI Technology, Fig. 2 Comparison of different methods for liver fibrosis staging. The “true” grade of liver fibrosis has been determined via liver biopsy and transformed into the METAVIR score. A comparison of MRE (a), FibroScan (b) and APRI test (c) on 141 patients

shows the superiority of the 3D MRE approach [3]. When assuming power-law behavior for the complex shear modulus, Gd and Gl can be reinterpreted into the previously mentioned structural parameter ˛. Obviously, disease progression changes the stiffness but does not change ˛ (d)

   Gl 2 ratio of viscosity over elasticity ˛ D atan  Gd for different grades of fibrosis as a function of a parameter which is proportional to 1=Gd . Obviously, disease progression is from low to high values of Gd

(i.e., the tissue stiffens, as seen in the Fig. 2a) while ˛ does not change. Thus, viscosity and elasticity are increasing in synchrony. This is not what is observed in breast cancer. Figure 3a shows the experimental setup for in vivo breast measurements. The patient is

E

410

Elastography, Applications Using MRI Technology 10

a

b

c

8 6 4 2 0

d

8

0,40

7

0,35

e

6

0,35

5 4 3

Histology benign malignant

α

0,25 0,20 0,15

2 0,10 1 0

0,05 0

50

100

150

200

250

250

∼1/Gd Elastography, Applications Using MRI Technology, Fig. 3 Results on Breast Cancer Characterization. Schematic of the experimental setup (a). The transducer is attached from the side and generates mechanical waves which traverse the breast. MR subtraction image in case of a ductal invasive carcinoma

(b). Map of Gd [kPa] (c) and map of Gl [kPa] (d). When assuming power-law behavior for the complex shear modulus, Gd and Gl can be reinterpreted into the previously mentioned structural parameter ˛ [13]. Malignant lesion obviously differs significantly from benign lesion when considering ˛

in prone position with the transducer attached from the side. This setting allows first performing standard MR mammography with contrast agent and applying afterwards MRE as an add-on. A selected example for a palpable invasive ductal carcinoma is presented in Fig. 3b. The subtraction image (with/without contrast agent) shows clearly the presence of a large strongly enhancing tumor. Figure 3c,d show the corresponding images of the dynamic modulus and the loss modulus. Interestingly, the large tumor is barely visible in the map of Gd but well circumscribed in the image of Gl . When arranging the measurements of a large group of benign and malignant lesions similar to Fig. 2d (see Fig. 3d), it becomes obvious that malignant tumors populate the high ˛ – low 1=Gd region. The difference to the data from liver fibrosis might be explained due to the fact that no strong neovasculature is installed during the development from low-grade to high-grade fibrosis which is clearly the case for malignant breast cancer.

Those very encouraging results should however be taken with a grain of salt: other pathological effects can equally enhance the stiffness of the liver (like inflammation). Thus, viscoelastic parameters are certainly a very interesting biomarker for the characterization of fibrosis. However, enhanced stiffness of the liver is not ONLY created by fibrotic effects, and MRE should rather be seen as one valuable additional physical parameter within the portfolio of diagnostic liver MRI or MR mammography. Discussion and Future Directions Many MRE research groups are currently exploring the further potential diagnostic value of the viscoelastic parameters for disease characterization in the areas of breast [7, 13], human brain [15], preclinical tumor characterization and liver tumors characterization. Overall, viscoelastic parameters are sensitive to architectural changes of tissue and seem to provide valuable additional clinical information for tissue

Electrical Circuits

characterization. As always, many side effects can influence the mechanical parameters (inflammatory effects, steatotic effects, fibrotic effects), and it is therefore most beneficial that MRE can be part of a broad spectrum of physical parameters which can be assessed during one MR examination. More fundamental oriented investigations try to infer from the scattering processes of shear waves micro-architectural information about the underlying medium. This information might be valuable for the characterization of the efficacy of antiangiogenic treatments at early stages of the therapy. Hence, MRE might become more than a sophisticated in vivo rheometer: it might turn into a tool for understanding microscopic structural properties.

411 11. Rouviere, O., Yin, M., Dresner, M.A., Rossman, P.J., Burgart, L.J., Fidler, J.L., et al.: MR elastography of the liver: preliminary results. Radiology 240(2), 440–448 (2006) 12. Sinkus, R., Lorenzen, J., Schrader, D., Lorenzen, M., Dargatz, M., Holz, D.: High-resolution tensor MR elastography for breast tumour detection. Phys. Med. Biol. 45(6), 1649– 1664 (2000) 13. Sinkus, R., Siegmann, K., Xydeas, T., Tanter, M., Claussen, C., Fink, M.: MR elastography of breast lesions: understanding the solid/liquid duality can improve the specificity of contrast-enhanced MR mammography. Magn. Reson. Med. 58(6), 1135–1144 (2007) 14. Van Houten, E.E., Paulsen, K.D., Miga, M.I., Kennedy, F.E., Weaver, J.B.: An overlapping subzone technique for MRbased elastic property reconstruction. Magn. Reson. Med. 42(4), 779–786 (1999) 15. Wuerfel, J., Paul, F., Beierbach, B., Hamhaber, U., Klatt, D., Papazoou, S., Zipp, F., Martus, P., Braun, J., Sack, I.: MR-elastography reveals degradation of tissue integrity in multiple sclerosis. Neuroimage 49(3), 2520–2525 (2010)

References 1. Chenevert, T.L., Skovoroda, A.R., O’Donnel, M., Emelianov, S.Y.: Elasticity reconstructive imaging by means of stimulated echo MRI. MRM 39, 482–490 (1998) 2. Huwart, L., Peeters, F., Sinkus, R., Annet, L., Salameh, N., ter Beek, L.C., et al.: Liver fibrosis: non-invasive assessment with MR elastography. NMR Biomed. 19(2), 173–179 (2006) 3. Huwart, L., Sempoux, C., Vicaut, E., Salameh, N., Annet, L., Danse, E., Peeters, F., ter Beek, L.C., Rahier, J., Sinkus, R., Horsmans, Y., Van Beers, B.E.: Magnetic resonance elastography for the noninvasive staging of liver fibrosis. Gastroenterol. 135(1), 32–40 (2008) 4. Landau, L., Lifschitz, E.: Theory of Elasticity, 3 edn. Butterworth-Heinemann, Oxford (1986) 5. Lerner, R.M., Huang, S.R., Parker, K.J.: Sonoelasticity images derived from ultrasound signals in mechanically vibrated tissues. Ultrasound Med. Biol. 16(3), 231–239 (1990) 6. Lewa, G.J.: Elastic properties imaging by periodical displacement NMR measurements (EPMRI). In: Proceedings of the Ultrasonics Symposium IEEE, 1–4 Nov 1994, Cannes, vol. 2, pp. 691–694 (1994) 7. McKnight, A.L., Kugel, J.L., Rossman, P.J., Manduca, A., Hartmann, L.C., Ehman, R.L.: MR elastography of breast cancer: preliminary results. AJR Am. J. Roentgenol. 178(6), 1411–1417 (2002) 8. Muthupillai, R., Lomas, D., Rossman, P., Greenleaf, J., Manduca, A., Ehman, R.: Magnetic resonance elastography by direct visualization of propagating acoustic strain waves. Science 26(29), 1854–1857 (1995) 9. Ophir, J., Cespedes, I., Ponnekanti, H., Yazdi, Y., Li, X.: Elastography: a quantitative method for imaging the elasticity of biological tissues. Ultrason. Imaging 13(2), 111–134 (1991) 10. Plewes, D.B., Betty, I., Urchuk, S.N., Soutar, I.: Visualizing tissue compliance with MR imaging. J. Magn. Reson. Imaging 5(6), 733–738 (1995)

Electrical Circuits Michael G¨unther Fachbereich Mathematik und Naturwissenschaften, Bergische Universit¨at Wuppertal, Wuppertal, Germany

Definition An electrical circuit is composed of electrical components, which are connected by current-carrying wires or cables. Using the network approach, such a real circuit is mathematically modeled by an electrical network consisting of basic elements (resistors, capacitors, inductors, and current and voltage sources) and electrically ideal nodes.

Description Such a network model of an electrical circuit is automatically generated in computer-aided electronicsdesign systems. An input processor translates a network description of the circuit into a netlist. The network equations are generated from the netlist by combining network topology with basic physical laws like energy or charge conservation and characteristic

E

412

Electrical Circuits

equations for the network elements. Usually, this automatic modeling approach tries to preserve the topological structure of the network and does not look for systems with a minimal set of unknowns. As a result, coupled systems of implicit differential and nonlinear equations, shortly, differential-algebraic equations (DAEs), are generated, which have to be simulated numerically. In the following, we will shortly discuss the three steps involved in numerical simulation of electrical circuits: modelling, analysis, and numerical integration. For a detailed discussion of this topic, we refer to the handbook article [9] and survey papers [7, 8].

Besides these purely topological relations, additional equations are needed for the subsystems to fix the state variables uniquely. These so-called characteristic equations describe the physical behavior of the network elements. One-port or two-terminal elements are described by equations relating their branch current and branch voltage. The characteristic equations for the basic elements resistor, inductor, and capacitor are derived by field theoretical arguments from Maxwell’s equations assuming quasistationary behavior. In doing so, one abstracts on Ohmic losses for a resistor, on generation of magnetic fluxes for an inductor, and on charge storage for a capacitor, by neglecting all other effects. The set of basic elements is completed by independent, that is purely time-dependent, current and voltage sources. Interconnections and semiconductor devices are modeled by companion circuits using multi-ports, which contain voltage-controlled charge and currentcontrolled flux sources to model dynamical behavior. Voltage-controlled current sources are used to describe the static current in pn-junctions and channels.

Modeling: The Network Approach In contrast to a field theoretical description based on Maxwell’s equations, which is not feasible due to the large complexity of integrated electric circuits, the network approach rests on integral quantities – the three spatial dimensions of the circuit are only considered by the network topology. The time behavior of the system is given by the network quantities, branch currents I.t/ 2 IRnI , branch voltages U.t/ 2 IRnI , and node voltages u.t/ 2 IRnu that describe the voltage drop of the nodes versus the ground node. Modified Nodal Analysis The electrical network is now fully described by both Kirchhoff’s laws and the characteristic equations. Principles and Basic Equations The network model consists of elements and nodes, Based on these relations, the method of modified and the latter are assumed to be electrically ideal. nodal analysis (MNA) is commonly used in industrial The composition of basic elements is governed by applications to generate the network equations: Kirchhoff’s laws which can be derived by applying KCL (1) is applied to each node except ground, and Maxwell’s equations in the stationary case to the net- the branch currents of all voltage-controlled elements are replaced by their current-defining characteristic work topology. equations. The element equations for all current-, Kirchhoff’s Current Law (KCL). The algebraic sum of charge- and flux-controlled elements like voltage currents traversing each cut set of the network must be sources and inductors are added. Finally, all branch equal to zero at every instant of time: voltages are converted into node voltages with the help of KVL (2). Splitting the incidence matrix A into the element related incidence matrices AC , AL , AR , AV , A  I.t/ D 0; (1) and AI for charge- and flux-storing elements, resistors, and voltage and current sources, one obtains from with a reduced incidence matrix A 2 f1; 0; 1gnunI , MNA the network equations in charge/flux-oriented which describes the branch-nodes connections of the formulation: network graph. Kirchhoff’s Voltage Law (KVL). The algebraic sum of voltages along each loop of the network must be equal to zero at every instant of time: A>  u.t/ D U.t/:

(2)

AC qP C AR r.A> R u; t/ C AL |L C AV |V C AI {.A> u; q; P |L ; |V ; t/ D 0;

P 

A> Lu

(3a)

D 0; (3b)

> A> P |L ; |V ; t/ D 0; V u  v.A u; q;

(3c)

Electrical Circuits

413

parasitics may make things worse and lead to problems q  qC .A> C u; t/ D 0; (3d) which are more ill-posed.

 L .|L ; t/ D 0 (3e) with node voltages u, branch currents through voltageand flux-controlled elements |V and |L , voltagedependent charges and fluxes q and , voltagedependent resistors r, voltage and current-dependent charge and flux functions qC and L , and controlled current and voltage sources { and v. At this point the reader may pose the question why charges and fluxes are introduced to model characteristic equations of energy-storing elements in a charge/flux-oriented way – and not classically via capacitors and inductors. The answer to this question contains modelling, physical, numerical, and software technical argument, as discussed in detail in G¨unther and Feldmann [7]. Why DAE and Not ODE Models? The charge/flux-oriented formulation of energy-storing elements and MNA network equations supply us with a first argument for using differential-algebraic equations in electrical circuit modeling. More arguments are revealed by inspecting the ODE approach as an alternative:

Generating a State-Space Model with a Minimal Set of Unknowns. Drawbacks of this approach include software engineering, modeling, numerical, and designeroriented arguments. The state-space form cannot be generated in an automatic way and may exist only locally. The use of independent subsystem modeling, which is essential for the performance of today’s VLSI circuits, is limited, and the advantage of sparse matrices in the linear algebra part is lost. Finally, the topological information of the system is hidden for the designer, with state variables losing their technical interpretation. Regularizing the DAE to an ODE Model by Including Parasitic Effects. It is commonly believed that the DAE character of the network equations is only caused by a high level of abstraction, based on simplifying modeling assumptions and neglection of parasitic effects. So one proposal is to regularize a DAE into an ODE model by including parasitic effects. However, this will yield singularly perturbed problems, which will not be preferable to DAE models in numerical respect. Furthermore, refined models obtained by including

Analysis: The DAE Index of Network Equations So we are faced with network equations of differentialalgebraic type when simulating electrical circuits. Before attacking them numerically, we have to reveal the analytical link between network topology and DAE index.

E Network Topology and DAE Index In the linear case, the two-terminal elements capacitor, inductor, and resistor are linear functions of the respective branch voltage with positive scalars defining the capacitance, inductance, and resistance. In other words, the elements are strictly passive. Generalizing this property to the nonlinear case, the strictly local passivity of nonlinear capacitors, inductors, and resistors corresponds to the positive definiteness (but not necessarily symmetry) of the so-called generalized capacitance, inductance, and conductance matrices

@qC .w; t/ ; @w

@ L .w; t/ ; @w

and

@r.w; t/ : @w

If this property of positive-definiteness holds, the network is called an RLC network. Let us first investigate RLC networks with independent voltage and current sources. To obtain the perturbation index of (3), we perturb the right-hand side of (3a–3c) by a slight perturbation ı D .ıC ; ıL ; ıV /> . The corresponding solution of the perturbed system is denoted by x ı WD .uı ; |Lı ; |Vı /> . Then one can show that the difference x ı  x between perturbed and unperturbed solution is bounded by the estimate  kx .t/  x.t/k  C  kx ı .0/  x.0/k C max kık ı

 2Œ0;t

C max

 2Œ0;t

> P kQCRV ıC k

C max kQN V>C ıPV k



 2Œ0;t

with a constant C and using orthogonal projectors QC , > QCRV , and QN V C onto ker A> C , ker .AC AR AV / , and > AC D 0 holds, the index ker QC> AV [5, 9]. Since QCRV does not rise, if also perturbations q and are allowed in the charge- and flux-defining equations (3d–3e). Thus the index of the network equations is one, if the following two topological conditions hold:

414

T1: There are no loops of only charge sources (capacitors) and voltage sources (no VC loops): ker QC> AV D f0g. T2: There are no cut sets of flux sources (inductors) and/or current sources (no LI cut sets): ker.AC AR AV /> D f0g: In this case, we deal with well-posed problems. However, we have to cope with ill-posed index-2 problems, if T1 or T2 is violated. The results hold also for RLC networks with a rather large class of nonlinear voltage and current sources, as shown by Est´evez Schwarz and Tischendorf [5]: The index depends only on the topology; in general, the index is one, and two only for special circuit configurations. Influence of General Sources Independent charge and flux sources, which may model ˛-radiation or external magnetic fields, can destroy the positive definiteness of generalized capacitance and inductance matrices: The index may now depend also on modeling parameters and operation conditions of nonlinear elements [8]. The same effects can be generated by controlled sources. Higher-index cases may arise, for example, if independent current/voltage sources in index-2 configurations are replaced by charge/flux sources or index-2 problems are coupled via controlled sources. Even if the network contains no nonlinear sources, circuit parameters may have an impact on structural properties of the network equations such as the DAE index. These results have an important practical consequence, since it does not allow to rely only on structural aspects when trying to cope with higher-index problems in circuit simulation. After deriving and analyzing the analytical properties of the DAE network equations in time domain, the third step remains to be discussed in numerical circuit simulation: numerical integration using DAE discretization schemes tailored to structure and index of the network equations.

Numerical Time Integration In the following we describe the conventional approach based on implicit linear multi-step methods and discuss the basic algorithms used and how they are implemented and tailored to the needs of circuit simulation. Special care is demanded of index-2 systems.

Electrical Circuits

Throughout this chapter we will assume that the network equations correspond to RLC networks, and the only allowed controlled sources are those which keep the index between 1 and 2, depending on the network structure. To simplify notation, we first rewrite the network equations (3) in charge/flux-oriented formulation in a more compact linear-implicit form: 0 D F.y.t/; P x.t/; t/ WD A  y.t/ P C f .x.t/; t/;

(4a)

0 D y.t/  g.x.t//;

(4b)

with x WD .u; |L ; |V /> being the vector of unknown network variables and y WD .q; /> . The Basic Algorithm The conventional approach can be split into three main steps: (a) computation of consistent initial values, (b) numerical integration of yP based on multi step schemes, and (c) transformation of the DAE into a nonlinear system and its numerical solution by Newton’s procedure. Since the third step is usually performed with methods which are not very specific for circuit simulation, we will not discuss it further here. Let us assume for the moment that the network equations are of index 1 – the index-2 case will be discussed later. (a) Consistent Initial Values. The first step in the transient analysis is to compute consistent initial values .x0 ; y0 / for the initial time point t0 . In contrast to performing a steady state (DC operating point) analysis, i.e., to solve F.0; x0 ; t0 / D 0 for x0 and then set y0 WD g.x0 /, one can extract the algebraic constraints using the projector QC onto ker A> C:

QC> .AR r.A> R u; t/ C AL |L C AV |V CAI {.u; |L ; |V ; t// D 0; (5a) v.u; |L ; |V ; t/  A> V u D 0: (5b) If the index-1 topological conditions hold, this nonlinear system uniquely defines for t D t0 the algebraic components QC u0 and |V;0 for given (arbitrary) differential components .I QC /u0 and |L;0 . The derivatives yP0 have then to be chosen such that AyP0 C f .x0 ; t0 / D 0 holds. (b) Numerical Integration. Starting from consistent initial values, the solution of the network equations

Electrical Circuits

415

is computed at discrete time points t1 ; t2 ; : : : ; by numerical integration with implicit linear multistep formulas: For a timestep hk from tk1 to tk D tk1 C hk , the derivative y.t P k / in (4b) is replaced by a linear -step operator k for the approximate yPk , which is defined by k D ˛k g.xk / C rk with a timestep-depending coefficient ˛k . The remainder rk contains values of y and yP for  previous time points. This direct approach was first proposed by Gear [6] for backward differentiation formulas (BDF methods). Since SPICE2 [11], most circuit simulators solve the network equations either with the trapezoidal rule (TR) Today, a combination of TR and BDF schemes, so-called TR-BDF schemes, are widely used. This name was first introduced in a paper by Bank et al. [1]. The aim is to combine the advantages of both methods: large timesteps and no loss of energy of the trapezoidal rule (TR) combined with the damping properties of BDF. (c) Transformation into a Nonlinear System of Equations. The numerical solution of the DAE system (4b) is thus reduced to the solution of a system of nonlinear equations F .˛k g.xk / C rk ; xk ; tk / D 0;

(6)

which is solved iteratively for xk by applying Newtons’s method in a predictor-corrector scheme. .0/ Starting with a predictor step xk (xk1 or some kind of extrapolated value from previous time point may be a reasonable choice), a new Newton .l/ .l/ .l1/ correction xk WD xk  xk is computed from a system of linear equations .l/

DF .l1/ xk D F .l1/ ; .l1/

F .l1/ WD F .˛k g.xk

.l1/

/ C rk ; xk

; tk /: (7)

Due to the structure of the nonlinear equations, the Jacobian DF .l1/ is given by .l1/

DF .l1/ D ˛k  FxP

C Fx.l1/ with

.l1/

.l1/ FxP

@g.xk D A @x .l1/

Fx.l1/ D

/

;

@f .xk ; tk / : @x

If the stepsize h is sufficiently small, the regularity of DF .l1/ follows from the regularity of the matrix pencil fA  @g.x/=@x; @f =@xg that is given at least for index-1 systems. Element Stamps and Cheap Jacobian In every Newton step (7), two main steps have to be performed: • LOAD part: First, the right-hand side F .l1/ of (7) and the Jacobian DF .l1/ have to be computed. • SOLVE part: The arising linear system is solved directly by sparse LU decomposition and forward/backward substitution. A characteristic feature of the implementation in circuit simulation packages such as SPICE is that modeling and numerical integration are interwoven in the LOAD part: First, the arrays for right-hand side and Jacobian are zeroed. In a second step, these arrays are assembled by adding the contributions to F and DF element by element: So-called element stamps are used to evaluate the time-discretized models for all basic elements. Adaptivity: Stepsize Selection and Error Control Variable integration stepsizes are mandatory in circuit simulation since activity varies strongly over time. A first idea for timestep control is based on estimating the local truncation error "yP of the next step to be performed, that is, the residual of the implicit linear multistep formulas if the exact solution is inserted. The main flaw of controlling "yP is that the user has no direct control on the really interesting circuit variables, that is, node potentials u and branch currents |L ; |V . An approach to overcome this disadvantage [12] is based on the idea to transform the local truncation error "yP for qP and P into a (cheap) estimate for the local error "x WD x.tk /  xk of x.t/. By expanding F .y.t/; P x.t/; t/ at the actual time point tk into a Taylor series around the approximate solution .yPk ; xk / and neglecting higher-order terms, one obtains an error estimate "x for x.tk /, which can be computed from the linear system



@f @g ˛k A C @x @x

 "x D A"yP

(8)

of which the coefficient matrix is the Jacobian of Newton’s procedure! Since the local error "x can be interpreted as a linear perturbation of x.tk /, if F is perturbed with the local truncation error "yP , the choice

E

416

Electrical Circuits

of "x is justified as an error estimate for numerical currents in VC loops and node voltages in LI cut integration. The key motivation to weight the local sets. truncation error via Newton’s method was to damp (b) Fixing the Weak Instability. For a variable-order, the impact of the stiff components on timestep control variable-stepsize BDF scheme, M¨arz and Tischen– which otherwise would yield very small timesteps. dorf [10] have shown that if the ratio of two While this aspect can be found in the textbooks, a succeeding stepsizes is bounded and the defect second aspect comes from the framework of chargeık , representing the perturbations in the kth step caused by the rounding errors and the defects oriented circuit simulation: Newton’s matrix brings arising when solving the nonlinear equations nusystem behavior into account of timestep control, such merically, is small enough, then the BDF approach as mapping integration errors of single variables onto is feasible and convergent for index-2 network those network variables, which are of particular interest equations. However, a weakly instable term of for the user. the type 1 The Index-2 Case max kDk ık k k0 h k Since most applications of practical interest yield network equations of index 2, numerical integration must arises in the error estimate of the global error. be enabled to cope with the network equations (3) that Here Dk denotes a projector that filters out the are not of Hessenberg type. Fortunately, the fine struchigher-index components of the defect. In contrast ture of the network equations allows for adapting BDF to index-2 systems of Hessenberg type, where an schemes to such systems, provided that (a) consistent appropriate error scaling is a remedy, this instabilinitial values are available and (b) a weak instability ity may affect all solution components in our case associated with an index-2 non-Hessenberg system is and may cause trouble for the timestep and error fixed. control. However, the instability can be fixed by (a) Computing Consistent Initial Values. The usual reducing the most dangerous part of the defect ık , way in circuit simulation to compute initial values that is, those parts belonging to the range of Dk . described above for index-1 problems may yield This defect correction can be done by generalizing inconsistent initial values in the index-2 case, since the back propagation technique, since the projector the hidden constraints – relating parts of the solucan be computed very cheaply by pure graphical tion to the time derivatives of the time-dependent means with the use of an index monitor. elements – are not observed. A solution to this problem was developed by Est´evez Schwarz [4], New Challenges which aims at being as near as possible to the In the last decades, numerical simulation of electrisolution of the standard algorithm for low index: cal circuits has reached some level of maturity: The In a first step, a linear system is setup and solved link between modeling and analytical properties of for corrections to this solution such that the hidden network models is well understood and successfully constraints are fulfilled; in a second step, these exploited by numerical integration schemes tailored corrections are added to the initial values found in to the special structure of network equations. Besides the standard algorithm to get consistent ones. The robustness, efficiency has been drastically improved hidden constraints can be easily derived from the by parallelization, domain decomposition (subcircuit information provided by an index monitor, develpartitioning), and multirate schemes. However, the oped by Est´evez Schwarz and Tischendorf [5]: It ongoing miniaturization of integrated circuits increases determines the index, identifies critical parts of the the complexity of circuits (both qualitatively and quancircuit and invokes special treatment for them in titatively) and defines future direction for research: order to avoid failures of the numerical integration, and gives hints to the user how to regularize the Quantitative Challenge: A Need for More Speedup [3]. problem in case of trouble and which network The LOAD part becomes more expensive for refined variables may be given initial values and which MOSFET models. One idea to speed up is to load must not. When the algorithm is applicable, then devices in parallel by multi-threaded stamping, which the variables to be corrected turn out to be branch has to avoid access conflicts. The SOLVE part becomes

Electromagnetics-Maxwell Equations

the bottleneck if too many parasitics fill up the Jacobian DF . The implemented direct solvers (mainly sparse LU decomposition) have to be replaced by better (multi-threaded) ones. To speed up the Newton loop, the number of matrix evaluations and decompositions has to be reduced and, at the same time, one has to avoid costly noncovergence. Some ideas to speed up time integration are to exploit more multirate techniques and use multi-threading for stepsize control and explicit schemes for as many steps as possible. In addition, there is hope to gain additional speedup by model order reduction techniques (replacing linear parasitic elements as well as nonlinear subcircuits by a reduced net with same input-output behavior) and GPU computing. However, to be successful here demands for substantial progress in nonlinear MOR and a paradigm shift in algorithm development.

417 2. Bartel, A., Pulch, R.: A concept for classification of partial differential algebraic equations in nanoelectronics. In: Bonilla, L.L., Moscoso, M., Platero, G., Vega, J.M. (eds.) Progress in Industrial Mathematics at ECMI 2006, Mathematics in Industry, vol. 12, pp. 506–511. Springer, Berlin (2007) 3. Denk, G. (2011) Personal communication. 4. Est´evez Schwarz, D.: Consistent initialization for differential-algebraic equations and its application to circuit simulation. Ph.D. thesis, Humboldt Universit¨at zu Berlin, Berlin (2000) 5. Est´evez Schwarz, D., Tischendorf, C.: Structural analysis for electrical circuits and consequences for MNA. Int. J. Circuit Theory Appl. 28, 131–162 (2000) 6. Gear, C.W.: Simultaneous numerical solution of differentialalgebraic equations. IEEE Trans. Circuit Theory CT-18, 89– 95 (1971) 7. G¨unther, M., Feldmann, U.: CAD based electric circuit modeling in industry I. Mathematical structure and index of network equations. Surv. Math. Ind. 8, 97–129 (1999) 8. G¨unther, M., Feldmann, U.: CAD based electric circuit modeling in industry II. Impact of circuit configurations and parameters. Surv. Math. Ind. 8, 131–157 (1999) 9. G¨unther, M., Feldmann, U., ter Maten, E.J.W.: Modelling and discretization of circuit problems. In: Schilders, W.H.A., ter Maten, E.J.W. (eds.) Handbook of Numerical Analysis. Special Volume Numerical Analysis of Electromagnetism, pp. 523–659. Elsevier/North Holland, Amsterdam (2005) 10. M¨arz, R., Tischendorf, C.: Recent results in solving index 2 different algebraic equations in circuit simulation. SIAM J. Sci. Comput. 18(1), 139–159 (1997) 11. Nagel, W.: SPICE 2 – A Computer Program to Simulate Semiconductor Circuits. MEMO ERL-M 520/University of California, Berkeley (1975) 12. Sieber, E.-R., Feldmann, U., Schultz, R., Wriedt, H.: Timestep control for charge conserving integration in circuit simulation. In: Bank, R.E., et al. (eds) Mathematical Modelling and Simulation of Electrical Circuits and Semiconductor Devices, pp. 103–113. Birkh¨auser, Basel (1994)

Qualitative Challenge: Refined Network Modelling [2]. In the network approach, all spatially distributed components are modeled by subcircuits of lumped basic elements. With ongoing miniaturization, these companion models lack physical meaning and become less and less manageable. One alternative modeling approach is to replace the companion model by a physically oriented PDE model, which bypasses a huge number of more or less artifical parameters of the companion model. This refined network modeling yields coupled systems of DAEs and PDEs, PDAEs for short, with a special type of coupling: The node potentials at the boundaries define boundary conditions for the PDE model, and the currents defined by the PDE model enter the network equations as additional current source terms. This PDAE modeling approach can be extended to multiphysical problems, for example, coupling the circuit behavior with thermal effects, but may yield different coupling structures. Simulating these PDAE models numerically involves again the whole simu- Electromagnetics-Maxwell Equations lation chain: modeling, analysis (well-posedness and sensitivity), and numerical approximation. Leszek F. Demkowicz Institute for Computational Engineering and Sciences (ICES), The University of Texas at Austin, Austin, TX, USA

References 1. Bank, R.E., Coughran, W.M., Fichtner, W., Grosse, E.H., Rose, D., Smith, R.K.: Transient simulation of silicon devices and circuits. IEEE Trans. Comput. Aided Des. CAD 4, 436–451 (1985)

Mathematics Subject Classification 65N30; 35L15

E

418

Electromagnetics-Maxwell Equations

Synonyms

0 < min  .x/  max < 1;

H.curl/-conforming elements; Edge elements

0 < min  .x/  max < 1; 0  .x/  max < 1 :

Short Definition Finite element method is a discretization method for Maxwell equations. Developed originally for elliptic problems, finite elements must deal with a different energy setting and linear dependence of Maxwell equations.

In more general versions, ; ;  can be replaced with tensors, possibly complex valued. In practice, they will also depend upon the (angular) frequency !. The load data include impressed (electric volume) current J imp and impressed (volume) charge imp that satisfy themselves the continuity equation: j!imp C r  J imp D 0 :

Description Maxwell Equations Maxwell equations (Heaviside’s formulation) include equations of Amp`ere(with Maxwell’s correction) and Faraday, Gauss’ electric and magnetic laws, and conservation of charge equation. In this entry we restrict ourselves to the time-harmonic version of the Maxwell equations which can be obtained by Fourier transforming the transient Maxwell equations or, equivalently, using e j!t ansatz in time. 8 r  E D j!.H / ˆ ˆ ˆ ˆ ˆ Faraday law ˆ ˆ ˆ ˆ ˆ imp ˆ ˆ r H D J C E C j!.E/ ˆ ˆ ˆ ˆ ˆ Amp`ere-Maxwell law ˆ ˆ ˆ ˆ 0, and R.0/ D 0. d It quickly follows that dt .S C I C R/ D 0 which implies that N must be constant. Further, the introduction of a small number of infectious individuals, given that N is large, leads to the following reasonable approximation of the model dynamics (at the start of

.ˇ   /I [S.0/ N ]. Consethe outbreak): dI dt quently, I.t/ D e .ˇ /t I0 accounts for changes in the infectious class at the start of the outbreak (exponential growth or decay). This type of approximation (finding expressions that capture the dynamics generated by a small number of infectious individuals) is routinely used to asses, the potential for an epidemic outbreak. We conclude that if ˇ > 1 the disease will take off (an epidemic outbreak), while if

ˇ 

R0 = 1.7

0

0

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 s

Epidemiology Modeling, Fig. 2 The s-i phase diagram is plotted under two different values of R0 (R0 D 1:7; 3:4)

computing R0 , in settings that involve the interactions between heterogenous individuals or subpopulations, are important [28, 40, 41, 43, 47, 58–61, 96]. Since the population under consideration is constant, the state variables can be re-scaled (e.g., s D S=N ). Letting s, i , and r denote the fraction of susceptible, infectious, and recovered, respectively, leads to the following relationship (derived by dividing the second equation by the first in Model (1)) between the s and i proportions:  di D 1 C : ds ˇs

< 1 the disease will die

out. ˇ , known as the Basic Reproductive Number or R0 , defines a threshold that determines whether or not an outbreak will take place (crossing the line R0 D 1). R0 , a dimensionless quantity, is the product of the average infectious period .1= / (window of opportunity) times the average infectiousness (ˇ) of the members of the small initial population of infectious individuals (I0 ). ˇ measures the average per-capita contribution of the infectious individuals in generating secondary infectious, per unit of time, within a population of mostly susceptibles (S.0/ N ). R0 is most often defined as the average number of secondary infectious generated by a “typical” infectious individual after its introduction in a population of susceptibles [40, 58]. Computing R0 is central in most instances to the study of the dynamics and control of infectious diseases (but see [45]). Hence, efforts to develop methods for

1

(2)

Figure 2 displays the s-i phase diagram for two different values of R0 (R0 D 1:7; 3:4). For each value of R0 , three different initial conditions are used to simulate an outbreak and, in each case, the corresponding orbits are plotted. The parameter values are taken from Brauer and Castillo-Chavez [15]. A glance at Model (1) allows us to show that s.t/ is decreasing and that limt !0 s.t/ D s1 > 0. The integration of (2) leads to the relationship: ln

h i s0 D R0 1  s1 ; s1

(3)

where 1  s1 denotes the fraction of the population that recovered with permanent immunity. Equation (3) is referred to as the final epidemic size relation [15,63]. Estimates of the proportions s0 and 1  s1 can be

Epidemiology Modeling

431

Epidemiology Modeling, Table 2 Parameter definitions State variables

Definitions

Parameters

Definitions

S E I T N

Susceptible Exposed (asymptomatic and noninfectious) Infectious (active TB) Treated still partially susceptible Total population N D S C E C I C T

 ˇ , d k,  ˇ, 0    1 p, 0  p  1

Recruitment of new susceptible Transmission rate per susceptible and infectious Natural and disease-induced mortalities Per-capita progression and treatment rates Transmission rate per treated and infectious Susceptibility to reinfection

obtained from random serological studies conducted before and immediately after an epidemic outbreak. Independent estimates for the average infectious period (1= ) for many diseases are found in the literature. The use of priori and posteriori serological studies can be combined with independent estimates of the disease’s infectious period to estimate ˇ via (3) (see [64]). Efforts to develop methods for connecting models to epidemiological data and for estimating model parameters have accelerated, in part, as a result of the 2003 SARS outbreak [30]. Estimates of a disease’s basic reproduction number are now routinely computed directly from data [32–34,36,37,62]. Efforts to identify final epidemic size relations like those in (3) have received considerable attention over the past few years as well (see [7, 17] and references therein). Most recently estimates of the basic reproductive number for A-H1N1 influenza were carried out by modelers and public health researchers at Mexico’s Ministry of Health [35]. These estimates helped the Mexican government plan its initial response to this influenza pandemic. The value of these estimates turned out to be central in studies of the dynamics of pandemic influenza [62].

Backward Bifurcation: Epidemics When R0 < 1 The question of whether epidemic outbreaks are possible when R0 < 1 (backward bifurcation) has led to the study of models capable of sustaining multiple endemic states, under what appear to be paradoxical conditions. The study of hysteresis has received considerable attention in epidemiology particularly, after relevant theoretical results on mathematical models of infectious diseases appeared in the literatures [24, 57, 68]. The model for the transmission dynamics of tuberculosis (TB) provides an interesting introduction to the relevant and timely issue of hysteresis

behavior [48]. A brief introduction to the epidemiology of TB is outlined before the model (in Feng et al. [48]) is introduced. Tuberculosis’ causative agent is mycobacterium tuberculosis. This mycobacterium, carried by about one third of the world human population, lives most often within its host, on a latent state and, as a result, this mycobacterium often becomes dormant after infection. Most infected individuals mount effective immune responses after the initial “inoculation” [5, 6, 13, 79]. An effective immunological response most often limits the proliferation of the bacilli and, as a result, the agent is eliminated or encapsulated (latent) by the host’s immune system. Tuberculosis was one of the most deadly diseases in the eighteenth and nineteenth centuries. Today, however, only about eight million individuals develop active TB each year (three million deaths) in the world, a “small” fraction in a world, where about two billion individuals live with this mycobacterium [91]. Latently infected individuals (those carrying the disease in a “dormant” state) may increase their own re-activation rate as a result of continuous exposure to individuals with active TB (exogenous re-activation). The relevance of exogenous re-activation on the observed TB prevalence patterns at the population level is a source of debate [48, 90, 93]. The model in Feng et al. [48] was introduced to explore the role that a continuous exposure to this mycobacterium may have in accelerating the average population TB progression rates [21, 23, 48, 90, 93]. It was shown that exogenous re-activation had indeed the potential for supporting backward bifurcations [48]. In order to describe a TB model that supports multiple positive endemic states, we proceed to divide the host population in four epidemiological classes: susceptible, exposed (latently infected), infectious, and treated. The possible epidemiological transitions of individuals in this population are captured in the second diagram in Fig. 1, while the definitions of the parameters and state variables are collected in Table 2.

E

432

Epidemiology Modeling

The generation of new E-individuals per unit of time (E-incidence) comes from two subpopulations and therefore, it involves the terms: BS D ˇS NI and BT D ˇT NI . The generation of new active cases, the result of reinfection, is modeled by the term BE D pˇE NI . The definitions in Table 2 and the assumptions just described lead to the following model for the transmission dynamics of TB under exogenous reinfection: dS dt dE dt dI dt dT dt

D   BS .t/  S; D BS .t/  BE .t/  . C k/E C BT .t/; (4) D kE C BE .t/  . C  C d /I; D I  BT .t/  T:

Model (4) indeed allows for the possibility of exogenous reinfection but only when p > 0. The basic reproduction number can be computed using various methods [28, 40, 43] all leading to

R0 D



 k  ˇ : C Cd Ck

(5)

R0 is the number of E individuals “generated” from contacts between S and typical I -individuals (when ) during every body is susceptible, i.e., when S.0/   the critical window of opportunity, that is, over the average length of the infectious period, namely, CˇCd . R0 is computed by multiplying the average infectious k ) period times the proportion of latent individuals ( Ck that manage to reach the active TB-stage. R0 > 1 means that the average number of secondary active TB cases coming from the S -population is greater than one, while R0 < 1 corresponds to the situation when the average number of secondary active TB cases generated from the S population is less than one. In the absence of reinfection, one can show that if R0  1 then I.t/ decreases to zero as t ! 1 while if R0 > 1 then I.t/ ! I1 > 0. In the first case, the infection-freestate .=; 0; 0; 0/ is globally asymptotically stable, while in the latter there exists

a unique locally asymptotically stable endemic state .S  > 0; E  > 0; I  > 0; T  > 0/. The dynamics of Model (4) are therefore “ generic” and illustrated in Fig. 4 (bifurcation diagram and simulations). In the generic case, the elimination of the disease is feasible as long as the control measures put in place manage to alter the system parameters to the point that no TB outbreak is possible under the new (modified) parameters. In summary, if the model parameters jump from the region of parameter space where R0 > 1 to the region where R0 < 1 then the disease is likely to die out. In the presence of exogenous reinfection (p > 0) the outcomes may no longer be “generic.” It was established (in Feng et al. [48]) that whenever R0 < 1 there exits a p0 2 .0; 1/ and an interval Jp D .Rp ; 1/ with Rp > 0 (p > p0 ) with the property that whenever R0 2 Jp , exactly two endemic equilibria exist. Further, only one positive equilibrium is possible whenever Rp D R0 and no positive equilibria exists if Rp < R0 . The branch of endemic equilibria bifurcating “backward” from the disease-free equilibrium at R0 D 1 is shown in Fig. 3 (left). Figure 3 (right) illustrates the asymptomatic behaviors of solutions when p > p0 and Rp < R0 < 1 (p D 0:4 and R0 D 0:87). A forward bifurcation diagram of endemic steady states is also plotted in Fig. 4 (left). Figure 4 is generated from the model in the absence of exogenous reinfection (p D 0). Figure 4 (right) displays the asymptomatic behavior of solutions when R0 D 1:08 under various initial conditions. The parameter values were taken from Feng et al. [48]. For an extensive review of TB models, see Castillo-Chavez and Song [22]. The identification of mechanisms capable of supporting multiple endemic equilibria in epidemic models was initially carried out in the context of HIV dynamics by Huang et al. [24, 45, 68]. These researchers showed that asymmetric transmission rates between sexually active interacting populations could lead to backward bifurcations. Hadeler and Castillo-Chavez [57] showed that in sexually active populations, with a dynamic core, the use of prophylactics or the implementation of a partially effective vaccine could actually increase the size of the core group. Further, such increases in the effective size of the core may generate abrupt changes in disease levels, that is, the system may become suddenly capable of supporting

Epidemiology Modeling

433

100

1200 Stable endemic state

10

1000

−1

10−2

I

I

800

600

I0 = 50 I0 = 300

400 10−3

I0 = 750 I0 = 1100

200 10−4 0.4

0.6

0.8

1

1.2 R0

1.4

1.6

E

I0 = 900

Unstable endemic state

1.8

0

2

Epidemiology Modeling, Fig. 3 Backward bifurcation (p D 0:4): a bifurcation diagram of endemic steady states is displayed (left). The numbers of infectious individuals as functions of

0

50

100

150 200 250 Time (years)

300

350

400

time with various I0 are plotted when R0 D 0:87 (right). The outcomes (I1 > 0 or I1 D 0) of the simulation depend on the initial condition (value of I.0/) 100

101.6

I0 = 1 I0 = 10 I0 = 50 I0 = 70 I0 = 100

90 80 70

101.5

I

I

60 50 40

101.4

30 20 10

101.3 1

1.5

2

2.5

R0

0

0

1

2

3

4 5 6 Time (years)

7

8

9

10

Epidemiology Modeling, Fig. 4 Forward bifurcation (p D 0): a bifurcation diagram of endemic steady states is displayed (left). The numbers of infectious individuals as functions of time with various I0 are plotted when R0 D 1:08 (right)

multiple endemic states (backward bifurcation). In the next section, control measures that account for the cost of interventions in an optimal way are introduced in the context of the TB model discussed in this section.

social sciences. Recent contributions using optimal control approaches (from influenza to drinking) have generated insights on the value of investing on specific public health policies [55, 76–78]. Efforts to assess the relative effectiveness of intervention measures aimed at reducing the number of latently and actively TB infectious individuals at a minimal cost and over finite Optimal Control Approaches: The Cost time horizons can be found in the literature [69]. We of Epidemics highlight the use of optimal control in the context The use of optimal control in the context of contagion of Model (4). Three, yet to be determined, control models has a long history of applications in life and functions (policies): ui .t/ W i D 1; 2; 3 are introduced.

434

Epidemiology Modeling R0=0.879733

u1(t )

1

B1=100, B2=10, B3=10 B1=100, B2=10, B3=100

0.5 0

0

5

10

15

20

25

30

35

40

45

50

0

5

10

15

20

25

30

35

40

45

50

0

5

10

15

20

25 30 Time (years)

35

40

45

50

u2(t )

1

0.5 0

u3(t )

1

0.5

2.3

x 104

E(t)

S(t)

2.2 2.1 2 1.9

0

50

6000

100

5000

80

4000

60

I(t)

0

3000

40

2000

20

1000 0

50

0 0

50

x 104

1000 2.5 600 400

2.3 2.2

200 0

without controls B1=100, B2=10, B3=10 B1=100, B2=10, B3=100

2.4 N(t)

T(t)

800

2.1 0

50

2 0

50 Time (years)

Epidemiology Modeling, Fig. 5 Low-risk TB community: the optimal controls (u1 .t /, u2 .t /, and u3 .t /) and state variables are displayed as functions of time. The black solid, blue dotted, and

red dashed curves represent the cases of without controls, (B1 D 100; B2 D 10; B3 D 10), and (B1 D 100; B2 D 10; B3 D 100) with controls, respectively (Figures taken from [29])

Epidemiology Modeling

435 R0=1.382438

u1(t )

1 B1=100, B2=10, B3=10 B1=100, B2=100, B3=100

0.5 0

0

5

10

15

20

25

30

35

40

45

50

u2(t )

1

E

0.5 0

0

5

10

15

20

5

10

15

20

25

30

35

40

45

50

25 30 Time (years)

35

40

45

50

u3(t )

1 0.5 0 0

2.5

x 104

15000

10000 8000

10000

1.5 1

I(t)

E(t)

S(t)

2

4000

5000

2000

0.5 0

0

0

50

0

50

50

without controls B1=100, B2=10, B3=10 B1=100, B2=100, B3=100

N(t)

2 T(t)

0

2.5

3000 2000

1.5

1000

0

0

x 104

4000

0

6000

50

1

0

50 Time (years)

Epidemiology Modeling, Fig. 6 High-risk TB community: the optimal controls (u1 .t /, u2 .t /, and u3 .t /) and state variables are displayed as functions of time. The black solid, blue dotted,

and red dashed curves represent the cases of without controls, (B1 D 100; B2 D 10; B3 D 10) and (B1 D 100; B2 D 100, B3 D 100) with controls, respectively (Figures taken from [29])

436

Epidemiology Modeling

These three policies are judged on their ability to reduce or eliminate the levels of latent- and activeTB prevalence in the population at a reduced cost. In this TB setup, exogenous reinfection plays a role and therefore the optimization process must account for such a possibility. It is important, therefore, to identify optimal strategies in low-risk TB communities where the disease is endemic, despite the existence of effective public health norms (R0 < 1) as well as in high-risk TB communities, R0 > 1 (the dominant scenario in parts of the world where TB is highly endemic). Three controls or time-dependent intervention policies yet to be computed are introduced as multipliers to the incidence and treatment rates: BS .t/ D ˇ.1  u1 .t//SI =N , BE .t/ D pˇ.1  u1 .t//EI =N , BE .t/ D ˇ.1  u2 .t//T I =N , and  u3 .t/. The first control, u1 .t/, works at reducing contacts with infectious individuals through policies of isolation, or social distancing, or through the administration (if available) of vaccines or drugs that reduce susceptibility to infection. The second control, u2 .t/, models the effort required to reduce or prevent the reinfection of treated individuals. This control is not identical to u1 .t/ since individuals with prior TB bouts are likely to react differently in the presence of activeTB cases. The treatment control, u3 .t/, models the effort directed at treating infected individuals. The goal of minimizing the number of exposed and infectious individuals while keeping the costs as low as possible requires access to data that is rarely available. Hence, the focus here is on the identification of solutions that only incorporate the relative costs associated with each policy or combination of policies. The identification of optimal policies is tied in to the minimization of a functional J (defined below), over the feasible set of controls (ui .t/ W i D 1; 2; 3), subject to Model (4) over a finite time interval Œ0; tf . The objective functional is given by the expression: Z

tf

J.u1 ; u2 ; u3 / D 0

C

ŒE.t/ C I.t/ C

B1 2 u1 .t/ 2

B3 2 B2 2 u2 .t/ C u3 .t/ dt 2 2

(6)

where the coefficients B1 ; B2 , and B3 model constant relative cost weight parameters. These coefficients account for the relative size and importance (including cost) of each integrand in the objective functional. It is standard to assume that the controls are nonlinear and

quadratic. The objective, therefore, is to find numerically the optimal control functions, u1 , u2 , and u3 that satisfy J.u1 ; u2 ; u3 / D minJ.u1 ; u2 ; u3 /; ˝

(7)

where ˝ D f.u1 ; u2 ; u3 / 2 .L2 .0; tf //3 jai  ui  bi ; i D 1; 2; 3g and ai ; bi ; i D 1; 2; 3 are lower and upper bounds for the controls, respectively. Pontryagin’s maximum principle [50, 85] is used to solve the optimality system, which is derived and simulated following the approaches in Choi et al. [29], and Lee et al. [76]. We manage to identify optimal control strategies through simulations when R0 > 1 and R0 < 1 using reasonable TB parameters [29]. The optimal controls and corresponding states are displayed in Figs. 5 and 6 under two distinct scenarios: under a low-risk TB community (R0 D 0:87) and under a high-risk TB community (R0 D 1:38). It is observed that the social distancing control, u1 .t/, is the most effective when R0 < 1, while the relapse control, u2 .t/, is the most effective when R0 > 1. Further, simulation results suggest that when R0 < 1, the control strategy cannot work without the presence of u1 .t/. Similarly, when R0 > 1, u2 .t/ must be present. With the presence of u1 .t/ when R0 < 1 and the presence of u2 .t/ when R0 > 1, the identified optimal control programs will effectively reduce the number of exposed and infectious individuals.

Perspective on Epidemiological Models and Their Use Epidemiological thinking has transcended the realm of epidemiological modeling and in the processes, it has found applications to the study of dynamic social process where contacts between individuals facilitate the buildup of communities that can suddenly (tipping point) take on a life of their own. This perspective has resulted in applications of the contagion model in the study of the dynamics of bulimia [54], or in the study of the spread of specific scientific ideas [11], or in the assessment of the emergence of new scientific fields [12]. Contagion models are also being used to identify population-level mechanisms responsible for drinking patterns [81, 87] or drug addiction trends [92]. Contagion models have also been

Epidemiology Modeling

applied to the study of the spread of fanaticism [22] or the building of collaborative learning communities [38]. It is still in the context of the study of disease dynamics and in the evaluation of specific public policy measures that most of the applications of epidemiological models are found. Efforts to understand and manage the transmission dynamics of HIV [19, 24, 70, 95] or to respond to emergencies like those posed by the 2003 SARS epidemic [30], or the 2009 AH1N1 influenza pandemic [35, 62], or to assess the potential impact of widely distributed rotavirus vaccines [88, 89] are still at the core of most of the research involving epidemiological mathematical models. The events of 9/11/2001 when our vulnerability to bioterrorism was exposed in fronts that included the deliberate release of biological agents has brought contagion and other models to the forefront of our battle against these threats to our national security (see [10]). A series of volumes and books [1, 3, 9, 15, 16, 20, 26, 27,36,39,56,71,94,97] have appeared over the past two decades that highlight our ever present concern with the challenges posed by the transmission dynamics and evolution of infectious diseases. The contagion approach highlighted here relies primarily on the use of deterministic models. There is, however, an extensive and comprehensive mathematical epidemiological literature that has made significant and far-reaching contributions using probabilistic perspectives [1, 2, 9, 36, 39]. The demands associated with the study of diseases like influenza A-H1N1 or the spread of sexually transmitted diseases (including HIV) in the context of social landscapes that change in response to knowledge, information, misinformation, or the excessive use of drugs (leading to drug resistance) have brought to the forefront of the use of alternative approaches including those that focus on social networks, into the study of infectious diseases [31, 42, 44, 82, 83]. Renewed interest in the characterization and study of heterogenous mixing patterns and their role on disease dynamics have also reemerged [14, 18, 25, 75, 80, 84]. Contagion models continue to contribute to our understanding of “contact” processes that change in response to behavioral decisions [49]. It is our hope that this idiosyncratic overview has captured the fundamental role that epidemiological models play and will continue to play in the study of human process of importance in life and social sciences.

437

References 1. Allen, L.J.S.: An Introduction to Stochastic Processes with Applications to Biology. Pearson Prentice Hall, Upper Saddle River, NJ (2003) 2. Anderson, H., Britton, T.: Stochastic Epidemic Models and Their Statistical Analysis. Lecture Notes in Statistics. Springer, New York (2000) 3. Anderson, R.M.: Population Dynamics of Infectious Disease: Theory and Applications. Chapman and Hall, London–New York (1982) 4. Anderson, R.M., May, R.M.: Infectious Disease of Humans. Oxford Science Publications, Oxford (1991) 5. Aparicio, J.P., Capurro, A.F., Castillo-Chavez, C.: Transmission and dynamics of tuberculosis on generalized households. J. Theor. Biol. 206, 327–341 (2000) 6. Aparicio, J.P., Capurro, A.F., Castillo-Chavez, C.: Markers of disease evolution: The case of tuberculosis. J. Theor. Biol. 215, 227–237 (2002) 7. Arino, J., Brauer, F., Driessche, P., Watmough, J., Wu, J.: A final relation for epidemic models. Math. Biosci. Eng. 4, 159–175 (2007) 8. Aron, J.L., May, R.M.: The population dynamics of Malaria. In: Anderson, R.M. (ed.) Population Dynamics of Infectious Diseases, Theory and Applications. Population and Community Biology Series. Chapman and Hall, London, New York (1982) 9. Bailey, N.T.J.: The Mathematical Theory of Infectious Diseases and Its Applications. Charles Griffin & Company Ltd., London and High Wycombe (1975) 10. Banks, H.T., Castillo-Chavez, C. (eds.): Bioterrorism: Mathematical Modeling Applications to Homeland Security. SIAM Series Frontiers in Applied Mathematics, Philadelphia (2003) 11. Bettencourt, L.M.A., Cintron-Arias, A., Kaiser, D.I., Castillo-Chavez, C.: The power of a good idea: quantitative modeling of the spread of ideas from epidemiological models. Phys. A 364, 513–536 (2006) 12. Bettencourt, L.M.A., Kaiser, D.I., Kaur, J., Castillo-Chavez, C.: Population modeling of the emergence and development of scientific fields. Scientometrics 75(3), 495–518 (2008) 13. Bloom, B.R.: Tuberculosis: Pathogenesis, Protection, and Control. ASM, Washington DC (1994) 14. Blythe, S., Busenberg, S., Castillo Chavez, C.: Affinity in paired event probability. Math. Biosci. 128, 265–284 (1995) 15. Brauer, F., Castillo Chavez, C.: Mathematical models in population biology and epidemiology. Texts in Applied Mathematics, vol. 40. Springer, Berlin–Heidelberg–New York (2001) 16. Brauer, F., Driessche, P.V.D., Wu, J.: Mathematical Epidemiology. Lecture Nodes in Mathematics, Mathematical Biosciences Subseries. Springer (2000) 17. Brauer, F., Feng, Z., Castillo-Chavez, C.: Discrete epidemic models. Math. Biosci. Eng. 7(1), 1–16 (2010) 18. Busenberg, S., Castillo-Chavez, C.: A general solution of the problem of mixing of subpopulations and its application to risk and age-structured epidemic models. IMA J. Math. Appl. Med. Biol. 8, 1–29 (1991)

E

438 19. Castillo-Chavez, C.: Mathematical and Statistical Approaches to AIDS Epidemiology. Lect. Notes Biomath., vol. 83. Springer, Berlin–Heidelberg–New York (1989) 20. Castillo-Chavez, C., Chowell, G.: Special issue: mathematical models, challenges, and lessons learned from the 2009 A/H1N1 influenza pandemic. Math. Biosci. Eng. 8(1), 1–246 (2011) 21. Castillo-Chavez, C., Feng, Z.: To treat or not to treat: the case of tuberculosis. J. Math. Biol. 35, 629–656 (1997) 22. Castillo-Chavez, C., Song, B.: Models for the transmission dynamics of fanatic behaviors. In: Banks, H.T., CastilloChavez, C. (eds.) Bioterrorism: Mathematical Modeling Applications to Homeland Security. SIAM Series Frontiers in Applied Mathematics, vol. 28, p. 240. Society for Industrial and Applied Mathematics, Philadelphia (2003) 23. Castillo-Chavez, C., Song, B.: Dynamical models of tuberculosis and their applications. Math. Biosci. Eng. 1(2), 361– 404 (2004) 24. Castillo-Chavez, C., Cooke, K., Huang, W., Levin, S.A.: Results on the dynamics for models for the sexual transmission of the human immunodeficiency virus. Appl. Math. Lett. 2(4), 327–331 (1989a) 25. Castillo-Chavez, C., Hethcote, H.W., Andreasen, V., Levin, S.A., Liu, W.M.: Epidemiological models with age structure, proportionate mixing, and cross-immunity. J. Math. Biol. 27, 233–258 (1989b) 26. Castillo-Chavez, C., Blower, S., Van den Driessche, P., Kirschner, D., Yakubu, A.A. (eds.) Mathematical Approaches for Emerging and Reemerging Infectious Disease: Models, Methods and Theory. Springer-verlag, Berlin– Heidelberg–New York (2001a) 27. Castillo-Chavez, C., Blower, S., Van den Driessche, P., Kirschner, D., Yakubu, A.A. (eds.) Mathematical Approaches for Emerging and Reemerging Infectious Disease: Models, Methods and Theory. Springer, Berlin–Heidelberg– New York (2001b) 28. Castillo-Chavez, C., Feng, Z., Huang, W.: On the computation of R0 and its role on global stability. In: Mathematical approaches for emerging and reemerging infectious diseases: an introduction. IMA, vol. 125, pp. 229–250. Springer, New York (2002) 29. Choi, S., Jung, E., Castillo-Chavez, C.: Optimal strategies for tuberculosis with exogenous reinfection (2010) (preprint) 30. Chowell, G., Fenimore, P.W., Castillo-Garsow, M.A., Castillo-Chavez, C.: Sars outbreaks in ontario, hong kong and singapore: the role of diagnosis and isolation as a control mechanism. J. Theor. Biol. 224, 1–8 (2003a) 31. Chowell, G., Hyman, J.M., Eubank, S., Castillo-Chavez, C.: Scaling laws for the movement of people between locations in a large city. Phys. Rev. E 68:661021–661027 (2003b) 32. Chowell, G., Ammon, C.E., Hengartner, N.W., Hyman, J.M.: Transmission dynamics of the great influenza pandemic of 1918 in Geneva, Switzerland: assessing the effects of hypothetical interventions. J. Theor. Biol. 241(2), 193– 204 (2006) 33. Chowell, G., Diaz-Duenas, P., Miller, J.C., Alcazar-Velazco, A., Hyman, J.M., Fenimore, P.W., Castillo-Chavez, C.: Estimation of the reproduction number of dengue fever from spatial epidemic data. Math. Biosci. 208, 571–589 (2007a)

Epidemiology Modeling 34. Chowell, G., Miller, M.A., Viboud, C.: Seasonal influenza in the United States, France and Australia: transmission and prospects for control. Epidemiol. Infect. 18, 1–13 (2007b) 35. Chowell, G., Bertozzi, S.M., Colchero, M.A., Lopez-Gatell, H., Alpuche-Aranda, C., Hernandez, M., Miller, M.A.: Severe respiratory disease concurrent with H1N1 influenza circulation. N. Engl. J. Med. 361, 674–679 (2009a) 36. Chowell, G., Hyman, J.M., Bettencourt, L.M.A., CastilloChavez, C. (eds.): Mathematical and Statistical Estimation Approaches in Epidemiology. Springer, Princeton and Oxford (2009b) 37. Cintron-Arias, A., Castillo-Chavez, C., Bettencourt, L.M., Lloyd, A.L., Banks, H.T.: Estimation of the effective reproductive number from disease outbreak data. Math. Biosci. Eng. 6(2), 261–283 (2009) 38. Crisosto, N.M., Kribs-Zaleta, C.M., Castillo-Chavez, C., Wirkus, S.: Community resilience in collaborative learning. Discrete Continuous Dyn. Sys. Ser. B 14(1), 17–40 (2010) 39. Daley, D.J., Gani, J.: Epidemic Modeling an Introduction. Cambridge University Press, Cambridge (1999) 40. Diekmann, O., Heesterbeek, J.A.P.: Mathematical Epidemiology of Infectious Disease: Model Building, Analysis and Interpretation. Wiley, Chichester (2000) 41. Diekmann, O., Heesterbeek, J.A.P., Metz, J.A.J.: On the definition and the computation of the basic reproductive ratio r0 in models for infectious diseases in the heterogeneous populations. J. Math. Biol. 28, 365–382 (1990) 42. Dorogovtsev, S.N., Mendes, J.F.F.: Mathematics in Population Biology. Evolution of Networks. Princeton University Press, Princeton and Oxford (2003) 43. Driessche, P.V., Watmough, J.: Reproduction numbers and sub-threshold endemic equilibria for compartmental models of disease transmission. Math. Biosci. 180, 29–48 (2002) 44. Duncan, J.W.: Six Degrees. W.W. Norton & Company, New York, London (2003) 45. Dushoff, J., Huang, W., Castillo-Chavez, C.: Backward bifurcations and catastrophe in simple models of fatal diseases. J. Math. Biol. 36, 227–248 (1998) 46. Ehrlich, P.R., Ehrlich, A.H.: The Population Explosion. Simon & Schuster, New York (1997) 47. Farrington, C.P.: On vaccine efficacy and reproduction numbers. Math. Biosci. 185, 89–109 (2003) 48. Feng, Z., Castillo-Chavez, C., Capurro, A.: A model for tb with exogenous re-infection. J. Theor. Popul. Biol. 5, 235– 247 (2000) 49. Fenichel, E.P., Castillo-Chavez, C., Ceddiac, M.G., Chowell, G., Gonzalez, P., Hickling, G.J., Holloway, G., Horan, R., Morin, B., Perrings, C., Springborn, M., Velazquez, L., Villalobos, C.: Adaptive human behavior in epidemiological models. Proc. Natl. Acad. Sci. U.S.A. 108(15), 6306–6311 (2011) 50. Fleming, W.H., Rishel, R.W.: Deterministic and Stochasitic Optimal Control. Springer, New York (1975) 51. Gladwell, M.: The Tipping Point. New Yorker, Princeton and Oxford (1996) 52. Gladwell, M.: The Mosquito Killer. New Yorker, Princeton and Oxford (2001) 53. Gladwell, M.: The Tipping Point: How Little Things Can Make a Big Difference. Back Bay Books/LittleLittle, Brown and Company, Time Warner Book Group, Boston (2002)

Epidemiology Modeling 54. Gonzalez, B., Huerta-Sanchez, E., Ortiz-Nieves, A., Vazquez-Alvarez, T., Kribs-Zaleta, C.: Am I too fat? Bulimia as an epidemic. J. Math. Psychol. 47, 515–526 (2003) 55. Gonzalez-Parra, P., Lee, S., Castillo-Chavez, C., Velazquez, L.: A note on the use of optimal control on a discrete time model of influenza dynamics. To appear in Math. Biosci. Eng. in Special Issue on H1N1 2009 Influenza Models 8(1), 183–197 (2011) 56. Gumel, A.B., Castillo-Chavez, C., Mickens, R.E., Clemence, D.P.: Contemporary Mathematics Mathematical Studies on Human Disease Dynamics Emerging Paradigms and Challenges. AMS, Providence (2005) 57. Hadeler, K.P., Castillo-Chavez, C.: A core group model for disease transmission. Math. Biosci. 128, 41–55 (1995) 58. Heesterbeek, J.A.P.: R0 . Thesis, CWI, Amsterdam (1992) 59. Heesterbeek, J.A.P.: A brief history of R0 and a recipe for its calculation. Acta Biotheor. 50, 189–204 (2002) 60. Heesterbeek, J.A.P., Roberts, M.G.: Threshold quantities for helminth infections. J. Math. Biol. 33, 425–434 (1995) 61. Heffernan, J.M., Smith, R.J., Wahl, L.M.: Perspectives on the basic reproductive ratio. J. R. Soc. Interface 2, 281–293 (2005) 62. Herrera-Valdez, M.A., Cruz-Aponte, M., Castillo-Chavez, C.: Multiple waves for the same pandemic: local transportation and social distancing explain the dynamics of the A-H1N1 epidemic during 2009 in mexico. To appear in Math. Biosci. Eng. in Special Issue on H1N1 2009 Influenza Models (2010) 63. Hethcote, H.W.: Qualitative analysis for communicable disease models. Math. Biosci. 28:335–356 (1976) 64. Hethcote, H.W.: Three basic epidemiological models. In: Levin, S.A., Hallam, T.G., Gross, L.J. (eds.) Applied Mathematical Ecology. Springer, Berlin–Heidelberg–New York (1989) 65. Hethcote, H.W.: The mathematics of infectious diseases. SIAM Rev. 42(4), 599–653 (2000) 66. Hethcote, H.W., Van Ark, J.W.: Modeling HIV Transmission and AIDS in the United States. Lect. Notes Biomath. vol. 95. Springer, Berlin–Heidelberg–New York (1992) 67. Hethcote, H.W., Yorke, J.A.: Gonorrhea Transmission Dynamics and Control. Lect. Notes Biomath., vol. 56. Springer, Berlin–Heidelberg–New York (1984) 68. Huang, W., Cooke, K., Castillo-Chavez, C.: Stability and bifurcation for a multiple group model for the dynamics of HIV/AIDS transmission. SIAM J. Appl. Math. 52(3), 835– 854 (1992) 69. Jung, E., Lenhart, S., Feng, Z.: Optimal control of treatments in a two strain tuberculosis model. Discrete Continuous Dyn. Sys. Ser. B 2, 473–482 (2002) 70. Kasseem, G.T., Roudenko, S., Tennenbaum, S., CastilloChavez, C.: The role of transactional sex in spreading HIV/AIDS in Nigeria: a modeling perspective. In: Gumel, A., Castillo-Chavez, C., Clemence, D.P., Mickens, R.E. (eds.) Mathematical Studies on Human Disease Dynamics: Emerging Paradigms and Challenges. American Mathematical Society, Providence (2006) 71. Keeling, M., Rohani, P.: Modeling infecitous diseases in human and animals. Princeton University Press, Princeton and Oxford (2008)

439 72. Kermack, W.O., McKendrick, A.G.: A contribution to the mathematical theory of epidemics. Proc. R. Soc. A 115, 700–721 (1927) (reprinted in Bull. Math. Biol. 53, 33–55) 73. Kermack, W.O., McKendrick, A.G.: A contribution to the mathematical theory of epidemics. Proc. R. Soc. Lond. 138, 55–83 (1932) 74. Kermack, W.O., McKendrick, A.G.: A contribution to the mathematical theory of epidemics. Proc. R. Soc. Lond. 141, 94–112 (1933) 75. Lajmanovich, A., Yorke, J.A.: A deterministic model for gonorrhea in a nonhomogeneous population. Math. Biosci. 28, 221–236 (1976) 76. Lee, S., Chowell, G., Castillo-Chavez, C.: Optimal control of influenza pandemics: the role of antiviral treatment and isolation. J. Theor. Biol. 265, 136–150 (2010a) 77. Lee, S., Jung, E., Castillo-Chavez, C.: Optimal control intervention strategies in low- and high-risk problem drinking populations. Socio-Econ. Plann. Sci. 44, 258–265 (2010b) 78. Lee, S., Morales, R., Castillo-Chavez, C.: A note on the use of influenza vaccination strategies when supply is limited. Math. Biosci. Eng. in Special Issue on H1N1 2009 Influenza Models 8(1), 171–182 (2011) 79. Miller, B.: Preventive therapy for tuberculosis. Med. Clin. North Am. 77, 1263–1275 (1993) 80. Morin, B., Castillo-Chavez, C., Hsu Schmitz, S., Mubayi, A., Wang, X.: Notes from the heterogeneous: a few observations on the implications and necessity of affinity. J. Biol. Dyn. 4(5), 456–477 (2010) 81. Mubayi, A., Greenwood, P.E., Castillo-Chavez, C., Gruenewald, P., Gorman, D.M.: Impact of relative residence times on the distribution of heavy drinkers in highly distinct environments. Socio-Econ. Plann. Sci. 43(1), 1–12 (2010) 82. Newman, M.: The structure and function of complex networks. SIAM Rev. 45, 167–256 (2003) 83. Newman, M., Barabasi, A.L., Watts, D.J.: The Structure and Dynamics of Networks. Princeton University Press, Princeton and Oxford (2006) 84. Nold, A.: Heterogeneity in disease-transmission modeling. Math. Biosci. 52, 227–240 (1980) 85. Pontryagin, L.S., Boltyanskii, V.G., Gamkrelidze, R.V., Mishchenko, E.F.: The Mathematical Theory of Optimal Processes. Wiley, New York (1962) 86. Ross, R.: The prevention of Malaria. John Murray, London (1911) 87. Sanchez, F., Wang, X., Castillo-Chavez, C., Gruenewald, P., Gorman, D.: Drinking as an epidemic–a simple mathematical model with recovery and relapse. Therapists Guide to Evidence Based Relapse Prevention, Princeton and Oxford (2007) 88. Shim, E., Castillo-Chavez, C.: The epidemiological impact of rotavirus vaccination programs in the United States and Mexico. In: Chowell, G., Hyman, J.M., Bettencourt, L.M.A., Castillo-Chavez, C. (eds.) Mathematical and Statistical Estimation Approaches in Epidemiology. Springer, New York (2009) 89. Shim, E., Feng, Z., Martcheva, M., Castillo-Chavez, C.: An age-structured epidemic model of rotavirus with vaccination. J. Math. Biol. 53(4), 719–746 (2006)

E

440 90. Smith, P.G., Moss, A.R.: Epidemiology of Tuberculosis, in Tuberculosis: Pathogenesis, Protection, and Control. ASM, Washington DC (1994) 91. Song, B., Castillo-Chavez, C., Aparicio, J.P.: Tuberculosis models with fast and slow dynamics: the role of close and casual contacts. Math. Biosci. 180, 187–205 (2002) 92. Song, B., Castillo-Garsow, M., Rios-Soto, K., Mejran, M., Henso, L., Castillo-Chavez, C.: Raves clubs, and ecstasy: the impact of peer pressure. J. Math. Biosci. Eng. 3(1), 1– 18 (2006) 93. Styblo, K.: Epidemiology of Tuberculosis. VEB Gustuv Fischer Verlag Jena, The Hague (1991) 94. Thieme, H.R.: Mathematics in Population Biology. Princeton University Press, Princeton and Oxford (2003) 95. Thieme, H.R., Castillo-Chavez, C.: How many infectionage dependent infectivity affect the dynamics of HIV/AIDS? SIAM J. Appl. Math. 53, 1447–1479 (1993) 96. White, L.F., Pagano, M., Morales, E.: A likelihood-based method for real-time estimation of the serial interval and reproductive number of an epidemic. Stat. Med. (2007). doi:10.1002/sim.3136 97. Zeng, D., Chen, H., Castillo-Chavez, C., Lober, W.B., Thurmond, M.: Infectious Diseases Bioinformatics and Biosurveillance, Integrated Series in Information Systems. Springer, New York (2011)

Error Estimates for Linear Hyperbolic Equations Chi-Wang Shu Division of Applied Mathematics, Brown University, Providence, RI, USA

Mathematics Subject Classification

Error Estimates for Linear Hyperbolic Equations

Short Definition We discuss error estimates for finite difference (FD), finite volume (FV), finite element (FE), and spectral methods for solving linear hyperbolic equations, with smooth or discontinuous solutions.

Description Hyperbolic equations arise often in computational mechanics and other areas of computational sciences, for example, they can describe various wave propagation phenomena, such as water waves, electromagnetic waves, aeroacoustic waves, and shock waves in gas dynamics. In this entry we are concerned with linear hyperbolic equations, which take the form ut C Aux D 0

(1)

together with suitable initial and boundary conditions in one spatial dimension. Here A either is a constant matrix or depends on x and/or t, and it is diagonalizable with real eigenvalues. Many of the numerical methods we discuss in this entry also work for nonlinear hyperbolic equations; however, their error estimates may become more complicated, especially for discontinuous solutions. We should mention that linear hyperbolic equations could also arise in second-order form ut t D Auxx

(2)

with a positive definite matrix A. Equation (2) can be solved directly or rewritten into the form (1) by introducing auxiliary variables. In this entry we will concentrate on the numerical methods for solving (1) Synonyms only. We will first describe briefly several major classes Discontinuous Galerkin method (DG); Finite differof numerical methods for solving linear hyperbolic ence method (FD); Finite element method (FE); Finite volume method (FV); Ordinary differential equation equations (1). We will then describe their error esti(ODE); Partial differential equation (PDE); Strong sta- mates, starting from the simpler situation of smooth bility preserving (SSP); Total variation diminishing solutions, followed by the more difficult situation of discontinuous solutions. (TVD) 65M15; 65M06; 65M60; 65M70

Error Estimates for Linear Hyperbolic Equations

Four Major Classes of Numerical Methods We will use the simple model equation

441

where the numerical flux fOjnC 1 D unj for the first2   order scheme (4) and fOjnC 1 D 12 unj C unj C1 for the 2

(3) second-order approximation (5). There are usually two approaches to increase the temporal order of accuracy. The first approach is to use to describe briefly four major classes of numerical an ordinary differential equation (ODE) solver, such as a Runge-Kutta or multistep method. In order to methods for solving linear hyperbolic equations. better approximate discontinuous solutions, a special class of ODE solvers, referred to as the total variation Finite Difference Methods diminishing (TVD) or strong stability preserving (SSP) Finite difference methods are standard numerical meth- time discretizations, is usually used; see [12]. The ods for solving hyperbolic partial differential equations second approach is to use a Lax-Wendroff procedure (PDEs); see, for example, [13]. [17], which writes a Taylor expansion in time, converts Assuming that we would like to solve (3) over the time derivatives to spatial derivatives by using the PDE, interval x 2 Œ0; 1 , the finite difference scheme would and then discretizes these spatial derivatives by finite start with a choice of grid points 0 D x1 < x2 <    < differences. xN D 1, which are usually assumed to be uniform with h D xj C1  xj D N1 , and a time discretization 0 D Finite Volume Methods t 0 < t 1 < t 2 <    , which is again assumed to be Finite volume methods are also standard numerical uniform with  D t nC1  t n for simplicity. We can methods for solving hyperbolic PDEs; see, for exthen write down the scheme satisfied by the numerical ample, [18]. Again, assuming that we would like to solution unj , which approximates the solution u.xj ; t n / solve (3) over the interval x 2 Œ0; 1 , the finite volume of the PDE. scheme would start with a choice of cells Ij D The simplest example is the upwind scheme .xj  1 ; xj C 1 / with x 1 D 0 and xN C 1 D 1. The mesh 2 2 2 2 size hj D xj C 1  xj  1 does not need to be uniform 2 2 and we denote h D max1j N hj . Time discretization unj  unj 1 ujnC1  unj C D 0; (4) is still 0 D t 0 < t 1 < t 2 <    , which is again assumed  h to be uniform with  D t nC1  t n for simplicity. We can then write down the scheme satisfied by the which is first-order accurate, that is, the error ejn D numerical solution uN n , which approximates the cell j u.xj ; t n /  unj is of the size O.hC/. Higher order in x average 1 R u.x; t n /dx of the exact solution u of hj Ij can be achieved by using a wider stencil. For example, the PDE. The simplest example is again the first-order replacing upwind scheme un  un ut C ux D 0

j 1

j

h in (4) by

uN nC1  uN nj j

unj C1  unj 1

(5) 2h would increase the spatial order of accuracy from one to two. In order to approximate discontinuous solutions, especially for nonlinear problems, the spatial discretization is usually required to be conservative, that is, ux jxDxj

fOjnC 1  fOjn 1 2

2

h

;



C

uN nj  uN nj 1 hj

D 0:

(6)

Notice that the finite volume scheme (6) is identical to the finite difference scheme (4) on a uniform mesh, if the cell average uN nj in the former is replaced by the point value unj in the latter. This is not surprising, since we can easily verify that 1 hj

Z u.x; t n /dx D u.xj ; t n / C O.h2 /; Ij

E

442

Error Estimates for Linear Hyperbolic Equations

that is, one can replace cell averages by point values at cell centers and vice versa up to second-order accuracy. There is no need to distinguish between finite difference and finite volume schemes up to second-order accuracy. However, for schemes higher than secondorder accuracy, finite volume schemes are different from finite difference schemes. The main advantage of finite volume schemes over finite difference schemes is that the former is more flexible for nonuniform meshes and unstructured meshes (in higher spatial dimensions). However, high-order finite volume schemes are more costly than finite difference schemes in multidimensions [22].

tion can again be realized by using standard ODE solvers, for example, those in [12], or by space-time discontinuous Galerkin methods. When we take the polynomial degree k D 0 and forward Euler time discretization, the discontinuous Galerkin method (7) becomes the standard first-order upwind finite difference (4) or finite volume scheme (6). Thus, the discontinuous Galerkin methods can also be considered as a generalization of first-order monotone finite difference or finite volume methods. The main advantages of discontinuous Galerkin methods include their flexibility in unstructured meshes, mesh and order adaptivity, and more complete stability analysis and error estimates.

Finite Element Methods Traditional finite element methods, when applied to hyperbolic equations, suffer from non-optimal convergence for smooth solutions and spurious oscillations polluted to smooth regions for discontinuous solutions. The more successful finite element methods for solving hyperbolic equations include the streamline diffusion methods [3] and discontinuous Galerkin methods [7]. We will only describe briefly the discontinuous Galerkin method when applied to the model equation (3). We use the same notations for the mesh as in the previous section. The finite element space is given by

Spectral Methods Another important class of numerical methods for solving hyperbolic equations is the class of spectral methods; see, e.g., [14]. Spectral methods seek approximations within a finite dimensional function space VN containing global polynomials or trigonometric polynomials of degree up to N . The numerical solution uN 2 VN is chosen such that the residue

RN D .uN /t C .uN /x

either is orthogonal to all functions in the space VN (spectral Galerkin methods) or is zero at preselected collocation points (spectral collocation methods). The ˚ Vh D v W vjIj 2 P k .Ij /I 1  j  N ; main advantage of spectral methods is their high-order accuracy. For analytic solutions, the error of spectral where P k .Ij / denotes the set of polynomials of degree methods can be exponentially small. However, spectral up to k defined on the cell Ij . The semi-discrete DG methods are less flexible for complex geometry. They method for solving (3) is defined as follows: find the are also rather difficult to design for complicated PDEs unique function uh 2 Vh such that, for all test functions and boundary conditions to achieve stability. vh 2 Vh and all 1  j  N , we have Error Estimates for Smooth Solutions Z Z d When the solutions of hyperbolic equations are uh .x; t/vh .x/dx  uh .x/@x vh .x/dx dt Ij smooth, it is usually easy to obtain error estimates Ij for the numerical schemes mentioned in the previous   C uh .xi C 1 ; t/vh .xi C 1 / 2 2 section.  C  uh .xi  1 ; t/vh .xi  1 / D 0: 2 2 Finite Difference, Finite Volume, and Spectral (7) Methods For finite difference and finite volume schemes, we Here, the inter-cell boundary value of uh (the so-called would first need to prove their stability. If the problem numerical flux) is taken from the left (upwind) side is defined on uniform or structured meshes with uh .xiC 1 ; t/, which ensures stability. Time discretiza- periodic or compactly supported boundary conditions, 2

Error Estimates for Linear Hyperbolic Equations

standard Fourier analysis can be applied to prove stability. For other boundary conditions, the theory of Gustafsson, Kreiss, and Sundstr¨om (the GKS theory) provides a tool for analyzing the stability of finite difference and finite volume schemes. For unstructured meshes, the analysis of stability would need to rely on the energy method in the physical space and can be quite complicated. We refer to, e.g., [13] for a detailed discussion of stability of finite difference and finite volume schemes. After stability is established, we would be left with the relatively easy job of measuring the errors locally, namely, by putting the exact smooth solution u of the PDE into the finite difference or finite volume scheme and measuring its remainder (the so-called local truncation error). The Lax equivalence theorem provides us with assurance that for a stable scheme, such easily measured local truncation error and the global error (the error between the exact solution of the PDE and the numerical solution) are of the same order. We again refer to, e.g., [13] for a detailed discussion. The same approach can also be applied to spectral methods. The proof of stability can be quite difficult, especially for collocation methods. We refer to [14] for more details.

443

method has the further complication of the inter-cell boundary terms, which, when not estimated carefully, may lead to a loss of optimal order in the error analysis. For linear hyperbolic equations, discontinuous Galerkin methods can be proved to provide an L2 error estimate of order at least O.hkC1=2 / when piecewise polynomials of degree k are used. In many cases the optimal error estimate O.hkC1 / can be proved as well. We refer to, e.g., [9, 15, 21, 24] for more details.

E Superconvergence Superconvergence refers to the fact that the error estimates can be obtained to be of higher order than expected, that is, higher than O.hkC1 / when piecewise polynomials of degree k are used. We will only discuss superconvergence results for discontinuous Galerkin methods for solving linear hyperbolic PDEs. One approach to obtain superconvergence is through the so-called negative Sobolev norms. We recall that the negative Sobolev norm is defined by

kukk D

.u; v/ ; v2H k ;v¤0 jjvjjk max

where H k is the space of all functions with finite k-th order Sobolev norm defined by

Finite Element Methods While the principle of error estimates for finite element methods is the same as that for finite difference, finite volume, and spectral methods, namely, stability analysis plus local error analysis, the procedure is somewhat different. We will again use the discontinuous Galerkin method as an example to demonstrate the procedure. First, the error u  uh is decomposed into two parts

where .a; b/ is our computational interval. In [8], it is proved that, for the discontinuous Galerkin methods solving linear hyperbolic PDEs, we have

u  uh D .u  P u/ C .P u  uh /;

ku  uh k.kC1/ D O.h2kC1 /:

where P u is a suitable projection of the exact solution u to the finite element space Vh . The estimate for the term P u  uh , which is within the finite element space Vh , is achieved by the stability of the discontinuous Galerkin method. The estimate on the term u  P u, which is an approximation error and has nothing to do with the discontinuous Galerkin method, can be obtained by standard finite element techniques [5]. Comparing with the error estimates for standard finite element methods, the analysis for the discontinuous Galerkin

That is, we achieve a superconvergence of order 2k C 1 when using the weaker negative norm. Together with a similar estimate for divided differences of the numerical solution and a post-processing procedure [2], we can obtain O.h2kC1 / convergence in the strong L2 norm for the post-processed solution on uniform meshes. There are extensive follow-up works after [8] to explore this superconvergence for more general situations, for example, with boundaries or with non uniform meshes.

jjvjj2k

D

k Z X `D0

b a



d `v dx `

2 dx;

444

Error Estimates for Linear Hyperbolic Equations

Another approach to obtain superconvergence is through the analysis of the super-closeness of the numerical solution to a specific projection of the exact solution or to the exact solution itself at certain quadrature points. The former is analyzed in, e.g., [4,25], with [25] establishing the superconvergence result kP u  uh k D O.hkC2 /; where P u is the Gauss-Radau projection of the exact solution. The latter is analyzed in, e.g., [1], establishing the superconvergence result .u  uh /.xj` / D O.hkC2 /; where xj` are the Gauss-Radau quadrature points in the cell Ij . The superconvergence results, besides being of their own value in revealing the hidden approximation properties of the discontinuous Galerkin methods, can also be used to design effective error indicators for adaptive methods. Error Estimates for Discontinuous Solutions Unlike elliptic or parabolic PDEs, the solutions to hyperbolic PDEs may be discontinuous. For linear PDEs such as (1), the discontinuities in the solution may come from the prescribed initial and/or boundary conditions. For such discontinuous solutions, the performance of high-order accurate schemes, such as the spectral method and high-order finite difference, finite volume, and finite element, will degrade dramatically. Convergence will be completely lost in the strong L1 norm, and it is at most first order in average norms such as the L1 norm. This problem exists already at the approximation level, namely, even the approximation to the discontinuous initial condition cannot be highorder for finite element and spectral methods. A simple example is the Fourier spectral solution for the linear equation (3) and a discontinuous initial condition. There are significant oscillations for the numerical solution near discontinuities, which are called the Gibbs phenomenon, and these spurious oscillations are polluted throughout the computational domain, causing first-order convergence even in smooth regions. Modern non-oscillatory schemes, e.g., the weighted essentially non-oscillatory (WENO) schemes

[23], can remove these spurious oscillations and produce sharp, monotone shock transitions. However, with transition point(s) across the discontinuity, which cannot be avoided by conservative shock-capturing schemes, the error measured by the L1 norm still cannot be higher than first order. Therefore, when measured by the errors in the Lp norms, a high-order accurate scheme seems to have little advantage over a first-order accurate scheme whenever the solution contains discontinuities. This would seem to be a major difficulty in justifying the design and application of high-order schemes for discontinuous problems. One possible way to address this difficulty is to measure the error away from the discontinuities. In such situations many high-order schemes, for example, upwind-based finite difference, finite volume, and discontinuous Galerkin schemes, can achieve the designed high-order of accuracy. For example, it is proved in [6, 27] that, for discontinuous Galerkin methods solving a linear hyperbolic PDE with a discontinuous but piecewise smooth initial condition, the designed optimal order of accuracy is achieved in a weighted L2 norm with the weight used to exclude a roughly O.h1=2 / neighborhood of the discontinuities. For many problems in applications, such high-order accuracy would be desirable and would justify the usage of high-order schemes. However, such measurement of error is not global, leaving open the theoretical issue whether a high-order scheme produces solutions which are truly globally high-order accurate. The proof of high-order accuracy away from the discontinuities is also difficult (see [6,27]), and for coupled hyperbolic systems, in regions between characteristics lines, the error may be only first order even though we measure it away from the discontinuities [19]. A major contribution of mathematics to the design and understanding of algorithms in such a situation is the discovery that many high-order schemes are still high-order accurate for discontinuous solutions, if we measure the error in the negative Sobolev norm. It can be proved, for example, in [16, 19] for finite difference methods, in [14, 20] for spectral methods, and in [26] for discontinuous Galerkin methods, that a high-order scheme is still high-order accurate for a linear hyperbolic PDE, measured in a suitable negative norm, for discontinuous solutions of linear hyperbolic PDEs. For example, a fourth-order accurate scheme

Error Estimates for Linear Hyperbolic Equations

is still fourth-order accurate measured in the jj  jj4 norm, and a spectral method is accurate of k-th order for any k in the negative jj  jjk norm. This, together with a post-processing technique, e.g., those in [2, 10, 11, 16, 19], can recover high-order accuracy in strong norms, such as the usual L2 or L1 norm, in smooth regions of the solution, for any sequence of numerical solutions which converges in the negative norm with high-order accuracy to a discontinuous but piecewise smooth solution. Thus, we can conclude that for a linear hyperbolic PDE with discontinuous but piecewise smooth solutions, a good computational strategy is still to use a high-order accurate numerical method. The numerical solutions may be oscillatory and converge poorly in strong norms, but they do converge in high-order accuracy measured in suitable negative norms. A good post-processor can then be applied to recover highorder accuracy in strong norms in smooth regions of the solution.

References 1. Adjerid, S., Baccouch, M.: Asymptotically exact a posteriori error estimates for a one-dimensional linear hyperbolic problem. Appl. Numer. Math. 60, 903–914 (2010) 2. Bramble, J.H., Schatz, A.H.: Higher order local accuracy by averaging in the finite element method. Math. Comput. 31, 94–111 (1977) 3. Brooks, A.N., Hughes, T.J.R.: Streamline upwind PetrovGalerkin formulations for convection dominated flows with particular emphasis on the incompressible Navier-Stokes equations. Comput. Methods App. Mech. Eng. 32, 199–259 (1982) 4. Cheng, Y., Shu, C.-W.: Superconvergence of discontinuous Galerkin and local discontinuous Galerkin schemes for linear hyperbolic and convection diffusion equations in one space dimension. SIAM J. Numer. Anal. 47, 4044–4072 (2010) 5. Ciarlet, P.G.: Finite Element Method For Elliptic Problems. North-Holland, Amsterdam (1978) 6. Cockburn, B., Guzm´an, J.: Error estimates for the RungeKutta discontinuous Galerkin method for the transport equation with discontinuous initial data. SIAM J. Numer. Anal. 46, 1364–1398 (2008) 7. Cockburn, B., Shu, C.-W.: Runge-Kutta discontinuous Galerkin methods for convection-dominated problems. J. Sci. Comput. 16, 173–261 (2001) 8. Cockburn, B., Luskin, M., Shu, C.-W., S¨uli, E.: Enhanced accuracy by post-processing for finite element methods for hyperbolic equations. Math. Comput. 72, 577–606 (2003) 9. Cockburn, B., Dong, B., Guzm´an, J.: Optimal convergence of the original DG method for the transport-reaction equation on special meshes. SIAM J. Numer. Anal. 46, 1250– 1265 (2008)

445 10. Gottlieb, D., Shu, C.-W.: On the Gibbs phenomenon and its resolution. SIAM Rev. 30, 644–668 (1997) 11. Gottlieb, D., Tadmor, E.: Recovering pointwise values of discontinuous data within spectral accuracy. In: Murman, E.M., Abarbanel, S.S. (eds.) Progress and Supercomputing in Computational Fluid Dynamics, pp. 357–375. Birkh¨auser, Boston (1985) 12. Gottlieb, D., Ketcheson, D., Shu, C.-W.: Strong Stability Preserving Runge-Kutta and Multistep Time Discretizations. World Scientific, Singapore (2011) 13. Gustafsson, B., Kreiss, H.-O., Oliger, J.: Time Dependent Problems and Difference Methods. Wiley, New York (1995) 14. Hesthaven, J., Gottlieb, S., Gottlieb, D.: Spectral Methods for Time-Dependent Problems. Cambridge University Press, Cambridge (2007) 15. Johnson, C., Pitk¨aranta, J.: An analysis of the discontinuous Galerkin method for a scalar hyperbolic equation. Math. Comput. 46, 1–26 (1986) 16. Lax, P.D., Mock, M.: The computation of discontinuous solutions of linear hyperbolic equations. Commun. Pure Appl. Math. 31, 423–430 (1978) 17. Lax, P.D., Wendroff, B.: Systems of conservation laws. Commun. Pure Appl. Math. 13, 217–237 (1960) 18. LeVeque, R.J.: Finite Volume Methods for Hyperbolic Problems. Cambridge University Press, Cambridge (2002) 19. Majda, A., Osher, S.: Propagation of error into regions of smoothness for accurate difference approximations to hyperbolic equations. Commun. Pure Appl. Math. 30, 671– 705 (1977) 20. Majda, A., McDonough, J., Osher, S.: The fourier method for nonsmooth initial data. Math. Comput. 32, 1041–1081 (1978) 21. Richter, G.R.: An optimal-order error estimate for the discontinuous Galerkin method. Math. Comput. 50, 75–88 (1988) 22. Shu, C.-W.: Essentially non-oscillatory and weighted essentially non-oscillatory schemes for hyperbolic conservation laws. In: Cockburn, B., Johnson, C., Shu, C.-W., Tadmor, E., Quarteroni, A. (eds.) Advanced Numerical Approximation of Nonlinear Hyperbolic Equations. Lecture Notes in Mathematics, vol. 1697, pp. 325–432. Springer, Berlin (1998) 23. Shu, C.-W.: High order weighted essentially non-oscillatory schemes for convection dominated problems. SIAM Rev. 51, 82–126 (2009) 24. Shu, C.-W.: Discontinuous Galerkin methods: general approach and stability. In: Bertoluzza, S., Falletta, S., Russo, G., Shu, C.-W. (eds.) Numerical Solutions of Partial Differential Equations. Advanced Courses in Mathematics CRM Barcelona, pp. 149–201. Birkh¨auser, Basel (2009) 25. Yang, Y., Shu, C.-W.: Analysis of optimal superconvergence of discontinuous Galerkin method for linear hyperbolic equations. SIAM J. Numer. Anal. 50, 3110–3133 (2012) 26. Yang, Y., Shu, C.-W.: Discontinuous Galerkin method for hyperbolic equations involving ı-singularities: negativeorder norm error estimates and applications. Numerische Mathematik. doi: 10.1007/s00211-013-0526-8 27. Zhang, Q., Shu, C.-W.: Error estimates for the third order explicit Runge-Kutta discontinuous Galerkin method for linear hyperbolic equation in one-dimension with discontinuous initial data. Numerische Mathematik (to appear)

E

446

Error Estimation and Adaptivity Mats G. Larson and Fredrik Bengzon Department of Mathematics and Mathematical Statistics, Ume˚a University, Ume˚a, Sweden

Synonyms

Error Estimation and Adaptivity

Basic Methodology Model Problem Let  Rd be an open bounded domain with polygonal boundary @ . Consider the weak problem: find u 2 V such that a.u; v/ D l.v/;

8v 2 V

(1)

A posteriori error estimation; Adaptive algorithms; where V is a suitable Hilbert space with norm k  kV . Here a.; / is a continuous bilinear form and l./ is Adaptive finite element methods a continuous linear functional. We assume that a.; / satisfies the following inf-sup condition. There is a constant ˛ > 0 such that

Definition An adaptive finite element method consists of a finite element method, an a posteriori error estimate, and an adaptive algorithm. The a posteriori error estimate is a computable estimate of the error in a functional of the solution or a norm of the error that is based on the finite element solution and not the exact solution. The adaptive algorithm seeks to construct a near optimal mesh for a certain computational goal and determines which elements should be refined based on local information provided by the a posteriori error estimate. This procedure is repeated until a satisfactory solution is obtained.

Overview Adaptive finite elements originate from the works by Babuˇska and Rheinboldt [3, 4] in the late 1970s. Since then this topic has been an active research area, and significant advances have been achieved by several contributors including error estimates based on local problems [1, 16], error estimates based on dual problems for stationary and time-dependent problems [10], dual-weighted residual-based error estimates [6], and convergence of adaptive finite element methods [15, 17]. Recent research directions include extensions to more complex applications and estimates that take uncertainty in models and parameters into account. For a more comprehensive review of these matters, we refer to [2, 5], and [18].

˛kukV  sup v2V

a.u; v/ ; kvkV

8u 2 V

(2)

Then, by virtue of the Lax-Milgram lemma (1) has a unique solution u 2 V . Many important differential equations can be formulated as the variational equation (1). For example, for Poisson’s equation u D f with homogeneous Dirichlet boundary conditions, u D 0 on @ , we have a.u; v/ D .ru; rv/, l.v/ D .f; v/, and V D H01 . / with norm k  k2V D a.; R /. Further, (2) holds with ˛ D 1. Here, .u; v/ D uv denotes the L2 inner product. Other equations that can be formulated in this fashion include the Navier-Lam´e equations for linear elasticity and the Stokes equations of laminar fluid flow. Finite Element Method Let K D fKg be a partition of the domain into elements of size hK D diam.K/, and let Vh  V be a finite dimensional subspace typically consisting of continuous piecewise polynomials of degree at most p on this mesh. Replacing V with Vh in (1), we obtain the finite element method: find uh 2 Vh such that a.uh ; v/ D l.v/;

8v 2 Vh

(3)

Introducing a basis f'j gnj D0 for Vh , we have uh D Pn j D1 uj 'j , where uj are n unknown coefficients that can be determined by solving the following linear system of equations: Au D b

(4)

Error Estimation and Adaptivity

447

where the entries of the n  n matrix A and the n  1 where v 2 Vh is a suitable interpolant of v, for invector b are given by Aij D a.'j ; 'i / and bi D l.'i /, stance, the Scott-Zhang interpolant. Also, ŒnK  ruh D respectively. nK C  ruh jK C C nK   ruh jK  denotes the jump of the normal derivative across the common edge of any two adjacent elements K C and K  , with outward unit Energy Norm A Posteriori Error Estimates Based normals nK C and nK  , respectively. This term arises on Residuals Let e D u  uh denote the error. Using the inf-sup since uh generally only has C 0 continuity. Further, we have the interpolation error estimate condition, we can estimate the error as follows: ˛kekV  sup v2V

 sup v2V

a.e; v/ l.v/  a.uh ; v/  sup kvkV kvkV v2V hR; vi D kRkV  kvkV

kv  vk2K C hK kv  vk2@K  C h2K krvk2N.K/ (11) where N.K/ is the union of all elements that share a (5) node with K. Combining the above results, we arrive at

and thus we obtain the abstract a posteriori error estimate (6) kekV  ˛ 1 kRkV 

hR.uh /; vi  C

X

h2K kf C uk2K

K2K

!1=2 2 Here, we introduced the weak residual R 2 V  , C 14 hK kŒnK  ruh k@Kn@ kvkV defined by hR; vi D l.v/  a.uh ; v/ for all v 2 V , where V  is the dual of V and h; i denotes the duality (12) pairing between V and V  . The dual norm kR.uh /kV  is not directly computable, due to the supremum, and Finally, dividing by kvkV , recalling that ˛ D 1 for therefore we shall seek estimates instead. We consider Poisson’s equation, and taking the supremum, we get two different approaches, where the first is based on a the following estimate: direct estimate in terms of computable residual quantities and the second is based on solving local problems. !1=2 X For simplicity, we restrict our attention to Poisson’s 2 K (13) kekV  kR.uh /kV   C equation. K2K

Estimates based on explicit residuals Using Galerkin orthogonality, or (3), followed by elementwise integration by parts, we obtain

hR.uh /; vi D l.v/  a.uh ; v/

(7)

D l.v  v/  a.uh ; v  v/

(8)

D

X

.f C u; v  v/K

K2K

 12 .ŒnK  ruh ; v  v/@K 

X

(9)

1=2

where K D hK kf C ukK C 12 hK kŒnK  ruh k@Kn@ is the element residual. See [2, 5] and [18], for further details. Estimates Based on Local Problems A more refined way of estimating the residual (see [1] and [14]) is to first compute a so-called equilibrated normal flux †n .uh / on each edge. This flux is an approximation of the normal flux n  ru that satisfies the condition

.f; 1/K C .†nK .uh /; 1/@K D 0;

8K 2 K

(14)

Then, we proceed as follows:

kf C ukK kv  vkK

K2K

hR.uh /; vi D

C 12 kŒnK  ruh k@Kn@ kv  vk@K

X

.f C u; v/K C .nK  ruh ; v/@K

K2K

(10)

(15)

E

448

Error Estimation and Adaptivity

D

X

.f C u; v/K C .nK  ruh

K2K

†nK .uh /; v/@K D

X

(16)

.rEK ; rv/K

(17)

K2K



X

krEK kK krvkK

(18)

K2K

Here, EK is the solution to an elementwise Neumann problem, EK D f C uh and nK  ruh D nK  ruh  †nK .uh /, which is solvable due to (14). We thus arrive at the a posteriori error estimate kekV  kR.uh /kV   C

X

!1=2 2 K

(19)

K2K

where K D krEK kK . Note that in this approach there is no unknown constant in the estimate, which is an advantage in practice. There is also an alternative approach for constructing estimators based on local problems that avoids constructing an equilibrated flux; see [16]. Energy Norm Error Estimates Based on Averaging the Gradient Suppose that we can compute an approximation rh uh of the gradient that is more accurate than the directly evaluated gradient ruh . Then a possible estimator for kru  ruh k is given by krh uh  ruh k. For instance, for piecewise linear elements, we can define rh uh as the L2 projection of the piecewise constant directly evaluated flux ruh on piecewise linear continuous functions, computed using a lumped mass matrix for efficiency. This estimator is known as the ZZ-indicator and was originally proposed in [19]. It has been shown that the averaged gradient approach yields reliable error estimators; see [7].

nomial order of the finite element basis functions locally. We can also have a combination of h- and p-refinement, the so-called hp-refinement. Loosely speaking, h-refinement is efficient when the regularity of the exact solution is low, whereas it is the opposite way around with p-refinement. In the following we shall concentrate on h-refinement and refer to [8] and the references therein for a thorough discussion of hprefinement. One principle for constructing an adaptive algorithm is that we seek a mesh such that the contribution to the error from each element is equal, the so-called equidistribution of the error. A basic approach to construct such a mesh is the following adaptive algorithm: 1. Solve (3) for uh . 2. Compute the element indicators K . 3. Mark elements for refinement based on fK gK2K . 4. Refine marked elements. This procedure is repeated until a stopping criterion is satisfied. For instance, until X

!1=2 2 K

 TOL

(20)

K2K

where TOL is a user-prescribed tolerance. In practice, other restrictions such as computer memory and computing time may also have to be taken into account. The marking of elements is carried out using a refinement criterion. The simplest criterion is to refine element K if K > ˇ maxK 0 2K K 0 with 0  ˇ  1 a user-defined parameter. For numerical stability it is important not to degrade the mesh quality (i.e., element shape) in successive refinements.

Goal-Oriented A Posteriori Error Estimates In practice, one is often interested in computing some specified quantities, for instance, the lift and drag of an airfoil or the average stress in a mechanical component. Adaptive Algorithms Such quantities may be expressed as functionals of the In principle, there are two ways to increase the accu- solution. In order to derive error estimates for funcracy of a finite element solution uh , namely: tional values, we use the so-called duality arguments • h-refinement where an auxiliary dual problem is used to connect • p-refinement the error in the goal functional to the residual of the h-refinement means using a mesh with locally smaller computed solution. The connection is given by a local elements and p-refinement means increasing the poly- weight that typically depends on derivatives of the

Error Estimation and Adaptivity

449

solution to the dual problem, multiplying the local residual. General references include [5,6,10], and [12]. Let m.u/ be a linear functional on V describing the quantity of interest and consider the dual or adjoint problem: find 2 V m.v/ D a.v; /;

8v 2 V

Duality-based techniques are also applicable to time-dependent problems. Here the dual problem is a backward-in-time problem, but the basic principle remains the same as in the stationary case; see [9, 11], and [13].

(21)

Given and choosing v D e in (21), we get the error Cross-References representation formula  A Posteriori Error Estimates of Quantities of Interest  Adaptive Mesh Refinement m.e/ D a.e; / D l. /  a.uh ; / D hR; i (22)  Finite Element Methods Note that this is an equality, which relates the error to a computable weighted residual. As before hR.uh /; i can be further manipulated using Galerkin References orthogonality and interpolation theory to yield a 1. Ainsworth, M., Oden, J.T.: A unified approach to a posposteriori estimates m.e/  C

X

K !K

(23)

2.

K2K

3.

Here, K is the element residual and !K a weight accounting for the dual information. For Poisson’s equation the weight takes the form 1=2

!K D h1 K k   kK C hK  C hK j jH 2 . /

k   k@K

4.

5.

(24) 6.

where  2 Vh is an interpolant, and we finally used an interpolation error estimate. The product K D K !K 7. defines the element indicator in an a duality-based adaptive algorithm. The dual weight !K determines 8. the contribution of the element residual K to the error estimate and thus contains information on which parts of the domain influence the error for a specific goal functional. An adaptive algorithm based on K can 9. therefore generate a mesh that is tailored to efficient computation of a specific functional. In practice, it is necessary to, at least approximately, compute the dual 10. weight, and different approaches have been developed in the literature; see [5]. 11. In order to derive error estimates in other norms than the energy norm, a dual problem is often used in combination with analytical or computational ap12. proaches to estimate the stability properties of the dual problem.

teriori error estimation using element residual methods. Numerische Mathematik 65, 23–50 (1993) Ainsworth, M., Oden, J.T.: A Posteriori Error Estimation in Finite Element Analysis. Wiley, New York (2000) Babuka, I., Rheinboldt, W.C.: A-posteriori error estimates for the finite element method. Int. J. Numer. Methods Eng. 12(10), 1597–1615 (1978) Babuska, I., Rheinboldt, W.C.: Error estimates for adaptive finite element computations. SIAM J. Numer. Anal. 15(4), 736–754 (1978) Bangerth, W., Rannacher, R.: Adaptive Finite Element Methods for Differential Equations. Birkh¨auser, Basel/Boston (2003) Becker, R., Rannacher, R.: An optimal control approach to a posteriori error estimation in finite element methods. Acta Numer. 10, 1–102 (2001) Carstensen, C., Funken, S.A.: Fully reliable localized error control in the fem. SIAM J. Sci. Comput. 21(4), 1465–1484 (1999) Demkowicz, L., Oden, J.T., Rachowicz, W., Hardy, O.: Toward a universal h-p adaptive finite element strategy, part 1. Constrained approximation and data structure. Comput. Methods Appl. Mech. Eng. 77(12), 79–112 (1989) Eriksson, K., Johnson, C.: Adaptive finite element methods for parabolic problems. i: a linear model problem. SIAM J. Numer. Anal. 28(1), 43–77 (1991) Eriksson, K., Estep, D., Hansbo, P., Johnson, C.: Introduction to adaptive methods for differential equations. Acta Numer. 4, 105–158 (1995) Estep, D., Larson, M., Williams, R.: Estimating the error of numerical solutions of systems of nonlinear reaction– diffusion equations. Mem. Am. Math. Soc. 696, 1–109 (2000) Giles, M.B., Suli, E.: Adjoint methods for pdes: a posteriori error analysis and postprocessing by duality. Acta Numerica 11, 145–236 (2002)

E

450 13. Hoffman, J., Johnson, C.: Computational Turbulent Incompressible Flow. Springer, Berlin/London (2007) 14. Ladeveze, P., Leguillon, D.: Error estimate procedure in the finite element method and applications. SIAM J. Numer. Anal. 20(3):485–509 (1983) 15. Morin, P., Nochetto, R.H., Siebert, K.: Convergence of adaptive finite element methods. SIAM Rev. 44(4), 631–658 (2002) 16. Morin, P., Nochetto, R.H., Siebert, K.G.: Local problems on stars: a posteriori error estimators, convergence, and performance. Math. Comput. 72, 1067–1097 (2003) 17. Stevenson, R.: Optimality of a standard adaptive finite element method. Found. Comput. Math. 7, 245–269 (2007) 18. Verfurth, R.: A Review of A Posteriori Error Estimation and Adaptive Mesh Refinement Techniques. Teubner, Stuttgart (1996) 19. Zienkiewicz, O.C., Zhu, J.Z.: A simple error estimator and adaptive procedure for practical engineering analysis. Int. J. Numer. Methods Eng. 24(2), 337–357 (1987)

Euler Equations: Computations Thomas Sonar Computational Mathematics, TU Braunschweig, Braunschweig, Germany

Euler Equations: Computations

Z

@' X @' fk .u/  dx dt D 0 (2) C @t @xk d

u .t;t Ct /

kD1

to hold for all compactly supported test functions '. We find all kinds of methods in use, namely, finite volume, finite difference, finite element, and spectral element methods, where the finite volume method is the one mostly used in aerodynamics. We therefore choose the finite volume method as a paradigmatic discretization method for the Euler equations.

The Finite Volume Method Like any other method, the finite volume method has to start with a weak formulation of the Euler equations. One considers a control volume  Rd with outward unit normal n and surface measure dS . Then the Euler equations are integrated over the .d C 1/-dimensional cylinder .t; t C t/  and apply Gauss’s integral theorem to obtain Z

Z

t Ct

.u.x; t C ıt/  u.x; t// d C

t

I

F .u/  n dS dt D 0:

(3)

@

Basic Facts

The link between this weak form and the weak form The Euler equations governing compressible inviscid (2) is given by Haar’s lemma; see [2]. In particular fluid flow are usually given in the form of a system of Morrey [5] (see also Kl¨otzler [4]) has proved a useful conservation laws: generalization of this kind for cuboids to be used as control volumes, while M¨uller [7] and Bruhn [1] have d @u @u X @fk .u/ extended this result to quite general control volumes. C C r  F .u/ D 0 (1) D @t @xk @t Introducing the cell average operator kD1 where d 2 f1; 2; 3g denotes the space dimension considered, u WD .; v; E/ is the vector of conserved variables, and fk are the fluxes. From the point of view of partial differential equations, the unsteady Euler equations constitute a hyperbolic system. However, if a steady state is reached, the system is hyperbolic only in supersonic regions. The properties of the Euler equations are very well documented in the literature; see, for instance, [3]. Since the Euler equations are derived from physical conservation principles, integral form is most important and is the starting point for numerical methods. The integral form is called weak formulation and requires

A. /u.t/ WD

1 j j

Z u.x; t/ d ;

we can reformulate (3) to result in 1 d A. /u.t/ D  dt j j

I F .u/  u dS dt D 0

(4)

@

where j i j is the measure of i . We now restrict ourselves to the two-dimensional case for the sake of ease of notation. If we cover a domain D  R2 with a conforming triangulation (see [6]) consisting of triangles i and if N.i / WD fj 2 N j i \ j

Euler Methods, Explicit, Implicit, Symplectic

451

is an edge of i g, then (4) can be formulated on a The time stepping can now be done with an appropriate ODE solver. single triangle as R d 1 P A. i /u.t/ D  j j j 2N.i / @ i \@ j dt P2 `D1 f` .u/nij;` dS:

References (5) 1. Bruhn, G.: Erhaltungss¨atze und schwache L¨osungen in der

Here, nij;` is the `th component of the unit normal vector at the edge i \ j which points outwards with respect to i . In order to tackle the line integral, we introduce nG Gauss points xij .s /;  D 1; : : : ; nG , on the edge i \ j and Gaussian weights ! . Then (5) yields X j@ i \ @ j j d A. i /u.t/ D  dt 2j j j 2N.i /

(

nG X 2 X

) ! f` .u.xij .s /; t//nij;` C O.h

2nG

/ : (6)

D1 `D1

In order to derive a numerical method, we introduce Ui .t/ as an approximation to A. i /u.t/ and introduce an approximate Riemann solver .ui ; uj I n/ 7! H.ui ; uj I n/ satisfying the consistency condition 8u W P2 H.u; uI n/ D `D1 f` .u/n` . There is a multitude of approximate Riemann solvers available and readily described in the literature; cp. [3]. It is most simple to work with piecewise constant cell averages Ui , but this results in a scheme being first order in space. Therefore, a recovery step is the most important ingredient in any finite volume scheme. From triangle i and a number of neighboring triangles, one constructs a polynomial pi , defined on i , with the properties pi  u D O.hr /; r 1 for all x 2 i at a fixed time t, and A. i /pi .t/ D A. i /Ui .t/ D Ui . The art to recover such polynomials includes TVD, ENO, and WENO techniques and is described in some detail in [6]. Instead of using Ui and Uj as arguments in the approximate Riemann solver, one employs pi and pj and arrives at X j@ i \ @ j j d Ui .t/ D  dt 2j j

Gasdynamik, Math. Methods Appl. Sci. 7, 470–479 (1985) ¨ 2. Haar, A.: J. Reine Angew.: Uber die Variation der Doppelintegrale, Math 149, 1–18 (1919) 3. Hirsch, C.: Numerical Computation of Internal and External Flows, vol. 2. Wiley, Chichester/New York/Brisbane/ Toronto/Singapore (1990) 4. Kl¨otzler, R.: Mehrdimensionale Variationsrechnung. Birkh¨auser, Basel/Stuttgart (1970) 5. Morrey, C.B.: Multiple integral problems in the calculus of variations and related topics, Ann. Scuola Norm. Pisa (III) 14, 1–61 (1960) 6. Morton, K.W., Sonar, T.: Finite volume methods for hyperbolic conservation laws. In: Acta Numerica, pp. 155– 238. Cambridge University Press, Cambridge/New York/ Melbourne (2007) 7. M¨uller, C.: Grundprobleme der Mathematischen Theorie Elektromagnetischer Schwingungen. Springer, Berlin/ Heidelberg/New York (1957)

Euler Methods, Explicit, Implicit, Symplectic Ernst Hairer and Gerhard Wanner Section de Math´ematiques, Universit´e de Gen`eve, Gen`eve, Switzerland

Euler’s methods for differential equations were the first methods to be discovered. They are still of more than historical interest, because their study opens the door for understanding more modern methods and existence results. For complicated problems, often of very high dimension, they are even today important methods in practical use.

Euler’s Legacy

A differential equation yP D f .t; y/ defines a slope yP at every point .t; y/ where f is defined. A solution curve ) (n G X   y.t/ must respect this slope at every point .t; y.t// ! H pi .xij .s /; t/; pj .xij .s /; t/; nij;` : (7) where y is defined. j 2N.i /

D1

E

452

Euler Methods, Explicit, Implicit, Symplectic

Euler Methods, Explicit, Implicit, Symplectic, Fig. 1 Riccati’s equation with initial value t0 D 1:5; y0 D 1:51744754; Euler polygons 1 1 and 32 (left); for h D 14 ; 18 ; 16 Taylor parabolas of order 2 1 (right) for h D 12 ; 14 ; 18 and 16

y˙ = t2 + y 2, Euler

y˙ = t2 + y 2, Taylor 2

1

1 h=1/8

h=1/4

t2, y2

h=1/32

h=1/16

1

t1, y1

1

h=1/4 h=1/2

−1 t0, y0

t0, y0

Example 1 Some slopes for Riccati’s differential equation yP D t 2 C y 2 are drawn in Fig. 1. We set y 0 the initial value y0 D 1:51744754 for t0 D 1:5, which is chosen such that the exact solution passes through the origin. Euler’s Method Euler, in Art. 650 of his monumental treatise on integral calculus [3], designs the following procedure: Choose a step size h and compute the “valores successivi” y1 , y2 , y3 ; : : : by using straight lines of the prescribed slopes on intervals of size h: tnC1 D tn C h;

ynC1 D yn C hf .tn ; yn / :

(1)

t1, y1

−1 t2, y2

ym

E5 E4

y1

e4

e1

E2

e3

e2

E3

y2 E1

y3 h t0

h t1

y4

h t2

t3

t4

yn tn

Euler Methods, Explicit, Implicit, Symplectic, Fig. 2 Cauchy’s convergence and existence proof

We observe in Fig. 1 (left) that these polygonal lines, or including additional higher order terms. The nufor h ! 0, converge apparently, although slowly, to the merical solution, displayed in Fig. 1 (right), converges much faster than the first order method. correct solution. Euler’s Taylor Methods of Higher Order Some pages later (in Art. 656 of [3]), Euler demon- Cauchy’s Convergence and Existence Proof strates how higher derivatives of the solution can be obtained by differentiating the differential equation, for Before trying to compute the unknown solution of example, a differential equation by numerical means, we must first prove its existence and uniqueness. This research yP D t 2 Cy 2 ) yR D 2t C2y yP D 2t C2yt 2 C2y 3 ; etc., has been started by Cauchy [1]. Cauchy’s proof is illustrated in Fig. 2: which allows to replace formula (1) by piece-wise • Consider for a small step size h the Euler polygons h y0 ; y1 ; : : : ; yn and for a still smaller step size e Taylor polynomials y1; : : : ; e y m on the same interval. the polygons y0 ; e • Suppose that f .t; y/ is continuous, which makes it h2 yRn ynC1 D yn C hyPn C uniformly continuous. 2Š

Euler Methods, Explicit, Implicit, Symplectic

• Therefore, for any  > 0 there is h so small such that jf .t; y/  f .s; z/j <  in compact domains (in our figure sketched as ellipses). • For the “local errors” ei we then obtain jei j < h. • A Lipschitz condition jf .t; y/  f .t; z/j  Ljy  zj allows to obtain jEi j  e L.tn ti / jei j. • Adding up these errors finally leads to je y m  yn j  C , where C depends only on the interval length tn  t0 and L. • This means that, for a step size sequence h1 ; h2 , h3 ; : : : tending to 0, the Euler polygons form a Cauchy sequence and must converge. Later, Cauchy published other convergence proofs based on Taylor series as well as the so-called Picard– Lindel¨of iteration.

Second Order Equations and Systems In the case of a higher order equation, Euler (in Art. 1082 of [4]) applies method (1) component-wise to the solution and lower order derivatives. For example, for the Van der Pol equation yR C .y 2  1/yP C y D 0 this would become yP D v vP D .1  y 2 /v  y which gives

453

explicit Euler method yields stable numerical solutions only for extremely small step sizes.

Implicit Euler Method Euler, who liked to modify his formulas in all possible directions, also arrived at the implicit Taylor methods. The first of these would be ynC1 D yn C hf .tnC1 ; ynC1 / or ynC1 D yn C hvnC1   2 vnC1 D vn C h .1  ynC1 /vnC1  ynC1 in the case of the Van der Pol equation. The polygons now assume the correct slope (or the correct velocity) at the end of each integration step. This requires the solution of (a system of) nonlinear equations at each step, which is usually performed with Newton’s method. In Fig. 3 (right) are presented 25 steps of the implicit Euler method, again with step size h D 0:3. We observe that the instability phenomenon of the explicit method has disappeared. This turns out to be a general property and, despite the difficult implementation of the numerical procedure, the implicit Euler method is the first of the methods which are applicable to very stiff problems (see Sect. IV.1 of [6]).

ynC1 D yn C hvn

Symplectic Euler Method vnC1 D vn C h..1  yn2 /vn  yn /: Figure 3 (left) represents such a numerical solution for a relatively large step size which seems to work reasonably. However, we observe after step number 10 a strange instability phenomenon. This phenomenon becomes more and more serious when  increases, which means that the equation becomes stiff. The same idea applies to systems of equations, which allows to treat initial value problems for such systems as well and also to extend Cauchy’s existence proofs. However, equations of higher dimensions, in particular describing fast chemical reactions or heat transfer, are very often extremely stiff, so that the

This method is important for Hamiltonian problems, which are of the form pP D Hq .p; q/;

qP D Hp .p; q/;

where the Hamiltonian H.p1 ; : : : ; pd ; q1 ; : : : qd / represents the total energy, qi are the position coordinates, and pi the momenta for i D 1; : : : ; d . Hp and Hq are the vectors of partial derivatives. We choose as example the harmonic oscillator with 2 2 H D p2 C q2 , which leads to the equations pP D q and qP D p, and which we can imagine as a body attached to an elastic spring. We show in Fig. 4 (left) some explicit Euler steps

E

454

Euler Methods, Explicit, Implicit, Symplectic

Euler Methods, Explicit, Implicit, Symplectic, Fig. 3 Van der Pol oscillator for  D 1; vectorfield and two exact solutions with y0 D 0:3 and 2, v0 D 0. 20 steps of explicit Euler with h D 0:3, y0 D 1:3; v0 D 0 (left); 25 steps of implicit Euler with h D 0:3, y0 D 2:5; v0 D 0 (right)

p

2

2

1

1 2 y

1

q1, p1

q0, p0

v

v

explicit Euler q 2 , p2

q1, p1 symplectic Euler

1

2

y

q0, p0 p

A1/2 A0 q1, p1

q2, p2 q 3 , p3 q

O

A1 q

O

q3, p3

Euler Methods, Explicit, Implicit, Symplectic, Fig. 4 Explicit Euler versus symplectic Euler at the harmonic oscillator with step size h D 0:5 (left); one step of the symplectic Euler method

qnC1 D qn C hpn ;

pnC1 D pn  hqn

with step size h D 0:75 applied to an initial set A0 (right; in dashed lines the exact solution)

or in general qnC1 D qn C hHp .pn ; qnC1 /

with step size h D 0:5 and initial values q0 D 0; p0 D 1. In the first step, the position q starts off from q0 D 0 with velocity p0 D 1 to arrive at q1 D 0:5, while the velocity p1 D p0 D 1 remains unchanged, because at q0 D 0 there is no force. With this unchanged velocity the voyage goes on from q1 to q2 D 1, and only here we realize that the force has changed. This physical nonsense leads to a numerical solution which spirals outward. An improvement is obtained by updating the velocity with the force at the new position, that is, to use

qnC1 D qn C hpn pnC1 D pn  hqnC1

pnC1 D pn  hHq .pn ; qnC1 /

(2)

(the lower polygon in Fig. 4). Symplecticity Following an idea of Poincar´e, we replace the initial value q0 ; p0 by an entire two-dimensional set A0 (see Fig. 4, right). The first formula of (2) transforms this set into a set A1=2 by a shear mapping, which preserves the lengths of horizontal strips, hence preserves the area of the set A. The second formula then moves A1=2 to A1 by a vertical shear mapping. Therefore, the area of A1 is precisely the same as the area of A0 . This property, which is not true for the explicit Euler method, neither for the implicit, is in general true for the symplectic

Exact Wavefunctions Properties

Euler method applied to all Hamiltonian systems. It is an indicator for the quality of this method, in particular for long time integrations. The great importance of this symplectic Euler method (see [2, 5]), which Euler did not consider, and of its second order companion, Verlet method, was only realized during the second half of the twentieth century, for example, for simulations in molecular dynamics. The same idea appears already in Newton’s Principia (1687), where it was used to justify Newton’s “Theorem 1,” the preservation of the angular momentum in planetary motion.

455

Short Definition The entry discusses the regularity and decay properties of the solutions of the stationary electronic Schr¨odinger equation representing bound states.

Description

¨ The Electronic Schrodinger Equation The Schr¨odinger equation forms the basis of nonrelativistic quantum mechanics and is of fundamental importance for our understanding of atoms and molecules. It links chemistry to physics and describes References a system of electrons and nuclei that interact by Coulomb attraction and repulsion forces. As proposed 1. Cauchy, A.L.: R´esum´e des Lec¸ons donn´ees a` l’Ecole Royale by Born and Oppenheimer in the nascency of quantum Polytechnique. Suite du Calcul Infinit´esimal, 1824. In: Gilain, Chr. (ed.) Equations Diff´erentielles Ordinaires. Johnson mechanics, the slower motion of the nuclei is mostly separated from that of the electrons. This results in the Reprint Corporation, New York (1981) 2. de Vogelaere, R.: Methods of integration which preserve the electronic Schr¨odinger equation, the problem to find contact transformation property of the Hamiltonian equa- the eigenvalues and eigenfunctions of the electronic tions. Tech. report. Department of Mathematics, University Hamilton operator of Notre Dame, Notre Dame (1956) 3. Euler, L.: Institutionum calculi integralis volumen primum, Petropoli impensis academiae imperialis scientiarum ; Enestr. 342, Opera Omnia, Ser. I, vol. 11 (1768) 4. Euler, L.: Institutionum calculi integralis volumen secundum, Petropoli impensis academiae imperialis scientiarum; Enestr. 366, Opera Omnia, Ser. I, vol. 12 (1769) 5. Hairer, E., Lubich, C., Wanner, G.: Geometric Numerical Integration. Structure-Preserving Algorithms for Ordinary Differential Equations. Springer Series in Computational Mathematics, vol. 31, 2nd edn. Springer, Berlin (2006) 6. Hairer, E., Wanner, G.: Solving Ordinary Differential Equations II. Stiff and Differential-Algebraic Problems. Springer Series in Computational Mathematics, vol. 14, 2nd edn. Springer, Berlin (1996)

Exact Wavefunctions Properties Harry Yserentant Institut f¨ur Mathematik, Technische Universit¨at Berlin, Berlin, Germany

Mathematics Subject Classification 35J10; 35B65

H D 

N N 1 1 X 1 X (1) i C V0 .x/ C 2 i D1 2 i;j D1 jxi  xj j i ¤j

written down here in dimensionless form, where

V0 .x/ D 

K N X X i D1 D1

Z jxi  a j

is the nuclear potential. The operator acts on functions with arguments x1 ; : : : ; xN 2 R3 , which are associated with the positions of the considered electrons. The a are the fixed positions of the nuclei and the positive values Z are the charges of the nuclei in multiples of the absolute electron charge. The operator is composed of three parts: The first part, built up from the Laplacians i acting on the positions xi of the single electrons, is associated with the kinetic energy of the electrons. The second, depending on the euclidean distance of the electrons from the nuclei, describes the interaction of the electrons with the nuclei, and the third one the interaction of the electrons among each other. The eigenvalues represent the energies that the system can attain. This entry is concerned with the mathematical

E

456

Exact Wavefunctions Properties

properties of the solutions of this eigenvalue problem, the electronic wavefunctions.

a.u; v/ D  .u; v/

(4)

holds for all test functions v 2 H 1 . The weak form (4) of the eigenvalue equation H u D u particularly fixes The Variational Form of the Equation the behavior of the eigenfunctions at the singularities The solution space of the electronic Schr¨odinger of the interaction potential and at infinity. For normed 1 equation is the Hilbert space H that consists of u, a.u; u/ is the expectation value of the total energy. the one time weakly differentiable, square integrable One can deduce from the estimate (3) that the total functions energy of the system is bounded from below. Hence one is allowed to define the constant u W .R3 /N ! R W .x1 ; : : : ; xN / ! u.x1 ; : : : ; xN / (2) ˇ ˚ ƒ D inf a.u; u/ ˇ u 2 D; kuk0 D 1 ; with square integrable first-order weak derivatives. The norm on H 1 is composed of the L2 -norm k  k0 and the H 1 -seminorm, the L2 -norm of the gradient. the minimum energy that the system can attain. Its In the language of physics, H 1 is the space of the counterpart is the ionization threshold. To prepare its wavefunctions for which the total position probability definition let remains finite and the expectation value of the kinetic ˇ ˚ ˙.R/ D inf a.u; u/ ˇ u 2 D.R/; kuk0 D 1 ; energy can be given a meaning. Let D be the space of the infinitely differentiable functions (2) with bounded support. The functions in D form a dense subset of H 1 . where D.R/ consists of those functions in D for which Let u.x/ D 0 for jxj  R. One can show that the constants †.R/ are bounded from above by the value zero. As K N X N X they are monotonely increasing in R, one can therefore 1 X Z 1 C V .x/ D  define the constant jxi  a j 2 jxi  xj j i D1 D1

i;j D1 i ¤j

† D lim †.R/  0; R!1

be the potential in the Schr¨odinger operator (1). The basic observation is that there is a constant > 0 such the energy threshold above which at least one electron that for all functions u and v in the space D introduced has moved arbitrarily far away from the nuclei, the above Z ionization threshold. We restrict ourselves here to the (3) case that ƒ < † , that is, that it is energetically more V u v dx  kuk0 krvk0 advantageous for the electrons to stay in the vicinity of holds. The proof of this estimate is based on the three- the nuclei than to fade away at infinity. This assumption dimensional Hardy inequality and Fubini’s theorem. implies that the minimum energy ƒ, the ground state energy of the system, is an isolated eigenvalue of finite The expression multiplicity and that the corresponding eigenfunctions, the ground states of the system, decay exponentially. a.u; v/ D .H u; v/ The condition, thus, means that the nuclei can bind all electrons, which is surely not the case for arbitrary 1 defines, therefore, a H -bounded bilinear form on configurations of electrons and nuclei, but of course D, where . ; / denotes the L2 -inner product. It can holds for stable atoms and molecules. The ionization be uniquely extended to a bounded bilinear form on threshold is the bottom of the essential spectrum of H 1 . In this setting, a function u ¤ 0 in H 1 is an the Schr¨odinger operator. This entry discusses the eigenfunction of the electronic Schr¨odinger operator properties of eigenfunctions for eigenvalues below the ionization threshold. (1) for the eigenvalue  if the relation

Exact Wavefunctions Properties

It should be noted that only solutions u 2 H 1 of (4) are physically admissible that have certain symmetry properties. These symmetry properties result from the Pauli principle, the antisymmetry of the full, spindependent wavefunctions with respect to the simultaneous exchange of the positions and spins of the electrons. Since the Hamilton operator (1) does not act on the spin variables and is invariant to the exchange of the electrons, the complete, spin-dependent eigenvalue problem can be decomposed into subproblems for the components of the full, spin-dependent wavefunction. This leads to modified eigenvalue problems in which the solution space H 1 is replaced by subspaces of H 1 underlying corresponding symmetry conditions. The results discussed here transfer without changes to these modified problems. A more detailed discussion of the eigenvalue problem along the lines given here can be found in [13]. Comprehensive survey articles on the spectral theory of Schr¨odinger operators are [5] and [10]. Hydrogen-Like Wavefunctions An exact solution of the electronic Schr¨odinger equation is only possible for one particular but very important case, the motion of a single electron in the field of a single nucleus of charge Z. The knowledge about these eigenfunctions, the weak solutions of the Schr¨odinger equation Z 1 u D u;  u  2 jxj

457

square integrable functions with square integrable firstorder derivative that can be continuously extended by the value f .0/ D 0 to r D 0 and that solve, on the interval r > 0, the radial Schr¨odinger equation 1 2

 f

00

` .` C 1/ C f r2

 

Z f D f: r

The functions satisfying these conditions are the scalar multiples of the functions  Zr   Zr  Zr `C1 .2`C1/ fn` .r/ D exp  2 ; Ln`1 2 n n n .2`C1/

where the Ln`1 .r/ are the generalized Laguerre polynomials of degree n  `  1 with index 2` C 1. The values n D 1; 2; 3; : : : are the principal quantum numbers, the values ` D 0; : : : ; n  1 the angular momentum quantum numbers, and the values m D `; : : : ; ` the magnetic quantum numbers. They classify the orbitals and explain the shell structure of the electron hull. The eigenvalues themselves depend only on the principal quantum number n, a degeneracy that appears in this form only for the Coulomb potential. More details on the hydrogen-like wavefunctions can be found in every textbook on quantum mechanics, for example in [11]. A treatment starting from the variational formulation of the eigenvalue problem is given in [13].

Exponential Decay The hydrogen-like wavefunctions show a behavior that is typical for the solutions of the electronic Schr¨odinger equation. They are strongly localized around the position of the nucleus and are moderately singular there where the particles meet. The study of the localization and decay properties of the wavefunctions began in Z2 ; n D 1; 2; 3; : : :  D  the early 1970s. A first simple result of this type, 2 n2 essentially due to O’Connor [8], is as follows. Let  <  and cluster at the ionization threshold † D 0. The † be an eigenvalue below the ionization threshold  1 assigned eigenfunctions can be expressed in terms of † and p u 2 H be an assigned eigenfunction. For  polar coordinates and composed of eigenfunctions of  < 2 .˙  /, the functions the form 1 x ! exp.jxj/ u.x/; x ! exp.jxj/ ru.x/ u.x/ D f .r/ Y`m .'; /; r where the Y`m are the spherical harmonics and the are then square integrable, that is, u and ru decay radial parts f W R>0 ! R the infinitely differentiable, exponentially in the L2 -sense. That is, the speed of

is basic for the qualitative understanding of chemistry and explains the structure of the periodic table to a large extent. These eigenfunctions have first been calculated by Schr¨odinger [9] in his seminal article. The eigenvalues are

E

458

Exact Wavefunctions Properties

decay depends on the distance of the eigenvalue  under consideration to the bottom † of the essential spectrum. The given bound is optimal in the sense that the decay rate  can in general not be improved further. This can be seen by the example of the hydrogen-like wavefunctions. The exponential decay of the wavefunctions in the L2 -sense implies that their Fourier transforms are real-analytic, that is, can be locally expanded into multivariate power series. The given result is only a prototype of a large class of estimates for the decay of the wavefunctions representing bound states. The actual decay behavior of the wavefunctions is direction-dependent and rather complicated. The in some sense final study is Agmon’s monograph [1]. Agmon introduced the Agmon distance, named after him, with the help of which the decay of the eigenfunctions can be described precisely. A detailed proof of the result above on the L2 -decay of the eigenfunctions can be found in [13]. More information and references to the literature are given in [5].

means, the function v is continuously differentiable on the whole R3N and its first order partial derivatives are H¨older continuous for all indices ˛ in the given interval. Outside the set of points where more than two particles (both electrons and nuclei) meet, the exponential factor even completely determines the singular behavior of the wavefunctions. As has been shown in [3], the regular part v of the wavefunctions is realanalytic outside this set. To reach the bound ˛ D 1, the ansatz has to be modified and an additional term covering three-particle interactions has to be added to the function F . With this modification, Fournais et al. [2] 1;1 , that is, that the first-order have shown that v 2 Cloc derivatives of v become Lipschitz-continuous.

Existence and Decay of Mixed Derivatives The regularity of the electronic wavefunctions increases in a sense with the number of electrons, the reason being that the interaction potential is composed of two-particle interactions of a very specific form. To describe this behavior, we introduce a scale of norms ¨ Holder Regularity that are defined in terms of the Fourier transforms of It cannot be expected that the solutions of the electronic the wavefunctions. Let Schr¨odinger equation are smooth at the singular points N N X Y of the interaction potential. Their singularities at these   2 1 C j! i j2 : j! i j ; Pmix .!/ D places are, however, less strong as one suspects at Piso .!/ D 1 C first view. This can again be seen by the example of the hydrogen-like wavefunctions. The systematic study of the H¨older regularity of the eigenfunctions of electronic Hamilton operators began with the work of Kato [6]. The most recent and advanced results of this type are due to Hoffmann-Ostenhof et al. [4] and Fournais et al. [2]. Hoffmann-Ostenhof et al. [4] and Fournais et al. [2] start from an idea that can be traced back to the beginnings of quantum mechanics and split up the wavefunctions u.x/ D exp.F .x// v.x/

i D1

i D1

The ! i 2 R3 forming together the variable ! 2 .R3 /N can be associated with the momentums of the electrons. The expressions j! i j are their euclidean norms, so that Piso .!/ is a polynomial of degree 2 and Pmix .!/ a polynomial of degree 2N . The norms describing the smoothness of the solutions are now given by the expression Z jjjujjj#;2 m D

Piso .!/m Pmix .!/# jb u.!/j2 d!:

#; m into an explicitly given first part essentially cover- They are defined on the Hilbert spaces Hmix that ing their singularities and a more regular function v. consist of the square integrable functions (2) for which these expressions remain finite. For nonnegative inChoosing teger values m and #, the norms measure the L2 norm of weak partial derivatives. The parameter m X 1 X F .x/ D  Z jxi  a j C jxi  xj j; measures the isotropic smoothness that does not distin2 i x1 > x2 > : : : > xs such that jRs .xi /j D 1 for i D 1; : : : ; s and Rs .xi C1 / D Rs .xi / for i D 1; : : : ; s  1 is used to show that Rs .z/ D Ts .1 C z=s 2 / is indeed the solution of problem (7). We notice that these properties are inherited from corresponding properties of the Chebyshev polynomials. As a consequence, the optimal sequence of fhi gsiD1 is given by hi D t=zi , where zi are the zeros of Rs .z/ and we have jRs .z/j  1 for z 2 Œls ; 0 with ls D 2s 2 (see Fig. 1). The fact that the maximal stability domain on the negative real axis increases quadratically with the number of stages s is crucial to Methods by Composition the success of stabilized Runge–Kutta methods. This approach first proposed by Saul’ev [30] and Guillou and Lago [15] is based on a composition of Euler Complexity and Cost Reduction steps (6) Assume that the accuracy requirement dictates a step size of t and that the Jacobian of problem (1) has gi D gi 1 C hi f .gi 1 /; i D 1; : : : ; s; y1 D gs ; eigenvalues close to the real negative axis with a spec(11) tral radius given by ƒ (possibly large). For a classical explicit Runge–Kutta method, the stability constraint where g0 D y0 , hi D i t, i D 1=zi , and zi are forces to take a step size t=N ' C =ƒ which leads the zeros of the shifted Chebyshev polynomials. The to N D tƒ=C function evaluations per step size gi are called the internal stages of the method. Without t. For example, for the explicit Euler method, this special ordering of the step sizes, internal instabilities cost reads N D tƒ=2. For an explicit stabilized such as roundoff error can propagate within a single Runge–Kutta method with a stability interval along the integration step t in such a way that makes the

Explicit Stabilized Runge–Kutta Methods Explicit Stabilized Runge–Kutta Methods, Fig. 2 Stability domain for shifted Chebyshev polynomials of degree 6. Undamped polynomial (upper figure) and damped polynomial with  D 0:95 (lower figure)

463

5

−60

−40

−20 −5

Chebyshev stab.( s=6)

5

−60

−40

Damped Chebyshev stab.( s=6)

E

−20 −5

numerical method useless [18] (recall that we aim at . Indeed, for the points xi 2 R where R.xi / D using a large number of internal stages, e.g., s 100). Ts .1 C xi =s 2 / D ˙1; the stability domain has zero A strategy to improve the internal stability of the width (see Fig. 2). If one sets method (11) based on a combination of large and small Euler steps has been proposed in [14]. 1  Ts .!0 C !1 z/; !0 D 1 C 2 ; Rs .z/ D T .! / s s 0 Methods by Recurrence Ts .!0 / First proposed by van der Houwen and Sommei!1 D 0 ; (14) jer [34], this approach uses the three-term recurrence Ts .!0 / relation (10) of the Chebyshev polynomials to define a numerical method given by then the polynomials (14) oscillate approximately between 1 C  and 1   (this property is called t g1 D g0 C 2 f .g0 /; “damping”). The stability domain along the negative s real axis is a bit shorter, but the damping ensures that 2t gi D 2 f .gi 1 / C 2gi 1  gi 2 ; i D 2; : : : ; s; a strip around the negative real axis is included in the s stability domain (see Fig. 2). Damping techniques also (12) y1 D gs ; allow to consider hyperbolic–parabolic problems. By where g0 D y0 . One verifies that applied to the test increasing the value of ; a larger strip around the problem y 0 D y, this method gives for the internal negative real axis can be included in the stability domains. This has been considered for explicit stabilized stages Runge–Kutta methods in [33, 36]. Recently damping 2 gi D Ti .1 C t=s /y0 ; i D 0; : : : ; s; (13) techniques have also been used to extend stabilized Runge–Kutta methods for stiff stochastic problems and produces after one step y D g D T .1Cz=s 2 /y : [4, 6]. 1

s

s

0

Propagation of rounding errors within a single step is reasonable for this method even for large values of s such as used in practical computation [34]. Higher-Order Methods Damping It was first observed by Guillou and Lago [15] that one should replace the stability requirement jRs .z/j  1; z 2 Œls ; 0 by jRs .z/j   < 1; z 2 Œls; ; ı , where ı is a small positive parameter depending on

Both problems, constructing optimal stability polynomials and deriving related Runge–Kutta methods, are considerably more difficult for higher order. First, we have to find a polynomial of order p; that is, R.z/ D p 1 C z C : : : C zpŠ C O.zpC1 /, and degree s such that

464

Explicit Stabilized Runge–Kutta Methods

Rs .z/ D 1 C z C : : : C

zp C ˛pC1 zpC1 C : : : C ˛s zs ; pŠ

jRs .z/j  1 for z 2 Œlsp ; 0 ; p

(15)

with ls as large as possible. The existence and uniqueness of such polynomials with maximal real negative stability interval, called optimal stability polynomials, for arbitrary values of p and s have been proved by Riha [29]. No elementary analytical solutions are known for these polynomials for p > 1. Lebedev [23] found analytic expressions for second-order optimal polynomials in terms of elliptic functions related to Zolotarev polynomials. Abdulle [1] gave a classification of the number of complex and real zeros of the optimal stability polynomials as well as bounds for the p error constant Cs D 1=.p C 1/Š  ˛pC1 . In particular, optimal stability polynomials of order p have exactly p complex zeros for even values of p and exactly p  1 complex zeros for odd values of p. In practice such polynomials are approximated numerically [2, 18, 19, 21, 25, 28]. As for first-order optimal stability polynomials, higher-order optimal stability polynomials enjoy a quadratic growth (with s) of the stability region along the negative real axis lsp ' cp  s 2 ;

DUMKA Methods DUMKA methods are based on the zeros of the optimal stability polynomials, computed through an iterative procedure [21]. Then, as suggested by Lebedev in [20, 22], one groups the zeros by pairs (if a zero is complex, it should be grouped with its complex conjugate),considers   quadratic polynomials of the form  1  zzi 1  zzj D 1 C 2˛i z C ˇi z2 , and represents them as gi WD gi 1 C t˛i f .gi 1 / giC1 WD gi C t˛i f .gi /  gi C1 WD giC1  t ˛i 

ˇi ˛i

.f .gi /  f .gi 1 //: (17)

One step of the method consists of a collection of twostage schemes (17). The above procedure allows to represent complex zeros and almost halves the largest Euler step. As for first-order explicit stabilized RK methods, special ordering of the zeros is needed to ensure internal stability. This ordering is done “experimentally” and depends on the degree of the stability polynomial [24]. An extension for higher order has been proposed by Medovikov [27] (order 3 and 4).

c2 D 0:82; c3 D 0:49; c4 D 0:34: (16) RKC Methods p Approximations of ls up to order p D 11 can be found RKC methods rely on introducing a correction to the first-order shifted Chebyshev polynomial to allow for in [2]. Several strategies for approximating the optimal second-order polynomials (Fig. 3). These polynomials, stability polynomials have been proposed. The introduced by Bakker [8], are defined by three main algorithms correspond to the DUMKA methods (optimal polynomials without recurrence Rs .z/ D as C bs Ts .w0 C w1 z/; (18) relation), the Runge–Kutta–Chebyshev (RKC) methods (nonoptimal polynomials with recurrence relation), and the orthogonal Runge–Kutta–Chebyshev where (ROCK) methods (near-optimal polynomials with recurrence relation). The construction of explicit 1 stabilized Runge–Kutta–Chebyshev methods is then based on composition (DUMKA-type methods), recurrence formulas (RKC-type methods), and a combination of composition and recurrence formulas 0 0 −60 −50 −40 −30 −20 −10 (ROCK-type methods). An additional difficulty for methods of order p > 2 is that the structure of the p stability functions 1CzC: : :C zpŠ CO.zpC1 / guaranties −1 the order p only for linear problems. Additional order conditions have to be built in the method to have order Explicit Stabilized Runge–Kutta Methods, Fig. 3 Secondp also for nonlinear problems. Only DUMKA- and order RKC polynomial of degree 9 (bold line). All internal stages are drawn (thin lines) ROCK-type methods exist for p > 2.

Explicit Stabilized Runge–Kutta Methods

465 00

as D 1  bs Ts .w0 /;

bs D

0

w1 D

Ts .w0 / ; Ts00 .w0 /

w0 D 1 C

Ts .w0 / ; .Ts0 .w0 //2  ; s2

 ' 0:15:

Polynomials (18) remain bounded by  ' 1=3 on their stability interval (except for a small interval near the origin). The stability intervals are approximately given by 0:65  s 2 and cover about 80 % of the stability intervals of the optimal second-order stability polynomials. For the internal stages, the polynomials Rj .z/ D aj C bj Tj .w0 C w1 z/; j D 0; : : : ; s  1; can be used. To have consistent internal stages, one must have Rj .0/ D 1 and thus aj D 1  bj Tj .w0 /. It remains to determine b0 ; : : : ; bs1 . If one requires the polynomials Rj .z/ for j 2 to be of second order at nodes t0 C ci t in the interval Œt0 ; t0 C t ; that is, Rj .0/ D 1; .Rj0 .0//2 D Rj00 .0/; then Rj .z/ D 1 C bj .Tj .w0 C w1 z/  Tj .w0 //; with bj D

00

Tj .w0 /

0 .Tj .w0 //2

ROCK Methods The orthogonal Runge–Kutta–Chebyshev methods (ROCK) [2, 3, 7] are obtained through a combination of the approaches of Lebedev (DUMKA) and van der Houwen and Sommeijer (RKC). These methods possess nearly optimal stability polynomials, are built on a recurrence relation, and have been obtained for order p D 2; 4. As the optimal stability polynomials of even order have exactly p complex zeros [1], the idea is to search, for a given p, an approximation of (15) of the form (20) Rs .z/ D wp .z/Psp .z/; where Psp .z/ is a member of a family of polynomials fPj .z/gj 0 orthogonal with respect to the weight funcw .z/2

tion pp 2 . The function wp .z/ is a positive polynomial 1z of degree p. By an iterative process, one constructs wp .z/ such that: • The zeros of wp .z/ are close to the p complex zeros of (15). • The polynomial Rs .z/ satisfies the pth order condition, that is,

for j 2. The parameters b0 ; b1 are free parameters (only first order is possible for R1 .z/ and R0 .z/ is zp Rs .z/ D wp .z/Psp .z/ D 1CzC: : :C CO.zpC1 /: constant) and the values b0 D b1 D b2 are suggested in pŠ [31]. Using the recurrence formula of the Chebyshev polynomials, the RKC method as defined by van der The theoretical foundation of such an approximation Houwen and Sommeijer [34] reads is a theorem of Bernstein [9], which generalizes the property of minimality and orthogonality of Chebyshev g1 D g0 C b1 w1 tf .g0 / polynomials to more general weight functions. For gi D g0 C i t.f .gi 1 /  ai 1 f .g0 // p D 2; 4, such families of polynomials (depending Ci .gi 1  y0 / C i .gi 2  y0 /; i D 2; : : : ; s on s) can be constructed with nearly optimal stability domains. Thanks to the recurrence relation of the y1 D gs ; (19) orthogonal polynomials fPj .z/gj 0 , a method based on recurrence formula can be constructed. where i D

2bi w1 2bi w0 bi ; i D ; i D ; i D 2; : : : ; s: bi 1 bi 1 bi 2

Second-order ROCK2 methods. We consider the polynomials (20) for p D 2 (Fig. 4). The three-term

1 3 −60

−50

−40

−30

−20

−10

0

0

−10

−3

−1

Explicit Stabilized Runge–Kutta Methods, Fig. 4 Second-order ROCK polynomial of degree 9 (thin line) with damping  D 0:95. All internal stages are drawn (bold lines). The optimal stability polynomial is displayed in dotted line

E

466

Explicit Stabilized Runge–Kutta Methods

recurrence formula associated with the polynomials For nonlinear problems, there are four additional order fPj .z/gj 0 conditions that are not encoded in the fourth-order stability polynomials [17, Sect. II.1]. This issue is overcome by using a composition of a s  4 stage Pj .z/ D .˛j z  ˇj /Pj 1 .z/  j Pj 2 .z/; method (based on recurrence relation) with a general fourth-order method having w4 .z/ as stability function is used to define the internal stages of the method such that the resulting one-step method has fourthorder accuracy for general problems. The Butcher g1 D y0 C ˛1 tf .g0 /; group theory [10] is the fundamental tool to achieve gi D y0 C ˛i tf .gi 1 /  ˇi gi 1  i gi 2 ; this construction. An interesting feature of the ROCK4 i D 2; : : : ; s  2: (21) methods is that their stability functions include a strip around the imaginary axis near the origin. Such a prop2 Then, the quadratic factor w2 .z/ D 1 C 2z C z erty (important for hyperbolic–parabolic equations) is represented by a two-stage “finishing procedure” does not hold for second-order stabilized Runge–Kutta methods [3]. similarly as in [22]: gs1 WD gs2 C tf .gs2 / gs? WD gs1 C tf .gs1 /     gs WD gs?  t 1  2 f .gs1 /  f .gs2 / :  (22) For y 0 D y, we obtain gj D Pj .z/g0

j D 0; : : : ; s  2

gs D w2 .z/Ps2 .z/ D Rs .z/y0 ;

(23)

where z D t. Fourth-order ROCK4 methods. We consider the polynomials (20) for p D 4. Similarly to (21), we use the three-term recurrence formula associated with the polynomials fPj .x/gj 0 to define the internal stages of the method g1 ; : : : ; gs4 (Fig. 5). For the finishing procedure, simply implementing successively two steps like (22) will only guarantee the method to be of fourth order for linear problems.

Implementation and Example Explicit stabilized RK methods are usually implemented with variable step sizes, variable stage orders, a local estimator of the error, and an automatic spectral radius estimation of the Jacobian matrix of the differential equation to be solved [3,7,27,32]. A code based on stabilized Runge–Kutta methods typically comprises the following steps. Algorithm 1. Selection of the stage number Given tn ; the current step size, compute an approximation of the spectral radius  of the Jacobian of q (1) and choose the stage number s such as s ' tn cp ; where cp is given by (16). 2. Integration with current stage number and step size Perform an integration step from yn ! ynC1 . 3. Error estimate and step size adjustment Compute the local error errnC1 . If errnC1  T ol, accept the step size and update the integration time t ! t C tn , compute a new step size tnC1 D

1 3 −20

−10

0

0

−10

−3

−1

Explicit Stabilized Runge–Kutta Methods, Fig. 5 Fourth-order ROCK polynomial of degree 9 (thin line) with damping  D 0:95. All internal stages are drawn (bold lines). The optimal stability polynomial is displayed in dotted line

Explicit Stabilized Runge–Kutta Methods

467

10

t

0

x

1

0

x

1

Explicit Stabilized Runge–Kutta Methods, Fig. 6 Integration of the Brusselator problem with the Dormand–Prince method of order 5 (DOPRI5) (left figure) and the ROCK4 method (right figure). A few intermediate stages are also displayed for the ROCK4 method

.errn ; errnC1 ; tn1 ; tn /, and go back to 1. If errnC1 > T ol, reject the step size, compute a new step size tn;new D .errn ; errnC1; tn1 ; tn /, and go back to 1. The function  is a step size controller “with memory” developed for stiff problems [16], and T ol is a weighted average of atol (absolute tolerance) and rtol (relative tolerance). If the spectral radius of the Jacobian is not constant or cannot be easily approximated, it is estimated numerically during the integration process through a power-like method that takes advantage of the large number of internal stages used for stabilized Runge–Kutta methods. Example 2 We consider a chemical reaction, the Brusselator, introduced by Prigogine, Lefever, and Nicolis (see, e.g., [17, I.1] for a description), given by the following reaction–diffusion equations involving the concentration of two species u.x; t/; v.x; t/ W  .0; T / ! R @u D a C u2 v  .b C 1/u C ˛u @t @v D bu  u2 v C ˛v: @t

A spatial discretization (e.g., by finite differences) of the diffusion operator leads to a large system of ODEs. For illustration purpose, we take D .0; 1/ and t 2 .0; 10/ (Fig. 6). We choose to compare the ROCK4

method with a classical efficient high-order explicit Runge–Kutta method, namely, the fifth-order method based on Dormand and Prince formulas (DOPRI5). We integrate the problem with the same tolerance for ROCK4 and DOPRI5 and check that we get the same accuracy at the end. The cost of solving the problem is as follows: number of steps (406 (DOPRI5) and 16 (ROCK4)) and number of function evaluations (2438 (DOPRI5) and 283 (ROCK4)).

E References 1. Abdulle, A.: On roots and error constants of optimal stability polynomials. BIT Numer. Math. 40(1), 177–182 (2000) 2. Abdulle, A.: Chebyshev methods based on orthogonal polynomials. Ph.D. thesis No. 3266, Department of Mathematics, University of Geneva (2001) 3. Abdulle, A.: Fourth order Chebyshev methods with recurrence relation. SIAM J. Sci. Comput. 23(6), 2041–2054 (2002) 4. Abdulle, A., Cirilli, S.: Stabilized methods for stiff stochastic systems. C. R. Math. Acad. Sci. Paris 345(10), 593–598 (2007) 5. Abdulle, A., Cirilli, S.: S-ROCK: Chebyshev methods for stiff stochastic differential equations. SIAM J. Sci. Comput. 30(2), 997–1014 (2008) 6. Abdulle, A., Li, T.: S-ROCK methods for stiff Ito SDEs. Commun. Math. Sci. 6(4), 845–868 (2008) 7. Abdulle, A., Medovikov, A.: Second order Chebyshev methods based on orthogonal polynomials. Numer. Math. 90(1), 1–18 (2001) 8. Bakker, M.: Analytical aspects of a minimax problem. Technical note TN 62 (in Dutch), Mathematical Centre, Amsterdam (1971) 9. Bernstein, S.: Sur les polynˆomes orthogonaux relatifs a` un segment fini. J. Math. 9, 127–177 (1930) 10. Butcher, J.: The effective order of Runge-Kutta methods. In: Morris, T.L.L. (ed.) Conference on the Numerical Solution of Differential Equations. Lecture Notes in Mathematics, vol. 109, pp. 133–139. Springer, Berlin (1969) ¨ 11. Courant, R., Friedrichs, K., Lewy, H.: Uber die partiellen Differenzenglechungen der mathematischen Physik. Math. Ann. 100(32), 32–74 (1928) 12. Dahlquist, G.G.: A special stability problem for linear multistep methods. Nord. Tidskr. Inf. Behandl. 3, 27–43 (1963) 13. Franklin, J.: Numerical stability in digital and analogue computation for diffusion problems. J. Math. Phys. 37, 305– 315 (1959) 14. Gentzsch, W., Schl¨uter, A.: Uber ein Einschrittverfahren mit zyklischer Schrittweiten¨anderung zur L¨osung parabolischer Differentialgleichungen. Z. Angew. Math. Mech. 58, 415– 416 (1978) 15. Guillou, A., Lago, B.: Domaine de stabilit´e associ´e aux formules d’int´egration num´erique d’´equations diff´erentielles, a` pas s´epar´es et a` pas li´es. Recherche de formules a` grand rayon de stabilit´e pp. 43–56 (1960)

468 16. Gustafsson, K.: Control-theoric techniques for stepsize selection in implicit Runge–Kutta methods. ACM Trans. Math. Softw. 20, 496–517 (1994) 17. Hairer, E., Nørsett, S., Wanner, G.: Solving Ordinary Differential Equations I: Nonstiff Problems. Springer Series in Computational Mathematics, vol. 8. Springer, Berlin (1993) 18. Houwen, P.V.: Construction of Integration Formulas for Initial Value Problems, vol. 19. North Holland, Amsterdam/New York/Oxford (1977) 19. Houwen, P.V., Kok, J.: Numerical Solution of a Minimax Problem. Report TW 124/71 Mathematical Centre, Amsterdam (1971) 20. Lebedev, V.: Explicit difference schemes with time-variable steps for solving stiff systems of equations. Sov. J. Numer. Anal. Math. Model. 4(2), 111–135 (1989) 21. Lebedev, V.: A new method for determining the zeros of polynomials of least deviation on a segment with weight and subject to additional conditions. part i, part ii. Russ. J. Numer. Anal. Math. Model. 8(3), 195–222; 397–426 (1993) 22. Lebedev, V.: How to solve stiff systems of differential equations by explicit methods. In: Marchuk, G.I. (ed.) Numerical Methods and Applications, pp. 45–80. CRC Press, Boca Raton (1994) 23. Lebedev, V.: Zolotarev polynomials and extremum problems. Russ. J. Numer. Anal. Math. Model. 9, 231–263 (1994) 24. Lebedev, V., Finogenov, S.: Explicit methods of second order for the solution of stiff systems of ordinary differential equations. Zh. Vychisl. Mat. Mat. Fiziki. 16(4), 895–910 (1976) 25. Lomax, H.: On the construction of highly stable, explicit numerical methods for integrating coupled ordinary differential equations with parasitic eigenvalues. NASA technical note NASATND/4547 (1968) 26. Markov, A.: On a question of Mendeleev. Petersb. Proc. LXII, 1–24 (1890). (Russian) 27. Medovikov, A.: High order explicit methods for parabolic equations. BIT 38, 372–390 (1998) 28. Metzger, C.: M´ethodes Runge–Kutta de rang sup´erieur a` l’ordre. Th`ese troisi`eme cycle Universit´e de Grenoble (1967) 29. Riha, W.: Optimal stability polynomials. Computing 9, 37– 43 (1972) 30. Saul’ev, V.: Integration of parabolic type equations with the method of nets. Fizmatgiz, Moscow (1960). (in Russian) 31. Sommeijer, B., Verwer, J.: A performance evaluation of a class of Runge–Kutta–Chebyshev methods for solving semi-discrete parabolic differential equations. Report NW91/80, Mathematisch Centrum, Amsterdam (1980) 32. Sommeijer, B., Shampine, L., Verwer, J.: RKC: an explicit solver for parabolic PDEs. J. Comput. Appl. Math. 88, 316– 326 (1998) 33. Torrilhon, M., Jeltsch, R.: Essentially optimal explicit Runge–Kutta methods with application to hyperbolicparabolic equations. Numer. Math. 106(2), 303–334 (2007) 34. Van der Houwen, P., Sommeijer, B.: On the internal stage Runge–Kutta methods for large m-values. Z. Angew. Math. Mech. 60, 479–485 (1980) 35. Verwer, J.: Explicit Runge–Kutta methods for parabolic partial differential equations. Appl. Numer. Math. 22, 359– 379 (1996)

Exponential Integrators 36. Verwer, J.G., Sommeijer, B.P., Hundsdorfer, W.: RKC timestepping for advection-diffusion-reaction problems. J. Comput. Phys. 201(1), 61–79 (2004) 37. Yuan, C.D.: Some difference schemes of solution of first boundary problem for linear differential equations with partial derivatives. Thesis cand.phys.math. Sc., Moscov MGU (1958). (in Russian)

Exponential Integrators Alexander Ostermann Institut f¨ur Mathematik, Universit¨at Innsbruck, Innsbruck, Austria

Mathematics Subject Classification 65L04; 65L06; 65J08

Definition Exponential integrators are numerical methods for stiff and/or highly oscillatory problems. Typically, they make explicit use of the matrix exponential and related functions.

Description Exponential integrators form a class of numerical methods for the time integration of stiff and highly oscillatory systems of differential equations. The basic idea behind exponential integrators is to solve a nearby problem exactly and to use this result for the solution of the original problem. Exponential integrators were first considered by Hersch [3], see [7, Sect. 6] for more details on their history. Below, we consider three typical instances of exponential integrators: (i) Exponential quadrature rules for linear evolution equations; (ii) Exponential integrators for semilinear evolution equations; (iii) Exponential methods for highly oscillatory problems.

Exponential Integrators

469

For more methods (exponential multistep and gen- Example 3 For s D p D 2 the order conditions (5) eral linear methods) and specific applications in sci- determine the weights ence and engineering, we refer to the review article [7]. c2 1 '1 .z/  '2 .z/; b1 .z/ D Notations For the numerical solution of the initial c2  c1 c2  c1 value problem c1 1 '1 .z/ C '2 .z/: (8) b2 .z/ D    0 c  c c  c1 2 1 2 (1) u .t/ D F t; u.t/ ; u.t0 / D u0 ; The particular choice c1 D 0 and c2 D 1 yields the we consider one-step methods that determine for a exponential trapezoidal rule. given approximation un u.tn / at time tn and a chosen step size hn the subsequent approximation unC1

u.tnC1 / at time tnC1 D tn C hn . Semilinear Evolution Equations Consider a semilinear problem of the form

Exponential Quadrature Rules The solution of the linear evolution equation u0 .t/ C Au.t/ D f .t/;

u.t0 / D u0 ;

u0 .t/ C Au.t/ D g.t; u.t//; (2)

u.t0 / D u0 ;

(9)

where A is the generator of a strongly continuous where A is the generator of a strongly continuous semigroup (or an N  N matrix). We assume that semigroup (or simply a matrix) satisfies the variation- (9) has a temporally smooth solution. Parabolic probof-constants formula lems can be written in this form either as an abstract evolution equation on a suitable function space or Z 1 N u.tnC1 / D ehn A u.tn /Chn e.1 /hn Af .tn Chn / d: as a system of ordinary differential equations in R 0 stemming from a spatial discretization. (3) This representation motivates the following numerical Exponential Runge–Kutta Methods scheme: For the solution of the semilinear problem (9) we cons sider the following class of (explicit) one-step methods X bi .hn A/f .tn Cci hn /; (4) unC1 D ehn A un Chn s X i D1 hn A un C hn bi .hn A/g.tn C ci hn ; Uni /; unC1 D e i D1 which is called an exponential quadrature rule with weights bi .hA/ and nodes ci . Expanding f in a i 1 X Taylor series at tn in (3) and (4), and comparing both Uni D eci hn A un Chn aij .hn A/g.tn Ccj hn ;Unj / expansions gives the order conditions j D1 (10) s j 1 X ci D 'j .z/; bi .z/ j D 1; : : : ; p (5) with c1 D 0 and Un1 D un . The coefficients aij and .j  1/Š bi are constructed from exponential and related funci D1 tions or (rational) approximations of such functions. for order p (see [7, Sect. 2] for details). Here, 'j .z/ Method (10) is called an s-stage exponential Runge– denote the entire functions Kutta method, see [6, 7]. For A D 0 it reduces to an explicit Runge–Kutta method with coefficients bi D Z 1 j 1 .1 /z .0/ and aij D aij .0/. b i d ; j 1: (6) e 'j .z/ D .j  1/Š 0 Example 4 The well-known exponential Euler method They satisfy 'j .0/ D 1=j Š and the recurrence relation 'j .z/  'j .0/ ; 'j C1 .z/ D z

'0 .z/ D ez :

unC1 D ehn A un C hn '1 .hn A/g.tn ; un /;

(7) is a first-order scheme with one stage (s D 1).

(11)

E

470

Exponential Integrators

Example 5 A one-parameter family of second-order It has order 2 and is computationally attractive since it methods with two stages (s D 2) is given by requires only one matrix function per step. b1 .z/ D '1 .z/ 

1 '2 .z/; c2

b2 .z/ D

Exponential Rosenbrock-type methods for nonautonomous problems are obtained by applying (15) to the autonomous form of the problem, see [9].

1 '2 .z/; c2

a21 .z/ D c2 '1 .c2 z/:

(12)

Exponential Rosenbrock-Type Methods Exponential Runge–Kutta methods integrate the nonlinearity g in an explicit way. Their step size is thus determined by the Lipschitz constant of g which results in small time steps if the linearization is out-dated. A remedy are Rosenbrock-type methods, see [8, 9, 11]. For the time discretization of the autonomous problem   (13) u0 .t/ D F u.t/ ; u.t0 / D u0 ;

Example 7 The exponential Rosenbrock–Euler method for non-autonomous problems is given by unC1 D un C hn '1 .hn Jn /F .tn ; un / C h2n '2 .hn Jn /vn ; (17a) with Jn D

@F .tn ; un /; @u

vn D

@F .tn ; un /: @t

(17b)

This scheme was already proposed by Pope [10]. exponential Rosenbrock-type methods use a continuous linearization of (13) along the numerical solution Highly Oscillatory Problems un , viz. Problems with purely imaginary eigenvalues of large   modulus are challenging for explicit and implicit meth(14a) u0 .t/ D Jn u.t/ C gn u.t/ ; ods: whereas the former simply lack stability, the latter @F tend to resolve all the oscillations in the solution. Jn D DF .un / D .un /; @u Consequently, both methods have to use small time     gn u.t/ D F u.t/  Jn u.t/; (14b) steps. Exponential integrators treat the (linear) oscillations exactly and can therefore use larger time steps. The with Jn denoting the Jacobian of F and gn the nonlinerror of the numerical solution is typically bounded ear remainder evaluated at un , respectively. The numerindependently of the highest frequencies arising in ical schemes make explicit use of these quantities. By applying an exponential Runge–Kutta method the problem. Applications range from Schr¨odinger to (13), we get an exponential Rosenbrock-type method equations with time-dependent potential to Newtonian equations of motion and semilinear wave equations. Below, we discuss two types of methods: Magnus inteunC1 D un C hn '1 .hn Jn /F .un / grators and trigonometric integrators. For more details s X   and other methods, we refer to [2, 7]. bi .hn Jn / gn .Uni /  gn .un / ; Chn i D2

Uni D un C hn ci '1 .ci hn Jn /F .un / Chn

i 1 X

  aij .hn Jn / gn .Unj /  gn .un / ;

j D2

Magnus Integrators Let A.t/ be a time dependent matrix. The exact solution of the initial value problem 0

.t/ D A.t/ .t/;

.0/ D

(15)

(18)

can be represented in the form

again with c1 D 0 and consequently Un1 D un . Example 6 The well-known exponential Rosenbrock– Euler method is given by unC1 D un C hn '1 .hn Jn /F .un /:

0

.tn C h/ D e n .h/ .tn /;

(19)

where the matrix n .h/ can be expanded in a so-called (16) Magnus series

Exponential Integrators

Z

471

q 00 .t/ D  2 q.t/ C g.q.t//; q.0/ D q0 ; q 0 .0/ D p0 ; (24)

h

n .h/ D

An ./ d 0



C

C

1 2 1 4

 An ./ d; An ./ d

Z h Z 0

 0

Z h Z  Z 0

1 12

0

  An ./ d; An ./ d; An ./ d



0

Z h Z

Z



An ./ d; 0

we use its equivalent formulation as a first-order system with q 0 .t/ D p.t/ and apply the variation-ofconstants formula to get





An ./ d; An ./

0

d

0

C  (20)

q.t/ D cos.t /q.0/ C 1 sin.t /p.0/ Z t   C 1 sin .t  s/ g.q.s// ds;

E

0

p.t/ D  sin.t /q.0/ C cos.t /p.0/ Z t   C cos .t  s/ g.q.s// ds:

(25)

0

with An .s/ D A.tn C s/ and ŒA; B D AB  BA denoting the matrix commutator. This representation Approximating the integrals by the trapezoidal rule motivates us to look for a numerical approximation yields a first numerical method .tnC1 / of the form nC1

nC1

De

n

qnC1 D cos.h /qn C 1 sin.h /pn n;

(21)

C 1 h 1 sin.h /g.qn /; 2

where n is an approximation to n .hn /. An extensive review on such Magnus integrators is given in [1].

(26)

pnC1 D  sin.h /qn C cos.h /pn   C 1 h cos.h /g.qn / C g.qnC1 / : (27)

2 Example 8 Truncating the series after the first term and approximating the integral by the midpoint rule If the solution of (24) has highly oscillatory compoyields the exponential midpoint rule nents, the use of filter functions is recommendable. We therefore consider the more general class of one-step hn A.tn Chn =2/ (22) schemes given by nC1 D e n;

which is a second-order scheme.

qnC1 D cos.h /qn C 1 sin.h /pn

Example 9 Truncating the series after the second term and using thepGaussian quadrature rule with nodes c1;2 D 1=2  3=6 yields a fourth-order scheme with   n D 1 hn A.tn C c1 hn / C A.tn C c2 hn / C

h

2

pnC1 D  sin.h /qn C cos.h /pn   C 1 h 0 g.˚ qn / C 1 g.˚ qnC1 / ; 2

2

p

C 1 h2 g.˚ qn /;

i

3 2 h A.tn C c2 hn /; A.tn C c1 hn / : 12 n

(28)

where ˚ D .h /,  D .h /, 0 D 0 .h / and (23) 1 D 1 .h / with suitable filter functions , , 0 and 1 , see [2, Chap. XIII]. The method is symmetric if and only if and are even, and This method requires two evaluations of A and one commutator in each time step. ./ D 1 ./ sinc ; 0 ./ D 1 ./ cos ; (29) Trigonometric Integrators Let be a symmetric positive definite matrix with where sinc  D sin =. For appropriate initial values, possibly large norm. For the numerical solution of the a symmetric one-step method (28), (29) is equivalent semilinear problem to the following two-step formulation

472

Extended Finite Element Method (XFEM)

qnC1  2 cos.h /qn C qn1 D h2 .h / g. .h /qn /: Extended Finite Element Method (XFEM) (30) Geometric properties of these methods are discussed Nicolas Mo¨es in [2]. Ecole Centrale de Nantes, GeM Institute, UMR CNRS 6183, Nantes, France Example 10 The particular choice  2

./ D sinc2 ;

   sinc 

./ D 1 C 1 sin2 3

2

Synonyms

yields a method with a small error constant, see [5].

Extended finite element method; also related to partition of unity Finite element method (PUFEM); Matrix Functions closely related to the Generalized finite element The implementation of exponential integrators requires method (GFEM); XFEM the numerical evaluation of matrix functions or products of matrix functions with vectors. The efficiency of the integrators strongly depends on how these apDefinition proximations are evaluated. For small scale problems, a review of current methods is given in [4]. For largeThe extended finite element method is an extension of scale problems, the use of Chebyshev methods, Krylov the finite element method allowing the introduction of subspace methods, Leja interpolation or contour intediscontinuous or special approximations inside finite gral methods is recommended, see [7, Sect. 4]. elements. This introduction is carried out with a socalled partition of unity technique. The discontinuities considered may be strong (in the field) or weak (in the References gradient). With the XFEM, the mesh no longer needs 1. Blanes, S., Casas, F., Oteo, J., Ros, J.: The Magnus expan- to conform to physical boundaries (cracks, material insion and some of its applications. Phys. Rep. 470, 151–238 terface, free surfaces, . . . ). The location of boundaries (2009) are stored independently of the mesh. The level set 2. Hairer, E., Lubich, C., Wanner, G.: Geometric Numerical representation is particularly well suited in conjunction Integration, Structure-Preserving Algorithms for Ordinary with the XFEM. Differential Equations. Springer, Berlin/Heidelberg (2006) 3. Hersch, J.: Contribution a` la m´ethode des e´ quations aux diff´erences. Z. Angew. Math. Phys. 9, 129–180 (1958) 4. Higham, N.J., Al-Mohy, A.H.: Computing matrix functions. Acta Numer. 19, 159–208 (2010) 5. Hochbruck, M., Lubich, C.: A Gautschi-type method for oscillatory second-order differential equations. Numer. Math. 83, 403–426 (1999) 6. Hochbruck, M., Ostermann, A.: Explicit exponential Runge–Kutta methods for semilinear parabolic problems. SIAM J. Numer. Anal. 43, 1069–1090 (2005) 7. Hochbruck, M., Ostermann, A.: Exponential integrators. Acta Numer. 19, 209–286 (2010) 8. Hochbruck, M., Lubich, C., Selhofer, H.: Exponential integrators for large systems of differential equations. SIAM J. Sci. Comput. 19, 1552–1574 (1998) 9. Hochbruck, M., Ostermann, A., Schweitzer, J.: Exponential Rosenbrock-type methods. SIAM J. Numer. Anal. 47, 786– 803 (2009) 10. Pope, D.A.: An exponential method of numerical integration of ordinary differential equations. Commun. ACM 8, 491– 493 (1963) 11. Strehmel, K., Weiner, R.: Linear-implizite Runge–Kutta Methoden und ihre Anwendungen. Teubner, Stuttgart (1992)

Overview The finite element approach is a versatile tool to solve elliptic partial differential equation like elasticity models (Laplacian-type operator). The mesh needs however to conform to physical boundaries across which the unknown field may be discontinuous. For instance, crack growth in elastic solids requires the mesh to follow the crack path, and remeshing at every stage of propagation is mandatory. The reason for this is that finite element approximations are continuous inside each elements. A discontinuity may thus only appear on the element boundaries. The idea behind the XFEM is to inject a field discontinuity right inside the element. In order to introduce discontinuous approximation inside the element, the XFEM uses the partition of unity concept. Basically, new degrees of freedom

Extended Finite Element Method (XFEM)

473

N1

are introduced which act on classical approximation functions multiplied by a discontinuous function across the boundary that needs to be modeled.

N2

N3

u= 0

Basic Methodology

N4

u= 1

Extended Finite Element Method (XFEM), Fig. 1 The 1D model problem with four finite elements

A Discontinuity in a 1D Diffusion Equation We consider the following 1D diffusion problem:

N1

N2−

N2+

N3

N4

u00 .x/ D 0 for x 2 D .0; 1/; u.0/ D 0; u.1/ D 1; (1) u=1 for which the solution is obviously u D x. Defining the u =0 space of trial and test functions, Extended Finite Element Method (XFEM), Fig. 2 The 1D model problem with a discontinuity located at node 2

U D fu 2 V; u.0/ D 0; u.1/ D 1g; U0 D fv 2 V; v.0/ D v.1/ D 0g;

(2)

where V is the space with the proper regularity for the solution, we may express the variational form associated to the strong form (1), Z

1

u 2 U;

0 0

u v dx D 0;

8v 2 U0 :

(3)

0

u00 .x/ D0 for x 2 =xc ; u.0/ D 0; u.1/ D 1; u0 .xc / Du0 .xcC / D 0:

(6)

Defining the proper spaces

Let us now discretize the domain using four finite elements of equal size. The finite element approximation, uh , has three degrees of freedom as well as test functions v h : 3 X

and enforcing traction free boundary conditions on the crack lips. The problem now reads

U c D fu 2 V 0 ; u.0/ D 0; u.1/ D 1g;

(7)

U0c D fv 2 V 0 ; v.0/ D v.1/ D 0g;

(8)

3 X

vi Ni .x/: where V 0 is the space with the proper regularity of the solution now allowing discontinuity across xc . The (4) variational principle is now Finite element approximation functions, Ni , are depicted in Fig. 1. The finite element problem is given Z 1 below and yields the exact solution: u 2 Uc; u0 v 0 dx D 0; 8v 2 U0c : (9)

uh .x/ D

ui Ni .x/CN4 .x/;

i D1

v h .x/ D

i D1

0

2

32 3 2 3 2 1 0 0 u1 4 1 2 1 5 4 u2 5 D 4 0 5 u3 0 1 2 1 3 2 3 2 1=4 u1 ) 4 u2 5 D 4 1=2 5 : 3=4 u3

Regarding the discretization, we first consider a crack located at node 2 (see in Fig. 2). Trial and test functions are (5)

We now consider that the field may be discontinuous at x D xc and enforce zero Neumann boundary conditions at xcC and xc . If u is a displacement field, this amounts physically to place a crack at x D xc

uh .x/ Du1 N1 .x/ C u2 N2 .x/ C u2C N2C .x/ C u3 N3 .x/ C N4 .x/;

(10)

v h .x/ Dv1 N1 .x/ C v2 N2 .x/ C v2C N2C .x/ C v3 N3 .x/;

(11)

E

474

Extended Finite Element Method (XFEM)

Extended Finite Element Method (XFEM), Fig. 3 Enrichment functions for a discontinuity located at a node (left) and in between two nodes (right)

N3H(x) 1

2

N2H(x)

2 1 6 1 1 6 4 0 0 0 0

)

2 3 0 0 0 u1 6 u2 7 6 0 7 0 0 7 7D6 7 76 1 1 5 4 u2C 5 4 0 5 u3 1 2 1 3 2 3 2 u1 0 6 u2 7 6 0 7 7 6 7 6 4 u2C 5 D 4 1 5 : u3 1 32

3

1

2

3

4

N2H(x)

leading to the finite element problem and the exact solution: 2

H(x)

H(x)

3 3 2 0 u1 6 u2 7 6 1=2 7 7 6 7D6 4 u3 5 4 1 5 : 1=2 a 2

)

3

(15)

Note that the upper 3  3 matrix is the same matrix as in (5) for which there was no crack. We now consider that the crack is no longer located at node 2, but in between node 2 and 3, the approximation is now (12) slightly different from (13) and reads uh .x/ Du1 N1 .x/ C u2 N2 .x/ C u3 N3 .x/ C N4 .x/

C aN2 .x/H.x/ C bN3 .x/H.x/: (16) Introducing the Heaviside (in fact generalized since it goes from 1 to C1) function, depicted in Fig. 3 In the above both nodes 2 and 3 are enriched because (left), the finite element approximation (10) may be their support is split by the crack. The approximation rewritten as functions are drawn in Fig. 3 (right). If one only enriches node 2 or 3, the jump in u and its derivative uh .x/ Du1 N1 .x/ C u2 N2 .x/ C u3 N3 .x/ C N4 .x/ across the discontinuity will not be independent. Assuming the cut on elements 2-3 is at 1/3 close to C aN2 .x/H.x/; (13) node 2, the stiffness matrix is given below. Note that since approximation functions are discontinuous, the where we have introduced the average and (half) jump integration on element 2-3 is performed in two parts: as new degrees of freedom: 32 3 2 u1 2 1 0 1 0 7 6 u2 7 6 1 2 1 2=3 1=3 u2C  u2C u2C C u2 76 7 6 ; aD : (14) u2 D 6 7 6 0 1 2 1=3 4=3 7 2 2 7 6 u3 7 6 4 1 2=3 1=3 2 1 5 4 a 5 We observe that in approximation (13), there are classical approximation functions N1 , N2 , N3 , and N4 as well as a so-called enriched approximation function which is the product of the classical approximation function N2 and the enrichment function H.x/. One may assemble the stiffness matrix, and it will give the proper solution. 2

2 6 1 6 4 0 1

1 0 2 1 1 2 0 1

2 3 1 0 u1 6 u2 7 6 0 7 0 7 76 7 D 6 7 1 5 4 u3 5 4 1 5 a 2 0 32

3

0 1=3 4=3 2 3 0 607 6 7 7 ) D6 617 405 1

1 2

2 3

b

3 u1 0 6 u2 7 6 1=2 7 6 7 6 7 6 u3 7 D 6 1=2 7 : 6 7 6 7 4 a 5 4 1=2 5 1=2 b 2

A Derivative Discontinuity in a 1D Diffusion Equation Consider the following problem: .a.x/u0 .x//0 D0 for x 2 =xc ;

a.x/ D 1;

(17)

Extended Finite Element Method (XFEM)

475

x 2 .0; xc /; a.x/ D ˛; x 2 .xc ; 1/ (18) u.0/ D0; u.1/ D 1; a.xc /u0 .xc / Da.xcC /u0 .xcC /:

R(x)

(19) 1

The solution to this problem is uD

˛x ; x 2 .0; xc /; 1 C .˛  1/xc

uD

x C .˛  1/xc ; x 2 .xc ; 1/: 1 C .˛  1/xc

2

3

4

Extended Finite Element Method (XFEM), Fig. 4 Ridge enrichment for jump in the derivative N1

N2

N3

N4

(20) u=0

The solution is continuous but suffers a derivative jump at x D xc . The poor regularity of standard finite element will take this into account without any effort if the interface xc is placed at a node. If, however, the interface falls inside an element, an enrichment is needed. In the XFEM spirit, there are two ways to proceed. The first one is to simply consider the approximation of type (13) and to introduce a Lagrange multiplier to force the discontinuity at point xc to be zero. Since the Heaviside enrichment provides both jumps in u and its derivative, the jump in derivative inside the element will be modeled. The second approach consists in replacing the Heaviside enrichment in (13) by a so-called ridge enrichment, R.x/, which is depicted in Fig. 4. Domain Discontinuity in a 1D Diffusion Equation Finally, we consider a last important type of discontinuity that may take place inside a finite element: the case of a domain discontinuity. We consider again Eq. (1), but the mesh is now too large for the domain as depicted in Fig. 5. The finite element approximation over the domain will again read as (4), but it will now only be integrated only over the domain .0; 1/ (gray zone). Note that even though node 4 lies outside the domain of interest, its area of action intersects the domain of interest. The degree of freedom associated to node 4 should thus be kept to define the approximation. The fact that displacement is prescribed inside the element (u.1/ D 1) may be taken into account by the use of a Lagrange multiplier.

u =1

Extended Finite Element Method (XFEM), Fig. 5 A finite element mesh larger than the domain of interest

Extension to 2D, 3D, and Level Set Representation In the previous section, we detailed how the XFEM was enriching finite elements to account for the presence of discontinuity. The decision of whether a node should be enriched is taken by looking at its support. The support of a node is the set of elements connected to it. If the finite element approximation is higher order, degrees of freedom are not only attached to nodes but also to element edges, faces, or elements themselves, i.e., mesh entities. The support of a mesh entity is the domain of influence of the approximation function associated to this entity. Here is a set of rules to decide whether an entity should be enriched or not. We use the term crack and material interface, but the rules below are more generally valid for the jump in a field or its derivative: 1. If the support of an entity is split in two parts by a crack, the entity will be enriched by the Heaviside function. 2. If the support of an entity is split in two parts by a material interface and this interface remains perfect (does not open), the entity may be enriched by the ridge function, provided at least one element in the support is split in two parts by the interface. 3. If the support of an entity is split in two parts by a material interface, the entity may be enriched by the Heaviside enrichment. Then a Lagrange multiplier may be introduced to enforce the law for the jump on the interface (and possibly no jump). 4. Rules 2 and 3 may not be applied at the same time.

E

476

Extended Finite Element Method (XFEM)

5. If the support of an entity does not intersect with the domain of interest, the degree of freedom associated to the entity should be discarded. A key ingredient in the success of the XFEM is the use of level set representation of boundaries. A level set is a signed distance function to the boundaries of interest. Numerically, they are discretized using nodal values and classical finite element approximations. The use of level set dramatically accelerates the search of entity to be enriched. Enrichment functions may also be quickly computed from this level set. For instance, the Heaviside enrichment is simply the sign of the level set, whereas the ridge enrichment uses the nodal level set values ( i ) in the following way: R.x/ D

n X

j i jNi .x/  j

i D1

n X i D1

i Ni .x/j;

functions. The final approximation of the displacement field is the classical approximation plus the Heaviside enrichment plus the tip enrichment: uh .x/ D

X

ui Ni .x/ C

i 2I

C

4 XX

X

aj Nj .x/H.x/

j 2J

bk;˛ Nk .x/F˛ .x/;

(22)

k2K ˛D1

where I is the set of nodes in the mesh, J the set of squared nodes, and K the set of circled nodes in Fig. 6. The (vectorial) degrees of freedom are ui , aj , and bk;˛ . The tip enrichment involves the four functions below which are able to capture the asymptotic behavior of the displacement field (at least for small strain (21) elasticity). Note that the first mode introduces the proper displacement discontinuity at the crack tip ( D C=  ):

where n is the number of nodes on the element. Another advantage of the level set representation is the fact that it opens the possibility to reuse the existing technology on level set techniques and fast marching methods to move boundaries.

p F1 .r; / D r sin. =2/; p F2 .r; / D r cos. =2/; p F3 .r; / D r sin. =2/ sin. /; p F4 .r; / D r cos. =2/ sin. /;

Crack Modeling (23) Consider the mesh depicted in Fig. 6. A crack is placed on the mesh. All nodes surrounded by a little square where r and are the polar coordinates at the tip of share in common the fact that their support is split the crack (Fig. 7). A crack may be located in 2D (and in two parts by the crack. These nodes will thus be 3D) by two level sets. The first level set, lsn , indicates enriched by the Heaviside enrichment. Regarding the nodes of the element where the crack tip is located, they cannot be enriched by the Heaviside function because their support is only cut but not fully split by the crack. These nodes will be enriched by so-called tip

H(x)=1

H(x)=−1 Extended Finite Element Method (XFEM), Fig. 6 A 2D mesh with a crack. Squared nodes are enriched with the Heaviside function and circled nodes with tip functions

Extended Finite Element Method (XFEM), Fig. 7 Two level sets locating a crack on a 2D mesh. The crack, in red, correspond to the set of points such that lsn D 0 and lst  0, whereas the crack tip corresponds to the point satisfying lsn D lst D 0

Extended Finite Element Method (XFEM)

477

the distance to the crack surface, whereas the level set lst indicates the distance to the front (measured tangentially to the crack). They are illustrated in Fig. 6. The polar coordinates are easy to compute using level set informations rD

q

lsn2 C lst2 ;

D arctan.lsn = lst /:

(24)

3.

4.

5.

References The partition of unity, which is the key ingredient to allow enriching the finite element approximation in the XFEM, GFEM, or other partition of unity-related methods (PUM), was introduced by [4]. Regarding the modeling of cracks, the partition of unity was first used to enrich crack tips by [1] and [8]. The Heaviside enrichment was introduced in [5]. An alternative way to view the Heaviside enrichment was later introduced by [3]. The use of level sets as a mean to store the location of interfaces enriched in the XFEM was first proposed in [10] for holes and inclusions and first used for cracks in 2D in [7] and 3D in [9]. Level set-based algorithm to grow 3D cracks were designed in [2]. The ridge enrichment was introduced in [6]. Beyond 2003, the literature on the PUMrelated approaches and their applications has been booming. It involves at this stage more than 3000 papers.

6.

7.

8.

9.

10. 1. Belytschko, T., Black, T.: Elastic crack growth in finite elements with minimal remeshing. Int. J. Numer. Methods Eng. 44, 601–620 (1999) 2. Gravouil, A., Mo¨es, N., Belytschko, T.: Non-planar 3D crack growth by the extended finite element and level sets?

Part II: level set update. Int. J. Numer. Methods Eng. 53(11), 2569–2586 (2002). doi:10.1002/nme.430. http://doi.wiley. com/10.1002/nme.430 Hansbo, A., Hansbo, P.: An unfitted finite element method, based on Nitsche’s method, for elliptic interface problems. Comput. Methods Appl. Mech. Eng. 191, 5537–5552 (2002). http://linkinghub.elsevier.com/retrieve/ pii/S0045782502005248 Melenk, J., Babuska, I.: The partition of unity finite element method: basic theory and applications. Comput. Methods Appl. Mech. Eng. 139, 289–314 (1996) Mo¨es, N., Dolbow, J., Belytschko, T.: A finite element method for crack growth without remeshing. Int. J. Numer. Methods Eng. 46(1), 131–150 (1999). doi:10. 1002/(SICI)1097-0207(19990910)46:13.0.CO;2-J. http://www3.interscience.wiley. com/journal/63000340/abstract Mo¨es, N., Cloirec, M., Cartraud, P., Remacle, J.F.: A computational approach to handle complex microstructure geometries. Comput. Methods Appl. Mech. Eng. 192(28–30), 3163–3177 (2003). doi:10.1016/S0045-7825(03)00346-3. http://linkinghub. elsevier.com/retrieve/pii/S0045782503003463 Stolarska, M., Chopp, D.L., Mo¨es, N., Belytschko, T.: Modelling crack growth by level sets in the extended finite element method. Int. J. Numer. Methods Eng. 51(8), 943–960 (2001). doi:10.1002/nme.201. http://doi. wiley.com/10.1002/nme.201 Strouboulis, T., Babuska, I., Copps, K.: The design and analysis of the generalized finite element method. Comput. Methods Appl. Mech. Eng. 181, 43–69 (2000) Sukumar, N., Mo¨es, N., Moran, B., Belytschko, T.: Extended finite element method for three-dimensional crack modelling. Int. J. Numer. Methods Eng. 48(11), 1549–1570 (2000). doi:10.1002/1097-0207(20000820)48: 113.0.CO;2-A. http://doi.wiley. com/10.1002/1097-0207(20000820)48:11h1549::AIDNME955i3.0.CO;2-A Sukumar, N., Chopp, D.L., Mo¨es, N., Belytschko, T.: Modeling holes and inclusions by level sets in the extended finite-element method. Comput. Methods Appl. Mech. Eng. 190(46–47), 6183–6200 (2001). doi:10.1016/S00457825(01)00215-8. http://linkinghub.elsevier.com/retrieve/ pii/S0045782501002158

E

F

Factorization Method in Inverse Scattering Armin Lechleiter Zentrum f¨ur Technomathematik, University of Bremen, Bremen, Germany

Synonyms Kirsch’s factorization method; Linear sampling method (ambiguous and to avoid – the name is very rarely used for the factorization method by now but has been employed more frequently after the first papers on the method appeared); Operator factorization method (rarely used)

a special test function belongs to the range of the square root of a certain operator that can be straightforwardly computed in terms of far-field data. This characterization yields a fast and easy-to-implement numerical algorithm to image scattering objects. A crucial ingredient of the proof of this characterization is a factorization of the measurement operator, which explains the method’s name. There are basically two variants of the method leading to different characterization criteria: If the far-field operator F is normal, the above characterization applies for the square root .F  F /1=4 of F itself; otherwise, one considers the square root of F] WD jReF j C ImF where ReF and ImF are the self-adjoint and non-self-adjoint part of F , respectively.

Overview Glossary/Definition Terms Factorization method Inverse scattering problem Far field operator Factorization Range identity

Definition The factorization method for inverse scattering provides an explicit and theoretically sound characterization for the support of a scattering object using multi-static far-field measurements at fixed frequency: A point z belongs to the scatterer if and only if

The factorization method was first introduced by Andreas Kirsch [15, 16] for time-harmonic inverse obstacle and inverse medium scattering problems where the task is to determine the support of the scatterer from multi-static far-field measurements at fixed frequency (roughly speaking, from measurements of the far-field pattern of scattered waves in several directions and for several incident plane waves). The method follows the spirit of the linear sampling method and can be seen as a refinement of the latter technique. Both methods try to determine the support of the scatterer by deciding whether a point z in space is inside or outside the scattering object. When the farfield operator F is normal, the factorization method’s criterion for this decision is whether or not special test functions z , parametrized by z and explicitly known

© Springer-Verlag Berlin Heidelberg 2015 B. Engquist (ed.), Encyclopedia of Applied and Computational Mathematics, DOI 10.1007/978-3-540-70529-1

480

for homogeneous background media, are contained in the range of the linear operator .F  F /1=4 . Indeed, when the point z is outside the scatterer then, z is not contained in the range of .F  F /1=4 , whereas z belongs to this range when z is inside the scatterer. The factorization method can be used for imaging by computing the norm of a possible solution gz to .F  F /1=4 gz D z using Picard’s criterion for many sampling points z from a grid covering a region of interest. Plotting these norms then yields a picture of the scattering object. If F fails to be normal, a variant of the method based on F] WD jReF j C ImF provides analogous analytical results and imaging algorithms. These algorithms are very efficient compared to other techniques solving inverse scattering problems since their numerical implementation basically requires the computation of the singular value decomposition of a discretization of the far-field operator F . A further attractive feature of the method is its independence of the nature of the scattering object; for instance, the factorization method yields the same object characterization and imaging algorithm for penetrable and impenetrable objects, such that a mathematical model describing the scatterer does not need to be known in advance. The analysis of the factorization method is based on functional analytic results on range identities for operator factorizations of the form F D H  TH . Under appropriate assumptions, these results state, roughly speaking, that the range of the square root of F equals the range of H  . Moreover, via unique continuation results and fundamental solutions, it is usually not difficult to show that the range of H  characterizes the scattering object, since the far-field z of a point source at z belongs to this range if and only if z belongs to the scattering object. Combining these two results hence provides a direct characterization of the scattering object in terms of the range of the square root of F . Differences to the Linear Sampling Method and Limitations The fundamental difference between the factorization method and the linear sampling method is that the latter one considers an operator equation for the measurement operator itself, while the factorization method considers the corresponding equation for the square root of this operator. Due to this difference, the factorization method is able to provide a mathematically

Factorization Method in Inverse Scattering

rigorous and exact characterization of the scattering object that is fully explicit and merely based on the measurement operator. Note that the linear sampling method does not share this feature, since, for points z inside the scatterer, the theorem that is usually employed to justify that method claims that there exist approximate solutions to a certain operator equation. It remains however unclear how to actually determine or to compute these approximate solutions. Several variants of the standard version of the linear sampling method are able to cope with this problem; see, e.g., [4] or [6]. To obtain a mathematically rigorous characterization of the scatterer’s support, the factorization method however requires the inverse scattering problem under investigation to satisfy several structural assumptions that are not required by the linear sampling method (or other sampling methods). The reason is a functional analytic result on range identities for operator factorizations that is the backbone of the method. First, the measurement operator F defined on a Hilbert space V (imagine the far-field operator defined on L2 of the unit sphere) needs to have a self-adjoint factorization of the form F D H  TH with a compact operator H W V ! X and a bounded operator T W X ! X  , where X is a reflexive Banach space. It is crucial that the outer operators of this factorization are adjoint to each other. Second, the middle operator T needs to be a compact perturbation of a coercive operator: T D T1 C T2 such that Re hT1 ; iX  X  ckk2X for all  2 X and some c > 0 and such that T2 is compact. There are several inverse scattering problems where at least one of these two conditions is violated. The first one does, for instance, not hold for near-field measurements when the wave number is different from zero. The coercivity assumption for the middle operator is violated, e.g., for electromagnetic scattering from a perfect conductor, for acoustic scattering from a scatterer that is partly sound-soft and partly sound-hard, and for scattering from an inhomogeneous medium that is partly stronger scattering and partly weaker scattering than the background medium. Consequently, providing theory that does not require either of the two conditions would be highly desirable. In the first years after the invention of the method in [15], the factorization method could only be applied to far-field inverse scattering problems where the far-field operator is normal. When the scatterer is absorbing, the far-field operator fails to be normal,

Factorization Method in Inverse Scattering

and it was an open problem whether the factorization method applies to such problems. This problem has been solved by decoupling real and imaginary parts of the measurement operator, yielding range identities for the square root of the auxiliary operator F] D .Re.F / Re.F //1=2 C Im .F / that is easily computed in terms of F (see [12, 20]). Applications of the Factorization Method in Inverse Scattering Even if the factorization method cannot be applied to all inverse scattering problems, there are many situations where the method provides the abovementioned characterization of the support of the scattering object. To list only a few of them, the method has been successfully applied to inverse acoustic obstacle scattering from sound-soft, sound-hard, or impedance obstacles; see [12, 15]; to inverse acoustic medium scattering problems, see [16]; to electromagnetic medium scattering problems, see [19, 20]; to inverse electromagnetic scattering problems at low frequency, see [11]; to inverse scattering problems for penetrable and impenetrable periodic structures, see [2, 3, 24]; to inverse problems in elasticity, see [8]; to inverse scattering problems in acousto-elasticity, see [21]; to inverse problems for stationary Stokes flows, see [25]; and to inverse scattering problems for limited aperture, see [20, Section 2.3]. Apart from inverse scattering, the factorization method has been applied to a variety of inverse problems for partial differential equations. The monograph of [20] and the review of [14] indicate a variety of other inverse problems treated by this method and also further references for the factorization method in inverse scattering. Finally, we mention that the factorization method is linked to other sampling methods as the linear sampling method; see [1, 4], and the MUSIC algorithm, and see [5, 18].

An Example: Factorization Method for Inverse Medium Scattering

481

Scattering from an Inhomogeneous Medium Time-harmonic scattering theory considers waves U.x; t/ D u.x/ exp.i!t/ with angular frequency ! > 0 and time dependence exp.i!t/. If we denote by c the space-dependent wave speed in R3 , and by c0 the constant wave speed in the background medium, then the wave equation c 2 U  @t t U D 0 reduces to the Helmholtz equation u C k 2 n2 u D 0

in R3

(1)

with (constant) wave number k D !=c0 > 0 and space-dependent refractive index n D c0 =c. In the following, we suppose that the refractive index equals one in the complement of a bounded Lipschitz domain D with connected complement; the domain D hence plays the role of the scattering object. A typical direct scattering problem is the following: For an incident plane wave ui .x/ D exp.ik x  /, x 2 R3 , of direction  2 S2 WD fx 2 R3 ; jxj D 1g, we seek a total field ut that solves (1). Moreover, the scattered field us D ut  ui needs to satisfy the Sommerfeld radiation condition   @  ik us D0 uniformly in lim jxj jxj!1 @jxj x 2 S2 : xO D (2) jxj Sommerfeld’s radiation condition acts as a boundary condition “at infinity” for the scattered field and guarantees uniqueness of solution to scattering problems on unbounded domains. Physically, this condition means that the scattered wave is created locally in D and propagates away from D. The scattering problem to find us when given ui and n2 is well posed in standard function spaces under reasonable assumptions on the refractive index; see [10]. Solutions to the exterior Helmholtz equation that satisfy the Sommerfeld radiation condition behave at infinity like an outgoing spherical wave modulated by a certain angular behavior,     as jxj ! 1; O C O jxj2 u.x/ Dˆ.x/ u1 .x/

e ikjxj In this section, we consider a time-harmonic inverse : ˆ.x/ WD medium scattering problem and explain in some detail 4jxj how the factorization method works. This material is mostly from [16, 18, 20]. We also indicate why there The function u1 2 L2 .S2 / is called the far-field pattern of u. exist several variants of the method.

F

482

Factorization Method in Inverse Scattering

In the following, we denote by u1 .x; O / the farfield pattern in the direction xO 2 S2 of the scattered wave caused by an incident plane wave of direction  2 S2 . The refractive index n2 is allowed to be real and positive or complex valued with positive real part and nonnegative imaginary part (further assumptions on n2 will be stated where they are required).

k 2 n2 v D k 2 .1n2 /f in R3 , subject to the Sommerfeld radiation condition (2).

The adjoint H  of the Herglotz operator can be used to characterize the scatterer’s support D: It holds that the far-field ˆ1 .x; O z/ D exp.ik xO  z/ of a point source ˆ.x  z/ D exp.ikjx  zj/=.4jx  zj/ at z 2 R3 belongs to the range of H  if and only if z 2 D. Due to the factorization from the last theorem, one would Inverse Problem and Factorization now like to link the range of H  with the range of In an inverse medium scattering problem with far-field (some power of) the measurement operator F to obtain data, one seeks to determine properties of the scatterer characterization results for the scatterer D. O / from the knowledge of the far-field pattern u1 .x; for all directions xO in a given set of measurement Two Characterization Results directions and all  in a given set of directions of inci- If the refractive index n2 is real, then F is a normal opdence. Particularly, the factorization method solves the erator and consequently possesses a complete system O / of eigenvectors f g following inverse scattering problem: Given u1 .x; j j 2N with associated eigenvalues for all xO 2 S2 and all  2 S2 , find the support D of the f g . Under suitable assumptions, this basis of j j 2N scattering object! Recall that D was defined to be the eigenvectors allows to prove that the test functions support of n2  1. ˆ1 .; z/ belong to the range of .F  F /1=4 – the square A central tool for the factorization method is the far- root of F – if and only if the point z belongs to D. field operator F , One key idea of the proof is that the orthonormal basis fj gj 2N of L2 .S2 / transforms into a Riesz basis 1=2 F W L2 .S2 / ! L2 .S2 / g 7! u1 .; / g./ ds./: fj Hj g of a suitable subspace of L2 .D/ due to the 2 S factorization of F . (This is a simplified statement; see Section 4 in [16] for the precise formulation.) Picard’s This is an integral operator with continuous kernel criterion yields the following characterization of the u1 .; /, and the theory on integral equations states that scatterer: F is a compact operator. By linearity of the scattering problem, F maps a density g to the far field of the z 2 R3 belongs to D if and only if scattered field for the incident Herglotz wave function ˇ ˇ2 1 ˇ X hˆ1 .; z/; j iL2 .S2 / ˇ Z < 1: (3) jj j vg .x/ D g./e ik  x ds./; x 2 R3 : j D1 Z

S2

The main assumptions on n2 for this result are that n2 is real valued, that n2 1 does not change sign, and that k 2 is not an interior transmission eigenvalue; see [7] for a definition. If the refractive index takes imaginary values inside D, which corresponds to an absorbing scattering object, then the far-field operator fails to be normal and the .F  F /1=4 -variant of the factorization method Theorem 1 (Factorization) The far-field operator does not work. However Grinberg and Kirsch [12, 18] can be factored as showed that, under suitable assumptions, the auxiliary operator F] D .Re.F / Re.F //1=2 C Im .F / allows  1=2 F D H TH; to prove that the ranges of F] and of H  are equal. Since the test functions ˆ1 .; z/ belong to the range where T W L2 .D/ ! L2 .D/ is defined by Tf D of H  if and only if z 2 D, one can then conclude 1=2 1 .R3 / solves v C that ˆ1 .; z/ belongs to the range of F] if and only k 2 .n2  1/ .f C vjD / and v 2 Hloc The restriction of a Herglotz wave function vg on the obstacle D yields a bounded linear operator H W ˇ L2 .S2 / ! L2 .D/, g 7! vg ˇD . Obviously, if we know O / W x; O  2 S2 g for all directions x; O  2 fu1 .x; 2 S , then we also know F . Therefore we reformulate our inverse scattering problem as follows: Given F , determine the support D of the scatterer!

Factorization Method in Inverse Scattering

483

Factorization Method in Inverse Scattering, Fig. 1 Reconstructions of the support of an inhomogeneous medium using the factorization method (Reproduced from [5]). (a)

The exact support of the scatterer (b) Reconstruction without artificial noise (c) Reconstruction with 5 % artificial noise

if z 2 D. Denote by f j gj 2N the eigenvalues of the compact and self-adjoint operator F] and by fj gj 2N the corresponding eigenvalues. Using Picard’s criterion we reformulate the characterization of D as follows:

than the noise level ı > 0. Consequently, one needs to regularize the Picard series. Several methods are available: Tikhonov regularization, see [9]; regularization by truncation of the series, see [22]; comparison techniques, see [13]; and noise subspace techniques, see [5]. Figure 1 shows reconstructions for a twodimensional inhomogeneous medium with piecewise constant index of refraction; n2 equals 10 inside the inclusion, shown in Fig. 1a, and 1 outside the inclusion. The wave number is k D 2, and the reconstruction uses 32 incident and measurement directions uniformly distributed on the unit circle. These examples are reproduced from [5] where further details can be found.

z 2 R3 belongs to D if and only if ˇ ˇ2 1 ˇ X hˆ1 .; z/; j iL2 .S2 / ˇ < 1: jj j j D1

(4)

The main assumptions for this result are that Re .n2  1/ does not change sign. The assumption that k 2 is not an interior transmission eigenvalue can be dropped for this variant of the method, but not for the .F  F /1=4 -variant from (3); see [23]. Discretization The criterion in (3) or (4) suggests the following algorithm to image the scattering object: Choose a discrete set of grid points in a certain test domain and plot the reciprocal of the series in (3) or (4) on this grid. Of course, in practice one can only plot a finite approximation to the infinite series afflicted with certain errors. Nevertheless, one might hope that plotting the reciprocal of the truncated series as a function of y leads to large and small values at points z inside and outside the scatterer D, respectively. However, the illposedness of the inverse scattering problem afflicts this imaging process, because we divide by small numbers j or j . For instance, if one only knows approximations ıj with jıj j j  ı and jı with kjı j k  ı, ˇ ˇ2 then the difference between ˇhˆ1 .; z/;  ı iˇ =jı j and

Key Results on Range Characterizations

The factorization method can be seen as a tool to pass the geometric information on the scattering object contained in the inaccessible operator H  of the factorization F D H  TH to the measurement operator F . To this end, there are basically three functional analytic frameworks that can be used. As in the first section, we assume here that F is a compact operator on a Hilbert space V , that H W V ! X is compact, and that T W X ! X  is bounded where X is a reflexive Banach space. The first variant of the factorization method, the socalled .F  F /1=4 -variant, requires F to be normal. In this case F possesses a complete basis of eigenvectors fj gj 2N such that F j D j j . The vectors j D j j 1=2 the corresponding exact value is in general much larger j Hj satisfy

F

484

Factorization Method in Inverse Scattering

hT

i;

j iX  X

D

i ıi;j ; ji j

i; j 2 N:

If T is a compact perturbation of a coercive operator, and if the eigenvalues fj gj 2N satisfy certain geometric conditions, then Theorem 3.4 in [15] proves that f j gj 2N is a Riesz basis of X . This is the key step to prove that the range of .F  F /1=4 equals the range of H  ; see [15, Theorem 3.6]. The second variant of the method, the so-called infimum criterion, was the first step towards the treatment of problems where F fails to be normal. In [17, Theorem 2.3], it is shown that if there exist positive numbers c1;2 such that c1 kT k2X   jhT ;

iX  X j  c2 kT k2X 

for all 2 X , then an element g 2 V belongs to the range of H  is and only if inf fjhF ; iV j ;  2 V; hg; iV D 1g > 0:

The drawback of this characterization is that the criterion whether or not a point belongs to the scatterer requires to solve an optimization problem. To get an image of the scattering object, one hence needs to solve an optimization problem for each sampling point in the grid. The third variant of the method relies on the auxiliary operator F] D .Re.F / Re.F //1=2 C Im .F /; see [12]. For this operator, the equality of the ranges of 1=2 F] and H  can be shown, e.g., under the conditions that T is injective, that the real part of T is a compact perturbation of a coercive operator and that the imaginary part of T is nonnegative; see [23].

Cross-References  Adjoint Methods as Applied to Inverse Problems  Inversion Formulas in Inverse Scattering  Inhomogeneous Media Identification  Linear Sampling  Optical Tomography: Theory

References 1. Arens, T.: Why linear sampling works. Inverse Probl. 20, 163–173 (2004) 2. Arens, T., Grinberg, N.: A complete factorization method for scattering by periodic structures. Computing 75, 111– 132 (2005) 3. Arens, T., Kirsch, A.: The factorization method in inverse scattering from periodic structures. Inverse Probl. 19, 1195–1211 (2003) 4. Arens, T., Lechleiter, A.: The linear sampling method revisited. J. Int. Eqn. Appl. 21, 179–202 (2009) 5. Arens, T., Lechleiter, A., Luke, D.R.: Music for extended scatterers as an instance of the factorization method. SIAM J. Appl. Math. 70(4), 1283–1304 (2009) 6. Audibert, L., Haddar, H.: A generalized formulation of the linear sampling method with exact characterization of targets in terms of farfield measurements. Inverse Probl. 30, 035011 (2014) 7. Cakoni, F., Gintides, D., Haddar, H.: The existence of an infinite discrete set of transmission eigenvalues. SIAM J. Math. Anal. 42(1), 237–255 (2010) 8. Charalambopoulus, A., Kirsch, A., Anagnostopoulus, K.A., Gintides, D., Kiriaki, K.: The factorization method in inverse elastic scattering from penetrable bodies. Inverse Probl. 23, 27–51 (2007) 9. Colton, D., Coyle, J., Monk, P.: Recent developments in inverse acoustic scattering theory. SIAM Rev. 42, 396–414 (2000) 10. Colton, D.L., Kress, R.: Inverse Acoustic and Electromagnetic Scattering Theory, 3rd edn. Springer, New York (2013) 11. Gebauer, B., Hanke, M., Schneider, C.: Sampling methods for low-frequency electromagnetic imaging. Inverse Probl. 24, 015,007 (2008) 12. Grinberg, N.: The operator factorization method in inverse obstacle scattering. Integral Eqn. Oper. Theory 54, 333–348 (2006) 13. Hanke, M., Br¨uhl, M.: Recent progress in electrical impedance tomography. Inverse Probl. 19, S65–S90 (2003) 14. Hanke, M., Kirsch, A.: Sampling methods. In: Scherzer, O. (ed.) Handbook of Mathematical Methods in Imaging, chap 12. Springer, New York (2011) 15. Kirsch, A.: Characterization of the shape of a scattering obstacle using the spectral data of the far-field operator. Inverse Probl. 14, 1489–1512 (1998) 16. Kirsch, A.: Factorization of the far field operator for the inhomogeneous medium case and an application in inverse scattering theory. Inverse Probl. 15, 413–429 (1999) 17. Kirsch, A.: New characterizations of solutions in inverse scattering theory. Appl. Anal. 76, 319–350 (2000) 18. Kirsch, A.: The MUSIC-algorithm and the factorization method in inverse scattering theory for inhomogeneous media. Inverse Probl. 18, 1025–1040 (2002) 19. Kirsch, A.: The factorization method for Maxwell’s equations. Inverse Probl. 20, S117–S134 (2004) 20. Kirsch, A., Grinberg, N.: The Factorization Method for Inverse Problems. Oxford Lecture Series in Mathematics and its Applications, vol. 36. Oxford University Press, Oxford (2008)

Fast Fourier Transform 21. Kirsch A, Ruiz A (2012) The Factorization Method for an inverse fluid-solid interaction scattering problem. Inverse Problems and Imaging 6:681–695 22. Lechleiter, A.: A regularization technique for the factorization method. Inverse Probl. 22, 1605–1625 (2006) 23. Lechleiter, A.: The Factorization method is independent of transmission eigenvalues. Inverse Probl. Imaging 3, 123–138 (2009) 24. Lechleiter, A., Nguyen, D.-L.: Factorization Method for Electromagnetic Inverse Scattering from Biperiodic Structures. SIAM J. Imaging Sci. 6, 1111–1139 (2013) 25. Lechleiter, A., Rienm¨uller, T.: (2013) Factorization method for the inverse stokes problem. Inverse Probl. Imaging 7, 1271–1293 (2013)

485

always approximated by a trigonometric polynomial. The tasks of summing the N -term polynomial at each of N points on a uniform grid, and of calculating each of its N coefficients (“frequencies”) by interpolation, are collectively known as the discrete Fourier transform (DFT). The forward and inverse DFT are the evaluation of fj D

N X kD1

   2 j ak exp i k N

j D 1; : : : ; N (1)

and the inverse ak D

   N 1 X 2 j fj exp i k N j D1 N

k D 1; : : : ; N (2)

Fast Fourier Transform

The forward DFT is just the summation of the Fourier series for a function f .x/ on a grid of N uniformly spaced points. (The series is in complex-valued exJohn P. Boyd ponential form rather the usual cosines and sines.) Department of Atmospheric, Oceanic and Space The inverse DFT is equivalent to approximating the Science, University of Michigan, Ann Arbor, usual Fourier coefficients of sophomore mathematics MI, USA by trapezoidal rule quadrature. (For nonperiodic f .x/, the trapezoidal rule has an error proportional to 1=N 2 , but if f .x/ is analytic and f .x/ D f .x C 2/, the acMathematics Subject Classification curacy of both the Fourier series and of the trapezoidal rule approximation of its coefficients is exponential 42A99; 42B99; 65D05; 65M70 in N [3].). The forward and inverse transforms are so similar that essentially the same algorithm can be applied to both. This is why we speak of the “FFT” in Synonyms the singular instead of the plural. (Briggs and Henson DFT fast cosine transform; Discrete fourier transform; [4] is a book-length account that greatly (and very readably) expands on our brief treatment of the FFT.) FCT; FFT; FST; Fast sine transform Both tasks can be written as a matrix-vector multiply. Let fE and aE denote N -dimensional vectors whose E Short Definition elements are the fj and ak , respectively. Let TE denote a square matrix of dimension N whose elements are The fast Fourier transform (FFT) is an algorithm    for summing a truncated Fourier series and also for 2 j ; j D 1; 2; : : : N; T D exp i k jk computing the coefficients (frequencies) of a Fourier N approximation by interpolation. k D 1; 2; : : : N (3)

Fourier Series, Fourier Transforms, and Trigonometric Interpolation

Then

E fE D TE aE

[MMT]

(4)

This way of calculating the grid point values (“samSince computers cannot manipulate an infinite number ples”) of a function f .x/ from the lowest N terms of quantities, Fourier series and Fourier transforms are of its Fourier series, or calculating the Fourier coef-

F

486

Fast Fourier Transform

ficients of the trigonometric polynomial that interpolates f .x/ at the N grid points, is called the Matrix Multiplication Transform (MMT) [3]. The cost is about 6N 2 real floating-point operations where we count both multiplications and additions equally and where

one complex-valued multiplication is the equivalent of four real-valued operations and one complex-valued addition costs as much as two real-valued operations. The FFT’s achievement is to perform the same operation very cheaply:

cost of N-pt. complex FFT  5N log2 ŒN  total real operations

(5)

For N D 1;024, for example, the MMT cost is about cheaper, in both operation count and storage, to apply 6;000N floating-point operations, whereas the FFT the strategy known variously as “partial summation” or “factored summation,” which are fancy labels or price is only 50N , a savings of a factor of 120! performing multidimensional transforms as a nested sequence of one-dimensional transforms [3]. The Curse of Conventions To illustrate the basic idea, the two-dimensional case is sufficient. Arrange the Fourier coefficients in Library FFT software usually evaluates sums over an M  N matrix aEE , ordered so that different x wave positive wave numbers only, as in (1), whereas the Ex Fourier series in mathematics textbooks and physics numbers correspond to different matrix rows. Let TE and engineering applications is almost always the sum and TEE y denote the one-dimensional transformation over both positive and negative wave numbers: matrices as defined above of dimensions M  M and N N , respectively. Defining T to denote matrix trans   N=21 X 2 j position without complex conjugation of the elements, am exp i m (6) fj D N the two-dimensional transform is mDN=2 The two forms are mathematically equivalent because exp.i Œ2=N mj / is invariant to the shift m ! m ˙ N , but the indexing is altered: am ! amN for m > N=2. Library software employs positive wave numbers as in (1) because this makes life easier for the computer programmer. Unfortunately, this convention requires the physicist to convert the FFT output into the conventional form (6). Matlab helpfully provides a routine fftshift, but it is usually necessary to do the conversion manually. One also must be very careful when taking derivatives (Chap. 9 of [3]). Some library software starts the sum at wave number zero whereas others, as here, begin with wave number one. Be vigilant!

Multidimensional Transforms and Partial Summation, Alias Factored Summation Transforms in d dimensions can be performed by constructing a complex-valued square matrix of dimension N d followed by a matrix-vector multiply, but this costs about 6N 2d floating-point operations and requires storage of N 2d numbers, both very expensive. It is far

E E fE D TE y

 T !T EE x E T aE

[2D MMT, factored summation]

(7)

The cost is about 6MN.M CN / versus 6M 2 N 2 for the single-giant-matrix approach. The savings is a factor of N=2 in two dimensions when M D N and a savings of a factor of N d 1 =d in d dimensions. It is important to note that the partial summation/factored summation trick is not restricted to Fourier transforms, but is applicable to any tensor product grid with a tensor product basis. (By a “tensor product” grid, we mean a lattice of points .xj ; yk / defined in two dimensions by taking all possible pairwise combinations of the one-dimensional grids xj ; j D 1; 2; : : : M and yk ; k D 1; : : : N ; similarly, a tensor product basis consists of all possible pairwise products j .x/ k .y/ from a pair of one-dimensional basis sets.)

Fast Fourier Transform

487

it is helpful to rewrite the one-dimensional transform in a form similar to a special case of the two-dimensional transform: special in that the second dimension (“y”) has just two basis functions and two grid points. First, introduce composite indices to split both j and k into two parts:

FFT Algorithm It is unnecessary to describe the FFT in detail because it is a perfect “black box.” Every numerical library and all systems like Maple, Mathematica, and Matlab contain highly optimized FFT routines. Consequently, it is never necessary for the user to write his own FFT software. Furthermore, the FFT is a direct, deterministic algorithm with no user-chosen parameters to fiddle with. The FFT is very “well conditioned” in the sense that there is little accumulation of roundoff error even for very large N . Still, it is worth noting that the algorithmic heart of the FFT is factored summation. To see this connection,

fj D

N X

 ak exp i k

kD1

f

j 0 CJN=2

D

N=2 1 X X

a

2k 0 K

KD0 k 0 D1

fQj 0 ;J D

D



2 j N

j D j 0 C JN=2; j 0 D 1; 2 : : : N=2; J D 0; 1 (8) k D 2k 0  K; k 0 D 1; 2 : : : N=2; K D 0; 1

(9)

where J D 0 corresponds to the first half of index j and J D 1 corresponds to the second half of index j . The index K D 0 selects the even values of k while K D 1 selects odd k. The DFT successively becomes

 (10)

    0  0  2 exp i 2k  K j C JN=2 N

(11)

     2 aQ m;K exp i .2m  K/ j 0 C JN=2 N KD0 mD1 N=2 1 X X

(12)

  2 2 2 2  iKj 0 C i 2mJN=2  iKJN=2 aQ m;K exp i 2mj 0 N N N N KD0 mD1 N=2 1 X X

where we have introduced

fQj 0 ;J 



fj 0 ; fj 0 CN=2 ;

J D0 J D1

 &

aQ m;K 

a2m ; a2m1 ;

KD0 K D1

(13)

Observing that exp.i 2mJ.N=2/.2=N // D exp .iKj 0 2 =N / is independent of k 0 and therefore exp.i mJ 2/ is one for all integers m and J and that can be taken outside the summation yields

X    N=2 1 X 02 02 Q fj 0 ;J D exp iKj exp .i KJ / aQ m;K exp i 2mj N mD1 KD0 N The double summation is identical in form to two-dimensional transform in which there are just two basis functions in the second coordinate y, indexed by K, evaluated at only two points in y,

(14)

indexed by J . We can therefore apply factored summation to evaluate this series as a pair of one-dimensional sums, saving roughly a factor of 2.

F

488

The crucial property that allows factored summation is that the exponential basis functions can be factored into products by the familiar identity exp.a C b/ D exp.a/ exp.b/. If N is a power of two, then this pairwise factorization can be applied again and again. One does not quite save a factor of 2 at each step because there are additional operations besides matrixvector multiplies, but the reduction in cost from 6N 2 to 5N log2 .N / is nonetheless dramatic. When N is a composite of products of prime numbers, i.e., N D 2m1 3m2 5m3 7m4 : : : where the exponents mj are nonnegative integers, it is still possible to apply factored summation to obtain a very fast transform. The “composite prime” FFT is slower than when N is a power of two, partly because the transform is inherently less efficient and partly also because the algorithm must identify the prime factors and their exponents in N . Consequently, it is very common for N to be chosen in the form N D 2m in applications, but there is only a modest loss of efficiency if N D 2m p where p is another small prime.

Fast Fourier Transform

property that f .x/ D f .x/ for all x is said to be “antisymmetric” or of “odd parity” and its Fourier series contains only sines. The fast cosine transform (FCT) and fast sine transform (FST) manipulate pure cosine and sine series, respectively, more efficiently than the FFT [14]. As explained in [3], Fourier series consisting only of odd cosines or only odd sines are common in applications. Specialized “quarter-wave” transforms have been developed by Swarztrauber [13] in his FFTPACK library. Written originally in Fortran (http:// www.netlib.org/fftpack/), C and Java (http://sites. google.com/site/piotrwendykier/software/jtransforms) translations are available. Because the FFT is so valuable in applications, many variants of the general, complex-valued FFT exist even though the underlying principles are the same. The whimsically named “Fastest Fourier Transform in the West” (FFTW) will time various options to determine which is best on your particular hardware [8].

Restrictions and Generalizations FFT on Parallel Computers The FFT evaluates sums which couple every grid point and basis function, which would seem bad for parallelism. However, intensive applications are usually multidimensional Fourier transform which, as noted earlier, are performed by factored summation as a series of one-dimensional transforms which can be done in parallel on different processors. Transposing the data before the next step of factored summation requires a lot of interprocessor communication [15]. On current architectures, these difficulties have not precluded very ambitious and highly parallel applications such as the three-dimensional pseudospectral flow simulations of Mininni et al. [12].

Variants: Fast Cosine Transform and Parity When a function f .x/ has the property that it is equal to its own reflection across the origin, that is, f .x/ D f .x/ for all x, the function is said to be “symmetric with respect to x D 0” or to possess the property of “even parity”; its Fourier series will contain only cosine functions. Similarly, a function with the

The FFT is applicable only when some restrictions are satisfied. First, the FFT only applies to a basis of exponentials, or equivalently, the sines and cosines of an ordinary Fourier series, and also to functions obtainable from sines and cosines by a smooth change of coordinate. The latter category includes Chebyshev polynomials and rational Chebyshev functions [3]. The FFT is not applicable to Legendre, Gegenbauer, and Jacobi polynomials except for the special case of Chebyshev polynomials. The second restriction is that the grid must be uniformly spaced. So-called nonuniform FFTs (NUFFTs) remove these restrictions. For example, [2] proposed an efficient NUFFT for summing N -term Fourier series “off-grid,” that is, at irregularly distributed points. His procedure is to pad the Fourier coefficients with zeros, take a conventional FFT with 2N or 3N terms, and then apply low-degree polynomial interpolation to the samples created by the FFT. The rationale is that most of the error in polynomial interpolation is in high Fourier wave numbers, but by construction, these are absent. This procedure is implicitly used in the US global spherical harmonic spectral weather forecasting model, which employs

Fast Fourier Transform

polynomial interpolation for “off-grid” interpolation at the irregularly spaced departure points in its semiLagrangian time advancement scheme. The high wave number coefficients are removed by filtering to prevent so-called aliasing effects, but a fringe benefit of the filtering is a great improvement in the accuracy of “off-grid” spectral approximation. Ware’s [16] review compares a variety of NUFFTs. As explained in [10], the fast multipole method (FMM) was originally devised to approximate the gravitational forces of a cluster of 100,000 stars by multipole series – a local Taylor expansion – with only a handful of terms. Boyd [1] and Dutt and Rokhlin [7] pointed out that the FMM can sum spectral series on both uniform and nonuniform grids even for basis sets, such as Legendre polynomials and associated Legendre functions, for which the FFT is inapplicable. FMM and closely related treecode algorithms are still an active research frontier and have been extended to radial basis functions [11]. Although nominally NUFFTs claim O.N log2 .N // performance, sometimes even an O.N / cost, these generalized FFTs have significant disadvantages compared to the original. First, the proportionality constant in front of the N log2 .N / factor is usually huge compared to the FFT proportionality factor of 5. Second, NUFFTs require additional approximations which make them inexact even in infinite-precision arithmetic and require users to choose parameters to control the trade-off between speed and accuracy. Good, robust library software is much harder to find for NUFFTs than for FFTs. Nevertheless, these FFT generalizations greatly extend the range of spectral applications.

History Gauss invented the FFT in 1805 to calculate the orbit of an asteroid. It was then forgotten for nearly a century until Carl Runge, best known for the Runge-Kutta family of time-integration methods, rediscovered the FFT in 1903. As reviewed by Carse and Urquhart [5], Sir Edmund Whittaker’s mathematical laboratory used printed forms to guide the student calculators, a form of computer programming for the FFT when computers were people. The algorithm was then forgotten a second time. The statistician Irving Good described forms of both partial summation and FFT in [9], but the fast Fourier transform exploded only after Cooley and

489

Tukey’s rediscovery [6]. The explosion has included not only FFT software and thousands of applications but also the development of special-purpose chips that hardwire the FFT for signal processing, data analysis, and digital music.

Applications The applications of the FFT are more numerous than the stars in the galaxy, and we can only mention a couple. Time series analysis employs the FFT to look for periodicities in a 100 years of weather data or a decade of stock market prices, automatically identifying oscillations and cycles. Another application is to solve partial differential equations and integral equations by Chebyshev polynomial and Fourier spectral methods [3]. The Chebyshev polynomials, Tn .x/, are just a Fourier cosine series in disguise, connected by the identity Tn .cos.t// D cos.nt/, and so can be manipulated by the fast cosine transform. Differentiation is most efficient in coefficient space using d=dx.exp.i kx// D i k exp.i kx/, but multiplication in a nonlinear term is most efficient using grid point values of the factors. In time-dependent problems, the FFT is used to jump back and forth between the grid point and coefficient (“nodal” and “modal”) representations at each and every time step. The global weather forecasting model for the United States is such a spectral model. However, it is only one example among a vast number of spectral method applications.

References 1. Boyd, J.P.: A fast algorithm for Chebyshev and fourier interpolation onto an irregular grid. J. Comput. Phys. 103, 243–257 (1992) 2. Boyd, J.P.: Multipole expansions and pseudospectral cardinal functions: a new generalization of the fast fourier transform. J. Comput. Phys. 102, 184–186 (1992) 3. Boyd, J.P.: Chebyshev and Fourier Spectral Methods, 2nd edn., 665p. Dover, Mineola (2001) 4. Briggs, W.L., Henson, V.E.: The DFT: An Owner’s Manual for the Discrete Fourier Transform. Society for Industrial and Applied Mathematics, Philadelphia (1995) 5. Carse, G.A., Urquhart, J.: Harmonic analysis. In: Horsburgh, E.M. (ed.) Modern Instruments and Methods of Calculation: A Handbook of the Napier Tercentenary Exhibition, pp. 220–248. G. Bell and Sons, London (1914)

F

490

Fast Marching Methods

6. Cooley, J.W., Tukey, J.W.: An algorithm for the machine calculation of complex fourier series. Math. Comput. 19, 297–301 (1965) 7. Dutt, A., Rokhlin, V.: Fast fourier transforms for nonequispaced data. SIAM J. Comput. 14, 1368–1393 (1993) 8. Frigo, M., Johnson, S.G.: The design and implementation of FFTW3. Proc. IEEE. 93(2), 216–231 (2005) 9. Good, I.J.: The interaction algorithm and practical fourier analysis. J. R. Stat. Soc. B 20, 361–372 (1958) 10. Greengard, L.: The numerical solution of the N-body problem. Comput. Phys. 4, 142–152 (1990) 11. Krasny, R., Wang, L.: Fast evaluation of multiquadric RBF sums by a Cartesian treecode. SIAM J. Sci. Comput. 33, 2341–2355 (2011) 12. Mininni, P.D., Rosenberg, D., Reddy, R., Pouquet, A.: A hybrid MPI-OpenMP scheme for scalable parallel pseudospectral computations for fluid turbulence. Parallel Comput. 37(6–7), 316–326 (2011) 13. Swarztrauber, P.N.: Vectorizing the FFTs. In: G. Rodrique (ed.) Parallel Computations, pp. 51–83. Academic, New York (1982) 14. Swarztrauber, P.N.: Symmetric FFTs. Math. Comput. 47, 323–346 (1986) 15. Swarztrauber, P.N.: Multiprocessor FFTs. Parallel Comput. 7, 197–210 (1987) 16. Ware, A.F.: Fast approximate Fourier transforms for irregularly spaced data. SIAM Rev. 40, 838–856 (1998)

Fast Marching Methods James A. Sethian Department of Mathematics, University of California, Berkeley, CA, USA Mathematics Department, Lawrence Berkeley National Laboratory, Berkeley, CA, USA

Fast Marching Methods Fast marching methods are computational techniques to efficiently solve the Eikonal equation given by jruj D K.x/ where x is a point in Rn , u is an unknown function of x, and K.x/ is a cost function known at every point in the domain. This is a first-order nonlinear partial differential equation and occurs in a spectrum of scientific and engineering problems, including such varied applications as wave propagation and seismic imaging, image processing, photolithography, optics, control theory, and robotic path planning.

In general, the Eikonal equation is a special form of the more general first-order static Hamilton–Jacobi equation given by H.x; u; Du/ D 0 in which the function H is known, but depends on x, the unknown function u, and the various first derivatives of u, denoted by Du. Here, we are interested in the so-called viscosity-solutions, which may be non-differentiable. To fully specify the Eikonal and Hamilton–Jacobi equations, boundary conditions are also provided which provide the solution u on a hypersurface in the domain. Three simple examples are given by: • The distance equation: jruj D 1;

uD0

on

where is a hypersurface in Rn . The solution u.x/ then gives the Euclidean distance from any point in Rn to the boundary set . • The Eikonal equation: jruj D K.x/;

u D 0 on



where K.x/ is an isotropic cost function defined at every point in the domain. Here, isotropic means that the cost of moving through the point x does not depend on the direction. The solution u.x/ then gives the path with the least total cost from any point in Rn to the boundary set . • The anisotropic Hamilton–Jacobi equation: jruj D K.x; ru/;

u D 0 on

where the cost function K.x; ru/ depends on the direction of motion through the point x. The solution u.x/ then gives the path with the least total cost from any point in Rn to the boundary set . Because of the nonlinear nature of these equations, it may seem that any numerical scheme must fundamentally rely on computing values of the solution at discrete mesh points by iteratively solving a list of coupled nonlinear equations. Fast marching methods exploit a fundamental ordering inherent in the equations themselves, and yield highly efficient numerical methods that avoid iteration entirely, and

Fast Marching Methods

491

have computational complexity of O.N log N /, where to determine the next Accepted grid point. Efficient implementation can be obtained using heap-sort data N is the total number of grid points in the mesh. structures. Consider now the problem of finding the true Dijkstra’s Method and Optimal Paths We begin discussing such efficient methods by first cheapest path in a two-dimensional domain: here, considering a discrete optimal trajectory problem on ui;j represents the cost of reaching and entering a network. Given a network and a cost associated the subdomain of the region represented by the cell with each node, the global optimal trajectory is the centered at grid point i; j . Executing Dijkstra’s method most efficient path from a starting point to some exit to find the optimal (cheapest/shortest) path from a set in the domain. Dijkstra’s classic algorithm [5] starting position to an exit set produces a solution to computes the minimal cost of reaching any node on a the PDE max.jux j; juy j/ D h K. It is easy to see that this produces a solution which network in O.N log N / operations. Since the cost can depend on both the particular node, and the particular remains on the links between the mesh points. Consider link, Dijkstra’s method applies to both isotropic and an easy version of this problem, in which one divides anisotropic control problems. The distinction is minor the unit square in cells, each of whose cost is 1, and for discrete problems, but significant for continuous the goal is to find the shortest path from (0, 0) to problems. Dijkstra’s method is a ‘one-pass’ algorithm; (1, 1). As the grid becomes finer and finer, Dijkstra’s each point on the network is updated a constant number method always produces a stairstep path with total cost of times to produce the solution. This efficiency comes 2 and does not converge to the correctpanswer, which from a careful use of the direction of information is a diagonal path with minimal cost 2, (see [13]). As h goes to zero, the true desired solution of this propagation and stems from the optimality principle. continuous Eikonal problem is given by the solution We briefly summarize Dijkstra’s method, since the ˇ ˇ1=2 ˇ ˇ flow of logic will be important in explaining fast to ˇu2 C u2 ˇ D K D 1. x y marching methods. For simplicity, imagine a rectangular grid of size h in two space dimensions, where the cost Kij > 0 is given for passing through a grid point xij D .ih; jh/. We first note that the minimal total cost uij of arriving at the node xij can be written in terms of Dijkstra-Like Solvers for Continuous Isotropic the minimal total cost of arriving at its neighbors: Control Nonetheless, algorithms which produce convergent apuij D min.ui 1;j ; ui C1;j ; ui;j 1 ; ui;j C1/ C Kij : (1) proximations to the true shortest path for continuous problems can be obtained: the causality property obThen, to find the minimal total cost of reaching a grid served above can serve as a basis for Dijkstra-like point from a given starting point, Dijkstra’s method di- methods for the Eikonal PDE. The first such method vides grid points into three classes: Far (no information was introduced by Tsitsiklis for isotropic control probabout the correct value of u is known), Accepted (the lems using first-order semi-Lagrangian discretizations correct value of u has been computed), and Considered on uniform Cartesian grids [18]. The fast marching (adjacent to Accepted). Begin by classifying all initial method was developed in [11], uses first-order upstarting points as Considered and assigned with an wind finite differences in the context of isotropic front initial value u = 0. All other points are classified as propagation to build a Dijkstra-like Eikonal solver. Far and assigned an infinite initial value. The algorithm A detailed discussion of similarities and differences proceeds by (1) moving the smallest Considered value of these approaches can be found in [17]. Sethian into the Accepted set, (2) moving its Far neighbors into and collaborators have later extended the fast marchthe Considered set, (3) recomputing all Considered ing approach to higher-order discretizations on grids neighbors according to formula 1, and then returning and meshes [14], more general anisotropic Hamilton– to (1) until all points become Known. This algorithm Jacobi–Bellman PDEs [15, 17], and quasi-variational has the computational complexity of O.N log.N //; inequalities [16]. the factor of log.N / reflects the necessity of mainWe now briefly discuss the finite difference approxtaining a sorted list of the Considered values ui;j imations behind Fast Marching Methods.

F

492

Fast Marching Methods

Fast Marching Methods, Fig. 1 Segmented fundus vessels [19] Navigating chemical accessibility space [7] 3D Navier–Stokes multiphase/multifluid with interface permeability and surface tension under external agitator [9, 10]

3. Recompute the values of u at all Considered neighbors of Trial by solving the piecewise quadratic Eq. 2. 4. Add point Trial to Accepted; remove from Considjruj D K.x/ ered. where K.x/ is the cost at point x in the domain. As a 5. Return to top until the Considered set is empty. two-dimensional example, we replace the gradient by This is the fast marching method given in [11]: the key to efficient implementation lies in a fast heap algorithm an upwind approximant of the form: to locate the grid point with the smallest value for u in " #1=2 set of Considered grid points. max.Dijx u; DijCx u; 0/2 C D Kij ; (2) y Cy max.Dij u; Dij u; 0/2 Beyond the Eikonal Equation: Dijkstra-like Solvers where we have used standard finite difference notation. for Continuous Anisotropic Control The fast marching method is as follows. Suppose The above solvers are special cases of the more general at some time the Eikonal solution is known at a set “Ordered Upwind Methods,” introduced by Sethian of Accepted points. For every not-yet accepted grid and Vladimirsky in [15–17] for full continuous optimal point with an Accepted neighbor, we compute a trial control problem, in which the cost function depends on solution to the above quadratic Eq. 2, using the given both position and direction. This corresponds to probvalues for u at accepted points and values of 1 at all lems in anisotropic front propagation, with applications other points. We now observe that the smallest of these to such areas as seismic exploration and semiconductor trial solutions must be correct, since it depends only processing. They showed how to produce the solution on accepted values which are themselves smaller. This uij by recalculating each uij at most r times, where r “causality” relationship can be exploited to efficiently depends only the equation and the mesh structure, but not upon the number of mesh points. and systematically compute the solution as follows: Building one-pass Dijkstra-like methods for general First, tag points in the boundary conditions as Accepted. Then tag as Considered all points one grid point optimal control is considerably more challenging than away and compute values at those points by solving it is for the Eikonal case, since characteristics no Eq. 2. Finally, tag as Far all other grid points. Then the longer coincide with gradient lines of the viscosity solution. Thus, characteristics and gradient lines may loop is: 1. Begin loop: Let Trial be the Considered point with in fact lie in different simplexes. This is precisely why the Eikonal solvers discussed above cannot be smallest value of u. 2. Tag as Considered all neighbors of Trial that are not directly applied in the anisotropic (non-Eikonal) case: Accepted. If the neighbor is in Far, remove it from it is no longer possible to de-couple the system by computing/accepting mesh points in ascending order. that set and add it to the set Considered.

Fast Marching Method Update Procedure We approximate the Eikonal equation

Fast Methods for Large Eigenvalues Problems for Chemistry

493

The key idea introduced in [15,16] is to use the local 7. Haranczyk, M., Sethian, J.A.: Navigating molecular worms inside chemical labyrinths. PNAS 106, 21472–21477 (2009) anisotropy of the cost function to limit the number of 8. Malladi, R., Sethian, J.A.: Fast methods for shape extraction points on the accepted front that must be examined in in medical and biomedical imaging. In: Malladi, R. (ed.) the update of each Considered point. Define F1 .F2 / to Geometric Methods in Biomedical Image Analysis, pp. 1– 13. Springer, Berlin/Heidelberg (2002) the maximum (minimum) of the cost function K.x; aE /, where aE is unit vector determining the motion. Then the 9. Saye, R., Sethian, J.A.: The Voronoi implicit interface method for computing multiphase physics. Proc. Natl. Acad. anisotropy ratio F1 =F2 can be used to exclude a large Sci. 108(49), 19498–19503 (2011) fraction of points on the Accepted Front in the update 10. Saye, R., Sethian, J.A.: Analysis and applications of the Voronoi implicit interface method. J. Comput. Phys. of any Considered Point; the size of this excluded 231(18), 6051–6085 (2012) subset depends on the anisotropy ratio. The result 11. Sethian, J.A.: A fast marching level set method for monotonare one-pass Dijkstra-like methods with computational ically advancing fronts. Proc. Natl. Acad. Sci. 93(4), 1591– complexity O.. FF21 /2 N log.N //. See [16] for proof of 1595 (1996) convergence to the viscosity solution and [16, 17] for 12. Sethian, J.A.: Fast marching level set methods for threedimensional photolithography development. In: Proceednumerous examples. Examples There are a large number of algorithmic extensions for fast marching methods, including higher-order versions [14] and parallel implementations. They have been used in many applications, including as reinitialization techniques in level set methods [1, 4], seismic inversion [3] and the computation of multiple arrivals in wave propagation [6], medical imaging [8], and photolithography development in semiconductor manufacturing. Here, we show three examples. On the left, variants of fast marching methods are used to segment out blood vessels in the eye. In the middle, a highdimensional version is used to find accessible pathways through a chemical structure. On the right, they form part of the core Voronoi step in Voronoi Implicit Interface Techniques to track problems involving multiple interacting multiphase physics.

References 1. Adalsteinsson, D., Sethian, J.A.: The fast construction of extension velocities in level set methods. J. Comput. Phys. 148, 2–22 (1999) 2. Andrews, J., Sethian, J.A.: Fast marching methods for the continuous traveling salesman problem. Proc. Natl. Acad. Sci. 104(4), 1118–1123 (2007) 3. Cameron, M., Fomel, S., Sethian, J.A.: Seismic velocity estimation using time migration velocities. Geophysics 73(5), VE205–VE210 (2008) 4. Chopp, D.: Some improvements of the fast marching method. SIAM J. Sci. Comput. 23, 230–244 (200) 5. Dijkstra, E.W.: A note on two problems in connection with graphs. Numer. Math. 1, 269–271 (1959) 6. Fomel, S., Sethian, J.A.: Fast phase space computation of multiple arrivals. Proc. Natl. Acad. Sci. 99(11), 7329–7334 (2002)

13.

14. 15.

16.

17.

18.

19.

ings of SPIE 1996 International Symposium on Microlithography, Santa Clara, Mar 1996 Sethian, J.A.: Level Set Methods and Fast Marching Methods, 2nd edn. Cambridge University Press, Cambridge/New York (1999) Sethian, J.A.: Fast marching methods. SIAM Rev. 41(2), 199–235 (1999) Sethian, J.A., Vladimirsky, A.V.: Ordered upwind methods for static Hamilton-Jacobi equations. Proc. Natl. Acad. Sci. 98(20), 11069–11074 (2001) Sethian, J.A., Vladimirsky, A.V.: Ordered upwind methods for hybrid control. In: Proceedings 5th International Workshop (HSCC 2002), Stanford, 25–27 Mar. LNCS, vol. 2289 (2002) Sethian, J.A., Vladimirsky, A.V.: Ordered upwind methods for static Hamilton-Jacobi equations: theory and algorithms. SIAM J. Numer. Anal. 41(1), 325–363 (2003) Tsitsiklis, J.N.: Efficient algorithms for globally optimal trajectories. IEEE Trans. Autom. Control 40, 1528–1538 (1995) Ushizima, D., Medeiros, F.: Retinopathy diagnosis from ocular fundus image analysis. Modeling and Analysis of Biomedical Image, SIAM Conference on Imaging Science (IS10), Chicago (2010)

Fast Methods for Large Eigenvalues Problems for Chemistry Yousef Saad Department of Computer Science and Engineering, University of Minnesota, Minneapolis, MN, USA

Introduction The core of most computational chemistry calculations consists of a large eigenvalue problem which is to be solved several times in the course of a complex nonlinear iteration. Other entries in this

F

494

encyclopedia discuss the origin of this problem which is common in electronic structures for example. The entries ( Large-Scale Electronic Structure and Nanoscience Calculations), ( Numerical Analysis of Eigenproblems for Electronic Structure Calculations), ( Density Functional Theory), ( Self-Consistent Field (SCF) Algorithms), ( Finite Difference Methods), ( Finite Element Methods for Electronic Structure), ( Numerical Approaches for High-Dimensional PDEs for Quantum Chemistry) discuss various related aspects of the problem and should be consulted for details. Here we focus specifically on the numerical solution of the eigenvalue problem. Due to the rich variety of techniques used in computational chemistry, one is faced with many different types of matrix eigenvalue problems to be solved. One common feature of these problems is that they are real symmetric and that the number of eigenvalues or eigenvectors to be computed is often large, being of the order of the total number of valence electrons in the system. Apart from this feature, the main differences between the problems arise from the type of discretization as well as the specific technique used. When plane-wave bases are used for example, the matrix is typically dense and it is often not formed explicitly but used in the form of a matrix-vector product subroutine which is invoked in the diagonalization routine. When real-space methods are used, for example, [2, 5, 6, 13], the matrix is sparse and is either explicitly available using some sparse matrix storage, see, for example, [11] or is again available in the form of a “stencil” operation, see, e.g., [6]. Realspace codes benefit from savings brought about by not needing to store the Hamiltonian matrix, although this may be balanced by the need to store larger vector bases. As was mentioned above, a common difficulty when solving the (discretized) eigenproblems lies in the large number of required eigenvalues/vectors to be computed. This number can in the thousands or tens of thousands in modern calculations. In addition to storage, maintaining the orthogonality of the basis vectors or just the approximate eigenvectors can be very demanding, often resulting in the most computationally expensive part of diagonalization codes. Another challenge is that the relative separation of the eigenvalues decreases as the matrix size increases, and this has an adverse effect on the rate of convergence of

Fast Methods for Large Eigenvalues Problems for Chemistry

the eigenvalue solvers. Preconditioning techniques are often invoked to remedy this. Among the oldest methods for computing eigenspectra is the well-known power method and its block generalization the subspace iteration developed by Bauer in the 1960s, see, for example, [10, 11]. Krylov subspace methods, among which are the Lanczos, and Arnoldi methods, appeared in the early 1950s but did not get the attention they deserved for various reasons until the 1970s and 1980s [10,11]. These are projection methods, that is, methods for extracting approximate eigenvectors from selected subspaces. They can be improved by adding polynomial acceleration shiftand-invert [10], or implicit restart [8]. Among other variations to the main scheme of the Krylov approach are Davidson’s method, Generalized Davidson’s method [9], or the Jacobi-Davidson approach [3].

Projection Methods and the Subspace Iteration Algorithm Numerical algorithms for extracting eigenvalues and vectors of large matrices often combine a few common ingredients. The following three can be found in the most successful algorithms in use today: (1) projection techniques, (2) preconditioning techniques, and (3) deflation and restarting techniques. This sections focuses on general projection-type methods and will discuss the subspace iteration algorithm. Projection Methods and the Rayleigh–Ritz Procedure These are techniques for extracting eigenvalues of a matrix from a given subspace. The general formulation of a projection process starts with two subspaces K and L of the same dimension. The subspace K is the subspace of “approximants,” that is, the approximate eigenvectors will be sought among vectors in this space. Let m be the dimension of this space. The subspace K is the subspace of constraints, i.e., it determines the constraints that need to be applied to extract the approximations from K. Since we have m degrees of freedom, we also need m constraints to determine eigenvectors, so L will be of dimension m as well. The projection process fill extract an approximate Q uQ such that: eigenpair ;

Fast Methods for Large Eigenvalues Problems for Chemistry

Q 2 C; uQ 2 KI

Q  A/Qu ? L .I

(1)

495

One of the main advantages of subspace iteration is its ease of implementation, especially in the symmetric case. The method is also easy to analyze mathematically. On the other hand, a known disadvantage of the method is that it is generally slower than its rivals obtained from Krylov subspaces. There are, however, some important uses of the method in practice. Because its analysis is rather simple, the method provides an attractive means for validating results obtained from a given subspace. This is especially true in the real symmetric (or complex Hermitian case). The method is often used with polynomial acceleration: Ak X replaced by Ck .A/X , where Ck is typically a Chebyshev polynomial of the first kind, see, for example, [11].

The special case when L D K is that of Orthogonal projection methods. The general case, when L ¤ K, corresponds to Oblique projection methods, which is not too common for the symmetric case. The question which we will address now is the following. We are given a subspace X which is known to contain good approximations to some of the eigenvectors of A and we would like to extract these approximations. A good way to do this is via the Rayleigh– Ritz process, which is a projection method onto the subspace X and orthogonally to X . In the above notation, this means that the process uses L D K D X . We start by building an orthonormal basis Q D Œq1 ; : : : ; qm  of X . Then we write an approximation Algorithm 2 Subspace Iteration with Projection in the form uQ D Qy and obtain y by writing the Q /Qu D 0 which yields Start: Choose an initial system of vectors orthgonality condition QH .A I X D Œx0 ; : : : ; xm  and an initial the projected eigenvalue problem: Q QH AQy D y:

Algorithm 1 Rayleigh–Ritz Procedure 1. 2. 3. 4.

Obtain an orthonormal basis Q of X Compute C D QH AQ (an m  m matrix) Obtain Schur factorization of C , C D YRY H Compute UQ D QY

When X is (exactly) invariant, then this procedure will yield exact eigenvalues and eigenvectors. Indeed Q /u D Qz for a certain z. since X is invariant, .A  I H Then Q Qz D 0 implies z D 0 and therefore Q /u D 0. This procedure is often referred to .A  I as subspace rotation in the computational chemistry literature. Indeed, Q, and UQ D QY , are both bases of the same subspace X . Subspace Iteration The original idea of subspace iteration is that of a projection technique (Rayleigh–Ritz) onto a subspace if the form Y D Ak X , where X is a matrix of size n  m, representing the basis of some initial subspace. As can be seen, this is a simple generalization of the power method, which iterates with a single vector. In practice, Ak is replaced by a suitable polynomial, for example, Chebyshev.

polynomial Ck . Iterate: Until convergence Do: Compute ZO D Ck .A/Xold . Orthonormalize ZO into Z. Compute B D Z H AZ and use the QR algorithm to compute the Schur vectors Y D Œy1 ; : : : ; ym  of B. Compute Xnew D Z Y . Test for convergence. If satisfied stop. Else select a new polynomial C 0 k0 and continue. EndDo

If the eigenvalues of A are labeled such that j1 j  j2 j     jm j > jmC1 j;    , then, in the symmetric real case and without acceleration (Ck .t/ D t k ) the error for the i -th approximate eigenvector uQ i , with i  m, will behave like jmC1 =i jk . The advantage of this approach when compared with alternatives in the context of DFT calculations is that the subspace iteration procedure can exploit the subspace calculated in the previous Self-Consistent Field (SCF) iteration. In fact, we can even make the subspace iteration a nonlinear process, as was suggested in [6]. In other words, the Hamiltonian is updated at each restart of the subspace iteration loop, instead of waiting for the eigenvalues to all converge. The savings in computational time with this approach can be substantial over conventional, general purpose, techniques.

F

496

Fast Methods for Large Eigenvalues Problems for Chemistry

Krylov Subspace Methods Krylov Subspace Methods are projection methods on Krylov subspaces, that is, on subspace of the form:

Km .A; v1 / D spanfv1 ; Av1 ;    ; Am1 v1 g

0

˛1 Bˇ2 B Tm D B @

1 ˇ2 C ˛2 ˇ3 C : :: :: :: C : : :A ˇm ˛m

(2) Further, denote by Vm the n  m matrix Vm D Œv1 ; : : : ; vm  and by em the mth column of the m  m identity matrix. After m steps of the algorithm, the where v1 is some initial vector and m is an integer. following relation holds: This is arguably the most important class of projection methods today, whether for solving linear systems of T : AVm D Vm Tm C ˇmC1 vmC1 em equations or for solving eigenvalue problems. Many variants of Krylov subspace methods exist depending on the choice of subspace L as well as other choices In the ideal situation, where ˇmC1 D 0 for a certain m, AVm D Vm Tm , and so the subspace spanned by the related to deflation and restarting. Note that Km is the subspace of vectors of the vi ’s is invariant under A, and the eigenvalues of Tm form p.A/v where p is a polynomial of degree not become exact eigenvalues of A. This is the situation exceeding m  1. If  is the degree of the minimal when m D n, and it may also happen for m n, in polynomial of v, then, Km D K for all m   and K highly unlikely cases referred to as lucky (or happy) is invariant under A. In fact dim.Km / D m iff   m. breakdowns [10]. In the generic situation, some of the eigenvalues of the tridiagonal matrix Hm will start approximating corresponding eigenvalues of A when m The Lanczos Algorithm becomes large enough. An eigenvalue Q of Hm is called The Lanczos algorithm is one of the best-known tech- a Ritz value, and if y is an associated eigenvector, then niques for diagonalizing a large sparse matrix A. In the vector Vm y is, by definition, the Ritz vector, that theory, the Lanczos algorithm generates an orthonor- is, the approximate eigenvector of A associated with Q If m is large enough, the process may yield good mal basis v1 ; v2 ; : : : ; vm , via an inexpensive three-term . approximations to the desired eigenvalues 1 ; : : : ; s recurrence of the form: of H , corresponding to the occupied states, that is, all occupied eigenstates. ˇj C1 vj C1 D Avj  ˛j vj  ˇj vj 1 : In practice, orthogonality of the Lanczos vectors, which is guaranteed in theory, is lost as soon as one of the eigenvectors starts to converge [10]. A number of In the above sequence, ˛j D vjH Avj , and ˇj C1 D schemes have been developed to remedy this situation kAvj  ˛j vj  ˇj vj 1 k2 . So the j th step of the in different ways; see [10] for a discussion. A common algorithm starts by computing ˛j , then proceeds to method, called partial reorthogonalization, consists of form the vector vOj C1 D Avj  ˛j vj  ˇj vj 1 , and modeling loss of orthogonality by building a scalar then vj C1 D vOj C1 =ˇj C1 . Note that for j D 1, the recurrence, which parallels the three-term recurrence formula for vO 2 changes to vO 2 D Av2  ˛2 v2 . The of the Lanczos vectors. As soon as loss of orthogonality algorithm is a form of the Gram-Schmidt process for is detected by this scalar recurrence, a reorthogonalcomputing an orthonormal basis of Km .A; v1 /. Indeed, ization step is taken which orthogonalizes the current if at step j we form Avj and try to orthogonalize it vj C1 against all previous vi ’s. This is the approach against v1 ; v2 ; vj 1 ; vj , we would discover that all the taken in the computational codes PROPACK [7] and coefficients required for the orthogonalization are zero PLAN [14]. In these codes, semi-orthogonality is enexcept the ones for vj and vj 1 . Of course, this result forced, that is, the inner product of two basis vectors is only guaranteed not to exceed a certain threshold, holds in exact arithmetic only. p Suppose that m steps of the recurrence are carried which is of the order of where is the machine epsilon [4]. out, and consider the tridiagonal matrix:

Fast Methods for Large Eigenvalues Problems for Chemistry

Since the eigenvectors are not individually needed in electronic structure calculation, one can think of not computing them but rather to just use a Lanczos basis Vm D Œv1 ; : : : ; vm  directly. This does not provide a good basis in general. A Lanczos algorithm with partial reorthogonalization can work quite well although it tends to require large bases. See [1] for important implementation aspects of a Lanczos procedure deployed within a real-space electronic structure calculation code. Implicit Restarts Implicit restarts techniques consist of combining two ingredients: polynomial acceleration and implicit deflation. The method, due to Lehoucq and Sorensen [8], exploits the intricate relationship between the QR algorithm for computing eigenvalues of matrices and polynomial filtering within Arnoldi’s procedure. Specifically, we can restart the Arnoldi algorithm, with v1 replaced by q.A/v1 where q is a polynomial of degree k, by performing the same m C k steps of a modified Arnoldi procedure. Davidson’s Approach Another popular algorithm for extracting the eigenpairs is the Davidson [9] method, which can be viewed as a preconditioned version of the Lanczos algorithm, in which the preconditioner is the diagonal of A. We refer to the generalized Davidson algorithm as a Davidson approach in which the preconditioner is not restricted to being a diagonal matrix. The Davidson algorithm differs from the Lanczos method in the way in which it defines new vectors to add to the projection subspace. Instead of adding just Avj , it preconditions a given residual vector ri D .A  i I /ui and adds it to the subspace (after orthogonalizing it against current basis vectors). The algorithm consists of an “eigenvalue loop,” which computes the desired eigenvalues one by one (or a few at a time), and a “basis” loop which gradually computes the subspace on which to perform the projection. Consider the eigenvalue loop which computes the i th eigenvalue and eigenvector of A. If M is the current preconditioner, and V D Œv1 ;    ; vk  is the current basis, the main steps of the outer (eigenvalue) loop are as follows: 1. Compute the i th eigenpair .k ; yk / of Ck D VkT AVk . 2. Compute the residual vector rk D .A  k I /Vk yk . 3. Precondition rk , i.e., compute tk D M 1 rk .

497

4. Orthonormalize tk against v1 ;    ; vk and call vkC1 the resulting vector, so VkC1 D ŒVk ; vkC1 . 5. Compute the last column-row of CkC1 D T VkC1 AVkC1 . The original Davidson approach used the diagonal of the matrix as a preconditioner, but this works only for special cases. For a plane-wave basis, it is possible to construct fairly effective preconditioners by exploiting the lowerorder bases. By this, we mean that if Ak is the matrix representation obtained by using k plane waves, we can construct a good approximation to Ak from Am , with m k, by completing it with a diagonal matrix representing the larger (undesirable) modes, see, for example, [12]. Note that these matrices are not explicitly computed since they are dense. This possibility of building lower-dimensional approximations to the Hamiltonian, which can be used to precondition the original matrix, constitutes an advantage of planewave-based methods. For real-space discretizations, preconditioning techniques are often based on filtering ideas and the fact that the Laplacian is an elliptic operator. The eigenvectors corresponding to the few lowest eigenvalues of r 2 are smooth functions, and so are the corresponding wave functions. When an approximate eigenvector is known at the points of the grid, a smoother eigenvector can be obtained by averaging the value at every point with the values of its neighboring points. Other preconditioners that have been tried resulted in mixed success. For example, the use of shift-and-invert [10] involves solving linear systems with A  I , where A is the original matrix, and the shift  is close to the desired eigenvalue (s). These methods would be prohibitively expensive in most situations of interest in DFT codes given the size of the matrix, and the number of times that A  I must be factored given the usually large number of eigenvalues to be computed. Real-space algorithms avoid the use of fast Fourier transforms by performing all calculations in real physical space instead of Fourier space. Fast Fourier transforms require global communication; as such, they tend to be harder to implement on message-passing distributed memory multiprocessor systems. The only global operation remaining in real-space approaches is that of the inner products which will scale well as long as the vector sizes in each processor remain relatively large.

F

498

Fast Multipole Methods

Acknowledgements Work supported by DOE DE-SC0001878 and by the Minnesota Supercomputer Institute.

Fast Multipole Methods

References

Per-Gunnar Martinsson Department of Applied Mathematics, University of Colorado, Boulder, CO, USA

1. Bekas, C., Saad, Y., Tiago, M.L., Chelikowsky, J.R.: Computing charge densities with partially reorthogonalized Lanczos. Comput. Phys. Commun. 171(3), 175–186 (2005) 2. Fattebert, J.L., Bernholc, J.: Towards grid-based o(n) density-functional theory methods: optimized nonorthogonal orbitals and multigrid acceleration. Phys. Rev. B 62, 1713–1722 (2000) 3. Fokkema, D.R., Sleijpen, G.L.G., van der Vorst, H.A.: Jacobi-Davidson style QR and QZ algorithms for the reduction of matrix pencils. SIAM J. Sci. Comput. 20(1), 94–125 (1998) 4. Golub, G.H., Van Loan, C.F.: Matrix Computations, 3rd edn. Johns Hopkins University Press, Baltimore (1996) 5. Heikanen, M., Torsti, T., Puska, M.J., Nieminen, R.M.: Multigrid method for electronic structure calculations. Phys. Rev. B 63, 245106–245113 (2001) 6. Kronik, L., Makmal, A., Tiago, M.L., Alemany, M.M.G., Jain, M., Huang, X., Saad, Y., Chelikowsky, J.R.: PARSEC the pseudopotential algorithm for real-space electronic structure calculations: recent advances and novel applications to nano-structure. Phys. Status Solidi (B) 243(5), 1063–1079 (2006) 7. Larsen, R.M.: PROPACK: a software package for the symmetric eigenvalue problem and singular value problems on Lanczos and Lanczos bidiagonalization with partial reorthogonalization, SCCM, Stanford University. http://sun. stanford.edu/ rmunk/PROPACK/ 8. Lehoucq, R., Sorensen, D.C.: Deflation techniques for an implicitly restarted Arnoldi iteration. SIAM J. Matrix Anal. Appl. 17, 789–821 (1996) 9. Morgan, R.B., Scott, D.S.: Generalization of Davidson’s method for computing eigenvalues of sparse symmetric matrices. SIAM J. Stat. Sci. Comput. 7, 817–825 (1986) 10. Parlett, B.N.: The Symmetric Eigenvalue Problem. Number 20 in Classics in Applied Mathematics. SIAM, Philadelphia (1998) 11. Saad, Y.: Numerical Methods for Large Eigenvalue Problems- Classics Edition. SIAM, Philadelpha (2011) 12. Teter, M.P., Payne, M.C., Allen, D.C.: Solution of Schr¨odinger’s equation for large systems. Phys. Rev. B 40(18), 12255–12263 (1989) 13. Torsti, T., Heiskanen, M., Puska, M.J., Nieminen, R.M.: MIKA: a multigrid-based program package for electronic structure calculations. Int. J. Quantum Chem. 91, 171–176 (2003) 14. Wu, K., Simon, H.: A parallel Lanczos method for symmetric generalized eigenvalue problems. Technical Report 41284, Lawrence Berkeley National Laboratory, 1997. Available on line at http://www.nersc.gov/research/SIMON/ planso.html

Short Definition The Fast Multipole Method (FMM) is an algorithm for rapidly evaluating all pairwise interactions in a system of N electrical charges. While the direct computation requires O.N 2 / work, the FMM carries out this task in only O.N / operations. A parameter in the FMM is the prescribed accuracy " to within which the electrostatic potentials and forces are computed. The choice of " affects the scaling constant implied by the O.N / notation. A more precise estimate of the time required (in two dimensions) is O.N log .1="// as " ! 0. More generally, the term “FMM” refers to a broad class of algorithms with linear or close to linear complexity for evaluating all pairwise interactions between N particles, given some pairwise interaction kernel (e.g., the kernels associated with elasticity, gravitation, wave propagation). An important application is the evaluation of the matrix-vector product x 7! Ax where A is a dense N  N matrix arising from the discretization of an integral operator. The classical FMM and its descendants rely on quadtrees or octtrees to hierarchically subdivide the computational domain. This tree structure enables the algorithms to adaptively refine the data structure to nonuniform charge distributions and makes them well suited for parallel implementations.

Introduction To introduce the concepts supporting fast summation techniques like the FMM, we will in this note describe a bare-bones algorithm for solving the problem addressed in the original work [10] of Greengard and Rokhlin, namely, the evaluation of all pairwise interactions between a set of N electrical charges in the plane. The basic technique has since [10] was published been substantially improved and extended. Analogous fast summation techniques have also been developed for related summation problems, most notably those

Fast Multipole Methods

499

associated with acoustic and electromagnetic scattering theory. These improvements and extensions are reviewed in section “Extensions, Accelerations, and Generalizations.”

( G.x; y/ D

 log.x  y/

x ¤ y;

0

x D y:

We introduce a vector q 2 CN and a matrix A 2 CN N via q.i / D qi ;

Notation

and

A.i; j / D G.x i ; x j /

i; j D 1; 2; 3; : : : ; N: We let fx i gN i D1 denote the locations of a set of electrical N charges and let fqi gi D1 denote their source strengths. We then seek to evaluate the matrix-vector product Our task is then to evaluate the potentials N X ui D g.x i ; x j / qj ; i D 1; 2; : : : ; N; (1) u D A q: (3) j D1

The real potentials ui defined by (1) are given by real where g.x; y/ is the interaction potential of electro- parts of the entries of u. statics in the plane ( g.x; y/ D

 log jx  yj

x ¤ y;

0

x D y:

(2) The General Idea

The key to rapidly evaluating the sum (1) is that the Let denote a kernel g.x; y/ defined by (2) is smooth when x and y (We omit the common scaling by are not close. To illustrate how this can be exploited, square that holds all points; see Fig. 1a. It will be convenient to use a complex notation. We let us first consider a simplified situation where we are N think of each source location x i as a point in the com- given a set of electrical sources fqj gj D1 at locations N plex plane and let G denote the complex interaction fy j gj D1 in one box  and seek the potentials these sources induce at some target locations fxi gM potential i D1 in a 1 2 .)

a

b

cσ Ωσ

cτ Ωτ

Ω

Fast Multipole Methods, Fig. 1 (a) Geometry of the full N -body problem. The domain is drawn in black and the points x i are gray. (b) The geometry described in sections

Ω

“The General Idea” and “Multipole Expansions.” The box  contains source locations (red) and contains target locations (blue)

F

500

Fast Multipole Methods

different box . In other words, we seek to evaluate matrix-vector product (3) by exploiting rank deficienthe sum cies in off-diagonal blocks of A. The algorithm will be introduced incrementally. N Section “Multipole Expansions” formalizes the discusX ui D g.x i ; y j / qj ; i D 1; 2; : : : ; M: (4) sion of the case where target and source boxes are j D1 separate. Section “A Single-Level Method” describes a method based on a single-level tessellation of the Since g is smooth in this situation, we can approximate domain. Section “Conceptual Description of a Multiit by a short sum of tensor products level Algorithm” provides a conceptual description of a multi-level algorithm with O.N / complexity, with P 1 details given in sections “A Tree of Boxes” and “The X g.x; y/  Bp .x/ Cp .y/; when x 2 ; y 2  ; Classical Fast Multipole Method.” pD0

(5) where P is a small integer called the interaction rank. (How to construct the functions Bp and Cp and how to choose P will be discussed in section “Multipole Expansions.”) As a result, an approximation to the sum (4) can be constructed via the two steps: qOp D

X

Cp .x j / qj ;

j 2I

Multipole Expansions

We start by considering a subproblem of (1) corresponding to the interaction between two disjoint subsets  and , as illustrated in Fig. 1b. Specifically, we seek to evaluate the potential at all points in (the “target points”) caused by sources in  . To formalize, p D 0; 1; 2; : : : ; P  1 let I and I be index sets pointing to the locations (6) inside each box so that, e.g.,

and ui 

P 1 X

i 2 I Bp .x i / qOp ;

i D 1; 2; : : : ; M:

(7)

pD0

While evaluating (4) directly requires M N operations, evaluating (6) and (7) requires only P .M C N / operations. The power of this observation stems from the fact that high accuracy is achieved even for small P when the regions  and are moderately well separated; cf. section “Error Analysis.” Using matrix notation, the approximation (5) implies that the M  N matrix A with entries A.i; j / D g.x i ; y j / admits an approximate rank-P factorization A  B C. Then clearly the matrix-vector product Aq can cheaply be evaluated via Aq  B .Cq/. In the problem (1), the summation problem that we are actually interested in, the sets of target locations, and source locations coincide. In this case, no one relation like (5) can hold for all combinations of target and source points. Instead, we are going to cut the domain up into pieces and use approximations such as (5) to evaluate interactions between distant pieces and use direct evaluation only for points that are close. Equivalently, one could say that we will evaluate the

,

x i 2  :

Our task is then to evaluate the sums vi D

X

G.x i ; x j / qj ;

i 2 I :

(8)

j 2I

In matrix notation, (8) is equivalent to the matrixvector product v D A.I ; I / q.I /:

(9)

We will next derive an approximation like (5) for the kernel in (8). To this end, let c  and c denote the centers of  and , respectively. Then, for y 2  and x 2 ,     G.x; y/ D log xy D log .xc  /.y  c  /     y  c D  log x  c   log 1  x  c 1   X 1 .y  c  /p D  log x  c  C ; (10) p .x  c  /p pD1

Fast Multipole Methods

501

where the series converges whenever jy  c  j < jx  c  j. Observe that the last expression in (10) is precisely of the form (5) with Cp .y/ D p1 .y  c  /p and Bp .x/ D .x  c  /p . When the sum is truncated after P  1 terms, the error incurred is roughly of size  P jy  c  j=jx  c  j . We define the outgoing expansion of  as the  P 1 vector qO D fqOp gpD0 where X 8 ˆ qO0 D qj ˆ ˆ ˆ ˆ j 2I  ˆ ˆ < X 1 p  x j  c  qj ; ˆ qOp D ˆ p ˆ j 2I ˆ ˆ ˆ ˆ : p D 1; 2; 3; : : : ; P  1:

 qO D Tofs  q.I /;

(11)

 The vector qO is a compact representation of the sources in  . It contains all information needed to P evaluate the field v.x/ D j 2I G.x; x j / qj when x is a point “far away” from  . It turns out to be convenient to also define an incoming expansion for . The basic idea here is that for x 2 , the potential

v.x/ D

X

1 X pD1

1 qO  .x  c  /p p

(12)

F

where Tofs  is a P N matrix called the outgoing-fromsources translation operator with the entries implied by (11). Analogously, (13) can upon truncation be

ifo O written vO D Tifo

; q , where T ; is the incoming-fromoutgoing translation operator. Finally, the targets-fromincoming translation operator is the matrix Ttfi

such

O , where v is an approximation to that v D Ttfi v

tfi the field v defined by (12); in other words T .i; p/ D .x i  c /p1 . These three translation operators are factors in an approximate rank-P factorization Tifo Tofs A.I ; I /  Ttfi

;  : N  P P  P P  N N  N

G.x; x j / qj D  log.x  c  / qO 0

j 2I

C

P 1 is the incoming expansion The vector vO D fvO p gpD0 for generated by the sources in  . It is a compact (approximate) representation of the harmonic field v defined by (12). The linear maps introduced in this section can advantageously be represented via matrices that we refer to as translation operators. Let N and N denote the number of points in  and , respectively. The map (11) can then upon truncation be written

(14)

A diagram illustrating the factorization (14) is given as Fig. 2.

Remark 1 The terms “outgoing expansion” and “incoming expansion” are slightly nonstandard. The corresponding objects were in the original papers called “Multipole Expansion” and “Local Expansion,” and 1 X  p these terms continue to be commonly used, even in v.x/ D x  c vOp : summation schemes where the expansions have nothpD0 ing to do with multipoles. Correspondingly, what we A simple computation shows that the complex numbers call the “incoming-from-outgoing” translation operator is often called the “multipole-to-local” or “M2L” fvO p g1 O p g1 pD0 can be obtained from fq pD0 via operator. 8 1 X 1 ˆ ˆ vO0 D qO0 log.c  c / C qOj .1/j ; ˆ ˆ ˆ .c  c /j  ˆ j D1 ˆ ˆ ˆ ˆ ! A Single-Level Method < 1 X i C j  1 1 vO D qO0 C qOj .1/j ˆ Having dealt with the simplified situation where the ˆ i j 1 i.c  c /i ˆ j D1 ˆ ˆ source points are separated from the target points in ˆ ˆ ˆ ˆ 1 section “Multipole Expansions,” we now return to the ˆ : : i Cj original problem (1) where the two sets of points are .c  c / (13) the same. In this section, we construct a simplistic

is a harmonic function on . In consequence, it has a convergent expansion

502

Fast Multipole Methods

The full vector of sources in σ. Vector of length Nσ (a long vector). qσ

A(Iτ,Iσ)

The full vector of potentials in τ. Vector of length Nτ (a long vector). vτ tfi

Tσofs ˆσ q

Tτ ifo Tτ,σ

The incoming expansion for τ. A compact representation of the potential. Vector of length k (a short vector).

The outgoing expansion for σ. A compact representation of the sources. Vector of length k (a short vector).

Fast Multipole Methods, Fig. 2 The outgoing and incoming expansions introduced in section “Multipole Expansions” are compact representations of the sources and potentials in the

a

vˆτ

source and target boxes, respectively. The diagram commutes to ifo ofs high precision since A.I ; I /  Ttfi

T ; T ; cf. (14)

b

Fast Multipole Methods, Fig. 3 (a) A tessellation of into m  m smaller boxes; cf. section “A Single-Level Method.” (b) Evaluation of the potential in a box . The target points in are

marked with blue dots, the source points in the neighbor boxes in Lnei

are marked with red dots, and the centers of the outgoing expansions in the far-field boxes Lfar

are marked ˝

method that does not achieve O.N / complexity but will help us introduce some concepts. Subdivide the box into a grid of m  m equisized 2 smaller boxes f gm

D1 as shown in Fig. 3a. As in section “Multipole Expansions,” we let for each box

the index vector I list the points inside and let

c denote the center of . The vector qO denotes the outgoing expansion of , as defined by (11). For a box , let Lnei

denote the list of neighbor boxes; these are the boxes that directly touch (there will be between 3 and 8 of them, depending on where

is located in the grid). The remaining boxes are collected in the list of far-field boxes Lfar

. Figure 3b illustrates the lists. The sum (1) can now be approximated via three steps: (1) Compute the outgoing expansions: Loop over all boxes . For each box, compute its outgoing ex pansion qO via the outgoing-from-sources translation operator:

qO D Tofs

q.I /:

Fast Multipole Methods

503

(2) Convert outgoing expansions to incoming expan- tree, one going upwards (from smaller boxes to larger) sions: Loop over all boxes . For each box, con- and one going downwards:

struct a vector uO called the incoming expansion. The upwards pass: In the upwards pass, the outgoing It represents the contribution to the potential in expansion is computed for all boxes. For a leaf box from sources in all boxes in the far field of and is

, the straight forward approach described in section given by X

“Multipole Expansions” is used. For a box that ifo  T ; qO : uO D has children, the outgoing expansion is computed  2Lfar

not directly from the sources located in the box, but (3) Compute near interactions: Loop over all boxes . from the outgoing expansions of its children, which Expand the incoming expansion and add the conare already available. tributions from its neighbors via direct summation: The downwards pass: In the downwards pass, the incoming expansion is computed for all boxes. This O C A.I ; I / q.I / u.I / D Ttfi is done by converting the outgoing expansions con u X structed in the upwards pass to incoming expansions A.I ; I / q.I /: C via the formula (13). The trick is to organize the  2Lnei

computation so that each conversion happens at its appropriate length scale. Some further machinery The asymptotic complexity of the method as the is required to describe exactly how this is done, number of particles N grows depends on how the 2 but the end result is that the FMM computes the number m is picked. If the number m of boxes is large, incoming expansion for a leaf box from the then Steps 1 and 3 are cheap, but Step 2 is expensive. 2 2=3 outgoing expansions of a set of O.log N / boxes that The optimal choice is m  N and leads to overall 4=3 are sufficiently well separated from the target that complexity O.N /. the expansions are all accurate; cf. Fig. 5.

Conceptual Description of a Multilevel Algorithm To achieve linear complexity in evaluating (1), the FMM uses a multilevel technique in which the computational domain is split into a tree of boxes; cf. Fig. 4. It evaluates the sum (1) in two passes over the

11 3

Once the upwards and downwards passes have been completed, the incoming expansion is known for all leaf boxes. All that remains is then to expand the incoming expansion into potentials and adding the contributions from sources in the near field via direct computations. In order to formally describe the upwards and downwards passes, we need to introduce two new translation

13

19

43

45

51

53

75

77

83

85

42

44

50

52

74

76

82

84

39

41

47

49

71

73

79

81

38

40

46

48

70

72

78

80

27

29

35

37

59

61

67

69

26

28

34

36

58

60

66

68

23

25

31

33

55

57

63

65

22

24

30

32

54

56

62

64

21

5 10

7 2

12

18

9

15

20

17

4 6 Level 1

8

14 Level 2

16 Level 3

Fast Multipole Methods, Fig. 4 A tree of boxes on with L D 3 levels. The enumeration of boxes shown is simply one of the many possible ones

F

504

Fast Multipole Methods

Upon truncating the series in (15) and (16) to the first P terms, we write (15) and (16) in matrix form using the outgoing-from-outgoing translation operator Tofo

; : and the incoming-from-incoming translation operator Tifi ; ,

O qO D Tofo

; q

and

 O : uO D Tifi ; u

ifi Both Tofo

; and T; are matrices of size P  P .

A Tree of Boxes

Fast Multipole Methods, Fig. 5 Illustration of how the FMM evaluates the potentials in a leaf box marked by its blue target points. Contributions to the potential caused by sources in itself (blue dots) or in its immediate neighbors (red dots) are computed via direct evaluation. The contributions from more distant sources are computed via the outgoing expansions centered on the ˝ marks in the figure

operators (in addition to the three introduced in section “Multipole Expansions”). Let be a box containing a smaller box  which in turn contains a set of sources.  Let qO denote the outgoing expansion of these sources around the center c  of . These sources could also be

represented via an outgoing expansion qO around the center c of . One can show that 8 qO D qO 0 ; ˆ ˆ ˆ 0 ˆ ! ˆ ˆ i < X

1 i  i 1 qOi D qO0 .c  c / C qO j ˆ j 1 i ˆ j D1 ˆ ˆ ˆ ˆ : .c  c /i j :

(15)

Split the square into 4L equisized smaller boxes, where the integer L is chosen to be large enough that each box holds only a small number of points. (The optimal number of points to keep in a box depends on many factors, but having about 100 points per box is often reasonable.) These 4L equisized small boxes form the leaf boxes of the tree. We merge the leaves by sets of 4 to form 4L1 boxes of twice the side length and then continue merging by sets of 4 until we recover the original box , which we call the root. The set consisting of all boxes of the same size forms what we call a level. We label the levels using the integers ` D 0; 1; 2; : : : ; L, with ` D 0 denoting the root and ` D L denoting the leaves. Given a box in the hierarchical tree, we next define some index lists; cf. Fig. 6; • The parent of is the box on the next coarser level that contains . of boxes whose • The children of is the set Lchild

parent is . • The neighbors of is the set Lnei

of boxes on the same level that directly touch . • The interaction list of is the set Lint

of all boxes  such that (1)  and are on the same level, (2)  and do not touch, and (3) the parents of  and do touch.

Analogously, now suppose that a set of sources that are The Classical Fast Multipole Method distant to give rise to a potential v in represented

by an incoming expansion uO centered around c . Then We now have all tools required to describe the classical  the corresponding incoming representation uO of v FMM in detail. centered around c  is given by Given a set of sources fqi gN i D1 with associated ! , the first step is to find a minimal locations fxi gN 1 i D1 X j .c  c /j i : uO i D uO j (16) square that holds all points. Next, subdivide into i a hierarchy of smaller boxes as described in section j Di

Fast Multipole Methods

505

11 39

41

47

49

71

73

70

72

59

61

58

60

13

19

21

20 40

46

48

38

29

35

37

27

28

34

36

26

35

9

23

25

31

33

55

57

22

24

30

32

54

56

17

16 Neighbor list of τ = 35

Interaction list of τ = 35

F Interaction list of τ = 9

Fast Multipole Methods, Fig. 6 Illustration of some index vectors called “lists” that were introduced in section “A Tree of Boxes.” For instance, the leftmost figure illustrates that Lnei 35 D f28; 29; 34; 36; 37; 40; 46; 48g (boxes are numbered as in Fig. 4)

“A Tree of Boxes.” Then fix an integer P that deter is available. The incoming expansion uO is then mines the accuracy (a larger P gives higher accuracy constructed via but also higher cost; cf. section “Error Analysis”). The

algorithm then proceeds in five steps as follows: O: uO D uO C Tifi

; u (1) Compute the outgoing expansions on the leaves: Loop over all leaf boxes . For each box, compute

(5) Construct the potentials on all leaf boxes: Loop its outgoing expansion qO via over all leaf boxes . For each box compute the potentials at the target points by expanding the

ofs qO D T q.I /: incoming expansion and adding the contributions from the near field via direct computation, (2) Compute the outgoing expansions on all parent boxes: Loop over all parent boxes ; proceed O C A.I ; I / q.I / u.I / D Ttfi

u from finer to coarser levels so that when a box is X processed, the outgoing expansions for its children C A.I ; I / q.I /:

are already available. The outgoing expansion qO  2Lnei

is then computed via

X

ifo Observe that the translation operators Tofo

; , T ; , and Tifi

; can all be pre computed since they depend only  2Lchild

on P and on the vectors c  c  . The tree structure of the boxes ensures that only a small number of values (3) Convert outgoing expansions to incoming expanof c  c  are encountered. sions: Loop over all boxes . For each box, collect

contributions to its incoming expansion uO from Remark 2 To describe how the FMM computes the cells in its interaction list, potentials at the target points in a given leaf box (cf. Fig. 5), we first partition the computational box: X 

O [ far with sources in the D near q uO D Tifo :

. Interactions

; S near int D C near-field  2L

 are evaluated via

 2Lnei

direct computations. To define the far-field, we first (4) Complete the construction of the incoming expan- define the “list of ancestors” Lanc as the list holding

sion for each box: Loop over all boxes ; proceed the parent, grandparent, great grandparent, etc., of . S S D  . Interactions with from coarser to finer levels so that when a box Then far

2Lanc  2Lint

 is processed, the incoming expansion for its parent sources in the far-field are evaluated via the outgoing

qO D



O Tofo

; q :

506

S S expansions of the boxes in the list 2Lanc .  2Lint

 These are the boxes marked “˝” in Fig. 5.

Fast Multipole Methods

the FMM has few competitors for nonuniform point distributions such as, e.g., the distributions arising from the discretization of a boundary integral equations.

Error Analysis The potentials computed by the FMM are not exact since all expansions have been truncated to P terms. An analysis of how such errors could propagate through the transformations across all levels is technically complicated and should seek to estimate both the worst-case error and the statistically expected error [5]. As it happens, the global error is in most cases similar to the (worst case) local truncation error, P which means p that it scales roughly as ˛ , where ˛ D p 2=.4  2/ D 0:5469    . As a rough estimate, we see that in order to achieve a given tolerance ", we need to pick P  log."/= log.˛/: As P increases, the asymptotic complexity of the 2D FMM is O.P N / (if one enforces that each leaf node holds O.P / sources). In consequence, the overall complexity can be said to scale as log.1="/ N as " ! 0 and N ! 1.

Adaptive Trees for Nonuniform Distributions of Particles For simplicity, the presentation in this brief entry has been restricted to the case of relatively uniform particle distributions for which a fully populated tree (as described in section “A Tree of Boxes”) is appropriate. When the particle distribution is nonuniform, locally adaptive trees perform much better. The basic FMM can readily be adapted to operate on nonuniform trees. The only modification required to the method described in section “The Classical Fast Multipole Method” is that some outgoing expansions need to be broadcast directly to target points, and some incoming expansions must receive direct contributions from source points in certain boxes [2]. Note: In situations where the sources are distributed uniformly in a box, the FMM faces competition from techniques such as P3 M (particle-particle/particlemesh). These are somewhat easier to implement and can be very fast since they leverage the remarkable speed of FFT-accelerated Poisson solvers. However,

Extensions, Accelerations, and Generalizations Extension to R3 In principle, the FMM described for problems in the plane can readily be extended to problems in R3 ; simply replace log jx  yj by 1=jx  yj, replace the McLaurin expansions by expansions in spherical harmonics, and replace the quadtree by an octtree. However, the resulting algorithm performs quite poorly (especially at high accuracies) for two reasons: (1) the typical number of elements in an “interaction list” grows from 27 in 2D to 189 in 3D. (2) The number of terms required in an outgoing or incoming expansion to achieve accuracy " grows from O.log.1="// in 2D to O.log.1="/2/ in 3D. Fortunately, accelerated techniques that use more sophisticated machinery for converting outgoing to incoming expansions have been developed [11]. The Helmholtz Equation One of the most important applications of the FMM is the solution of scattering problems via boundary integral equation techniques. For such tasks a sum like (1) needs to be evaluated for a kernel associated with the Helmholtz equation or the closely related timeharmonic version of the Maxwell equations. When the computational domain is not large compared to the wavelength (say at most a few dozen wavelengths), then an FMM can be constructed by simple modifications to the basic scheme described here. However, when the domain becomes large compared to the scattering wavelength, the paradigm outlined here breaks down. The problem is that the interaction ranks in this case depend on the size of the boxes involved and get prohibitively large at the higher levels of the tree. The (remarkable) fact that fast summation is possible even in the short wavelength regime was established in 1992 [18]. The high-frequency FMM of [18] relies on data structures that are similar to those used in the basic scheme described here, but the interaction mechanisms between (large) boxes are quite different. A version of the high-frequency FMM that is stable in all regimes

Fast Multipole Methods

507

was described in [3]. See also [4]. It was shown in Let us close with a practical note. While it is not [6] that close to linear complexity can be attained that daunting of an endeavor to implement an FMM while relying on rank deficiencies alone, provided that with linear or close to linear asymptotic scaling, it is different tessellations of the domain are implemented. another matter entirely to write a code that actually achieves high practical performance – especially for problems in three dimensions and any problem involvOther Interaction Potentials (Elasticity, ing scattering on domains that are large compared to Stokes, etc.) Variations of the FMM have been constructed for most the wave-length. This would be an argument against of the kernels associated with the elliptic PDEs of using FMMs were it not for the fact that the algorithms mathematical physics such as the equations of elas- are very well suited for black box implementation. ticity [7], the Stokes and unsteady Stokes equations Some such codes are available publicly, and more [9], the modified Helmholtz (a.k.a. Yukawa) equations are expected to become available in the next several years. Before developing a new code from scratch, it is [12], and many more. See [14, 17] for details. usually worthwhile to first look to see if a high-quality code may already be available. Kernel-Free FMMs While FMMs can be developed for a broad range of kernels (cf. section “Other Interaction Potentials (Elasticity, Stokes, etc.)”), it is quite labor intense References to re-derive and re-implement the various translation operators required for each special case. The so-called 1. Beatson, R., Greengard, L.: A short course on fast multipole kernel-free FMMs [8, 19] overcome this difficulty by methods. Wavelets, multilelvel methods and elliptic PDEs. Oxford Uiversity Press, 1–37 (1997) setting up a common framework that works for a broad 2. Carrier, J., Greengard, L., Rokhlin, V.: A fast adaptive range of kernels. Matrix Operations Beyond the Matrix-Vector Product The FMM performs a matrix-vector multiply x 7! Ax involving certain dense N  N matrices in O.N / operations. It achieves the lower complexity by exploiting rank deficiencies in the off-diagonal blocks of the matrix A. It turns out that such rank deficiencies can also be exploited to perform other matrix operations, such as matrix inversions, construction of Schur complements, and LU factorizations, in close to linear time. The so-called H-matrix methods [13] provide a general framework that can be applied in many contexts. Higher efficiency can be attained by designing direct solvers specifically for the linear systems arising upon the discretization of certain boundary integral equations [16].

Practical Notes and Further Reading

3.

4.

5.

6.

7.

8.

9.

We have provided only the briefest of introductions to 10. the vast topic of Fast Multipole Methods. A fuller treatment can be found in numerous tutorials (e.g., [1, 15]), 11. survey papers (e.g., [17]), and full-length textbooks (e.g., [4, 14]).

multipole algorithm for particle simulations. SIAM J. Sci. Stat. Comput. 9(4), 669–686 (1988) Cheng, H., Crutchfield, W.Y., Gimbutas, Z., Greengard, L.F., Ethridge, J.F., Huang, J., Rokhlin, V., Yarvin, N., Zhao, J.: A wideband fast multipole method for the Helmholtz equation in three dimensions. J. Comput. Phys. 216(1), 300–325 (2006) Chew, W.C., Jin, J.-M., Michielssen, E., Song, J.: Fast and Efficient Algorithms in Computational Electromagnetics. Artech House, Boston (2001) Darve, E.: The fast multipole method i: error analysis and asymptotic complexity. SIAM J. Numer. Anal. 38(1), 98– 128 (2000) Engquist, B., Ying, L.: A fast directional algorithm for high frequency acoustic scattering in two dimensions. Commun. Math. Sci. 7(2), 327–345 (2009) Fu, Y., Klimkowski, K.J., Rodin, G.J., Berger, E., Browne, J.C., Singer, J.K., Van De Geijn, R.A., Vemaganti, K.S.: A fast solution method for three-dimensional many-particle problems of linear elasticity. Int. J. Numer. Methods Eng. 42(7), 1215–1229 (1998) Gimbutas, Z., Rokhlin, V.: A generalized fast multipole method for nonoscillatory kernels. SIAM J. Sci. Comput. 24(3), 796–817 (2002) Greengard, L., Kropinski, M.C.: An integral equation approach to the incompressible navier-stokes equations in two dimensions. SIAM J. Sci. Comput. 20(1), 318–336 (1998) Greengard, L., Rokhlin, V.: A fast algorithm for particle simulations. J. Comput. Phys. 73(2), 325–348 (1987) Greengard, L., Rokhlin, V.: A new version of the fast multipole method for the Laplace equation in three dimensions. Acta Numer. 6, 229–269 (1997). Cambridge University Press, Cambridge

F

508 12. Zhang, B., Huang, J., Pitsianis, N.P., Sun, X.: Revision of FMM-Yukawa: An adaptive fast multipole method for screened Coulomb interactions. Comput. Phys. Commun. 181(12), 2206–2207 (2010) 13. Hackbusch, W.: A sparse matrix arithmetic based on H-matrices; part I: introduction to H-matrices. Computing 62, 89–108 (1999) 14. Liu, Y.J.: Fast Multipole Boundary Element Method: Theory and Applications in Engineering. Cambridge University Press, Cambridge (2009) 15. Liu, Y.J., Nishimura, N.: The fast multipole boundary element method for potential problems: a tutorial. Eng. Anal. Boundary Elem. 30(5), 371–381 (2006) 16. Martinsson, P.G., Rokhlin, V.: A fast direct solver for boundary integral equations in two dimensions. J. Comp. Phys. 205(1), 1–23 (2005) 17. Nishimura, N.: Fast multipole accelerated boundary integral equation methods. Appl. Mech. Rev. 55(4), 299–324 (2002) 18. Rokhlin, V.: Diagonal forms of translation operators for the helmholtz equation in three dimensions. Appl. Comput. Harmonic Anal. 1(1), 82–93 (1993) 19. Ying, L., Biros, G., Zorin, D.: A kernel-independent adaptive fast multipole algorithm in two and three dimensions. J. Comput. Phys. 196(2), 591–626 (2004)

Fer and Magnus Expansions Sergio Blanes1 , Fernando Casas2 , Jos´e-Angel Oteo3 , and Jos´e Ros4 1 Instituto de Matem´atica Multidisciplinar, Universitat Polit`ecnica de Val`encia, Val`encia, Spain 2 Departament de Matem`atiques and IMAC, Universitat Jaume I, Castell´on, Spain 3 Departament de F´ısica Te`orica, Universitat de Val`encia, Val`encia, Spain 4 Departament de F´ısica Te`orica and IFIC, Universitat de Val`encia-CSIC, Val`encia, Spain

Fer and Magnus Expansions

in terms of exponentials of combinations of the coefficient matrix A.t/. Equation (1) is a first-order linear homogeneous system of differential equations in which Y .t/ is the unknown n-dimensional vector function. In general, Y0 ; Y , and A are complex-valued. The scalar case, n D 1, has the general solution Z Y .t/ D exp

t

 dx A.x/ Y0 :

(2)

t0

This expression is still valid for n > 1 if the matrix A is constant, or the commutator ŒA.t1 /; A.t2 /  A.t1 /A.t2 /  A.t2 /A.t1 / D 0, for any pair of values of t, t1 , and t2 , or, what is essentially R equivalent, A.t/ and its primitive commute: ŒA.t/; A.t/dt D 0. In the general case, there is no compact formula for the solution of (1), and the Fer and Magnus proposals endeavor to complement (2) in two different directions. If we attach a matrix factor to the expo

Rt nential, Y .t/ D exp t0 dx A.x/ M.t; t0 /Y0 , then the Fer expansion [1] gives an iterative multiplicative prescription to find M.t; t0 /. Alternatively, if we add a term to the argument in the exponential, namely,

Rt Y .t/ D exp t0 dx A.x/ C M.t; t0 / Y0 , then the Magnus expansion [2] provides M.t; t0 / as an infinite series. A salient feature of both Fer and Magnus expansions stems from the following fact. When A.t/ 2 g, a given Lie algebra, if we express Y .t/ D U.t; t0 /Y0 , then U.t; t0 / 2 G, the corresponding Lie group. By construction, the Magnus and Fer expansions live, respectively, in g and G. Furthermore, this is also true for their truncation to any order. In many applications, this mathematical setting reflects important features of the problem.

Synonyms Continuous Baker–Campbell–Hausdorff expansion

Definition

The Magnus Expansion Magnus proposed an exponential representation of the solution of (1) in the form

  Y .t/ D exp ˝.t/ Y0 ; (3) The Fer and Magnus expansions provide solutions to the initial value problem where ˝.0/ D O, and for simplicity, we have taken t0 D 0. dY D A.t/Y; Y .t0 / D Y0 ; t 2 R; Y .t/ 2 Cn ; The noncommutativity of A.t/ for different values dt of t makes the differential equation for ˝.t/ highly (1) nonlinear, namely, A.t/ 2 Cnn ;

Fer and Magnus Expansions

˝ 0 .t/ D A.t/ C

509

1 X Bk .1/k kŠ

.j /

where the operators Sn can be calculated recursively:

kD1

Sn.j / .t/

ktimes

‚ …„ ƒ  Œ˝.t/; Œ   Œ˝.t/; A.t/    :

D

nj X 

.j 1/ ˝m .t/; Snm .t/ ; 2  j  n  1;

mD1

(4)

D Œ˝n1 .t/; A.t/ ; Here, Bk are Bernoulli numbers, and the prime stands .n1/times ‚ …„ ƒ for derivative with respect to t. In spite of being much .n1/ Sn .t/ D Œ˝1 .t/; Œ   Œ˝1 .t/; A.t/    : (9) more complicated than (1), it turns out that (4) can be dealt with by introducing the series expansion After integration, we reach the final result in the 1 X form ˝.t/ D ˝n .t/; (5) Z t nD1 A. /d ; ˝1 .t/ D 0 which constitutes the Magnus expansion or Magnus Z series. Every term ˝k involves k-fold products of n1 X Bj t .j / A matrices. By introducing (5) into (4) and equating ˝n .t/ D S . /d ; n  2: (10) jŠ 0 n j D1 terms of the same order in powers of A, we obtain explicit expressions for each ˝k . The first three .k/ terms read The expression of Sn can be inserted into (10), thus Z t arriving at A.t1 / dt1 ; ˝1 .t/ D 0 Z t n1 X X Bj Z t Z t1 Œ˝k1 .s/; Œ˝k2 .s/; ˝n .t/ D 1 j Š k CCk Dn1 0 ˝2 .t/ D dt1 dt2 ŒA.t1 /; A.t2 / ; j D1 j 1 2 0 0 k1 1;:::;kj 1 Z t Z t1 Z t2 1 Œ   Œ˝kj .s/; A.s/     ds; n  2: (11) ˝3 .t/ D dt1 dt2 dt3 .ŒA.t1 /; ŒA.t2 /; 6 0 0 0 A.t3 / C ŒA.t3 /; ŒA.t2 /; A.t1 // :

Sn.1/ .t/

(6) Each term ˝n .t/ in the Magnus series is a multiple integral of combinations of n  1 nested commutators It is possible to show that the Magnus expansion containing n operators A.t/. It is worth noticing that converges for values of t such that this recursive procedure is well adapted to be implemented in algebraic symbolic languages. Z t kA.s/k ds < ; (7) 0 Some Applications of the Magnus Expansion In practice, one can rarely build up the whole Magnus in terms of the Euclidean norm. series and has to deal with a truncated version of it. The main advantage of these approximate solutions Magnus Expansion Generator is that they still share with the exact solution imThe generation of explicit formulae for higher-order portant qualitative properties, at variance with other terms in the Magnus series becomes quickly a difficult conventional approximation techniques. For instance, task [3]. Instead [4], they can be generated in a recur- in classical mechanics, the symplectic character of sive way by substituting equation (5) into (4). Equating the time evolution is preserved at every order of apthe terms of the same order, one gets proximation. Similarly the unitary character of the time evolution operator in quantum mechanics is also ˝10 .t/ D A.t/; preserved. Since the 1960s, the Magnus expansion has been n1 X Bj Sn.j / .t/; n  2; ˝n0 .t/ D (8) successfully applied [5] as a perturbative tool in jŠ j D1 numerous areas of physics and chemistry, from

F

510

Fer and Magnus Expansions

atomic and molecular physics to nuclear magnetic Inspection of the expression of AnC1 in (15) reresonance, quantum electrodynamics, and quantum veals an interesting feature of the Fer expansion. If computing. we substitute A by "A, where " is a real parameter, then we observe that FnC1 is of the same order in " as An , and then an elementary recursion shows n that the matrix An starts with a term of order "2 The Fer Expansion (correspondingly, the operator Fn contains terms of 2n1 and higher). This should greatly enhance Following Fer, we seek a solution of (1) (again with order " the rate of convergence of the product in (14) to the t0 D 0) in the factorized form exact solution. Z t  It is possible to show that the Fer expansion conY .t/ D exp dx A.x/ M1 .t/Y0 verges at least for values of t such that 0   Z t  exp F1 .t/ Y1 .t/; M1 .0/ D I; (12) kA.s/kds < 0:8604065; (16) 0

where Y1 .t/  M1 .t/Y0 . Substitution into (1) yields the differential equation for the matrix M1 .t/ or equiv- a bound smaller than the one corresponding to the alently for Y1 .t/ Magnus expansion. Y10 .t/ D A1 .t/Y1 .t/;

Y1 .0/ D Y0 ; (13) Z 1 An Example A1 .t/ D eF1 .t / A.t/eF1 .t /  dx exF1 .t / A.t/exF1 .t / : 0

The above procedure can be repeated to yield a sequence of iterated matrices Ak . After n steps, we have the following recursive scheme, known as the Fer expansion: Y .t/ D eF1 .t / eF2 .t /    eFn .t / Yn .t/; Yn0 .t/

D An .t/Yn .t/;

Yn .0/ D Y0 ;

(14) n D 1; 2; : : :

with Fn .t/ and An .t/ given by Z

t

FnC1 .t/ D

An .s/ds; A0 .t/ D A.t/; n D 0; 1; 2 : : : ;

To illustrate the main features of the Magnus and Fer expansions, the following 2  2 complex coefficient matrix is considered next: Q D i A.t/

AnC1 .t/ D e

Z

1



An .t/e

FnC1 .t /

dx exFnC1 .t / An .t/exFnC1 .t /

1 ! 2 0 i !t

ˇe

ˇ ei !t  12 !0

 ;

(17)

where ˇ, !, and !0 are real parameters, ! ¤ !0 . To improve the accuracy and the convergence of the expansions, a linear transformation is carried out in Q advance so as to integrate first the diagonal piece of A, namely, AQd  .i !0 =2/ diag.1; 1/. Then one ends up with system (1) and coefficient matrix 

0 FnC1 .t /



A.t/ D iˇ

0 ei.!!0 /t

ei.!!0 /t 0

 ;

(18)

to which we apply the recursive procedures (9–10) and (14–15). We compute up to 11 terms of the Magnus sej times 1 ‚ …„ ƒ ries (5) and the first two iterations of the Fer expansion, X j ŒFnC1 .t/; Œ   ŒFnC1 .t/; D .1/j F1 and F2 in (14). These are then applied to the initial .j C 1/Š j D1 condition Y0 D .1; 0/T for ˇ D 0:4, ! D 4, !0 D 1 An .t/    ; n D 0; 1; 2 : : : (15) to get Y .t/ at the final time tf D 5. Finally, the lower component of Y .tf / is compared with the exact result. Truncation of the expansion after n steps yields an The corresponding absolute errors as a function of t are approximation to the exact solution Y .t/. depicted in Fig. 1. 0

Fer and Magnus Expansions

M1,F1

0.01

10−4 F2

Error

Fer and Magnus Expansions, Fig. 1 Error in the solution Y .t / as a function of t . The curves have been obtained by the truncated Magnus and Fer expansions applied to (1) with coefficient matrix (18) for ˇ D 0:4, ! D 4, and !0 D 1. Lines coded as Mn stand for the result achieved by the truncated Magnus expansion with n terms, whereas F2 corresponds to the Fer expansion with two terms. F1 and M1 yield the same result

511

10−6

M5

10−8

F

M11 10−10

0

5

10

15

t

Note that for both expansions, taking into account more terms gives more accurate approximations, and this is so even for values of t outside the (rather conservative) convergence bounds provided by (7) and (16): t D =ˇ D 7:853 and t D 0:8604=ˇ D 2:151, respectively. On the other hand, Fer’s second-order approximation already provides results comparable to the fifth-order Magnus approximation, although it is certainly more difficult to compute.

For the Fer expansion, an analysis shows that k Fk .h/ D O.h2 1 /; k D 1; 2; : : :, and so F1 ; F2 in (14) suffice to build methods up to order 6 in h. Whereas for the Magnus expansion, one gets ˝1 D O.h/; ˝2k D O.h2kC1 /; ˝2kC1 D O.h2kC3 /; k D 1; 2; : : :, and then, ˝1 ; ˝2 in (5) suffice to build methods up to order 4 in h. Next, one has to approximate the integrals in (6) or (15) using appropriate quadrature rules. It turns out that their very structure allows one to approximate all the multivariate integrals up to a given order just by evaluating A.t/ at the nodes of a univariate quadrature Fer and Magnus Expansions as Numerical [6], and this can be done in many different ways. Integrators A procedure to obtain methods which can be easily adapted for different quadrature rules uses the averaged The Fer and Magnus expansions can also be used (or generalized momentum) matrices as numerical methods for solving (1). To obtain Y .t/ Z from Y0 , one follows a time-stepping advance procei 1 tn Ch  t  t1=2 A.t/dt A.i / .h/  i dure. For simplicity, we consider a constant time step, h tn h D t=N , and with tj D j h; j D 0; 1; 2 : : : ; N ,   k we compute approximations yj to the exact values X 1 i Dh bj cj  Aj C O.hpC1 /; Y .tj /. To obtain yj , we apply either the Fer or Mag2 j D1 nus expansions in each subinterval Œtj 1 ; tj ; j D 1; 2 : : : ; N to the initial condition yj 1 . The process (19) involves three steps. First, the expansions are truncated according to the order in h we want to achieve. Second, for i D 0; 1; : : :, where t1=2 D tn C h=2 and Aj D the multivariate integrals in the truncated expansions A.tn C cj h/. Here, bj , cj ; j D 1; : : : ; k, are the are replaced by conveniently chosen approximations. weights and nodes of a particular quadrature rule of Third, the exponentials of the matrices have to be order p, to be chosen by the user. The first order in the Fer and Magnus expansions, computed. We briefly consider the first two issues, .0/ .h//, leads to a second-order approximation exp.A while assume the user is provided with an efficient in the time step h. If the midpoint rule is used, tool to compute the matrix exponential or its action on we obtain a vector.

512

Fer and Magnus Expansions

  ynC1 D exp hA.tn C h=2/ yn :

(

(20)

.A1 C A2 / A.0/ ' h2p A.1/ ' h123 .A2  A1 /

where

Fourth-Order Fer and Magnus Integrators ( p   A1 D A tn C . 12  p63 /h With A.0/ and A.1/ in (19), one can obtain fourth(23)   order methods which usually provide a good A2 D A tn C . 12 C 63 /h balance between good performance and moderate complexity. From (21) and (22), there is the possibility of A fourth-order Magnus integrator is given by constructing commutator-free methods. For instance,   ynC1 D exp A.0/ C ŒA.1/ ; A.0/  yn :

(21)

In turn, a fourth-order Fer integrator reads    ynC1 D exp A.0/ exp ŒA.1/ ; A.0/   1 C ŒA.0/ ; ŒA.0/ ; A.1/  yn : 2

   1 .0/ 1 .0/ A C2A.1/ exp A 2A.1/ yn ; 2 2 (24) which is also a fourth-order method. 

ynC1 D exp

A Numerical Example We apply the previous numerical schemes to solve (1) (22) with A.t/ D ˇ.t/B C .t/C: (25)

As far as the A.n/ matrices are concerned, if, for Here, B; C are n  n noncommuting constant matriexample, we choose the Gauss–Legendre quadrature ces and ˇ.t/; .t/ scalar functions. For the sake of illustration, we take rule, we have oscillatory

stiff −1

−1 RK−4

−2

−2 M−F−2

F−4

−3

−3

M−F−2

−4

LOG10(ERROR)

LOG10(ERROR)

F−4 M−4 RK−4 −5 CF−4

M−4 −4

−5 CF−4

−6

−6

−7

−7

−8 0.5

1

1.5

2

LOG10(No. STEPS)

Fer and Magnus Expansions, Fig. 2 Error vs. the number of steps for the numerical solution of (1) with A.t / in (25). M-F-2 stands for the second-order method (20). Fourth-order

−8 0.5

1

1.5

2

LOG10(No. STEPS)

methods are coded as M-4, Magnus (21); F-4, Fer (22); CF-4, commutator-free (24); RK-4, standard Runge-Kutta

Filon Quadrature

BD

1 tridiag.1; 2; 1/; ın2

513

C D

  1 diag xj2 ; 2

ın D 20=n; xj D 10  j ın , and initial   conditions Y .0/ D .a1 ; : : : ; an /T , with aj D 10 exp .xj  2/2 . Let f .t/ D 1  12 sin t and g.t/ D 1  12 cos t. Firstly, we consider ˇ.t/ D f .t/; .t/ D g.t/, which corresponds to a stiff problem. Next, we take ˇ.t/ D if .t/; .t/ D ig.t/, which originate oscillatory solutions. We integrate until t D 2, with n D 100. Figure 2 shows the error in norm of Y .2/ as a function of the number of steps in double logarithmic scale. Schemes (21), (22), and (24) are implemented with the quadrature rule (23). The results of the second-order method (20) and the standard fourthorder Runge-Kutta (RK-4) method are also included for comparison.

3. Iserles, A., Munthe-Kaas, H.Z., Nørsett, S.P., Zanna, A.: Liegroup methods. Acta Numer. 9, 215–365 (2000) 4. Klarsfeld, S., Oteo, J.A.: Recursive generation of higherorder terms in the Magnus expansion. Phys. Rev. A 39, 3270– 3273 (1989) 5. Blanes, S., Casas, F., Oteo, J.A., Ros, J.: The Magnus expansion and some of its applications. Phys. Rep. 470, 151–238 (2009) 6. Iserles, A., Nørsett, S.P.: On the solution of linear differential equations in Lie groups. Phil. Trans. R. Soc. Lond. A 357, 983–1019 (1999) 7. Blanes, S., Casas, F., Ros, J.: High order optimized geometric integrators for linear differential equations. BIT 42, 262–284 (2002) 8. Hochbruck, M., Lubich, C.: On Magnus integrators for timedependent Schr¨odinger equations. SIAM J. Numer. Anal. 41, 945–963 (2003)

Filon Quadrature Further Developments Although only numerical methods up to order four have been treated here, higher-order integrators within these families exist which are more efficient than standard (Runge–Kutta) schemes for a number of problems [5, 7]. Methods (20) and (21) have also been used for the time integration of certain parabolic partial differential equations previously discretized in space. For the time-dependent Schr¨odinger equation, in particular, it has been shown that these schemes retain their full order of convergence if the exact solution is sufficiently regular, even when khA.t/k can be of arbitrary size [8]. The success of Magnus methods applied to the numerical integration of (1) has motivated several attempts to generalize them for solving nonlinear differential equations [5].

Daan Huybrechs Department of Computer Science, K.U. Leuven, Leuven, Belgium

Mathematics Subject Classification 65D30 (Numerical integration); 41A60 (Asymptotic approximations, asymptotic expansions (steepest descent, etc.))

Synonyms Filon-type quadrature

Cross-References  Exponential Integrators  Lie Group Integrators  Symplectic Methods

Short Definition

Filon quadrature is an efficient method for the numerical evaluation of a class of highly oscillatory integrals, in which the integrand has the form of a smooth and non-oscillatory function multiplying a highly osReferences cillatory function. The latter is most commonly a 1. Fer, F.: R´esolution de l’´equation matricielle UP D p U par trigonometric function with a large frequency. The produit infini d’exponentielles matricielles. Bull. Classe Sci. method is based on substituting the non-oscillatory Acad. R. Bel. 44, 818–829 (1958) function by an interpolating polynomial and integrat2. Magnus, W.: On the exponential solution of differential ing the result exactly. The most important advantage equations for a linear operator. Commun. Pure Appl. Math. of Filon quadrature is that the accuracy and the cost 7, 649–673 (1954)

F

514

Filon Quadrature

of the scheme are independent of the frequency of the integrand. Moreover, interpolating also derivatives at a well-chosen set of points leads to an error that decays rapidly with increasing frequency. However, one important assumption is that the moment problem can be solved efficiently, i.e., that polynomials times the oscillatory function can be integrated either analytically or numerically by other means.

Description Model Form Filon quadrature applies, for example, to oscillatory integrals of the form Z

Z

b

wi D

li .x/e i !g.x/ dx:

(3)

a

Composite rules based on piecewise polynomial interpolation are obtained by subdividing the interval Œa; b and applying (2) repeatedly, typically with small n. In both cases, convergence can be obtained by ensuring convergence of the interpolation process. A generalization based on interpolation of derivatives of f leads to quadrature rules using derivatives (see [9]): I Œf   QŒf  WD

di n X X

wi;j f .j /.xi /:

(4)

i D1 j D1

b

I Œf  D

(1) Particular choices of quadrature points result in high asymptotic order of accuracy, in the sense that for increasing !, it holds that where f and g are smooth functions of x on a bounded interval Œa; b. The complex exponential can be replaced by other highly oscillatory functions, with suitable modifications to the method. Common examples I Œf   QŒf  D O .! s / ; !!1 (5) include other trigonometric functions, Airy functions, and Bessel functions. The model form (1) has an explicit frequency parameter !. This simplifies the where s > 0 depends on the order of the derivatives analysis of the scheme for increasing frequency, but that are interpolated (see section “Convergence Analit is not a critical feature when considering Filon ysis” below). This property makes Filon quadrature quadrature for a particular oscillatory integral. Rele- ideally suited for the efficient and highly accurate vant properties are that the integrand can be written as evaluation of highly oscillatory integrals. the product of a non-oscillatory function, in this case Suitable interpolation points to use for the asympf .x/, and an oscillatory function, in this case e i !g.x/, totic property (5) to hold are found from the asymptotic and that the latter can be explicitly identified. Finally, analysis of the oscillatory integral. They generally unbounded intervals are possible as long as the integral include: is convergent or if suitable regularization is applied. • The end points of the integration interval, in this case a and b • So-called stationary points of g.x/: zeros of g 0 .x/ Overview of the Method in Œa; b Filon quadrature was introduced by L. N. G Filon in • Any point of discontinuity or any kind of singularity 1928 [2]. Modern versions are based on polynomial of f and/or g in Œa; b interpolation of the non-oscillatory function f . With interpolation points denoted by xi , i D 1; : : : ; n, this leads to a quadrature rule of a classical form Asymptotic Error Analysis Oscillatory integrals are a classical topic in asymptotic n X analysis [18]. Integrals of the form (1), with smooth wi f .xi /; (2) functions f and g, admit a Poincar´e-type asymptotic I Œf   Q1 Œf  WD i D1 expansion of the form f .x/e i !g.x/ dx;

a

but with the weights depending on !. They are given by integrals of the cardinal polynomials of Lagrangian interpolation, li .x/, which satisfy li .xj / D ıi j :

I Œf  

1 X kD0

ak Œf  ! bk ;

(6)

Filon Quadrature

515

where the ak s are linear functionals of f and the bk s form a strictly increasing sequence of positive rational values. The coefficients ak are independent of !. They depend on function values and derivatives of f at a small set of points, with coefficient akC1 depending on derivatives of one order higher than those of ak and with a0 most often depending only on function values of f . Some or all of the coefficients can be obtained explicitly via the integration by parts, the method of stationary phase, or the (complex plane) method of steepest descent [18]. The coefficients bk are determined by the highest-order stationary point of g in the interval Œa; b. A stationary point  is a root of g, i.e., g 0 ./ D 0, and its order corresponds to the number of vanishing derivatives:  has order r if g .j / ./ D 0, j D 1; : : : ; r, and g .rC1/ ./ ¤ 0. Then, bk D .k C 1/=r. In particular, b0 D 1=r and   I Œf  D O ! 1=r : Whether using derivatives or not, in both cases of Filon quadrature, the function f is approximated by a polynomial p and I Œf   QŒf  D I Œp. The integral I Œp itself admits an asymptotic expansion, with coefficients depending on values and derivatives of p at the critical points. If a number of values and derivatives of p agree with those of f , as is the case when p interpolates f , then the first few coefficients of the expansion of I Œf  and I Œp may agree. We then have

I Œf   QŒf  D I Œf   I Œp 

1 X

ak Œf  p! bk

kD0

D

1 X

ak Œf  p! bk ;

kDK

with the integer K  0 depending on the number of derivatives of f being interpolated. The asymptotic property (5) of Filon quadrature follows. In the particular case of an integral without stationary points, we have r D 1, and interpolation of values of f at the points a and b yields  an error of Filon quadrature that scales as O ! 2 for large !. Each additional derivative interpolated at both end points increases the asymptotic order by r D 1.

Convergence Analysis For fixed values of !, the asymptotic expansion (6) in general diverges. Filon quadrature may converge for increasing n depending on the (stable) convergence of the underlying interpolation process. Convergence can also be achieved via subdivision, but that approach is suboptimal for large !. For that reason, most convergent schemes are based on adding known stable interpolation points to Filon quadrature, such as the Chebyshev points, achieving spectral convergence [1, 5, 12]. Analysis of the interpolation error is most widespread in literature and readily yields an estimate of the integration error. Yet, the resulting estimates are usually pessimistic for large !. Optimal convergence estimates taking into account both large ! and large n simultaneously have not been described in detail. Typical in Filon quadrature is the use of derivatives. The error of interpolation of an increasing number of derivatives at the end points was analyzed by Melenk [14].

The Moment Problem Successful application of Filon quadrature rests on the identification of the oscillatory factor of the integrand, which in the case of model form (1) is simply e i !g.x/. Furthermore, in order to apply the method, one has to be able to calculate moments of the oscillator, i.e., integrals of polynomials times the oscillator. This is explicit in expression (3) for the weights of Filon quadrature. Moments can be explicitly computed for polynomial functions g at least up to degree 3 in terms of special functions. The computation of moments can be avoided by using moment-free Levin-type methods [15,16]. Alternatively, moments can be computed using other highly oscillatory quadrature methods such as the numerical method of steepest descent [6]. Origins of the Method Filon quadrature originated in the work of L. N. G. Filon Rin 1928 [2]. He considered an integral of the b form a sin kx .x/dx and proposed piecewise cubic interpolation of .x/, similar to Simpson’s rule. Explicit expressions were derived for the weights, and convergence was achieved by subdividing and thereby improving the approximation of . Later papers on this

F

516

Finite Difference Methods

topic, e.g., by Luke [13], Flinn [3] and Tuck [17], re- 5. Huybrechs, D., Olver, S.: Superinterpolation in highly oscillatory quadrature. Found. Comput. Math. (2011). main focused on analyzing and improving convergence doi:10.1007/s10208-011-9102-8 of the interpolation. 6. Huybrechs, D., Vandewalle, S.: On the evaluation of The asymptotic analysis of Filon quadrature was highly oscillatory integrals by analytic continuation. SIAM J. Numer. Anal. 44(3), 1026–1048 (2006). initiated by Iserles [7] and Iserles and Nørsett [9]. This doi:10.1137/050636814 led to Filon-type quadrature schemes with essentially 7. Iserles, A.: On the numerical quadrature of arbitrarily high asymptotic order, using derivatives or highly-oscillating integrals I: Fourier transforms. suitably scaled finite differences [8]. The use of firstIMA J. Numer. Anal. 24(3), 365–391 (2004). doi:10.1093/imanum/24.3.365 order derivatives had been proposed earlier, e.g., by Kim et al. [11], and other schemes had been devel- 8. Iserles, A., Nørsett, S.P.: On quadrature methods for highly oscillatory integrals and their implementation. BIT 44(4), oped with favorable asymptotic properties, though typ755–772 (2004). doi:10.1007/s10543-004-5243-3 ically without asymptotic analysis – see the review [4] 9. Iserles, A., Nørsett, S.P.: Efficient quadrature of highly oscillatory integrals using derivatives. Proc. R. Soc. Lond. and references therein. Competitive methods achieving A 461, 1383–1399 (2005). doi:10.1098/rspa.2004.1401 high asymptotic order were developed concurrently, in10. Iserles, A., Nørsett, S.P.: Quadrature methods cluding Levin-type methods [15] and numerical steepfor multivariate highly oscillatory integrals using est descent-based methods [6]. derivatives. Math. Comput. 75(255), 1233–1258 (2006).

Limitations and Extensions One limitation of Filon-type quadrature is the assumed ability to compute moments, which is needed in order to compute the weights of the quadrature rule (see Section “The Moment Problem”). Filon-type quadrature is easily extended to higher dimensions [10] and to other types of oscillators. Derivatives in the formulation can be replaced by finite differences, with asymptotic order of accuracy maintained if the spacing is inversely proportional to the frequency [8].

Cross-References  Numerical Steepest Descent  Levin Quadrature

References 1. Dom´ınguez, V., Graham, I.G., Smyshlyaev, V.P.: Stability and error estimates for Filon-Clenshaw-Curtis rules for highly-oscillatory integrals. IMA J. Numer. Anal. 31(4), 1253–1280 (2011) 2. Filon, L.N.G.: On a quadrature formula for trigonometric integrals. Proc. R. Soc. Edinb 49, 38–47 (1928) 3. Flinn, E.A.: A modification of Filon’s method of numerical integration. J. Assoc. Comput. Mach. 7, 181–184 (1960) 4. Huybrechs, D., Olver, S.: Highly oscillatory quadrature. In: Engquist, B., Fokas, A., Hairer, E., Iserles, A. (eds.) Highly Oscillatory Problems, pp. 25–50. Cambridge University Press, Cambridge (2009)

doi:10.1090/S0025-5718-06-01854-0 11. Kim, K.J., Cools, R., Ixaru, L.G.: Quadrature rules using first derivatives for oscillatory integrands. J. Comput. Appl. Math. 140(1–2), 479–497 (2002) 12. Ledoux, V., Van Daele, M.: Interpolatory quadrature rules for oscillatory integrals. J. Sci. Comput. 53, 586–607 (2012). doi:10.1007/s10915-012-9589-4 13. Luke, Y.L.: On the computation of oscillatory integrals. Proc. Camb. Philos. Soc. 50, 269–277 (1954) 14. Melenk, J.M.: On the convergence of Filon quadrature. J. Comput. Appl. Math. 234(6), 1692–1701 (2010). doi:10.1016/j.cam.2009.08.017 15. Olver, S.: Moment-free numerical integration of highly oscillatory functions. IMA J. Numer. Anal. 26(2), 213–227 (2006). doi:10.1093/imanum/dri040 16. Olver, S.: Fast, numerically stable computation of oscillatory integrals with stationary points. BIT 50, 149–171 (2010) 17. Tuck, E.O.: A simple Filon-trapezoidal rule. Math. Comput. 21(98), 239–241 (1967) 18. Wong, R.: Asymptotic Approximation of Integrals. SIAM, Philadelphia (2001)

Finite Difference Methods Gunilla Kreiss Division of Scientific Computing, Department of Information Technology, Uppsala University, Uppsala, Sweden

Introduction During the last 60 years, there has been a tremendous development of computer techniques for the solution

Finite Difference Methods

517

of partial differential equations (PDEs). This entry will give a basic introduction to finite difference method, which is one of the main techniques. The idea is to replace the PDE and the unknown solution function, by an algebraic system of equations for a finite dimensional object, a grid function. We will consider simple model equations and introduce some basic finite difference methods. These model problems have closed form solutions but are suitable for discussing techniques, properties, and concepts. However, most phenomena in real life are modeled by partial dierential equations, which have no closed form solution. For further reading we refer to [1–5].

Initial Value Problem for PDEs

A Simple Discretization of the Heat Equation Introduce a grid, which is a set of discrete points in the domain of the problem, and a grid function .xj ; tn / D .j h; nk/ and unj ; j D 0; 1; 2; : : : ; N; n D 0; 1; : : : : Here N > 0 is a positive (large) integer and h D 1=N is called the space step. Note that xN D 1. Correspondingly k > 0 is called the time step. We want the grid function to approximate the solution to the PDE at the grid points, that is, unj  u.xj ; tn /. An obvious choice for the grid function at time level n D 0 is (4) u0j D f .xj /:

In a finite difference method, the algorithm for computing all other parts of the grid function is based on replacing the derivatives of the PDE by differences. (1) A simple example is to use

To begin with we consider the so-called heat equation: ut D uxx :

This equation is the simplest possible parabolic PDE. Here  is a positive constant and u.x; t/ is the sought solution. Here we think of x as a spatial variable and t as time. We use the notation ut to denote differentiation with respect to t. Accordingly uxx denotes differentiation twice with respect to x. Equation (1) can be used to model diffusive phenomena such as the evolution of the temperature. To make the problem complete, we need to specify the domain. Here we will consider t  0 and 0  x  1. An initial condition at t D 0, u.x; 0/ D f .x/;

(2)

as well as boundary conditions at x D 0; 1 are needed. To begin with we consider the spatially periodic problem (with period 1), that is, we extend the solution to 1 < x < 1 under the condition

ujnC1  unj

D

k

unj C1  2unj C unj 1 h2

:

(5)

If the grid function is the restriction to the grid of a smooth function of x and t, it is straight forward to use Taylor expansion and convince oneself that the lefthand side approximates a time derivative and the righthand side a second derivative with respect to x. Combining (5) with the periodicity condition, we can formulate an algorithm for computing the grid function at time-level n D 1; 2; : : : , ujnC1 D unj C

k n .u  2unj C unj 1 /; h2 j C1

j D 1; : : : N; un0 D unN ;

uN C1 D un1 :

(6) (7)

This is an explicit method, that is, the solution at each (3) new time level can be computed directly without solving a system of equations. It is also a 1-step method, The periodic condition is very useful for understanding meaning that only one old time level is involved in aspects of numerical methods, as well as properties each time step. In Fig. 1a we have plotted discrete soof the model, which are not related to boundary phe- lutions at three different time levels. Note the diffusive nomena. In computations we only need to consider behavior of the solution. In Fig. 1b we display solutions at t D 1 for three different discretizations. Here we 0  x  1. u.x; t/ D u.x C 1; t/:

F

518

Finite Difference Methods

Finite Difference Methods, Fig. 1 Numerical solution of the heat equation with initial data u.x; 0/ D 4 e .10.x0:5// C sin.6x/. (a) Solutions at t D 0 (line), t D 1 (dots), and t D 4.C/. (b) Solutions at t D 1 with step sizes, h; k D 0:1; 0:2 (line), h; k D 0:05; 0:1 (dashed), and h; k D 0:025; 0:05 (dotted). (c) Discrete solutions with h; k D 0:0125; 0:025 at t D 0:5 (dashed) and t D 1 (dotted)

2

2

a

1.5

1.5

1.5

1

1

1

0.5

0.5

0.5

0

0

0

−0.5

−0.5

−0.5

−1 0.5

note that as the discrete solution seems to converge as h; k ! 0. Convergence and fast convergence rate of the discrete solution are essential. To be able to discuss this topic, we introduce the accuracy concept. By applying a difference approximation to a restriction to the grid of a smooth solution to the PDE, and performing Taylor expansion, we can discuss accuracy. For the finite difference method (5), we define the truncation error as the remainder, that is,  unj unC1 j k



unj C1  2unj C unj 1 h2

D ut  uxx C O.k C h2 / D O.k C h2 /: We say that the order of accuracy is 1 in time and 2 in space. As long as the truncation error tends to zero as h; k ! 0, we say that the approximation is consistent. However, a small truncation error is not the only concern. In Fig. 1c we have plotted the solutions obtained by the same method as before with an even finer grid. Even though the method is consistent, the result is useless.

1

c

−1

−1 0

jn D

2

b

0

0.5

0

1

0.5

1

the solution at later time levels. In particular, the bound needs to be uniform as the grid is refined. To measure the solution we will use the norm of a U n , the grid function at time-level n,

kU n k2 D

N X

hjunj j2 :

1

This norm is a discrete counterpart to the L2 norm of a continuous function at t D tn . Usually we want both temporal and spatial errors to be small, and as accuracy increases we reduce h and k in a coordinated way. We shall therefore in the following definition assume that h D h.k/. In the following we will give the definition for 1-step methods. Definition 1 Suppose a numerical method for specified time k and space step h D h.k/ for a linear PDE gives approximations unj . The numerical method is stable if for each time T , there is a constant CT such that kU n k  CT kU 0 k;

8k > 0; n 

T : k

Stability For scalar problems with constant coefficients and no To understand the behavior in Fig. 1c, we need the lower-order terms, we require CT D 1. important concept of stability. Loosely speaking, a A fundamental result is the Lax-Richtmyer theorem method is stable if a perturbation in the solution, introduced at some time level, causes a bounded change in stating that a consistent numerical method can only

Finite Difference Methods

converge if the method is stable. Therefore tools for analyzing stability are essential. The most straightforward tool is the von Neumann analysis, where a single frequency initial data is considered, u0j D a0 e i !hj . Due to linearity the finite difference method will at later time levels yield unj D an e i !hj . For the method above we get a difference equation for the amplitude coefficients of the following form:    4k 2 !h an H) anC1 D 1  2 sin h 2 n   4k 2 !h an D 1  2 sin a0 : h 2

519

Taylor expanding around .xj ; tn C k=2/ yields a truncation error jn D O.h2 C k 2 / for this approximation. To obtain an algorithm we add initial data (4) and periodic boundary conditions (7) as before. In this case the method is implicit, that is, the algorithm for computing the approximation at each new time level n C 1 involves solving a linear system of equations. In one space dimension the system is tri-diagonal and direct solution is the most efficient approach. In higher space dimensions the corresponding systems will be sparse and banded, and iterative solvers are often used. To analyze the stability of this method, we apply the von Neumann analysis. The ansatz unj D an e i !hj in (9) yields, after collecting terms related to the different time levels,

   The von Neumann condition for stability requires k !h 2 sin : that the amplitude for all possible frequencies ! should .1 C ı/ anC1 D .1  ı/ an ; ıD2 h2 2 at all later times be bounded by the initial amplitude, jan j  ja0 j. In this case the von Neumann condition is Here ı  0 for any combination of k > 0; h > 0, satisfied precisely if and !. For nonnegative ı we have j1  ıj=j1 C ıj  1, and thus the method is stable for any combination of ˇ ˇ ˇ 2k ˇ positive k; h. In particular k  h can be used, as ˇ ˇ (8) ˇ h2 ˇ  1: opposed to the k  h2 for explicit methods. Therefore, it is usually more efficient to use an implicit method We say that (5) is a stable method under the condi- for parabolic PDEs, at least if a sufficiently good linear tion (8). In Fig. 1a, b the condition is satisfied, while solver is available. in Fig. 1c it is violated. We note that stability requires k  h2 as the grid is refined. This is typical when an explicit method is used for a parabolic PDE, and the Finite Difference Methods for the requirement leads to very many time steps when a fine Advection Equation spatial grid is used. In this section we consider the advection equation, sometimes called the one-way wave equation, A Second-Order Unconditionally Stable Method for the Heat Equation ut C aux D 0; 0  x  1; t  0: (10) Note that the approximation of the time derivative in (5) is actually a second-order accurate approximation of the time derivative at .xj ; tn C k=2/. The accu- Here a is a constant, which for ease of notation, we racy of the previous method can therefore be improved will assume to be positive. As in the previous section, by including terms on the right-hand side to achieve we consider the spatially periodic case. This equation a second-order approximation at the same point. is a model for hyperbolic PDEs. The resulting approximation, the Crank-Nicolson We will introduce two approximations, the so-called method, is upwind method and the Lax-Wndroff method. The upwind method has its name from the fact that when  n nC1 n n n a > 0 the characteristic curves of the equation have uj  uj 1 uj C1  2uj C uj 1 D a positive slope in the x; t plane, indicating that infork 2 h2 mation propagates from left to right. We say that the ! nC1 ujnC1 C unC1 C1  2uj j 1 “upwind” direction is to the left. The approximation of C : (9) h2 the spatial derivative uses values to the left. If a > 0

F

520

Finite Difference Methods

the upwind direction would be to the right, and a one- explicit, and the stability conditions require k and h to sided spatial difference approximation with bias to the decrease at similar rates. This is typical of good explicit right must be used instead. methods for hyperbolic problems and is acceptable from an efficiency perspective. Therefore, hyperbolic

ak problems are usually solved by explicit methods. (11) unj  unj 1 ; D unj  unC1 j In Fig. 2 we have plotted solutions for the case a D h 6

1 with initial data u.x; 0/ D e .4.x0:5// at t D 1. Note ak n nC1 n n uj D uj   uj 1 u that for this problem the exact solution is u.x; 1/ D 2h j C1 u.x; 0/. Note that the second-order method converges  2

1 ak n n n faster. uj C1  uj C uj 1 : C 2 h (12)

Extensions

They are accurate of orders 1 and 2 (in both space and time), respectively. We apply the von Neumann Realistic models are more complicated then the equaanalysis by making the ansatz unj D an e i !hj , yielding tions above. However, the idea to replace derivatives by finite differences is still applicable, and the stability   ak i !h concept is central. In general high order is better / an ; anC1 D 1 .1  e h than low order. In the variable coefficient case, a first   2 ! indication of stability of a method can be obtained ak !h ak anC1 D 1i an ; by freezing coefficients (Example: consider using the sin.!h/C2 sin h h 2 Lax-Wendroff method for ut D a.x/ux where a0  a.x/  a1 . The relevant frozen coefficient problems respectively. By some trigonometric manipulations, we are u D ˛u where ˛ is constant, a  ˛  a .), and t x 0 1 find that the two methods are stable precisely if requiring the von Neumann stability condition to be satisfied for each relevant constant coefficient problem. ˇ ˇ ˇ ak ˇ ak ˇ ˇ  1; ˇ ˇ  1; 0 h h Initial Boundary Value Problems respectively. Note that the sign of a is essential for Let us replace the periodic condition (7) for the heat the stability of the upwind method. Both methods are equation by 1.5

1.5

1

1

0.5

0.5

0

0

−0.5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

−0.5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Finite Difference Methods, Fig. 2 Solution at t D 1 of ut Cux D 0 using the Lax-Wendroff method (left) and the upwind method (right), with h D 0:05 (rings), 0.025 (star with ring), 0.0125 (star), and k D 0:8h. Exact solution is also included (solid)

Finite Difference Methods in Electronic Structure Calculations

u.0; t/ D g.t/; ux .1; t/ D 0:

(13)

521

 18unj C 9ujn1  2ujn2 11unC1 j 6h

A discrete version of (13) to be combined with (5) D ut .xj ; tnC1 / C O.k 3 /: (16) or (9) is (14) These formulas can then be combined with high-order un0 D g.tn /; unN D unN 1 : For the advection equation (10) with a > 0 a spatial implicit approximations at time level n C 1 to boundary condition at x D 0 of the form u.0; t/ D yield high-order methods. For the advection equation and other hyperbolic g.t/ completes the model. For the upwind method this problems, explicit methods are desirable. A common condition is easily introduced in the discrete algorithm approach is to combine high-order spatial discretizan by setting u0 D g.tn / and then using (11) for j D tion with, for example, explicit Runge-Kutta methods. 1 : : : :N . In the Lax-Wendroff case, the (12) cannot be For these and other temporal discretizations, see [3]. used for j D N . A possible approach is to use a lowerorder accurate formula in this point. Von Neumann analysis is still relevant. If the method is not von Neumann stable, the discrete References initial boundary value problem cannot be expected 1. Gustafsson, B.: High Order Difference Methods for Time to be stable. A stability theory that explicitly includes Dependent PDE. Springer, Berlin (2008) boundary treatment is outside the scope of this entry, 2. Gustafsson, B., Kreiss, H.-O., Oliger, J.: Time Dependent Problems and Difference Methods. Wiley, New York (1995) but can be found in [1, 2, 5]. More Finite Difference Approximations By including more grid function values, we can approximate higher derivatives or decrease the truncation error. With five values in the spatial direction, we can, for example, get ui C2 C 8ui C1  8ui 1 C un2 12h

Finite Difference Methods in Electronic Structure Calculations

D ux .xi / C O.h /; 4

ui C2  4ui C1 C 6ui  4ui 1 C un2 h4

Jean-Luc Fattebert Lawrence Livermore National Laboratory, Livermore, CA, USA

D uxxxx .xi / C O.h2 /: These formulas can be derived by solving a system of equations for the coefficients, but they can also be found in the literature. In the temporal direction, care has to be taken to retain stability. For parabolic problems the so-called backward differentiation formulas yield implicit methods with good stability properties up to order 6. The second-and third-order methods in this family are  4unj C un1 3unC1 j j 2h

3. LeVeque, R.: Finite Difference Methods for Ordinary and Partial Differential Equations. SIAM, Philadelphia (2007) 4. Morton, K.W., Mayers, D.F.: Numerical Solution of Partial Differential Equations. Cambridge University Press, Cambridge (1994) 5. Strikwerda, J.C.: Finite Difference Schemes and Partial Differential Equations, 2nd edn. SIAM, Philadelphia (2004)

Definition The finite difference method is a numerical technique to calculate approximately the derivatives of a function given by its values on a discrete mesh.

Overview

Since the development of quantum mechanics, we know the equations describing the behavior (15) of atoms and electrons at the microscopic level.

D ut .xj ; tnC1 / C O.k 2 /;

F

522

Finite Difference Methods in Electronic Structure Calculations

The Schr¨oedinger equation ( Schr¨odinger Equation for Chemistry) is however too difficult to solve for more than a few particles because of the highdimensional space of the solution – 3N for N particles. So various simplified models have been developed. The first simplification usually introduced is the Born– Oppenheimer approximation ( Born–Oppenheimer Approximation, Adiabatic Limit, and Related Math. Issues) in which atomic nuclei are treated as classical particles surrounded by quantum electrons. But many more approximations can be introduced, all the way up to classical molecular dynamics models, where interacting atoms are simply described by parameterized potentials depending only on the respective atomic positions. The choice of an appropriate model depends on the expected accuracy and the physical phenomena of interest. For phenomena involving tens or hundreds of atoms and for which a quantum description of the electronic structure is needed – to properly describe chemical bonds making/breaking, or hydrogen bonds for instance – a very popular model is  Density Functional Theory (DFT). In DFT, the 3N-dimensional Schr¨oedinger problem is reduced to an eigenvalue problem in a threedimensional space, the Kohn–Sham (KS) equations. The electronic structure is described by N electronic wave functions (orbitals) which are the eigenfunctions corresponding to the N lowest eigenvalues of a nonlinear effective Hamiltonian HKS . Another simplification often introduced in DFT is the use of so-called pseudopotentials (see, e.g., [12]). These are effective potentials modeling the core of an atom, that is, the nuclei and the core electrons which do not participate to chemical bonds, assuming these core electrons do not depend on the chemical environment. Besides reducing the number of electronic wave functions to compute, the benefit of using pseudopotentials is to remove the singularity 1=r of the Coulomb potential associated to a nuclei. Indeed these potentials are built in such a way that the potential felt by valence electrons is as smooth as possible. This opens the way for various numerical methods to discretize DFT equations, in particular the finite difference (FD) method which is the object of this entry. The FD method (see, e.g., [4]) started being used in the 1990s as an alternative to the traditional plane waves (PW) (or pseudo-spectral) method used in the physics and material sciences communities [1, 3]. The

PW method had been a very successful approach to deal with DFT calculations of periodic solids. Besides being a good basis set to describe free electrons or almost free electrons as encountered in metallic systems and being a natural discretization for periodic systems, its mathematical properties of spectral convergence help reduce the size of the basis set in practical calculations. With growing computer power, researchers in the field started exploring real-space discretizations in order to facilitate distributed computing on large parallel computers. A simple domain decomposition leads to natural parallelism in real space: For p processors, the domain ˝ is split into a set of p spatial sub-domains of equal sizes and shapes, and each subdomain is associated to a processor. Each processor is responsible to evaluate operations associated to the local mesh points and ghosts values are exchanged between neighboring sub-domains to enable FD stencil evaluations at sub-domains boundaries [1]. In a FD approach, it is also easy and natural to impose various boundary conditions besides the typical periodic boundary conditions. It can be advantageous to use Dirichlet boundary conditions for the Coulomb potential for charged systems or polarized systems in lower dimension. The value at the boundary can be set by a multipole expansion of the finite system. This cancels out Coulomb interactions between periodic images. A real-space discretization also opens the door to replacing the simple Coulomb interaction with more complicated equations which model, for example, the presence of an external polarizable continuum, such as continuum solvation models [6]. Local mesh refinement techniques can also be used to improve numerical accuracy [5]. Like PW, a FD approach provides an unbiased discretization and accuracy can be systematically improved be reducing mesh spacing. Many of the advantages of real-space algorithms can be translated in some way into PW approaches. But doing so is not always natural, appropriate, or computationally interesting. It appears that one of the greatest potential for real-space methods is in O.N / complexity algorithms. The discussion in this entry is restricted to parallelepiped domains. This is appropriate to treat most solid-state applications where the computational domain has to coincide with a cell invariant under the crystal structure symmetry. For finite systems surrounded by vacuum, this is also a valid approach as long as the domain is large enough so that boundary

Finite Difference Methods in Electronic Structure Calculations

523

conditions do not affect the results. From a computational point of view, parallelepiped domains allow for the use of structured meshes which facilitates code implementation and improves numerical efficiency, allowing in particular, FD discretizations and matrix-free implementations.

nal potential. EII is the energy of interaction between atomic cores. The ground state of a physical system is represented by the orbitals that minimize (1) under the constraints (2). It can be found by solving the associated Euler–Lagrange equations – Kohn–Sham (KS) equations –

Equations

HKS

j

  1 D  r 2 C VH .e / C xc .e / C Vext 2

j

For a molecular system composed of Na atoms located D j j ; (4) a at positions fRE I gN I D1 in a computational domain ˝, the KS energy functional ( Density Functional Theory) which must be solved for the N lowest eigenvalues can be written (in atomic units)

j and corresponding eigenfunctions j . The Hartree potential VH represents the Coulomb potential due to   Z N h i X 1 the electronic charge density e and is obtained by N  2 E a fi EKS f i gN i D1 ; fRI gI D1 D i .r/  r 2 solving a Poisson problem. xc D ıExc Œe =ıe is the ˝ i D1 exchange and correlation potential. Z Z e .r1 /e .r2 / 1 From the eigenfunctions j ; j D 1; : : : ; N , one d r1 d r2  i .r/d r C 2 ˝ ˝ jr1  r2 j could construct the single-particle density matrix Z  X CEXC Œe  C i .r/.Vext i /.r/d r ˝ fi i .r/ i .r0 / (5) .r; O r0 / D h i i N a (1) CEII fRE I gI D1 : For a FD discretization, (5) becomes a finite dimensional matrix of dimension M given by the number of nodes on the mesh. Even if this matrix Z becomes sparse for very large problems, the number  (2) i .r/ j .r/ D ıij : of nonzero elements is prohibitively large. It is usually ˝ cheaper to compute and store the N eigenfunctions Equation 1 uses the electronic density e defined at corresponding to occupations numbers fi > 0 each point in space by without ever building the single-particle density matrix. N X e .r/ D fi j i .r/j2 (3) Finite Differences Discretization i D1 Let us introduce a uniform real-space rectangular grid where fi denotes the occupation of orbital i . In (1), ˝h of mesh spacing h – assumed to be the same in all the first term represents the kinetic energy of the directions for simplicity – that covers the computation electrons, the second the electrostatic energy of interac- domain ˝. The wave functions, potentials, and election between electrons. EXC models the exchange and tronic densities are represented by their values at the correlation between electrons. This term is not known mesh points rij k . Integrals over ˝ are performed using exactly and needs to be approximated ( Density Func- the discrete summation rule with the orthonormality constraints

Z tional Theory). Exchange and correlation functional of X u.r/d r  h3 u.rij k /: the type local density approximation (LDA) or general˝ r 2˝ ized gradient approximation (GGA) are typically easy ij k h to implement and computationally efficient in an FD For a function u.r/ given by its values on a set framework. Vext represents the total potential produced by the atomic nuclei and includes any additional exter- of nodes, the traditional FD approximation wi;j;k to

F

524

Finite Difference Methods in Electronic Structure Calculations

the Laplacian of u at a given node ri;j;k is a linear combination of values of u at the neighboring nodes p X  cn u.xi C nh; yj ; zk / wi;j;k D

.Vps /.r/ D vl .r/ .r/ p Z X  C vnl;i .r/EiKB vnl;i .r0 / .r0 /d r0 i D1

˝

(8)

nDp

Cu.xi ; yj C nh; zk / C u.xi ; yj ; zk C nh/



(6)

where the coefficients fcn g can be computed from the Taylor expansion of u near ri;j;k . Such an approximation has an order of accuracy 2p, that is, for a sufficiently smooth function u, wi;j;k will converge to the exact value of the derivative at the rate O.h2p / as the mesh spacing h ! 0. High-order versions of this scheme have been used in electronic structure calculations [3]. As an alternative, compact FD schemes (Mehrstellenverfahren [4]) have been used with success in DFT calculations [1]. For example, a fourthorder FD scheme for the Laplacian is based on the relation ) ( X X 1 u.r/ 24u.r0/  2 r2˝h ; r2˝h ; u.r/ p 6h2 krr0 kDh krr0 kD 2h  X 1 2 48.r 2 u/.r0 / C 2 r2˝h ; .r u/.r/ 72 krr0 kDh ) X C .r 2 u/.r/ C O.h4 /; (7) r2˝h ; p krr0 kD

5

2h

valid for a sufficiently differentiable function u.r/. This FD scheme requires p only values at grid points not further away than 2h. Besides its good numerical properties, the compactness of this scheme reduces the amount of communication in a domain-decompositionbased parallel implementation. In practice, this compact scheme consistently improves the accuracy compared to a standard fourth-order scheme.

energy (Ha)

D

where EiKB are normalization coefficients. The radial function vl contains the long range effect and is equal to Z=r far enough from the core charge Z. The functions vnl;i .r/ are the product of a spherical harmonics Y`m by a radial function which vanishes beyond some critical radius. To reduce the local potential vl to a short-range potential vls , we use the “standard trick” of adding to each atom a Gaussian charge distribution (with spherical symmetry, centered at the atomic position) which exactly cancels out the ionic charge. The sum of the charge distributions added to each atom is then subtracted from the electronic density used to compute the Hartree potential and leads to an equivalent problem. The correction added to the local atomic potential makes it short range, while the integral of the charge used to compute the Hartree potential becomes 0. Since the functions vls and vnl;i are nonzero only in limited regions around their respective atoms, the evaluation of the dot products between potentials and electronic wave functions on a mesh can take advantage of this property to reduce computational cost. An example of pseudopotential is represented in Fig. 1.

0

−5

non-local l=0 non-local l=1 local l=2 screened local l=2

−10 0

Pseudopotentials on a Mesh Accurate calculations can be performed on a uniform mesh by modeling each atomic core with a pseudopotential. For instance, a separable nonlocal Kleinman–Bylander (KB) potential Vps .r; r0 / in the form

1

2

3

distance (Bohr)

Finite Difference Methods in Electronic Structure Calculations, Fig. 1 Example of norm-conserving pseudopotential: chlorine. The local potential (before and after adding compensating Gaussian charge distribution) is shown as well as the radial parts of the nonlocal projectors l D 0; 1

Finite Difference Methods in Electronic Structure Calculations

525

With periodic boundary conditions, the total energy of a system should be invariant under spatial translations. A finite mesh discretization breaks this invariance. To reduce energy variations under spatial translations, the pseudopotentials need to be filtered. Filtering can be done in Fourier space using radial Fourier transforms [1]. Filtering can also be done directly in real space. In the so-called double-grid method [11], the potentials are first evaluated on a mesh finer than the one used to discretize the KS equations before being interpolated onto the KS mesh. In order to get smoother pseudo-wave functions and increase the mesh spacing required for a given calculation, one can relax the norm-conserving constraint when building pseudopotentials. FD implementations of the projected augmented wave (PAW) method [10], and the ultrasoft pseudopotentials [9] were proposed. While these approaches reduce the requirements on the mesh spacing, their implementations are much more complex than standard norm-conserving pseudopotential methods and they require the use of additional finer grids to represent some core functions within each atom.

with various solvers, either in self-consistent iterations or direct energy minimization algorithms ( Self-Consistent Field (SCF) Algorithms). The full approximation scheme (FAS), a multigrid approach for solving nonlinear problems, has also been used in FD electronic structure calculations to directly solve the nonlinear KS equations using coarse grid approximations of the full eigenvalue problem [14].

Forces, Geometry Optimization, and Molecular Dynamics Calculating the ground-state electronic structure of a molecular system is usually only the first step toward calculating other physical properties of interest. For instance to optimize the geometry of a molecular system or to simulate thermodynamic properties by molecular dynamics ( Calculation of Ensemble Averages), the electronic structure is just a tool to calculate the forces acting on atoms in a particular configuration. Knowing the ground-state electronic structure for E Na , one can coma given atomic configuration fRg I D1 pute the force acting on the ion I by evaluating the derivative of the total energy with respect to the atomic Real-Space Solvers coordinates RE I . Using the property that the set f i gN i D1 Among the various algorithms proposed for solving the minimizes the functional E, one shows that KS equations ( Fast Methods for Large Eigenvalues Problems for Chemistry), algorithms developed for E Na FI D rREI EKS .f i gN i D1 ; fRgI D1 / PW can be adapted and applied to FD discretizations. @ The two approaches use similar number of degrees E Na D EKS .f i gN (9) i D1 ; fRgI D1 / of freedom to represent the wave functions and thus @REI have similar ratios between the number of degrees of freedom and the number of wave functions to (Hellmann–Feynman forces, [8]). Since the mesh is incompute. However, some aspects are quite different dependent of the atomic positions, the wave functions between the two approaches and affect in particular do not depend explicitly on the atomic positions and the only quantities that explicitly depend on RI are their implementation. PW discretizations make use of the fact that the the atomic potentials. Thus, in practice, the force on Laplacian operator is diagonal in Fourier space not atom I can be computed by adding small variations to only to solve for the Hartree potential, but also to RE I in the x, y, and z directions, and computing finite precondition steepest descent corrections used to op- differences between the values of EKS evaluated for timize wave functions. For FD, the most scalable and shifted potentials but with the electronic structure that efficient solver for a Poisson problem is the multigrid minimizes EKS at RI , that is, without recomputing the method (see, e.g., [2]). Solving a Poisson problem on wave functions. The FD method is not variational: The energy does a mesh composed of O.N / nodes is achieved in O.N / not systematically decrease when one refines the disoperations with a very basic multigrid solver. The multigrid method has also been used as a cretization mesh. Energy can converge from below preconditioner to modify steepest descent directions (see Fig. 2). Usually the energy does not need to and speed up convergence [1, 7]. Preconditioned be converged to high precision in DFT calculations. steepest descent directions can be used in combination One typically relies on systematic errors introduced by

F

Finite Difference Methods in Electronic Structure Calculations −30.07 energy h=0.17 energy h=0.34 force h=0.17 force h=0.34

total energy (Ha)

−30.072 −30.074 −30.076 −30.078 −30.08 −30.082 3.5

3.6

3.7 3.8 3.9 4 distance Cl-Cl (Bohr)

0.08 0.07 0.06 0.05 0.04 0.03 0.02 0.01 0 −0.01 −0.02 −0.03 −0.04

force (Ha/Bohr)

526

E Na EKS Œfi gN i D1 ; fRI gI D1    Z N X 1 2 1  D 2 Sij j .r/  r i .r/d r 2 ˝ i;j D1 Z Z 1 e .r1 /e .r2 / d r1 d r2 C EXC Œe  C 2 ˝ ˝ jr1  r2 j N Z X Sij1 j .r/.Vext i /.r/d r: C i;j D1 ˝

(10)

4.1

Finite Difference Methods in Electronic Structure Calculations, Fig. 2 Energies and forces for C l2 molecule as function of the distance between the two atoms for two different meshes

discretization which only shifts the energy up or down. As illustrated in Fig. 2 for the case of a C l2 molecule, other physical quantities of interest, such as force in this case, can converge before the energy. By repeating the process of calculating the electronic structure, deriving the forces and moving atoms according to Newton’s equation for many steps, one can generate molecular dynamics trajectories. As an alternative to computing the ground-state electronic structure at every step, the Car–Parrinello molecular dynamics approach can be used. It was also implemented for a FD discretization [13].

with e .r/ D 2

N X

Sij1 j .r/i .r/

(11)

i;j D1

R and Sij D ˝ j .r/i .r/. Here we assume that all the orbitals are occupied with two electrons. This formulation does not reduce computational complexity since, for instance, the cost of orthonormalization is just shifted into a more complex calculation of the residuals for the eigenvalue problem. However, the flexibility gained by removing orthogonality constraints on the wave functions enables the possibility of adding locality constraints: One can impose a priori that each orbital is nonzero only inside a sphere of limited radius and appropriately located [7]. This is quite natural to impose on a real-space mesh and lead to O.N / degrees of freedom for the electronic structure. This approach is justified by the maximally localized Wannier functions’ representation of the electronic structure ( Linear Scaling Methods). Cutoff radii of 10 Bohr or less lead to practical accuracy for insulating system (with a finite band gap). While other ingredients are necessary to obtain a truly O.N / complexity algorithm, real-space truncation of orbitals is the key to reduce computational complexity in meshbased calculations.

O(N) Complexity Algorithms Probably the main advantage of FD over PW is the ability to truncate electronic wave functions in real space to obtain O.N / complexity algorithms ( Linear Scaling Methods). Typical implementation of DFT solvers require O.N 3 / operations for N electronic orbitals, while memory requirements grow as O.N 2 /. The O.N 2 / growth comes from the fact that the number of degrees of freedom per electronic wave function is proportional to the computational domain size References – one degree of freedom per mesh point – since quantum wave function live in the whole domain. 1. Briggs, E.L., Sullivan, D.J., Bernholc, J.: Real-space The O.N 3 / scaling of the solver is due to the fact multigrid-based approach to large-scale electronic structure calculations. Phys. Rev. B 54(20), 14,362–14,375 that each function needs to be orthogonal to all the (1996) others. 2. Briggs, W.L., Henson, V.E., McCormick, S.F.: A Mltigrid The first step to reduce scaling is to rewrite (1) in Tutorial, 2nd edn. Society for Industrial and Applied Mathterms of nonorthogonal electronic wave functions ematics, Philadelphia (2000)

Finite Element Methods 3. Chelikowsky, J.R., Troullier, N., Saad, Y.: Finite-differencepseudopotential method: electronic structure calculations without a basis. Phys. Rev. Lett. 72(8), 1240–1243 (1994) 4. Collatz, L.: The Numerical Treatment of Differential Equations. Springer, Berlin (1966) 5. Fattebert, J.L.: Finite difference schemes and block Rayleigh quotient iteration for electronic structure calculations on composite grids. J. Comput. Phys. 149, 75–94 (1999) 6. Fattebert, J.L., Gygi, F.: Density functional theory for efficient ab initio molecular dynamics simulations in solution. J. Comput. Chem. 23, 662–666 (2002) 7. Fattebert, J.L., Gygi, F.: Linear-scaling first-principles molecular dynamics with plane-waves accuracy. Phys. Rev. B 73, 115,124 (2006) 8. Feynman, R.: Forces in molecules. Phys. Rev. 56, 340–343 (1939) 9. Hodak, M., Wang, S., Lu, W., Bernholc, J.: Implementation of ultrasoft pseudopotentials in large-scale grid-based electronic structure calculations. Phys. Rev. B 76(8), 85108 (2007) 10. Mortensen, J., Hansen, L., Jacobsen, K.: Real-space grid implementation of the projector augmented wave method. Phys. Rev. B 71(3), 035109 (2005) 11. Ono, T., Hirose, K.: Timesaving double-grid method for real-space electronic-structure calculations. Phys. Rev. Lett. 82(25), 5016–5019 (1999) 12. Pickett, W.E.: Pseudopotential methods in condensed matter applications. Comput. Phys. Rep. 9, 115–198 (1989) 13. Schmid, R.: Car-Parrinello simulations with a real space method. J. Comput. Chem. 25(6), 799–812 (2004) 14. Wijesekera, N.R., Feng, G., Beck, T.L.: Efficient multiscale algorithms for solution of selfconsistent eigenvalue problems in real space. Phys. Rev. B 75(11), 115101 (2007)

527

sought as a finite linear combination of compactly supported, typically piecewise polynomial, basis functions, associated with a subdivision of the computational domain into a large number of simpler subdomains (finite elements).

Overview

The historical roots of the finite element method can be traced back to a paper by Richard Courant published in 1943; cf. [7]. The method was subsequently rediscovered by structural engineers and was termed the finite element method. Since the 1960s the finite element method has been developed into one of the most general and powerful class of techniques for the numerical solution of differential equations; see, for example, [3, 4, 6, 10–13]. Reasons for its popularity include the potential to approximate large classes of partial differential equations in general geometries, the availability of rigorous analysis of stability and convergence of the method, a wide choice of possible finite element spaces to obtain stable and accurate discretizations, and the potential for the development of adaptive algorithms with error control based on sharp a posteriori error bounds. There are two, conceptually different, approaches to the construction of finite element methods. The first of these, named after the Russian/Soviet mechanical engineer and mathematician Boris Grigoryevich Galerkin (1871–1945), is termed the Galerkin principle and is used in the solution of boundary-value problems for Finite Element Methods differential equations. The second approach, bearing the names of Lord Rayleigh (1842–1919) and Walther Endre S¨uli Ritz (1878–1909), is referred to as a the Rayleigh– Mathematical Institute, University of Oxford, Ritz principle and is associated with the finite element Oxford, UK approximation of energy-minimization problems such as those that arise in mechanics and the calculus of variations. The two approaches are related in that the Synonyms Galerkin principle can be viewed as a stationarity condition at the minimizer in a variational problem. FEM; Finite element approximation The Galerkin principle is however more general than the Rayleigh–Ritz principle; for example, non-selfadjoint elliptic boundary-value problems have an asDefinition sociated Galerkin principle, even though there is no natural energy functional that is minimized by the The finite element method is a numerical technique solution of the boundary-value problem. As was noted for the approximate solution of differential equations by Courant in his 1943 paper cited above, “Since and variational problems. The approximate solution is Gauss and W. Thompson, the equivalence between

F

528

boundary-value problems of partial differential equations on the one hand and problems of the calculus of variations on the other hand has been a central point in analysis. At first, the theoretical interest in existence proofs dominated and only much later were practical applications envisaged by two physicists, Lord Rayleigh and Walther Ritz; they independently conceived the idea of utilizing this equivalence for numerical calculation of the solutions, by substituting for the variational problems simpler approximating extremum problems in which but a finite number of parameters need be determined.” In Courant’s work the finite number of parameters were obtained via the Rayleigh–Ritz method with compactly supported basis functions that were chosen to be piecewise linear over a triangulation of the domain of definition of the analytical solution to the problem. This was a significant innovation compared to earlier efforts by Galerkin. On the other hand, while Courant adopted the Rayleigh– Ritz principle as the starting point for the construction of his method, Galerkin did not associate his technique with the numerical solution of minimization problems and viewed it as a method for the approximate solution of differential equations, by what is now referred to as the method of weighted residuals.

The Galerkin Principle Consider a differential equation, symbolically written as Lu D f , where L is a differential operator whose domain, D.L/, and range are contained in a Hilbert space H with inner product .; /. Suppose further that f is a given element in the range of L, and u 2 D.L/ is the unknown solution to the equation that is to be approximated. It is assumed that u exists and that it is unique. For any v 2 D.L/, the residual of v is defined by R.v/ WD f  Lv. Clearly, R.v/ D 0 if and only if v D u. Equivalently, .R.v/; w/ D 0 for all w 2 H if and only if v D u. Galerkin’s method is based on considering a linear subspace Hn of H of finite dimension n, contained in D.L/, and seeking a Galerkin approximation un 2 Hn to u by demanding that .R.un /; w/ D 0 for all w 2 Hn . The last equality, referred to as Galerkin orthogonality, can be restated as follows: un 2 Hn satisfies .Lun ; w/ D .f; w/ for all w 2 Hn . Under suitable assumptions on L and Hn , the Galerkin approximations un 2 Hn , n D 1; 2; : : : , thus defined exist and are unique, and the sequence

Finite Element Methods

.un /1 nD1 D.L/ H converges to u in the norm k  k of H in the sense that limn!1 ku  un k D 0, where k  k is defined by kwk WD .w; w/1=2 . A practical shortcoming of Galerkin’s method as stated above is that, in order to ensure that Lun 2 H, the linear spaces Hn , n D 1; 2; : : : are required to be contained in the domain, D.L/, of the differential operator L, and this leads to excessive demands on the regularity of the elements of Hn . In the construction of a finite element method, this difficulty is overcome by converting the differential equation, in tandem with the given boundary condition(s), into a weak form. Suppose, to this end, that ˝ Rd is a bounded open set in Rd with a Lipschitz-continuous boundary @˝. We shall denote by L2 .˝/ the linear space of square-integrable real-valued functions v defined on ˝, equippedR with the inner product .; / defined by .v; w/ WD ˝ v.x/ w.x/ dx and the induced norm kvk WD .v; v/1=2 . Let H m .˝/ denote the Sobolev space consisting of all functions v 2 L2 .˝/ whose (weak) partial derivatives @˛ v belong to L2 .˝/ for all ˛ D .˛1 ; : : : ; ˛d / such that j˛j WD ˛1 C    C ˛d  m, equipped with the norm kvkH m .˝/ WD  P ˛ 2 1=2 ; H01 .˝/ will denote the set of ˛ Wj˛jm k@ vk all v 2 H 1 .˝/ that vanish on @˝. Let a and c be realvalued functions, defined and continuous on ˝, such that c.x/  0 for all x 2 ˝, and there exists a positive constant c0 such that a.x/  c0 for all x 2 ˝. Let b be a d -component vector function whose components are real-valued and continuously differentiable on ˝. Assume, for simplicity, that r  b D 0 on ˝ and c.x/  c0 for all x 2 ˝. For f 2 L2 .˝/, we consider the boundary-value problem Lu W D r  .a.x/ru/ C b.x/  ru C c.x/u D f .x/ for x 2 ˝, with u D 0 on @˝.

(1)

By multiplying the differential equation in (1) with v 2 H01 .˝/, integrating over ˝, integrating by parts in the first term, and noting that the integral over @˝ that results from the partial integration vanishes, we obtain the following weak formulation of the boundary-value problem (1): find u 2 H01 .˝/ satisfying A.u; v/ D `.v/

8v 2 H01 .˝/;

where, for any w; v 2 H01 .˝/,

(2)

Finite Element Methods

529

a

b

Finite Element Methods, Fig. 1 (a) Finite element triangulation of the computational domain ˝. Vertices on @˝ are denoted by solids dots, and vertices internal to ˝ by circled solid dots.

Z A.w; v/ WD

a.x/rw  rv C .b.x/  rw/ v ˝

Cc.x/ wv dx

and `.v/ D .f; v/:

We shall describe the finite element approximation of (1), based on (2), in the special case when d D 2 and ˝ is a bounded open polygonal domain in R2 .

Finite Element Approximation We consider a triangulation of ˝ R2 , by subdividing ˝ into a finite number of closed triangles Ti , i D 1; : : : ; M , whose interiors are pairwise disjoint, and for each i; j 2 f1; : : : ; M g, i ¤ j , for which Ti \ Tj is nonempty, Ti \ Tj is either a common vertex or a common edge of Ti and Tj (cf. Fig. 1a). Let hT be the longest edge of a triangle T in the triangulation, and let h be the largest among the hT . Let Sh denote the linear space of all real-valued continuous functions vh defined on ˝ whose restriction to any triangle in the triangulation is an affine function. Let, further, Sh;0 WD Sh \ H01 .˝/. The finite element approximation of (2) is as follows: find uh 2 Sh;0 such that A.uh ; vh / D `.vh /

8vh 2 Sh;0 :

(3)

Let xi , i D 1; : : : ; L, be the vertices in the triangulation (cf. Fig. 1a), and let N D N.h/ denote the dimension of the finite element space Sh;0 . Let further f'j W j D 1; : : : ; N g denote the so-called nodal basis for Sh;0 , where the basis functions are defined by 'j .xi / D ıij , i D 1; : : : ; L, j D 1; : : : ; N . A typical piecewise linear nodal

(b) Piecewise linear nodal basis function associated with an internal vertex in a triangulation

basis function is shown in Fig. 1b. Thus, there exists a vector P U D .U1 ; : : : ; UN /T 2 RN such N that uh .x/ D j D1 Uj 'j .x/. The substitution of this expansion into (3) and taking vh D 'k , k D 1; : : : ; N, yield the system of linear algebraic equations PN j D1 A.'j ; 'k /Uj D `.'k /, k D 1; : : : ; N . By recalling the definition of A.; /, we see that the matrix  N T A.'j ; 'k / j;kD1 A WD of this system of linear equations is sparse and positive definite. The unique solution U D .U1 ; : : : ; UN /T 2 RN of the linear system yields the computed approximation uh to the analytical solution u on the given triangulation of ˝. As Sh;0 is a linear subspace of H01 .˝/, v D vh is a legitimate choice in (2). By subtracting (3) from (2), with v D vh , we can restate Galerkin orthogonality as: A.u  uh ; vh / D 0

8vh 2 Sh;0 :

(4)

By the assumptions on the coefficients a, b, and c stated above, A.v; v/  c0 kvk2H 1 .˝/ for all v 2 H01 .˝/, and there exists a positive constant c1 , such that A.w; v/  c1 kwkH 1 .˝/ kvkH 1 .˝/ for all w; v 2 H01 .˝/. Thus, by noting (4), we deduce the following result, known as C´ea’s lemma: ku  uh kH 1 .˝/  c1 c0 infvh 2Sh;0 ku  vh kH 1 .˝/ ; which expresses the fact that the finite element solution uh 2 Sh;0 is the nearbest approximation to the exact solution u 2 H01 .˝/ from the finite element subspace Sh;0 ; in the special case when c1 =c0 D 1 (which will occur, e.g., if a D c D c0 D const: > 0 and b D 0), the finite element solution uh 2 Sh;0 is the best approximation to the exact solution u 2 H01 .˝/ from the finite element space Sh;0 in the norm of the space H 1 .˝/. When c1 =c0 1, the numerical solution uh is typically a poor approximation to u in

F

530

the k  kH 1 .˝/ norm. The approximation and stability properties of the classical finite element method (3) can be improved in such instances by modifying in a consistent manner the definitions of A.; / and `./ through the addition of “stabilization terms.” The resulting finite element methods are referred to as stabilized finite element methods, a typical example being the streamline-diffusion finite element method; cf. [11]. C´ea’s lemma is a key tool in the analysis of finite element methods. Assuming, for example, that u 2 H 2 .˝/ \ H01 .˝/ and denoting by Ih the finite element interpolant of u defined by Ih u.x/ WD P N j D1 u.xj /'j .x/, where xj , j D 1; : : : ; N , are the vertices in the triangulation that are internal to ˝ (cf. Fig. 1a), by C´ea’s lemma, ku  uh kH 1 .˝/  cc10 ku  Ih ukH 1 .˝/ . Assuming further that the triangulation is shape regular in the sense that there exists a positive constant c , independent of h, such that for each triangle in the triangulation, the ratio of the longest edge to the radius of the inscribed circle is bounded below by c , arguments from approximation theory imply the existence of a positive constant c, O independent of h, such that ku  Ih ukH 1 .˝/  chkuk O H 2 .˝/ . Therefore, the following a priori error bound holds: ku  Q uh kH 1 .˝/  chkuk H 2 .˝/ , with cQ D cO c1 =c0 . Thus, as the triangulation is successively refined by letting h ! 0C , the sequence of finite element approximations uh converges to the analytical solution u in the H 1 .˝/ norm. It is also possible to derive a priori error bounds in other norms [4, 6].

Mixed Finite Element Methods and the Inf-Sup Condition Many problems in fluid mechanics, elasticity, and electromagnetism are modeled by systems of partial differential equations involving a number of dependent variables, which, due to their disparate physical nature, need to be approximated from different finite element spaces. If the finite element method is to have a unique solution and the method is to be stable, the choice of the finite element spaces in such mixed finite element methods in which the approximations to the various components of the vector of unknowns are sought cannot be arbitrary and need to satisfy a certain compatibility condition, called the inf-sup condition or Babuˇska–Brezzi condition (or Ladyzhenskaya– Babuˇska–Brezzi (LBB) condition); cf. [4, 5].

Finite Element Methods

Nonconforming and Discontinuous Galerkin Finite Element Methods There are instances when demanding continuity of the finite element approximation uh over the entire computational domain ˝ is too restrictive, either because the analytical solution exhibits steep layers, which are poorly approximated by continuous piecewise polynomial functions, or, as is the case in certain minimization problems in the calculus of variations that exhibit a Lavrentiev phenomenon, because the solution cannot be approached arbitrarily closely by a sequence of Lipschitz-continuous functions. In nonconforming finite element methods, the finite element space Sh in which the approximate solution is sought is still a subset of L2 .˝/, but the interelement continuity requirement is relaxed, for example, to continuity at selected points along edges (as is the case for the affine Crouzeix–Raviart element, for which continuity at midpoints of edges is imposed). In the case of discontinuous Galerkin finite element methods, no interelement continuity is generally demanded, and the fact that the finite element solution, which is then a discontinuous piecewise polynomial function, is meant to approximate a continuous analytical solution is encoded in the definition of the method through additional terms that penalize interelement jumps in the finite element solution. A Posteriori Error Analysis and Adaptivity A priori error bounds and asymptotic convergence results are of little practical use from the point of view of precise quantification of approximation errors on specific triangulations. A possible alternative is to perform a computation on a chosen triangulation and use the computed approximation to (i) quantify the approximation error a posteriori, and (ii) identify parts of the computational domain where the triangulation was inadequately chosen, necessitating local adaptive refinement or coarsening (h-adaptivity); cf. [2]. It is also possible to locally vary the degree of the piecewise polynomial function in the finite element space (p-adaptivity) and adjust the triangulation by moving/relocating the grid points (r-adaptivity). The h-adaptive loop for a finite element method has the form: SOLVE ! ESTIMATE ! MARK ! REFINE. In other words, first a finite element approximation is computed on a certain fixed triangulation of the computational domain. Then, in the second step, an a posteriori error bound is used to estimate the error

Finite Element Methods

531

6.000 4.750 3.500

0.8 0.6

2.250 1.000 Max: 6.000 Min: 1.000

0.4 0.2 0.0

F

-0.2 -0.4 -0.6 -0.8 -0.4

-0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

0.30 0.20 0.10 0.00 -0.10 -0.20 -0.30 -0.4

-0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

0.30 0.20 0.10 0.00 -0.00 -0.20 -0.30

Finite Element Methods, Fig. 2 An hp-adaptive finite element mesh (top), using piecewise polynomials with degrees 1; : : : ; 6 (indicated by the color coding), in a discontinuous Galerkin finite element approximation of the compressible Navier–Stokes

equations, and the computed density (middle) and x-momentum (bottom) for flow around the NACA0012 airfoil, with an angle of attack of 2ı , Mach number Ma = 0.5, and Reynolds number Re = 5,000 (By courtesy of Paul Houston)

in the computed solution: a typical a posteriori er- and f is a given right-hand side, is of the form ku  ror bound for a second-order elliptic boundary-value uh kH 1 .˝/  C jjjR.uh /jjj, where C is a computable problem Lu D f , where L is an elliptic operator constant, jjj  jjj is a certain norm, depending on the

532

problem, and R.uh / D f  Luh is the computable residual, which measures the extent to which the computed numerical solution uh fails to satisfy the partial differential equation Lu D f (cf. [1]). In the third step, on the basis of the a posteriori error bound, selected triangles in the triangulation are marked as those whose size is inadequate (i.e., too large or too small, relative to a prescribed local tolerance). Finally, the marked triangles are refined or coarsened. The process is then repeated either until a fixed termination criterion is reached (e.g., C jjjR.uh /jjj < TOL, where TOL is a preset global tolerance) or until the computational resources are exhausted. A similar adaptive loop can be used in p-adaptive finite element methods, except that the step REFINE is then interpreted as adjustment (i.e., increase or decrease) of the local polynomial degree, which, instead of being a fixed integer over the entire triangulation, may vary from triangle to triangle. Combinations of these strategies are also possible. Simultaneous h- and p-adaptivity is referred to as hp-adaptivity; it is particularly easy to incorporate into discontinuous Galerkin finite element algorithms thanks to the simple communication at interelement boundaries (cf. Fig. 2).

Outlook Current areas of active research in the field of finite element methods include the construction and mathematical analysis of multiscale finite element methods (e.g., homogenization problems cf. [8]), specialized methods for partial differential equations with highly oscillatory solutions (e.g., highwavenumber Helmholtz’s equation and Maxwell’s equation), stabilized finite element methods for nonself-adjoint problems, finite element approximation of transport problems (semi-Lagrangian, Lagrange– Galerkin, and moving-mesh finite element methods), finite element approximation of partial differential equations on manifolds, finite element methods for high-dimensional partial differential equations that arise from stochastic analysis, the construction and analysis of finite element methods for partial differential equations with random coefficients, and the approximation of the associated problems of uncertainty quantification, as well as finite element

Finite Element Methods

methods for adaptive hierarchical modeling and domain decomposition. There has also been considerable progress in the field of iterative methods, particularly Krylov subspace methods, and preconditioning associated with the solution of systems of linear algebraic equations with large sparse matrices that arise from finite element discretizations (cf. [9]), as well as the development of finite element software. Free and open-source finite element packages include the following: CalculiX, Code Aster, deal.II, DUNE, Elmer, FEBio, FEniCS, FreeFem++, Hermes, Impact, IMTEK Mathematica Supplement, OOFEM, OpenFOAM, OpenSees, and Z88. There are also a number of commercial finite element packages, including Abaqus, ANSYS, COMSOL Multiphysics, FEFLOW, LS-DYNA, Nastran, and Stresscheck.

References 1. Ainsworth, M., Oden, J.T.: A Posteriori Error Estimation in Finite Element Analysis. Pure and Applied Mathematics. Wiley, New York (2000) 2. Bangerth, W., Rannacher, R.: Adaptive Finite Element Methods for Differential Equations. Lectures in Mathematics ETH Z¨urich. Birkh¨auser Verlag, Basel (2003) 3. Braess, D.: Finite Elements: Theory, Fast Solvers, and Applications in Elasticity Theory, 3rd edn. Cambridge University Press, Cambridge (2007) 4. Brenner, S.C., Scott, L.R.: The Mathematical Theory of Finite Element Methods. Texts in Applied Mathematics, vol. 15, 3rd edn. Springer, New York (2008) 5. Brezzi, F., Fortin, M.: Mixed and Hybrid Finite Element Methods. Springer Series in Computational Mathematics, vol. 15. Springer, New York/Berlin/Heidelberg (1991) 6. Ciarlet, P.G.: The Finite Element Method for Elliptic Problems. Classics in Applied Mathematics, vol. 40. SIAM, Philadelphia (2002) 7. Courant, R.: Variational methods for the solution of problems in equilibrium and vibrations. Bull. Am. Math. Soc. 49(1), 1–23 (1943) 8. Weinan, E., Engquist, B.: Multiscale modeling and computation. Notice AMS 50(9), 1062–1070 (2003) 9. Elman, H.C., Silvester, D.J., Wathen, A.J.: Finite Elements and Fast Iterative Solvers: With Applications in Incompressible Fluid Dynamics. Numerical Mathematics and Scientific Computation. Oxford University Press, New York (2005) 10. Ern, A., Guermond, J.-L.: Theory and Practice of Finite Elements. Applied Mathematical Sciences, vol. 159. Springer, New York (2004) 11. Johnson, C.: Numerical Solution of Partial Differential Equations by the Finite Element Method. Dover, Mineola (2009). Reprint of the 1987 edition

Finite Element Methods for Electronic Structure 12. Schwab, C.: p- and hp-Finite Element Methods: Theory and Applications in Solid and Fluid Mechanics. Numerical Mathematics and Scientific Computation. The Clarendon/Oxford University Press, New York (1998) 13. S¨uli, E.: Lecture Notes on Finite Element Methods for Partial Differential Equations. Oxford Centre for Nonlinear Partial Differential Equations. Mathematical Institute, University of Oxford. http://www.maths.ox.ac.uk/groups/ oxpde/lecture-notes (2012)

Finite Element Methods for Electronic Structure John E. Pask Lawrence Livermore National Laboratory, Livermore, CA, USA

Definition The computation of the electronic wavefunctions and associated energies or electronic structure of atoms, molecules, and condensed matter lies at the heart of modern materials theory. We discuss the solution of the required large-scale nonlinear eigenvalue problems by modern finite element and related methods.

Overview The properties of materials – from color to hardness, from electrical to magnetic, from thermal to structural – are determined by quantum mechanics, in which all information about the state of a system is contained in the wavefunction (see entry  Schr¨odinger Equation for Chemistry). However, the wavefunction of a collection of M nuclei and N electrons is a function of 3.M CN / variables (coordinates), which is completely intractable to store or compute for all but the simplest systems of a few atoms. In order to make progress, therefore, approximations are required. For applications involving more than a few atoms, some form of Hartree-Fock (HF) (see entry  Hartree–Fock Type Methods) or density functional theory (DFT) (see entry  Density Func-

533

tional Theory) approximation is commonly employed, reducing the required problem to a nonlinear eigenvalue problem for N three-dimensional eigenfunctions fi .x/g and associated eigenvalues f"i g corresponding to N single-particle orbitals and associated energies (see entries  Variational Problems in Molecular Simulation and  Numerical Analysis of Eigenproblems for Electronic Structure Calculations). The solution of this problem remains, however, a formidable task. Large problems can require the solution of thousands of eigenfunctions with millions of degrees of freedom each – thousands of times over in the course of a molecular dynamics simulation. Numerous methods of solution have been developed over the course of decades [10]. Here, we discuss one of the more recent developments: the application of modern finite element [12] and related methods to the solution of the Kohn-Sham equations [7] of DFT, with the goal of increasing the range of physical systems which can be investigated by such accurate, quantum mechanical means. Due to the correspondence of the resulting eigenproblems, the methods discussed here apply to the HF equations as well. Like standard planewave and Gaussian or Slater-orbital-based methods [10], the FE method is a variational expansion approach in which approximate solutions are obtained by discretizing the continuous problem in a finite dimensional basis. Like the planewave method, the FE method is systematically improvable, allowing rigorous control of approximation errors by virtue of the polynomial nature of the basis. This is in marked contrast to Gaussian or Slater-orbital-based methods which require careful tuning for each particular problem and can suffer from ill- conditioning when pushed to higher accuracies [3]. While the planewave basis is global, however, with each function overlapping every other at every point in the domain, the FE basis is strictly local (compactly supported), with each function overlapping only its nearest neighbors. This leads to sparse matrices and allows for efficient, large-scale parallel implementation. By virtue of its systematically improvable basis and strict locality, the FE method combines significant advantages of both planewave and finite-difference-based approaches (see entry  Finite Difference Methods in Electronic Structure Calculations).

F

534

Finite Element Methods for Electronic Structure

i .xj / D ıij :

Finite Element Bases Finite element bases consist of strictly local piecewise polynomials [12]. A simple example is shown in Fig. 1: a one-dimensional (1D), piecewise-linear basis on domain ˝ D .0; 1/. In this case, the domain is partitioned into three elements ˝1 –˝3 ; in practice, there are typically many more so that each element encompasses only a small fraction of the domain. The basis in Fig. 1 illustrates the key properties of all such nodal bases, whether of higher dimension or higher polynomial order. The basis functions are strictly local, i.e., nonzero over only a (typically) small fraction of the domain. This leads to sparse matrices and efficient parallel implementation. Within each element, the basis functions are simple, low-order polynomials, which leads to computational efficiency, generality, and systematic improvability. The basis functions are C 0 in nature, i.e., continuous but not smooth, greatly facilitating construction in higher dimensions, in complex geometries in particular. Finally, the basis is cardinal, i.e., Finite Element Methods for Electronic Structure, Fig. 1 1D piecewise-linear FE bases. (a) General. (b) Dirichlet. (c) Periodic. (d) Bloch periodic

a

b

c

d

(1)

By virtue of this property, an FE expansion f .x/ D P i ci i .x/ has the property f .xj / D cj so that the expansion coefficients have a direct, “real-space” meaning. This eliminates the need for computationally intensive transforms, such as Fourier transforms in planewave-based solution methods [10], and facilitates preconditioning such as multigrid in grid-based approaches [3]. Figure 1a shows a general C 0 linear basis, capable of representing any piecewise linear function (having the same polynomial subintervals) exactly. To solve a problem subject to Neumann boundary conditions f 0 .0/ D f 0 .1/ D 0, one would use such a basis in a weak formulation (as discussed below) containing that condition as a natural boundary condition – i.e., one contained in the problem formulation rather than trial space from which the solution is drawn. To solve a problem subject to Dirichlet boundary conditions f .0/ D f .1/ D 0, as occur in atomic and molecular calculations, one would employ a basis as

Finite Element Methods for Electronic Structure

in Fig. 1b, omitting boundary functions, thus enforcing the condition as an essential boundary condition – i.e., one contained in the trail space. To solve a problem subject to periodic boundary conditions, f .0/ D f .1/ and f 0 .0/ D f 0 .1/, as occur in condensed matter calculations, one would use a basis as in Fig. 1c. In this case, since the basis satisfies i .0/ D i .1/ but not i0 .0/ D i0 .1/, the value-periodic condition f .0/ D f .1/ is enforced as an essential boundary condition, while the derivative-periodic condition f 0 .0/ D f 0 .1/ is enforced as a natural one [11]. Finally, to solve a problem subject to Bloch-periodic or Floquet boundary conditions, f .1/ D e i ka f .0/ and f 0 .1/ D e i ka f 0 .0/, as occurs in solid-state electronic structure, one would use a basis as in Fig. 1d, i.e., associate functions at domain boundaries after multiplying by Bloch phase factor e i kxj ; where k is the Bloch wavevector, xj is the coordinate of the node associated with boundary function j , and a is the length of the domain (in this case, 1). Here again, the value-periodic condition is enforced as an essential boundary condition, while the derivative-periodic condition is enforced as a natural one [13]. Higher-order FE bases are constructed by increasing polynomial completeness in each element: three quadratics in each element for quadratic completeness, four cubics for cubic completeness, etc. And with higher-order completeness comes the possibility of higher-order smoothness. For example, with cubic completeness, one can construct a standard C 0 Lagrange basis, C 1 Hermite basis, or C 2 Bspline basis, as shown in Fig. 2. Lagrange bases are C 0 for all polynomial orders p, Hermite bases are C .p1/=2 , and B-splines are C .p1/ . Note, however, that with increased smoothness comes decreased locality: e.g., while one Lagrange function overlaps six of its neighbors, the others overlap only three, whereas each Hermite function overlaps five neighbors and each B-spline function overlaps six. This difference becomes even more pronounced in higher dimensions, leading to less sparseness in discretizations and more communications in parallel implementations. Moreover, cardinality is lost for smoother bases, leading to less efficient nonlinear operations (such as constructing the charge density and evaluating exchange-correlation functionals in electronic structure), complications in imposing boundary conditions, and constraints on meshes in higher dimensions. However, for sufficiently smooth

535

problems, such higher-order continuity can yield greater accuracy per degree of freedom (DOF) [12] and more accurate derivatives, as required for forces and certain exchange-correlation functionals in electronic structure. Both C 0 (e.g., [8,11,14]) and smoother (e.g., [5, 15, 16]) bases have been employed in electronic structure. Whether C 0 or smoother, however, what is clear from much work in the area is that traditional low-order (e.g., linear) FE bases are not sufficient for electronic structure due to the relatively high accuracies required, e.g., errors on the order of one part in 105 as opposed to tenths of percents typical in engineering applications. Due to the accuracies required, higher-order bases, which afford higher accuracy per DOF [12], are generally necessary. Higher-dimensional FE bases are constructed along two main lines. Simplest, and most common in the context of smoother bases is a tensor product of 1D basis functions: ˚ij k .x; y; z/ D i .x/j .y/k .z/. However, while simple, and advantageous with respect to separability, this results in more basis functions per element than required for a given polynomial completeness. To reduce DOFs per element, 3D bases can also be defined directly: e.g., specify n nodes on the 3D element and a 3D polynomial of n terms, then generate the basis by requiring cardinality at each node. Standard “serendipity” bases [12] are of this type, for example. If adaptivity is a prime concern, as in all-electron calculations where wavefunctions are highly oscillatory in the vicinity of nuclei, then tetrahedral elements can be advantageous (e.g., [8,14]). For smoother, pseudopotential-based calculations, hexahedral (rectangular solid) elements generally afford a more efficient path to higher orders and accuracies (e.g., [11, 15]).

¨ Schrodinger and Poisson Equations We now consider the solution of the Schr¨odinger and Poisson equations required in the solution of the Kohn-Sham equations. We consider a general C 0 basis so that the development applies to smoother bases as well. For isolated systems, such as atoms or molecules, the solution of the required Schr¨odinger andPoisson problems is relatively straightforward: a sufficiently large domain is chosen so that the wavefunctions vanish on the boundary and the potential either

F

536

Finite Element Methods for Electronic Structure

a

b

c

Finite Element Methods for Electronic Structure, Fig. 2 1D piecewise-cubic FE bases. (a) C 0 Lagrange. (b) C 1 Hermite. (c) C 2 B-spline

vanishes or can be determined efficiently by other means (e.g., multipole expansion). The equations are then solved subject to Dirichlet boundary conditions. Condensed matter systems (i.e., solids and liquids) on the other hand are modeled as infinite periodic systems, and extra care is required in imposing appropriate boundary conditions and handling divergent lattice sums. We consider the latter case here. In a perfect crystal, the electrostatic potential is periodic, i.e., V .x C R/ D V .x/ (2) for all lattice vectors R, and the solutions of the Schr¨odinger equation satisfy Bloch’s theorem

.x C R/ D e i kR .x/

(3)

for all lattice vectors R and wavevectors k (see entry  Mathematical Theory for Quantum Crystals). Hence,

to find solutions in the infinite crystal, we need only consider a finite unit cell. ¨ Schrodinger Equation We consider the Schr¨odinger problem in a unit cell, subject to boundary conditions consistent with Bloch’s theorem:  12 r 2

CV`

C VO n`

D"

in ˝;

(4)

Finite Element Methods for Electronic Structure

537

X vLa .x a Rn /haL vLa .x0  a Rn /; (7) VO n` .x; x0 /D n;a;L

where n runs over all lattice vectors, a runs over atoms in the unit cell, and L indexes projectors on each atom, the nonlocal term in (4) is VO n`

X

D

vLa .x   a  Rn /haL

n;a;L

Z

d x0 vLa .x0   a  Rn / .x0 /;



Finite Element Methods for Electronic Structure, Fig. 3 Parallelepiped unit cell (domain) ˝, boundary , surfaces 1 – 3 , and associated lattice vectors R1 –R3

.x C R` / D e i kR` .x/; r .x C R` /  nO D e

i kR`

x 2 ` ;

O r .x/  n;

(5)

where the integral is over all space (R3 ). Rewriting the integral over all space as a sum over all unit cells and using the Bloch periodicity of , the integral in (8) can be transformed to the unit cell so that the nonlocal term in (4) becomes VO n`

D

XX

x 2 ` ; (6)

where is the wavefunction, " is the energy eigenvalue, VO D V ` C VO n` is the periodic potential, a sum of local and nonlocal parts, ` and R` are the surfaces and associated lattice vectors of the boundary , and nO is the outward unit normal at x, as shown in Fig. 3. (Atomic units are used throughout.) The nonlocal term arises in pseudopotential-based calculations [10] wherein the nucleus and core electrons are replaced by a smooth, effective potential having smooth valence wavefunctions with the same energies as the original “all-electron” potential, in order to facilitate computations. Since core electrons are tightly bound to the nuclei and largely inert with respect to chemistry, this is often an excellent approximation and is widely used. When it is not sufficient, “all-electron” calculations are necessary, which are more computationally intensive and in which the above nonlocal term does not occur. For generality, we retain the nonlocal term here. Since the problem is posed in the finite unit cell, nonlocal operators require special consideration. In particular, if as is typically the case for ab initio pseudopotentials, the domain of definition is all space (i.e., the infinite crystal), the operators must be transformed to the relevant finite subdomain (i.e., the unit cell). For a separable potential of the usual form [10]

(8)

e i kRn vLa .x   a  Rn /haL

a;L

n



d x0

Z ˝

X

e i kRn0 vLa .x0  a Rn0 / .x0 /:

n0

(9) Having transformed the relevant operators to the unit cell, the differential formulation (4)–(6) is then recast in weak form in order to accommodate the use of a C 0 basis and incorporate the derivative-periodic condition (6) [11]: find the scalars " 2 R and functions 2 W such that Z ˝

 1 2 rv

Z

r

v

D"

C v V `

C v  VO n`

d x 8v 2 W;

dx (10)

˝

where W D fw 2 H 1 .˝/ W w.xCR` / D e i kR` w.x/; x 2 ` g: Discretization in a C 0 basis then proceeds in the usual way. Let D

n X j D1

cj j

and v D i ;

F

538

Finite Element Methods for Electronic Structure

where fi gniD1 is a complex C 0 basis satisfying the Bloch-periodic condition (5) and fcj g are complex coefficients so that and v are restricted to a finite dimensional subspace Wn W. From (10) then, we arrive at a generalized eigenproblem determining the approximate eigenvalues " and eigenfunctions D P c  of the required problem: j j j X

Hij D

Z ˝

Hij cj D "

j

X

Sij cj ;

Z

Z rv  rV d x D 4



˝

where V D fv 2 H 1 .˝/ W v.x/ D v.xCR` /; x 2 ` g. Subsequent discretization in a real periodic FE basis j then leads to a symmetric linear system determining P the approximate solution V .x/ D j cj j .x/ of the required problem: X

(11)

j

1 ri 2

 rj C i V ` j C i VO n` j d x; (12)

Z Sij D ˝

i j d x:

v.x/ d x 8v 2 V; (19)

˝

Lij cj D fi ;

(20)

ri .x/  rj .x/ d x;

(21)

j

Z Lij D 

˝

Z

fi D 4

(13)

i .x/.x/ d x:

(22)

˝

For a separable potential of the usual form (7), transformed to the unit cell as in (9), the nonlocal term in Kohn-Sham Equations (12) becomes In the pseudopotential approximation [10], the KohnSham equations of density functional theory are Z X given by aj d x i VO n` j D fLai haL .fL / ; (14) ˝

a;L

Z fLai

D ˝

d x i .x/

X

e i kRn vLa .x

 12 r 2   a  Rn /: (15)

n

i .x/

C VOeff

i .x/

i .x/;

(23)

VOeff D VI` C VOIn` C VH C Vxc ; X VI;a .x/; VI` D

Since the Bloch-periodic basis fi g is complex valued, both H and S are Hermitian for a separable potential of the form (7). Furthermore, since both FE basis functions i and projectors vLa are localized in space, both H and S are sparse.

VOIn`

Poisson Equation The Poisson solution proceeds along the same lines as the Schr¨odinger solution. In this case, the required problem is

Vxc D Vxc .xI e /; X e D  fi i .x/

r 2 V .x/ D 4.x/ in ˝;

D "i

a i

XZ

D

Z VH D 

n` .x; x0 / d x0 VI;a

(24) (25) i .x

0

/;

(26)

a

d x0

e .x0 / ; jx  x0 j

(27) (28)

i .x/;

(29)

i

(16) where i and "i are the Kohn-Sham eigenfunctions and n` are the local and nonlocal eigenvalues, VI;a and VI;a V .x/ D V .x C R` /; x 2 ` ; (17) parts of the ionic pseudopotential of atom a, e is the nO  rV .x/ D nO  rV .x C R` /; x 2 ` ; (18) electronic charge density, the integrals extend over all space (R3 ), and the summations extend over all where V .x/ is the potential energy of an electron in atoms a and states i with occupations fi (see entry the charge density .x/ and the domain ˝, bounding  Density Functional Theory). (For simplicity, we surfaces ` , and lattice vectors R` are again as in Fig. 3. omit spin and crystal momentum indices and consider The weak formulation of (16)–(18) is then [11]: find the case in which the external potential arises from V 2 V such that the ions.) The nonlocal part VOIn` and exchange-

Finite Element Methods for Electronic Structure

539

correlation potential Vxc are determined by the choice of pseudopotentials and exchange-correlation functional, respectively. VI` is the Coulomb potential arising from the ions, and VH is that arising from the electrons (Hartree potential). We retain the nonlocal term for generality; in all-electron calculations, it is omitted. Since the eigenfunctions i depend on the effective potential VOeff in (23), which depends on the electronic density e in (27) and (28), which depends again on the eigenfunctions i in (29), the Kohn-Sham equations constitute a nonlinear eigenvalue problem (see entry  Numerical Analysis of Eigenproblems for Electronic Structure Calculations). They are commonly solved via “self-consistent” (fixed point) iteration (see entry  Self-Consistent Field (SCF) Algorithms). The process is generally as follows: An initial electronic charge density ein is constructed (e.g., by overlapping atomic charge densities). The effective potential VOeff is computed from (24). The eigenstates i are computed by solving the associated Schr¨odinger equation (23). Finally, a new electronic density e is computed from (29). If e is sufficiently close to ein , then selfconsistency has been reached; otherwise, a new ein is constructed from e (and possibly previous iterates), and the process is repeated until self-consistency is achieved. For isolated systems, such as atoms and molecules, the construction of VOeff is straightforward: all sums and integrals are finite. In condensed matter systems, however, modeled by infinite crystals, individual terms diverge due to the long-range 1=r nature of the Coulomb

Etot D

X i

C 12

Z fi “

dx d xd x0

 1 2 i .x/. 2 r / i .x/ 

e .x/e .x0 / C jx  x0 j

1 2

potential, and the sum is only conditionally convergent [2]; hence, some extra care is required to obtain welldefined results efficiently. We consider the latter case here. In an infinite crystal, VI` and VH are divergent, and the total Coulomb potential VC D VI` C VH within the unit cell depends on ions and electrons far from the unit cell due to the long-range 1=r nature of the Coulomb interaction. Both difficulties may be overcome, however, by replacing long-range ionic potentials by the short-ranged charge densities which generate them and incorporating long-range interactions into boundary conditions on the unit cell (e.g., [11]). By construction, the local ionic pseudopotentials VI;a of each atom a vary as Za =r (or rapidly approach this) outside their respective pseudopotential cutoff radii rc;a , where Za is the effective ionic charge and r is the radial distance. They thus correspond, by Poisson’s equation, to charge densities I;a strictly localized within rc;a (or rapidly approaching this). The total ionic charge density I D P a I;a .x/ is then a short-ranged sum, unlike the sum of ionic potentials. Having constructed the ionic charge density in the unit cell, the total charge density  D I Ce may then be constructed and the total Coulomb potential VC D VI` C VH may be computed at once by a single Poisson solution subject to periodic boundary conditions: r 2 VC .x/ D 4.x/; (30) whereupon VOeff may be evaluated as in (24). Having solved the Kohn-Sham equations, the ground state total energy is then given by

Z

X a;a0 ¤a

where Za is the ionic charge of atom a at position  a , "xc is determined by the choice of exchangecorrelation functional, and, as in (25)–(29), the integrals extend over all space, and the summations extend over all atoms a and a0 , and states i with occupations fi (see entries  Variational Problems in Molecular Simulation and  Density Functional Theory). In an

d x e .x/VI` .x/ C Za Za0  j a   a0 j

Z

X

Z fi

dx

 O n` i .x/VI

i .x/

i

d x e .x/"xc .xI e /;

(31)

infinite crystal, the total energy per unit cell may be obtained by restricting the integrals over x and summation on a to the unit cell, while the integral over x0 and summation on a0 remain over all space. In terms of total density  and Coulomb potential VC , however, this can be reduced to a local expression, with all integrals and summations confined to the unit cell [11]:

F

540

Etot D

Finite Element Methods for Electronic Structure

X i

Z fi "i C ˝

 ` d x e .x/Veff .x/  12 .x/VC .x/

e .x/"xc .xI e /  Es ;

(32)

where Es is the ionic self-energy computed from potentials VI;a and densities I;a . Figure 4 shows the convergence of the FE total energy and eigenvalues to exact values as the number of elements is increased in a self-consistent GaAs calculation at an arbitrary k point [11], where “exact” values were obtained from a well-converged planewave calculation. FE results are for a series of uniform meshes from 888 to 323232 in the fcc primitive cell using a cubic serendipity basis. The variational nature and optimal convergence of the method are clearly manifested: the error is strictly positive and rapidly establishes an asymptotic slope of  6 on the log-log scale, indicating an error of order h6 , consistent with analysis for linear elliptic problems [12] predicting an error of order h2p for a basis of order p. Conditions under which such optimal rates obtain for nonlinear elliptic problems also have been recently elucidated (see entry  Numerical Analysis of Eigenproblems for Electronic Structure Calculations).

Recent Developments Over the course of the past decade, the application of the FE method to electronic structure problems has matured to the point that FE codes are now competitive with the most mature and highly optimized planewave codes in a number of contexts, particularly in largescale parallel calculations. However, planewave methods in condensed matter and Gaussian- or Slater-type orbital methods in quantum chemistry remain most widely used in practice. While this is in part due to the high maturity and optimization of the more established codes, there is another key reason: FEbased methods can require an order of magnitude or more DOFs (basis functions) than standard planewave or atomic-orbital-based methods to attain the required accuracies, leading to greater storage requirements and increased floating point operations. Other such local, real-space approaches such as finite-difference (see entry  Finite Difference Methods in Electronic Structure Calculations) and wavelet-based methods [1] suffer from the same disadvantage, in the condensed

matter context in particular, where atoms are distributed throughout the unit cell and periodic boundary conditions apply. The DOF disadvantage can be mitigated by going to higher-order, e.g., fourth-order FE [8], 8th or 12th-order finite differences [3], and seventh-order wavelets [4]. However, as order is increased, locality is decreased, leading to less sparsity, higher cost per DOF, and less efficient parallelization. Another issue faced by real-space methods is preconditioning (see entry  Fast Methods for Large Eigenvalues Problems for Chemistry). In the planewave basis, the Laplacian is diagonal and so is readily invertible to provide efficient preconditioning. In real-space representations, such preconditioning comes at greater cost, e.g., by multigrid (see entry  Finite Difference Methods in Electronic Structure Calculations) or other smoothing (see entry  Fast Methods for Large Eigenvalues Problems for Chemistry). In the context of FE methods, recent progress on reducing or eliminating the DOF disadvantage altogether has come along three main lines: first, and most straightforward is reducing DOFs by going to higher polynomial order, typically via “spectral elements” [6] to maintain matrix conditioning. This approach affords the additional advantage of producing a standard rather than generalized discrete eigenproblem, which is more efficient to solve (see entry  Fast Methods for Large Eigenvalues Problems for Chemistry). Another approach has been to build known atomic physics into the FE basis via partition-of-unity FE techniques [13], thus increasing efficiency and decreasing size of the required basis substantially. Initial results along these lines have shown order-of-magnitude reductions in DOFs relative to current state-of-the-art planewave methods. A third direction, related to the previous, has been to reduce DOFs by building not just known atomic physics but atomic-environment effects also into the basis, using a discontinuous Galerkin framework [9] with subdomain Kohn-Sham solutions as a basis. By virtue of the DG framework, this approach produces standard rather than generalized eigenproblems also. Initial results along these lines have shown DOF reductions down to the level of minimal Gaussian basis sets while retaining strict locality and systematic improvability. Given their current competitiveness and continuing advance along multiple lines, finite element based electronic structure methods stand to play an increasing

Finite Fields

541

Finite Element Methods for Electronic Structure, Fig. 4 Error EFE  EEXACT of finite element (FE) self-consistent total energy and Kohn-Sham eigenvalues at an arbitrary k point for GaAs in the local density approximation

F

role in computational materials science in the years 10. Martin, R.M.: Electronic Structure: Basic Theory and Practical Methods. Cambridge University Press, Cambridge ahead, as ever larger-scale parallel computers become (2004) available, in particular. 11. Pask, J.E., Sterne, P.A.: Finite element methods in ab initio Acknowledgements This work was performed under the auspices of the US Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.

12. 13.

References 14. 1. Arias, T.A.: Multiresolution analysis of electronic structure: semicardinal and wavelet bases. Rev. Mod. Phys. 71(1), 267–311 (1999) 2. Ashcroft, N.W., Mermin, N.D.: Solid State Physics. Holt, Rinehart and Winston, New York (1976) 3. Beck, T.L.: Real-space mesh techniques in densityfunctional theory. Rev. Mod. Phys. 72(4), 1041–1080 (2000) 4. Genovese, L., Neelov, A., Goedecker, S., Deutsch, T., Ghasemi, S.A., Willand, A., Caliste, D., Zilberberg, O., Rayson, M., Bergman, A., Schneider, R.: Daubechies wavelets as a basis set for density functional pseudopotential calculations. J. Chem. Phys. 129(1), 014109 (2008). doi:10.1063/1.2949547 5. Hernandez, E., Gillan, M.J., Goringe, C.M.: Basis functions for linear-scaling first-principles calculations. Phys. Rev. B 55(20), 13485–13493 (1997) 6. Karniadakis, G.E., Sherwin, S.: Spectral/hp Element Methods for Computational Fluid Dynamics, 2nd edn. Oxford University Press, New York (2005) 7. Kohn, W., Sham, L.J.: Self-consistent equations including exchange and correlation effects. Phys. Rev. 140, A1133– A1138 (1965) 8. Lehtovaara, L., Havu, V., Puska, M.: All-electron density functional theory and time-dependent density functional theory with high-order finite elements. J. Chem. Phys. 131(5), 054103 (2009) 9. Lin, L., Lu, J., Ying, L., Weinan, E.: Adaptive local basis set for Kohn-Sham density functional theory in a discontinuous Galerkin framework I: total energy calculation. J. Comput. Phys. 231(4), 2140–2154 (2012)

15.

16.

electronic structure calculations. Model. Simul. Mater. Sci. Eng. 13(3), R71–R96 (2005) Strang, G., Fix, G.J.: An Analysis of the Finite Element Method. Prentice-Hall, Englewood Cliffs (1973) Sukumar, N., Pask, J.E.: Classical and enriched finite element formulations for Bloch-periodic boundary conditions. Int. J. Numer. Methods Eng. 77(8), 1121–1138 (2009) Suryanarayana, P., Gavini, V., Blesgen, T., Bhattacharya, K., Ortiz, M.: Non-periodic finite-element formulation of KohnSham density functional theory. J. Mech. Phys. Solids 58(2), 256–280 (2010) Tsuchida, E., Tsukada, M.: Large-scale electronic-structure calculations based on the adaptive finite-element method. J. Phys. Soc. Jpn. 67(11), 3844–3858 (1998) White, S.R., Wilkins, J.W., Teter, M.P.: Finite-element method for electronic-structure. Phys. Rev. B 39(9), 5819– 5833 (1989)

Finite Fields Harald Niederreiter RICAM, Austrian Academy of Sciences, Linz, Austria

Definition Terms/Glossary Finite field Algebraic structure of a field with finitely many elements Galois field Alternative name for finite field Finite prime field Finite field with a prime number of elements

542

Primitive element Generator of the cyclic multiplicative group of nonzero elements of a finite field Normal basis Ordered basis of a finite extension field consisting of all conjugates of a fixed element Irreducible polynomial Polynomial that cannot be factored into polynomials of smaller degrees Primitive polynomial Minimal polynomial of a primitive element Permutation polynomial Polynomial that permutes the elements of a finite field Discrete logarithm Analog of the logarithm function for the multiplicative group of a finite field Pseudorandom numbers Deterministically generated sequence that simulates independent and uniformly distributed random variables

Structure Theory A finite field is an algebraic structure satisfying the axioms of a field and having finitely many elements. Historically, the first-known finite fields were the residue class rings Z=pZ of the ring Z of integers modulo a prime number p. The field Z=pZ is called a finite prime field. It has p elements and does not contain a proper subfield. The basic properties of finite prime fields were already known in the eighteenth century through the work of Fermat, Euler, Lagrange, and others. The theory of general finite fields was initiated by Galois in a famous paper published in 1830. By the end of the nineteenth century, this theory was well developed. Finite fields became a crucial structure in discrete mathematics via many important applications that were found in the twentieth century. This article covers fundamental facts about finite fields as well as a selection of typical applications of finite fields. Basic resources on finite fields are the books Lidl and Niederreiter [3] and Shparlinski [6]. Applications of finite fields are covered in Lidl and Niederreiter [2] and Mullen and Mummert [4]. The first important issue about finite fields is that of their cardinality. A finite field F has a prime characteristic p, that is, we have p  a D 0 for all a 2 F: Furthermore, F contains Z=pZ as a subfield. Since F can also be viewed as a vector space over Z=pZ of finite dimension n, say, it follows that the cardinality of F is p n and hence a prime power. Conversely, for any prime power q, there exists a finite field of cardinality

Finite Fields

q. In fact, finite fields of the same cardinality q are isomorphic as fields. Thus, we can speak of the finite field with q elements, and we denote it by Fq . Some authors honor the pioneering work of Galois by calling Fq the Galois field with q elements and using the alternative notation GF.q/, but we will employ the more common notation Fq . The finite field Fq can be explicitly constructed for any prime power q. If q D p for some prime number p, then Fp is isomorphic to Z=pZ. If q D p n for some prime number p and some integer n  2, then Fq is isomorphic to the residue class ring Fp Œx=f .x/Fp Œx of the polynomial ring Fp Œx modulo an irreducible polynomial f .x/ over Fp of degree n. An irreducible polynomial over Fp of degree n exists for every prime number p and every positive integer n. The finite field Fq is also isomorphic to the splitting field of the polynomial x q  x over its prime subfield Fp . In fact, Fq consists exactly of all roots of the polynomial x q  x 2 Fp Œx in a given algebraic closure of Fp . The cardinality of every subfield of Fq has the form p d , where the integer d is a positive divisor of n. Conversely, if d is a positive divisor of n, then there exists exactly one subfield of Fq with p d elements. For any finite field Fq , the set Fq of nonzero elements of Fq forms a group under multiplication. It is an important fact that the group Fq is cyclic. Any generator of the cyclic group Fq is called a primitive element of Fq . The number of primitive elements of Fq is given by .q1/, where  is Euler’s totient function.

Bases Given a finite field Fq , we can consider its extension field of degree n, which is the finite field Fq n with q n elements. Then Fq n can also be viewed as a vector space over Fq of dimension n. Thus, a natural way of representing the elements of Fq n is in terms of an ordered basis of Fq n over Fq . Various types of convenient bases are available for this purpose. A polynomial basis of Fq n over Fq is an ordered basis of the form f1; ˛; ˛ 2 ; : : : ; ˛ n1 g, where ˛ 2 Fq n is a root of an irreducible polynomial over Fq of degree n. A normal basis of Fq n over Fq is an ordered basis 2 n1 of the form fˇ; ˇ q ; ˇ q ; : : : ; ˇ q g with a suitable ˇ 2 Fq n . A normal basis exists for every extension Fq n =Fq . Normal bases are useful for implementing fast

Finite Fields

543

arithmetic in Fq n . Other types of bases are obtained by polynomial-time algorithm is currently available for making use of the trace function TrF=K of F D Fq n this factorization problem, but several algorithms are over K D Fq , which is defined by practicable in reasonable ranges for q and the degree of f .x/. Standard computer algebra packages contain n1 also factorization algorithms for polynomials over fiX i TrF=K . / D q for all  2 F: nite fields. A straightforward argument shows that it i D0 suffices to consider algorithms for monic polynomials with no multiple roots. For any ordered basis A D f˛1 ; : : : ; ˛n g of Fq n over The classical factorization algorithm in this context Fq , there exists a unique dual basis, that is, an ordered is the Berlekamp algorithm. Given a monic polynomial basis B D fˇ1 ; : : : ; ˇn g of Fq n over Fq which satisfies f .x/ over Fq of degree n  1 with no multiple roots, TrF=K .˛i ˇj / D ıij for 1  i; j  n, where ıij is the we construct the n  n matrix B D .bij /0i;j n1 over Kronecker symbol. The dual basis of a normal basis is Fq via the congruences again a normal basis. An ordered basis A of Fq n over Fq is self-dual if A is its own dual basis. The extension n1 X Fq n =Fq has a self-dual basis if and only if either q is bij x j .mod f .x// for 0  i  n  1: x iq  even or both q and n are odd. j D0

Irreducible Polynomials and Factorization For any finite field Fq , the number of monic irreducible polynomials over Fq of degree n is given P by .1=n/ d jn .n=d /q d , where the sum is over all positive divisors d of n and  is the M¨obius function. This implies easily that for any positive integer n, there exists an irreducible polynomial over Fq of degree n. If f .x/ is an irreducible polynomial over Fq of degree n, then f .x/ has a root ˛ 2 Fq n and all roots of f .x/ are 2 n1 given by the n distinct elements ˛; ˛ q ; ˛ q ; : : : ; ˛ q of Fq n . In particular, the splitting field of f .x/ over Fq is equal to Fq n . Furthermore, the extension Fq n =Fq is a Galois extension with a cyclic Galois group. A polynomial over Fq of degree n  1 is called a primitive polynomial over Fq if it is the minimal polynomial over Fq of a primitive element of Fq n . A primitive polynomial over Fq is automatically monic and irreducible, but not every monic irreducible polynomial over Fq is primitive. The number of primitive polynomials over Fq of degree n is given by .q n  1/=n. Any nonconstant polynomial f .x/ over a finite field Fq can be factored into a product of an element of Fq , which is in fact the leading coefficient of f .x/, and of finitely many monic irreducible polynomials over Fq . This factorization is unique up to the order of the factors. For various applications it is important to compute this factorization efficiently. No deterministic

If I is the n  n identity matrix over Fq and r is the rank of the matrix B  I , then the number of distinct monic irreducible factors of f .x/ over Fq is given by nr. To determine these irreducible factors, we have to consider the homogeneous system of linear equations h.B  I / D 0 with an unknown vector h 2 Fnq . With any solution h D .h0 ; h1 ; : : : ; hn1 /, we associate a polynomial h.x/ D h0 C h1 x C    C hn1 x n1 2 Fq Œx. The factorization is completed by computing a certain number of greatest common divisors of the form gcd.f .x/; h.x/  c/ for some c 2 Fq . A factorization algorithm that is particularly effective for finite fields of small characteristic is the Niederreiter algorithm. Let f D f .x/ 2 Fq Œx be as above and consider the differential equation f q H .q1/ .h=f / D hq , where H .q1/ is the HasseTeichm¨uller derivative of order q  1 and h D h.x/ is an unknown polynomial over Fq . Since H .q1/ and h 7! hq are Fq -linear operators, the differential equation can be linearized and is equivalent to the homogeneous system of linear equations h.N  I / D 0, where N is an n  n matrix over Fq that can be determined from f . If g1 ; : : : ; gm are the distinct monic irreducible factors of f over Fq , then the solutions of the differential equation form the m-dimensional vector space V .f / over Fq with basis fl1 f; : : : ; lm f g, where li D gi0 =gi is the logarithmic derivative of gi for 1  i  m. Given an arbitrary basis of V .f /, there is a systematic procedure for extracting the irreducible factors g1 ; : : : ; gm . In the case of great practical interest

F

544

Finite Fields

where Fq is of characteristic 2 and f is sparse, the Niederreiter algorithm has the enormous advantage over the Berlekamp algorithm that it leads to a sparse system of linear equations which can be solved much faster than a general system of linear equations. There are also probabilistic factorization algorithms for polynomials over finite fields, the classical algorithm of this type being the Cantor-Zassenhaus algorithm. A detailed discussion of factorization algorithms for polynomials over finite fields can be found in the book of von zur Gathen and Gerhard [1].

Permutation Polynomials A permutation polynomial of Fq is a polynomial f .x/ over Fq for which the induced map c 2 Fq 7! f .c/ is a permutation of Fq . Permutation polynomials are of interest in combinatorics and also in cryptography where bijective maps are used for encryption and decryption. A monomial ax n with a 2 Fq and n  1 is a permutation polynomial of Fq if and only if gcd.n; q  1/ D 1. For a 2 Fq and n  1, the Dickson polynomial bn=2c

Da;n .x/ D

X

j D0

! nj n .a/j x n2j nj j

is a permutation polynomial of Fq if and only if gcd.n; q 2 1/ D 1. According to Hermite’s criterion, a polynomial f .x/ over Fq is a permutation polynomial of Fq if and only if f .x/ has exactly one root in Fq and for each integer t with 1  t  q  2 which is not divisible by the characteristic of Fq the reduction of f .x/t modulo x q  x has degree  q  2. If f .x/ 2 Fq Œx has degree  2 and satisfies the property that every irreducible factor of .f .x/  f .y//=.x  y/ in Fq Œx; y is reducible over some algebraic extension of Fq , then f .x/ is a permutation polynomial of Fq . It follows from Hermite’s criterion that if d  2 is a divisor of q  1, then there is no permutation polynomial of Fq of degree d . If d  2 is an even integer and q is odd and sufficiently large relative to d , then there is no permutation polynomial of Fq of degree d . If f .x/ 2 ZŒx is a permutation polynomial of Fp D Z=pZ for infinitely many prime numbers p when considered modulo p, then f .x/ is a composition of binomials ax n C b and Dickson polynomials.

Applications to Cryptology Finite fields are important in various areas of cryptology such as public-key cryptosystems, symmetric cryptosystems, digital signatures, and secret-sharing schemes. A basic map in this context is discrete exponentiation, where we take a primitive element g of a finite field Fq and assign to each integer r with 0  r  q  2 the element g r 2 Fq . This map can be efficiently computed by the well-known square-andmultiply algorithm. The inverse map is the discrete logarithm to the base g which assigns to each c 2 Fq the uniquely determined integer r with 0  r  q  2 and g r D c. Various cryptographic schemes are based on the complexity assumption that the discrete logarithm is hard to compute for many large finite fields Fq . Historically, the first scheme using discrete exponentiation was Diffie-Hellman key exchange. Here Fq and g are publicly known. If two participants A and B want to establish a common key for secret communication, they first select arbitrary integers r and s, respectively, with 2  r; s  q 2, and then A sends gr to B, whereas B transmits g s to A. Now they take g rs as their common key, which A computes as .g s /r and B as .g r /s . If q is chosen in such a way that the discrete logarithm to the base g 2 Fq is hard to compute, then this scheme can be regarded as secure. Further cryptographic schemes based on the difficulty of computing discrete logarithms include the ElGamal public-key cryptosystem, the ElGamal digital signature scheme, the Schnorr digital signature scheme, the DSS (Digital Signature Standard), and the Schnorr identification scheme. Finite fields are also instrumental in other cryptographic applications such as the AES (Advanced Encryption Standard), the McEliece public-key cryptosystem, the Niederreiter public-key cryptosystem, the Courtois-FiniaszSendrier digital signature scheme, elliptic-curve cryptosystems, and the Shamir threshold scheme. A detailed discussion of applications of finite fields to cryptology can be found in the book of van Tilborg [7].

Applications to Pseudorandom Number Generation A sequence of pseudorandom numbers is generated by a deterministic algorithm and should simulate a sequence of independent and uniformly distributed

Finite Volume Methods

random variables on the interval Œ0; 1. Pseudorandom numbers are employed in various tasks of scientific computing such as simulation methods, computational statistics, and the implementation of probabilistic algorithms. Finite fields are eminently useful for the design of algorithms generating pseudorandom numbers. A general family of such algorithms is that of nonlinear congruential methods. Here we work with a large finite prime field Fp and generate a sequence y0 ; y1 ; : : : of elements of Fp by the nonlinear recurrence relation ynC1 D f .yn / for n D 0; 1; : : :, where f .x/ is a polynomial over Fp of degree at least 2. Corresponding pseudorandom numbers in Œ0; 1 are obtained by setting xn D yn =p for n D 0; 1; : : :. Preferably, the feedback polynomial f .x/ is chosen in such a way that the sequence y0 ; y1 ; : : :, and therefore the sequence x0 ; x1 ; : : :, is purely periodic with least period p. A typical choice is f .x/ D ax p2 Cb, where a; b 2 Fp are such that x 2  bx  a is a primitive polynomial over Fp . Another general family of algorithms for pseudorandom number generation is that of shift-register methods. In practice, these methods are based on kthorder linear recurring sequences over the binary field F2 . For a given k  2, the largest value of the least period of a kth-order linear recurring sequence over F2 is 2k  1. This value is achieved if and only if the minimal polynomial of the linear recurring sequence is a primitive polynomial over F2 of degree k. To derive pseudorandom numbers in Œ0; 1 from linear recurring sequences over F2 , procedures such as the digital multistep method and the GFSR (generalized feedback shift-register) method are employed. Finite fields that are not prime fields are used in digital methods for pseudorandom number generation. Here we consider a finite field F2m with an integer m  2 (typically m D 32 or m D 64). A sequence 0 ; 1 ; : : : of elements of F2m is generated by a nonlinear recurrence relation with a feedback polynomial over F2m . If fˇ1 ; : : : ; ˇm g is an ordered basis of the vector space F2m over F2 , then we have the unique Pm .j / representation n D j D1 cn ˇj for n D 0; 1; : : :

545

integration and global optimization. We refer to the book Niederreiter and Xing [5] for a detailed discussion of constructions of low-discrepancy sequences based on finite fields.

References 1. von zur Gathen, J., Gerhard, J.: Modern Computer Algebra, 2nd edn. Cambridge University Press, Cambridge (2003) 2. Lidl, R., Niederreiter, H.: Introduction to Finite Fields and Their Applications, revised edn. Cambridge University Press, Cambridge (1994) 3. Lidl, R., Niederreiter, H.: Finite Fields. Cambridge University Press, Cambridge (1997) 4. Mullen, G.L., Mummert, C.: Finite Fields and Applications. American Mathematical Society, Providence (2007) 5. Niederreiter, H., Xing, C.P.: Rational Points on Curves Over Finite Fields: Theory and Applications. Cambridge University Press, Cambridge (2001) 6. Shparlinski, I.E.: Finite Fields: Theory and Computation. Kluwer Academic Publishers, Dordrecht (1999) 7. van Tilborg, H.C.A.: Fundamentals of Cryptology. Kluwer Academic Publishers, Boston (2000)

Finite Volume Methods Jan Martin Nordbotten Department of Mathematics, University of Bergen, Bergen, Norway

Mathematics Subject Classification 65M08; 65N08; 74S10; 76M12; 78M12; 80M12

Synonyms Conservative finite differences; Control Volume Method, CV method; Finite Volume Method, FV method (FVM)

.j /

with all cn 2 F2 . Now a sequence of pseudorandom P .j / j for numbers in Œ0; 1 is defined by xn D m j D1 cn 2 n D 0; 1; : : : . Finite fields play a crucial role in the construction of low-discrepancy sequences, which are sequences of points in a multidimensional unit cube that are used for special computational tasks such as numerical

Short Definition The finite volume method (FVM) is a family of numerical methods that discretely represent conservation laws. The FVM uses the exact integral form of the

F

546

Finite Volume Methods

conservation law on a covering partition of the domain Here, p is the fluid pressure, and the identity tensor is (usually forming a grid). Different FVMs are distin- denoted I. guished by their approximation of the surface integrals We will return to these examples later, but recall that appearing on domain boundaries. the advective type systems exemplified by Eqs. (1)–(2) or (1) and (4) are known as hyperbolic conservation laws, while the system exemplified by Eqs. (1) and (3) are known as parabolic conservation laws. In the steady Description state, parabolic conservation laws are referred to as elliptic conservation laws. Conservation Laws From the integral form of the conservation law, Conservation of a quantity u, which in applications usually denotes mass, components of linear momen- one can derive the usual differential form of the contum, or energy, is written for a spatial volume !, servation laws by noting that the integral form holds for any volume ! and assuming that the solution is assumed to be fixed in time, as continuous. This allows us to remove the integrals after Z Z Z d application of the generalized Stokes theorem to the u dV C n  q dS D r dV (1) surface integral, to obtain dt ! @! ! This integral equation states the physical observation that for the conserved quantity, the time rate of change of the quantity (first term) is balanced by the transfer across the boundary of the volume (second term, where q is the flux and n is the outward normal vector to the surface) and the addition or removal of the quantity by any internal sources or sinks (third term, where r is the source per volume). For clarity, vectors and tensors are distinguished from scalars by being typeset in bold. A mathematical introduction to conservation laws can be found in, e.g., [7]. The conservation laws are frequently supplemented by constitutive relationships, particular to the physical application, of which a common example is advection q D f .u/

(3)

where  is a positive scalar or symmetric, positive definite tensor coefficient. For the momentum balance equation, a typical rate law is that for an incompressible, inviscid and irrotational fluid: qD

u  u 2

Cp I

(5)

We stress that this differential form of the conservation law is equivalent to the integral form only when all the terms are well defined. For physical systems involving discontinuous solutions, Eq. (5) must either be considered in the sense of distributions or it is necessary to return to the fundamental conservation principle as expressed by Eq. (1). The limited validity of Eq. (5) is the motivation for constructing discretization methods directly for Eqs. (1)–(4) and leads to the methods known as finite volume methods. Alternative approaches include discretizing Eqs. (2)–(5) using, e.g., finite element or finite difference methods. (2)

where f is a known function – in the linear case, simply f D ug, where g is not a function of u. Another typical example is diffusive first-order rate laws (e.g., the laws of Fick and Darcy for mass and Fourier for energy): q D ru

@u Crq Dr @t

(4)

Finite Volume Grids Finite volume methods represent the integral form of the conservation equation (1) exactly on a finite number of volumes, hence the name. Typically, the volumes are chosen as a nonoverlapping partition of the domain, and we will only consider this situation henceforth. Thus, for a domain ˝, we assume that it is divided into N volumes !i (in three dimensions, areas, or line segments in 2D or 1D, respectively) which are nonoverlapping and whose union is ˝. We will refer to the volumes as cells and the edges between two cells i and j as a face, denoted @!i;j . For convenience, we will let @!i;j be void if i and j do not have a common edge. The structure of cells and faces form the finite volume grid.

Finite Volume Methods

547

Note that by definition, the face flux is of exactly the same magnitude regardless of which cell it is evaluated for, e.g., Fi;j D Fj;i . This observation is important in that it is required for the conservation equation to be handled consistently. Equations (7) and (8) now provide us with the discrete conservation structure which is the backbone of all finite volume methods: Z X dU i C Fi;j D r dV (9) j!i j j dt !i

Finite Volume Methods, Fig. 1 The domain ˝ shown in thick solid black line together with the finite volume grid !i (thinner solid black lines corresponding to faces between cells). Additionally, cell centers are indicated by gray dots, while a dual grid is indicated by gray dashed lines

While this construction is sufficient for many applications, we keep in mind that for some equations, it may be necessary to refer to a cell center (typically chosen as some internal point, possibly the centroid), which is then denoted x i . In the construction of some methods, it is also convenient to define a dual grid, which has the cell centers as vertexes, and is typically constrained such that each face of the finite volume grid has exactly one edge of the dual grid passing through it. These notions are all summarized for a sample two-dimensional grid in Fig. 1. To construct the finite volume method, we now return to Eq. (1). Considering this equation for each cell, such that ! D !i , we obtain

In this equation, the face fluxes Fi;j are inherently unknown, as they depend on the continuous flux q through Eq. (8). This situation is equivalent to that of the conservation laws (1), in that the fundamental conservation property must be supplemented by a constitutive relationship. By analogy to different physical systems being distinguished by different constitutive laws, as evidenced by Eqs. (2)–(4), various finite volume methods are distinguished by their approximation of these constitutive laws. In other words, for a given constitutive law, the finite volume method is defined by its approximate flux Fi;j D Fi;j .U /

(10)

Here, bold-face U indicates the vector of all solution variables Ui , and we recall that the flux relationships are again constrained such that the face flux is unique, Fi;j D Fj;i . By analogy to linear (Eqs. (2) and (3)) and nonlinear (Eqs. (2) and (4)) constitutive laws, we refer to linear and nonlinear finite volume methods by whether the approximate flux relationship (10) is linear Z Z Z d u dV C ni  q dS D r dV (6) or nonlinear. Furthermore, it is often desirable that the dt !i flux relationship is in some sense local (equivalently @!i !i termed compact), in which case the flux for face @!i;j Let Ui denote the average of the solution over a cell, will depend either only on cells i and j or possibly j!i j be the volume of cell, and ni;j be the normal vector also their immediate neighbors. The latter case allows from cell i to j . Then, we can write Eq. (6) as for higher-order methods to be constructed. X dU i C j!i j j dt

Z

Z

(7) Approximation of Surface Fluxes As can be expected from the example constitutive laws (2)–(4), the form of the discrete flux approximations Here, it is implied that the sum is taken over all cells (10) in general must be adapted to the particular conj sharing an edge with cell i . Finally, we introduce the stitutive law. From the perspective of applications, this face flux is an attractive feature of finite volume methods, in that the method can relatively easily be tailored to the parZ ni;j  q dS (8) ticular application. From a mathematical perspective, Fi;j D @!i;j it leads to a wide variety of methods, which are in ni;j  q dS D

@!i;j

r dV

!i

F

548

Finite Volume Methods

general supported by less unified theoretical development than, e.g., the finite element methods. The full breadth of flux approximations is too comprehensive to summarize here; therefore, we restrict our attention to a few particular cases: a generic approach known as Finite Volume Finite Element methods (FVFE, also known as Control-Volume Finite Element (CVFE)), as well as examples of typical specialized methods for the hyperbolic and parabolic cases. Finite Volume Finite Element Methods A generic approach to define the approximate relationship (10) is to combine a polynomial interpolation of the solution U together with the appropriate flux relationship [8]. In this case, let the (possibly nonlinear and differential) constitutive law be given as an operator

q D N .u/

(11)

Returning to Fig. 1, we now focus our attention to the nodal values and the dual grid depicted in gray. On this dual grid, we recognize that we can define piece-wise polynomial basis functions in the finite element sense. For the simplest case where the dual grid consists of simplexes and the polynomials are first order, these basis functions will be the lowest-order Lagrange basis functions (piece-wise linear functions). We denote the basis function which takes the value 1 at x i and zero at all other cell centers as i . Associating the (cell average) solutions Ui with the nodal points x i , we can now define an interpolation of the solution over the full domain according to X

solution is not regular, other approaches are usually preferred. These include hyperbolic conservation laws, where discontinuous solutions are common, and also parabolic conservation laws in cases where the parameters of the constitutive laws are discontinuous. Finite Volume Methods for Hyperbolic Conservation Laws Hyperbolic conservation laws are important in applications, as both the Euler equations (obtained from (4), by including equations for conservation of mass and energy), and also transport problems including oil recovery and traffic, are modeled by these equations. These equations are also the field of some of the earliest and most famous applications of finite volume methods. We consider the model problem given by Eq. (2). Since hyperbolic conservation laws have a strictly local flux expression, it is sufficient for low-order methods to consider only the case where the face flux is approximated as strictly on its neighbor  dependent  cells, e.g., Fi;j D Fi;j Ui ; Uj : For simplicity, we will therefore limit the discussion in this section to one spatial dimension; multidimensional problems are commonly treated dimension by dimension. We use the convention that cells are numbered left to right. In this case, we have j D i C 1, and several important schemes are on this form. These include: • Central difference method:

Fi;j

  f .Ui / C f Uj D 2

This simple method is, unfortunately, unconditionally unstable when used with an explicit time step (see next section) and is therefore essentially not Combining Eqs. (8), (11), and (12), we obtain a flux used in practice. expression for an arbitrary constitutive law as • The Lax-Friedrichs method: ! Z     X x Ui  Uj f .Ui / C f Uj ni;j  N Uk k .x/ dS Fi;j D C Fi;j D @!i;j 2 2t k uO .x/ 

i

Ui

i .x/

(12)

The FVFE method is attractive in its simple definition and can naturally be extended to higher-order basis functions in the approximation uO . The FVFE method also provides a useful link to the FE methods and in particular the Petrov-Galerkin method. However, the accuracy of the flux approximation relies on regularity of the solution u. For applications where the

This method is motivated by the observation that the unstable central difference method becomes conditionally stable with explicit time steps if it is regularized by a penalty term proportional to the change in the solution and the ratio of spatial to temporal time step (denoted x and t, respectively).

Finite Volume Methods

549

• Upstream weighted method:

Fi;j

8 df ˆ < f .Ui / if 0 du D df ˆ : f .Uj / if 0; (2)

where ˇ is a real parameter. This equation is a simple and classic case of the nonlinear reaction–diffusion equation (1). Fisher [1] first proposed the above wellknown equation, encountered in various fields of

Fisher’s Equation

551

science, as a model for the propagation of a mutant gene with u.x; t/ displaying the density of advantage. The equation is generally referred to as Fisher’s equation being of high importance to describe different mechanisms. In the model equation, the growth of the mutant gene population originates from the diffusion and nonlinear terms. Considering only small differences, the same physical event similar to the population is observed on neutrons in a reactor. Therefore, the Fisher’s equation is also used as a model for the evolution of the neutron population in a nuclear reactor. Nowadays, the equation has been used as a basis for a wide variety of models for different problems. As the population density in a habitat is bounded, the initial condition must satisfy the following inequality: 0  u.x; 0/  1;

1 < x < 1;

(3)

where the unity is used for convenience. The boundary conditions are taken as lim u.x; t/ D 0;

x!˙1

t  0:

(4)

When the solution domain is restricted to [a; b], the above physical boundary conditions are returned into the following artificial boundary conditions respectively: u.a; t/ D u.b; t/ D 0; t  0 (5) and u.a; t/ D 1;

u.b; t/ D 0;

t  0:

(6)

Many researchers have studied the mathematical properties of the Fisher’s equation. Kolmogoroff et al. [2] in their pioneering study, also known as KPP equation, paid their attention to the Fisher’s equation. In that paper, they showed that for each initial condition of the form (3), equation (2) has a unique solution bounded for all times as the initial distribution. Both Fisher [1] and Kolmogoroff et al. [2] also showed that a progressive wave solution with minimum speed is admitted by the problem. Various properties of the Fisher’s equation have been analyzed using very wide range of numerical methods [3–10]. In this study, a sixth-order finite difference scheme in space and a fourth-order Runge–Kutta (RK4) scheme in time were implemented for computing solutions of the Fisher equation. The combination

of the present scheme with the RK4 provides an efficient and highly accurate solution for such realistic problems.

The FD6 Schemes Spatial derivatives are evaluated by a sixth-order finite difference (FD6) scheme. The spatial derivative u0i at point i can be approximated by, .R C L C 1/-point stencil, .R C L/-order finite-difference scheme as R 1 X 0 ui D aj CL ui Cj ; h j DL

F 1  i  N;

(7)

where h D xi C1  xi is the spacing of uniform mesh. The above formula involves .R C L C 1/ constants, a0 ; a1 ; a2 : : : ; aRCL , which need to be known at point i . R and L indicates number of points in the right-hand side and the left-hand side for the taken stencil, respectively. R is equal to L for the considered stencil at internal points, but this is not the case for the boundary nodes. N is the number of grid points. The coefficients aj were determined with Taylor series expansion of (7). Thus, the scheme using seven points, hereafter referred to as FD6, is of order 6. The coefficients aj for the first derivatives in the FD6 scheme can be given at internal and boundary nodes in Ref. [11]. First-order spatial derivative terms can be rewritten into matrix form as (8) U 0 D AU with the system matrix A. The second-order spatial derivatives are obtained by applying the first-order operator twice, i.e., U 00 D AU 0 ;

(9)

where U D .u1 ; u2 ; : : : ; uN /T . For the approximate solutions of (2) with the boundary conditions (3) and (4) using the FD6 method, first the interval Œa; b is discretised such that a D x1 < x2 < ::: < xN D b. After application of the FD6 technique to (2), the equation can be reduced into a set of ordinary differential equations in time. Then, the governing equation becomes d ui D P ui ; dt

(10)

552

Fisher’s Equation

where P indicates a spatial nonlinear operator. Each spatial derivative on hand side of (10) was computed using method and then (10) was solved RK4 scheme.

differential the rightthe present using the

decreases rapidly (see Fig. 1). After the peak reaches its minimum value, the reaction starts to dominate the diffusion slowly. Then, the peak value goes up as seen in Fig. 2. Example 2 Consider the initial profile 8 10.xC1/ ; x < 1 1

Numerical Illustrations To show and introduce the physical behavior of the Fisher’s equation, the FD6-RK4 is utilized with the initial and boundary conditions. All computations were carried out using some MATLAB codes, and the parameter values are ˛ D 0:1, ˇ D 1, h D 0:025, and t D 0:0005.

for the current test problem. Similar behaviors and relations between the diffusion and reaction have been observed as is the case in previous example. In this case, however, effects of diffusion and reaction are seen to be very small. Effect of diffusion is dominant near the corners. As seen in Figs. 3 and 4, the effects of Example 1 The initial pulse profile diffusion near the critical points change the physical behavior from sharpness to smoothness, and in the u.x; 0/ D sech2 .10x/ longer term, it can be expected to get smoother and is taken as the initial condition for our first numerical smoother. experiment. In Fig. 1, the short-time behaviors of the solution are illustrated. At the beginning of the process, since the diffusion term uxx is negative and Conclusion it has a large absolute value and the reaction term u.1  u/ is very small, the effect of diffusion dominates The Fisher’s equation has been successfully introover the effect of reaction. Therefore, the peak value duced and solved by implementing a high-order finite 1

1 t=0 t=0.1 t=0.2 t=0.3 t=0.4 t=0.5

0.9 0.8

0.8 0.7

0.6

0.6

0.5

0.5

u

u

0.7

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1

0 −2

−1.5

−1

−0.5

0 x

0.5

1

Fisher’s Equation, Fig. 1 Solutions at early times

1.5

t=0 t=1 t=2 t=3 t=4 t=5 t=10

0.9

2

0 −6

−4

−2

0 x

2

4

6

Fisher’s Equation, Fig. 2 Behavior of solutions at later times

Fitzhugh–Nagumo Equation

553

out here can also be done in two and three space dimensions.

1 0.9

t=0 t=0.1 t=0.2 t=0.3 t=0.4 t=0.5

0.8 0.7

References

u

0.6 0.5 0.4 0.3 0.2 0.1 0 −3

−2

−1

0 x

1

2

3

Fisher’s Equation, Fig. 3 Solutions at early times 1 t=0 t=1 t=2 t=3 t=4 t=5

0.9 0.8 0.7

u

0.6 0.5 0.4 0.3 0.2 0.1 0 −6

−4

−2

0 x

2

4

6

1. Fisher, R.A.: The wave of advance of advantageous genes. Ann. Eugen 7, 355–369 (1936) 2. Kolmogoroff, A., Petrovsky, I., Piscounoff, N.: Study of the diffusion equation with growth of the quantity of matter and its application to biology problems. Bull. de l.universit´e d.´etat a` Moscou Ser. Int. Sect. A 1, 1–25 (1937) 3. Carey, G.F., Shen, Y.: Least-Squares finite element approximation of Fisher’s reaction–diffusion equation. Numer. Method Partial Differ. Equ. 11, 175–186 (1995) 4. Gazdag, J., Canosa, J.: Numerical solution of Fisher’s equation. J. Appl. Probab. 11, 445–457 (1974) 5. Al-Khaled, K.: Numerical study of Fisher’s reaction– diffusion equation by the Sinc collocation method. J. Comput. Appl. Math. 137, 245–255 (2001) 6. Mitchell, A.R., Manoranjan, V.S.: Finite element studies of reaction–diffusion. In: Whitehad, J.R. (ed.) MAFELAP. The Mathematics of Finite Elements and Applications, vol. IV, p. 17. Academic, London (1981) 7. Olmos, D., Shizgal, B.D.: A pseudospectral method of solution of Fisher’s equation. J. Comput. Appl. Math. 193, 219–242 (2006) 8. Qiu, Y., Sloan, D.M.: Numerical solution of Fisher’s equation using a moving mesh method. J. Comput. Phys. 146, 726–746 (1998) 9. Zhao, S., Wei, G.W.: Comparison of the discrete singular convolution and three other numerical schemes for solving Fisher’s equation. SIAM J. Sci. Comput. 25, 127–147 (2003) 10. Dag, I., Sahin, A., Korkmaz, A.: Numerical investigation of the solution of Fisher’s equation via the B-spline Galerkin method. Numer. Method Partial Differ. Equ. 26, 1483–1503 (2010) 11. Sari, M., Gurarslan, G., Zeytinoglu, A.: High-order finite difference schemes for the solution of the generalized Burgers–Fisher equation. Int. J. Numer. Method Biomed. Eng. 27, 1296–1308 (2011)

Fisher’s Equation, Fig. 4 Behavior of solutions at later times

Fitzhugh–Nagumo Equation difference method. The first test problem is related to pulse disturbance. The effects of diffusion are observed much clearer in this problem at the beginning of the process. The second problem has an initial step profile having two corners. Carey and Shen [3] observed oscillations near these sharp points. We have not met such oscillations and obtained stable solutions. The presented results are seen to be in agreement with the literature. The introduction and discussion carried

Serdar G¨oktepe Department of Civil Engineering, Middle East Technical University, Ankara, Turkey

Mathematics Subject Classification 35K57; 92C30

F

554

Fitzhugh–Nagumo Equation

Synonyms Fitzhugh–Nagumo Equation (FHN); Hodgkin–Huxley Model (HH Model)

Short Definition The Fitzhugh–Nagumo equation (FHN) is a set of nonlinear differential equations that efficiently describes the excitation of cells through two variables.

Description Motivation Hodgkin and Huxley laid the groundwork for the constitutive modeling of electrophysiology of excitable cells with their pioneering quantitative model of electrophysiology developed for the squid giant axon six decades ago. In this celebrated work [6], the local evolution of the transmembrane potential ˆ, difference between the intracellular potential and the extracellular potential, is described by the differential equation P C Iion D Iapp where Cm is the membrane Cm ˆ capacitance; Iion WD INa C IK C IL denotes the sum of the sodium (Na), potassium (K), and leakage currents (L); and Iapp is the externally applied current. The current due to the flow of an individual ion is modeled by the ohmic law I˛ D g˛ .ˆ  ˆ˛ / where g˛ D gO ˛ .tI ˆ/ denotes the voltage- and timedependent conductance of the membrane to each ion and ˆ˛ are the corresponding Nernst potentials for

˛ D Na, K, L. In the Hodgkin–Huxley (HH) model, based on the voltage-clamp experiments, the potassium conductance is assumed to be described by gK D gN K n4 where n is the potassium activation and gN K is the maximum potassium conductance. The sodium conductance, however, is considered to be given by gNa D gN Na m3 h with gN Na being the maximum sodium conductance, m the sodium activation, and h denotes the sodium inactivation. The evolution of the gating variables m; n, and h is then modeled by first-order kinetics equations with voltage-dependent coefficients. The diagram in Fig. 1 (left) depicts the action potential ˆ calculated with the original HH model. The time evolution of the three gating variables m; n, and h shown in Fig. 1 (right) illustrates dynamics of the distinct activation and inactivation mechanisms. The original HH model was significantly simplified by Fitzhugh [4] who categorized the original four transient parameters fˆ; m; n; hg as the fast variables fˆ; mg and the slow variables fn; hg. Since the sodium activation m evolves as fast as ˆ (see Fig. 1), it is approximated by its voltage-dependent steady-state value m O 1 .ˆ/. Moreover, Fitzhugh observed that the sum of the slow variables fn; hg remains constant (Fig. 1 (right)), during the course of an action potential [7]. These two observations reduced the number of parameters from four to two, which are the fast action potential ˆ and the slow gating variable n. The phasespace analysis of the reduced two-variable system has indicated that the nullcline of ˆ is cubic, while the n-nullcline nO 1 .ˆ/ is monotonically increasing. These observations led Fitzhugh to the generalization of these models toward phenomenological twovariable formulations.

120

Gating Variables

100

Φ [mV]

80 60 40 20 0 −20 0

2

4

6

8 10 12 14 16 18 20 t [ms]

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

m n

h

0

2

4

6

8 10 12 14 16 18 20 t [ms]

Fitzhugh–Nagumo Equation, Fig. 1 Action potential ˆ generated with the original HH model (left). The gating variable transients m; n; and h during the course of the action potential (right) [5]

Fitzhugh–Nagumo Equation

555

1 Formulation and Analysis P D c Œ r G ./CI  ; rP D  Œ  C b r  a  : (4) The evolution of the above-introduced two variables c fˆ; ng can be motivated by the following second-order These equations are now being referred to as the nonlinear equation of an oscillating variable : Fitzhugh–Nagumo (FHN) equation. On the experimental side, Nagumo et al. [8] contributed essentially to R C c g./ P C  D 0 (1) the understanding of (4) by building the corresponding circuit to model the cell. The graphical representation of the FHN equation in the phase space is depicted in with the quadratic damping factor g./ WD  2  1 as Fig. 2 (left) where the trajectories (solid lines) illustrate suggested by Van der Pol [9]. Through the Li´enard’s the solutions of (4) for different initial points .0 ; r0 / transformation (filled circles) and the parameters a D 0:7; b D 0:8; c D 3, and I D 0. While the N-shaped cubic Z  1 Q dQ polynomial (red dashed line) shows the -nullcline, g./ r WD Œ P C c G./  with G./ WD c 0 on which P D 0, the line with negative slope (blue 1 dashed line) denotes the r-nullcline where rP D 0 (2) D 3   ; in Fig. 2. The intersection point of the nullclines is 3 N rN / where both P D 0 and called the critical point .; r P D 0 characterizing the resting state. The stability of the second-order equation (1) can be transformed into the resting state, which is located at (N  1:2; rN  a system of two first-order equations: 0:625) in Fig. 2 (left), can be analyzed by linearizing the nonlinear FHN equation (4) about the critical point; 1 (3) that is, P D c Œ r  G./  and rP D   : c "

While the fast variable , the potential, has a cubic nonlinearity allowing for regenerative self-excitation through a fast positive feedback, the slow variable r, the recovery variable, has a linear dynamics providing slow negative feedback. By introducing a stimulus I and two additional terms a and b r, Fitzhugh has recast the van der Pol equations (3) into what he referred to as the Bonhoeffer–van der Pol model:

P

#

rP

"

D AN

 r

"

#

where AN WD

#

c .1   2 /

c

1=c

b=c

(5) Nr ;N

and  WD   N and r WD r  rN . The eigenvalues of the coefficient matrix AN determine whether the critical point is stable or unstable. The characteristic equation of the coefficient matrix can be expressed as 2  I1  C N and I2 WD det.A/ N are the I2 D 0 where I1 WD tr.A/ 1.5

1 1

r

r

0.5

0

0

–0.5

–1 –2

0.5

–1.5

–1

–0.5

0

0.5

φ

1

1.5

2

–0.5 –2

–1.5

–1

–0.5

0

0.5

1

1.5

2

φ

Fitzhugh–Nagumo Equation, Fig. 2 The phase space of the FHN equation (4) where the trajectories (solid lines) illustrate the solutions for distinct initial points .0 ; r0 / and the two different values of the stimulus I D 0 (left) and I D 0:4 (right)

F

556

principal invariants of AN. For the given parameters and the critical point, we have I1 D c .1  N 2 /  b=c < 0, I2 D b .N 2  1/ C 1 > 0, and  WD I12  4I2 < 0, which indicate that the critical point is stable [3]. Apparently, the stability of the resting state of the FHN equation depends primarily on the stimulus I that shifts the quadratic  -nullcline vertically, thereby changing the position of the critical point and its characteristics. Setting I D 0:4 in (4), the phase-space representation of the FHN equation becomes Fig. 2 (right). Clearly, upon addition of the negative stimulus, the  -nullcline shifts upward and the resting point becomes unstable. This results in the stable limit cycle, closed trajectory, which characterizes the limiting response of adjacent trajectories as time approaches to infinity as depicted in Fig. 2 (right). A negative criterion, i.e., nonexistence, of the limit cycles can be expressed through the Bendixson criterion that makes use of the Gauss’ integral theorem on simply connected regions [3]. This oscillatory behavior that can be generated by the FHN equation allows us to model pacemaker cells that create rhythmical electrical impulses as the sinoatrial node in the right atrium. Apparently, the FHN equation can be used to model a broad class of dynamic phenomena where the underlying complex mechanisms can be uncovered through fundamental methods of dynamics. In electrophysiology, the FHN equation inspired many researchers [1,2] to model computationally inexpensive and physiologically relevant models based on this equation.

Fokker-Planck Equation: Computation 7. Keener, J.P., Sneyd, J.: Mathematical Physiology. Springer, New York (2009) 8. Nagumo, J., Arimoto, S., Yoshizawa, S.: An active pulse transmission line simulating nerve axon. Proc. IRE 50(10), 2061–2070 (1962) 9. Van der Pol, B.: On relaxation oscillations. Philos. Mag. 2(11), 978–992 (1926)

Fokker-Planck Equation: Computation Per L¨otstedt Department of Information Technology, Uppsala University, Uppsala, Sweden

Mathematics Subject Classification 35Q84; 65M08; 65M20; 65M75

Synonyms Forward Kolmogorov equation

Definition The Fokker-Planck equation has the following form [2, 5]: @p @ @2 C vi p  aij p D 0; @t @xi @xi @xj

References 1. Aliev, R.R., Panfilov, A.V.: A simple two-variable model of cardiac excitation. Chaos Solitons Fractals 7(3), 293–301 (1996) 2. Clayton, R.H., Panfilov, A.V.: A guide to modelling cardiac electrical activity in anatomically detailed ventricles. Prog. Biophys. Mol. Biol. 96(1–3), 19–43 (2008) 3. Edelstein-Keshet, L.: Mathematical Models in Biology, 1st edn. Society for Industrial and Applied Mathematics, Philadelphia (2005) 4. Fitzhugh, R.: Impulses and physiological states in theoretical models of nerve induction. Biophys. J. 1, 455–466 (1961) 5. G¨oktepe, S., Kuhl, E.: Computational modeling of cardiac electrophysiology: a novel finite element approach. Int. J. Numer. Methods Eng. 79, 156–178 (2009) 6. Hodgkin, A., Huxley, A.: A quantitative description of membrane current and its application to excitation and conduction in nerve. J. Physiol. 117, 500–544 (1952)

(1)

where the drift vi and diffusion coefficients aij are functions of xi and t. We have adopted the Einstein summation convention with implicit summation over equal indices in terms. The scalar, time dependent solution p is often a probability density function for the system to be in a certain state x in the state space ˝. If the dimension of the state space is N , then the state is a vector with N real elements, x 2 RN , and i; j D 1 : : : N; in (1). Usually, the diffusion coefficient is symmetric with aij D aj i .

Overview In conservation form, the equation for p.x; t/ is

Fokker-Planck Equation: Computation

557

where pNk is the average of p in !k and the size of !k is wk (the area in two dimensions (2D) and the volume in three dimensions (3D)). If sk consists of m straight where F is the probability current or flux vector with edges in 2D or flat surfaces in 3D or other faces with a the elements Fi depending on p. With p interpreted constant nk in higher dimensions, then the equation for as a probability density, it is natural that with certain the time evolution of pNk is boundary conditions on ˝, the probability is conm served. Comparing with (1) we find that 1 X @ pNk C nk`  Fk` sk` D 0: (7) @t wk   `D1 @p @ @ Fi D vi p  aij p D vi  aij p  aij : @xj @xj @xj The size of the `:th face of !k is sk` , the normal nk` is (3) constant there, and Fk` is the average of F on the face. There is a related stochastic differential equation This average must be approximated using the averages according to Itˆo for the random variable vector X with of p in the surrounding cells. the elements Xi ; i D 1 : : : N , see [2]. The equation is A Cartesian mesh in 2D has the cells !ij ; i; j D 1; : : : ; M; with the constant step size sk` D h and the (4) averages pij . Then F D .Fx ; Fy /T and n D .nx ; ny /T , dXi D hi .X; t/ dt C gij .X; t/ d Wi ; and the equation for pij is where Wi is a Wiener process. With 4 @ 1 X 1 pij C 2 .nx Fx C ny Fy /` h vi D hi ; aij D gi k gj k ; (5) @t h `D1 2 (8) @ 1 Fx;i C1=2;j C Fy;i;j C1=2 D pij C the relation to (1) is that the probability density func@t h   Fx;i 1=2;j  Fy;i;j 1=2 D 0; tion for X to be x at time t is p.x; t/ solving (1). Given the Fokker-Planck equation, the corresponding where, e.g., Fx;i C1=2;j is Fx evaluated at the face stochastic equation is not unique since many gij may between !ij and !i C1;j . An approximation of the flux satisfy (5). function F in (3) is needed at the four edges of the cell !ij using pij . A simple and stable approximation is an upwind scheme for the drift term and a centered Numerical Solution scheme for the diffusion term as in [1]. Then on the The Fokker-Planck equation can be discretized in space face .i C 1=2; j /, the drift term is approximated by @p @Fi @p C C r  F D 0; D @t @xi @t

(2)

on a mesh with a finite difference, finite volume, or finite element method. The advantage with the finite volume method presented here is that if p in ˝ is preserved, then the computed solution also has that property. Let ˝ be partitioned into K computational cells !k ; k D 1 : : : K; with the boundary sk and normal nk pointing outward from ! and integrate the conservation form (2) over !k . Then according to Gauss’ integration formula Z Z Z @p @Fi @ d! C d! D p d! @t !k !k @t !k @xi Z Z @ C r  F d! D wk pNk C nk  F dS D 0; @t !k sk (6)

 .vx p/i C1=2;j 

vx;i C1=2;j  0 vx;i C1=2;j pij ; vx;i C1=2;j pi C1;j ; vx;i C1=2;j < 0

or

8 vx;i C1=2;j .3pij  pi 1;j /=2; ˆ ˆ < vx;i C1=2;j  0 .vx p/i C1=2;j  v .3pi C1;j  pi C2;j /=2; ˆ ˆ : x;i C1=2;j vx;i C1=2;j < 0: (9) The first approximation in (9) is first-order accurate in h, and the second one is second-order accurate. The diffusion term is a sum of two derivatives in the x and y directions in the flux (3). At the face .i C 1=2; j /, the two terms in Fx to be approximated are @ @ axx p C axy p: @x @y

(10)

F

558

Fokker-Planck Equation: Computation

D The solution can be computed as the eigenvector of A1 D limt !1 A.t/ with eigenvalue 0. A method to compute a few eigenvectors and their eigenvalues is the @ Arnoldi method in the software package ARPACK [4]. axx p  .qxx;i C1;j  qxx;ij /= h; Assume that the conditions at the boundary @˝ of @x @ (11) ˝ are such that there is no probability current across axy p  .qxy;i C1;j C1 C qxy;i;j C1 @y  the boundary. Then n  F D 0 at @˝ with the outward   qxy;i;j 1 C qxy;i C1;j 1 /=4h: normal n. The total probability in ˝ is constant since by (6) we have The other derivatives in the diffusive flux are approxiZ Z Z mated in a similar manner. The resulting stencil for the @ @ @F p d˝ C n  F dS D p d˝ D 0; y x C @y in 2D is then diffusive part of @F @t ˝ @t ˝ @x @˝ (16)   qxx;i C1;j C qxx;i 1;j  2qxx;ij C qyy;i;j C1 and R if the initial probability at t D 0 is scaled such that  1 ˝ p.x; 0/ d˝ D 1, then this equality holds true for all qxy;i C1;j C1 C qxy;i 1;j 1 times t > 0. The finite volume discretization inherits Cqyy;i;j 1  2qyy;ij C 2   this property.  qxy;i C1;j 1 C qxy;i 1;j C1 = h2 : (12) The sum of the fluxes in (7) over all cells vanishes because the same term appears in the flux of the cell to Let p be the solution vector at time t with pNk ; k D the left of a face and in the flux of the cell to the right 1 : : : K; as components. Then the discretization of (2) of the face but with opposite signs. Also, the fluxes at can be written with a K  K matrix A faces on the boundary vanishes. If w is the constant vector with the sizes of the cells wk , then it follows @p C Ap D 0: (13) from (13) that @t A second-order approximation is with qxx;ij axx;ij pij ; qxy;ij D axy;ij pij

K X m X There are alternatives how to discretize the time derivanC1 n nk`  Fk` sk` D wT Ap D 0 tive at the time points t D t C t with the kD1 `D1 constant time step t. For stability, an implicit method n n n n is preferred. With p.t /  p and A.t / D A , three and (7) can be written such possibilities are K K X m X X @ @ nC1 n nC1 nC1 p N w C nk`  Fk` sk` D wT p D 0: k k D p  tA p ; p @t @t kD1 `D1 (14) kD1 pnC1 D pn  12 t.AnC1 pnC1 C An pn /; (17) 4 n 1 n1 2 nC1 nC1 nC1 D 3p  3p  3 tA p : p T The total probability w p is preserved by the finite The first scheme is the Euler backward method and is volume discretization as it is in the analytical solufirst-order accurate in t. The second method is the tion (16). The size vector w is a left eigenvector of A trapezoidal method, and the third method is a backward with eigenvalue 0. The corresponding right eigenvector differentiation formula. These two schemes are of when t ! 1 is p1 in (15). second order. All methods are unconditionally stable If the dimension of the state space N is high, then when the real part of the eigenvalues of a constant A the Fokker-Planck equation cannot be solved numeris nonpositive. There is a system of linear equations ically by the finite volume method due to the curse to solve for pnC1 in every time step. Since the system of dimensionality. The number of unknowns K grows matrix is sparse, an iterative method such as GMRES exponentially with N and quickly becomes too large to or BiCGSTAB is the preferred choice of method. be manageable on a computer. Suppose that the mesh 1 The solution p of the steady-state problem when is Cartesian with M mesh points in each dimension. t ! 1 satisfies Then K D M N and with M D 100 and N D 10, K will be 1020 , and almost a zettabyte of memory is 1 1 A p D 0: (15) needed only to store the solution. Then a Monte Carlo

Framework and Mathematical Strategies for Filtering or Data Assimilation

approximation of p or its moments is possible using the stochastic differential equation (4). Generate R realizations of the process by solving (4) numerically, e.g., by the Euler-Maruyama method, see [3]. Then for each trajectory, save the position x at t in a mesh, or update the moments to obtain an approximation of p.x; t/ or its moments. The convergence is slow and p proportional to 1= R as it is for all Monte Carlo methods, but for large N it is the only feasible alternative. The conclusions in [6] for a similar problem is that the Monte Carlo method is more efficient computationally if N & 4.

559

Definition Filtering is a numerical scheme for finding the “best” statistical estimate of hidden true signals through noisy observations. For very high-dimensional problems, the best estimate is typically defined based on the linear theory, in the sense of minimum variance [21].

Overview

In the past two decades, data assimilation has been an active research area in the atmospheric and ocean sciences community with weather forecasting and cliCross-References matological state reconstruction as two direct applications. In this field, the current practical models for the  Finite Volume Methods prediction of both weather and climate involve general  Quasi-Monte Carlo Methods circulation models where the physical equations for  Simulation of Stochastic Differential Equations these extremely complex flows are discretized in space and time and the effects of unresolved processes are parameterized according to various recipes; the result References of this process involves a model for the prediction 1. Ferm, L., L¨otstedt, P., Sj¨oberg, P.: Conservative solution of of weather and climate from partial observations of the Fokker-Planck equation for stochastic chemical reactions. an extremely unstable, chaotic dynamical system with BIT 46, S61–S83 (2006) 2. Gardiner, H.: Handbook of Stochastic Methods, 3rd edn. several billion degrees of freedom. These problems typically have many spatiotemporal scales, rough turSpringer, Berlin (2004) 3. Kloeden, P.E., Platen, E.: Numerical Solution of Stochastic bulent energy spectra in the solutions near the mesh Differential Equations. Springer, Berlin (1992) scale, and a very large dimensional state space, yet real4. Lehoucq, R.B., Sorensen, D.C., Yang, C.: ARPACK Users’ Guide: Solution of Large-Scale Eigenvalue Problems with time predictions are needed. Two popular practical data assimilation apImplicitly Restarted Arnoldi Methods. SIAM, Philadelphia (1998) proaches that were advocated for filtering such high5. Risken, H.: The Fokker-Planck Equation. Springer, Berlin dimensional, nonlinear problems are the ensemble (1996) 6. Sj¨oberg, P., L¨otstedt, P., Elf, J.: Fokker-Planck approximation Kalman filters [15] and the variational methods of the master equation in molecular biology. Comput. Vis. [14]. Recently, most operational weather prediction Sci. 12, 37–50 (2009) centers, including the European Center for MediumRange Weather Forecasts (ECMWF), the UK Met Office, and the National Centers for Environmental Framework and Mathematical Strategies Prediction (NCEP), are adopting hybrid approaches, taking advantage from both the ensemble and for Filtering or Data Assimilation variational methods [12, 20, 34]. Despite some successes in the weather forecasting application for John Harlim assimilating abundant data, collected from radiosonde, Department of Mathematics and Department of scatterometer, satellite, and radar measurements, Meteorology, Pennsylvania State University, State these practical methods are very sensitive to model College, PA, USA resolution, ensemble size, observation frequency, and the nature of the turbulent signals [26] and are Synonyms suboptimal in the sense of nonlinear filtering since they were based on linear theory. Furthermore, there is an Data assimilation; State estimation inherent difficulty in accounting for model error in the

F

560

Framework and Mathematical Strategies for Filtering or Data Assimilation

state estimation of such complex multiscale processes. This is a prototypical situation in many applications due to our computational limitation to resolve the smaller-scale processes. To complicate this issue even more, in general, some of the available observations couple information from many spatial scales. For example, observations of pressure or temperature in the atmosphere mix slow vortical and fast gravity wave processes. Given all these practical constraints, finding the optimal nonlinear filtered solutions is an extremely difficult task. Indeed, for continuous-time observations, the optimal filter solution is characterized by a timedependent conditional density function that satisfies an infinite dimensional stochastically forced partial differential equation, known as the Kushner equation [24]. A theoretically well-established approach to approximate this conditional density is the Monte-Carlobased technique called particle filter [4]. However, it is inherently difficult to utilize this approach to sample high-dimensional variables [6, 9]. Therefore, it becomes important to develop practically useful mathematical guidelines to mitigate these issues in filtering high-dimensional nonlinear problems. This is the main emphasis of the book Filtering Complex Turbulent Systems [26].

Practical Filtering Methods

In general, the solutions of the filtering problem in (1)–(2) are characterized by conditional distributions, p.um jvm /, which are obtained by applying Bayes’ theorem sequentially, p.um jvm / / p.um /p.vm jum /:

(3)

Here, p.um / denotes a prior (or background) distribution of state u at time tm . If we assume that the prior error estimate is Gaussian, unbiased, and uncorrelated with the observation error, then we can write   1 p.um / / exp  .um  uN bm /> .Pmb /1 .um  uN bm / 2   1 b  exp  J .um / ; (4) 2 where Pmb D EŒ.um  uN bm /.um  uN bm />  denotes the prior error covariance matrix at time tm , which characterizes the error of the mean estimates, uN bm . In (3), the conditional density, p.vm jum /, denotes the observation likelihood function associated with the observation model in (2), that is,  1  p.vm jum // exp  .vm  g.um //> R1 .vm  g.um // 2   1 o (5)  exp  J .um / : 2

The posterior (or analysis) mean and covariance esHere, we discuss two popular practical methods for timates, uN am and Pma , are obtained by maximizing the filtering high-dimensional problems: the ensemble posterior density in (3), which is equivalent to solving Kalman filter [15] and the variational approaches [14]. the following optimization problem, These numerical methods were developed to solve the following discrete-time canonical filtering problem, (6) min J b .um / C J o .um /; um

umC1 D f .um /;

(1)

for um . These posterior statistics are fed into the model (2) in (1) to estimate the prior statistical estimates at the b next time step tmC1 , uN bmC1 , and PmC1 , when observawhere um denotes the hidden state variable of interest tions become available. at discrete time tm , which is assumed to evolve as If the dynamical and the observation operators f in (1); here, f denotes a general nonlinear dynamical and g are linear and the initial statistical estimates operator that can be either deterministic or stochastic. fuN a0 ; P0a g are Gaussian, then the unbiased posterior The observation, vm , is modeled as in (2) with an mean and covariance estimates are given by the observation operator g that maps the true solution Kalman filter solutions [21]. For general nonlinear um to the observation space and assumed to be cor- problems, the minimization problem in (6) is nontrivial rupted by i.i.d. Gaussian noises, with mean zero and when the state vector um is high dimensional; the major difficulty is in obtaining accurate prior statistical variance R. vm D g.um / C "m ;

"m 2 N .0; R/;

Framework and Mathematical Strategies for Filtering or Data Assimilation

estimates uN bm and Pmb . The ensemble Kalman filter (EnKF) empirically approximates these prior statistical solutions with an ensemble of solutions and uses the Kalman filter formula to obtain the posterior statistics, assuming that these ensemble-based prior statistics are Gaussian [15]. Inplementation-wise, there are many different ways to generate the posterior ensembles for EnKF [1, 10, 19]. Alternatively, the variational approach solves the minimization problem in (6), often by assuming that the matrix Pmb D B in (4) to be time independent [14]. The variational approach solves the following optimization problem, min J .um0 / C b

um0

T X

J o .umj /;

(7)

j D0

for the initial condition um0 , accounting for observations at times ftmj ; j D 0; : : : ; T g and constraining umj to satisfy the model in (1). This method (also known as the strongly constrained 4D-VAR) is typically solved with an incremental approach that relies on linear tangent and adjoint models, and it is sensitive to the choice of B [33]. To alleviate this issue, many operational centers such as the ECMWF, UK Met Office, and NCEP are adopting hybrid methods [12,20,34] that use an ensemble of solutions to estimate Pmb in each minimization step. Notice that both EnKF and 4D-VAR assume Gaussian prior model in (4) and likelihood observation function in (5) to arrive to the optimization problems in (6), (7) before approximating the solutions of these minimization problems. Therefore, it is intuitively clear that these Gaussian-based methods, which can only be optimal for linear problems, are suboptimal filtering methods for nonlinear problems. Indeed, a recent comparison study suggested that one should not take the covariance estimates from these two methods seriously [25]; at their best performance, only their mean estimates are accurate.

Model Error In the presence of model error, the true operators f and g are unknown. In multiscale dynamical systems, errors in modeling f and g are typically due to the practical limitation in resolving the smaller-scale processes and the difficulty in modeling the interaction between

561

the multiscale processes. For example, the predictability of the atmospheric dynamics in the Tropics remains the poorest, and the difficulties are primarily caused by the limited representation of tropical convection and its multiscale organization in the contemporary convection parameterization [31]. Many practically used methods to mitigate model error are also Gaussian-based methods. Most of these methods were designed to estimate only one of the model error statistics, either the mean or covariance, imposing various assumptions on the other statistics that are not estimated. For example, classical approaches proposed in [13] estimate the mean model error (which is also known as the forecast bias), assuming that the model error covariance is proportional to the prior error covariance from the imperfect model. An alternative popular approach is to inflate the prior error covariance statistics, either with empirically chosen [3, 17] or with adaptive [2, 5, 8, 18, 30] inflation factors. All of these covariance inflation methods assume unbiased forecast error (meaning that there is no mean model error). In the 4DVAR implementation, model error is accounted under various assumptions: Gaussian, unbiased, and with covariance proportional to the prior error covariance statistics [33]. Such formulation, known as the weak 4D-VAR method, is numerically expensive [16].

Contemporary Approaches to Account for Model Error Recently, reduced stochastic filtering approaches to mitigate model error in multiscale complex turbulent systems were advocated in [26, 28]. The main conclusion from the studies reported in [26] is that one can obtain accurate filtered mean estimates with judicious choices of reduced stochastic models. Several ideas for simple stochastic parameterization to account for model error induced by ignoring the smaller-scale processes were described in idealistic settings as well as in more complicated geophysical turbulent systems (see [26] and the references therein). In fact, recent rigorous mathematical analysis in [7] showed the existence (and even the uniqueness in a linear setting) of a reduced stochastic model that simultaneously produces optimal filter solutions and equilibrium (climate) statistical solutions. Here, the optimal filtering is in the sense that the mean and covariances are as accurate

F

562

Framework and Mathematical Strategies for Filtering or Data Assimilation

as the solutions from filtering with the perfect model. Another important implication based on the study in [26] is the “stochastic superresolution” [11, 22], which is a practical method that judiciously utilizes the aliasing principle (that is typically avoided in designing numerical solvers for differential equation) to extract information from sparse observations of compressed multiscale processes. These theoretical results and conceptual studies have provided many evidences that clever stochastic parameterization method is an appealing strategy for accurate practical filtering of multiscale dynamical systems in the presence of model error. Based on the author’s knowledge and viewpoint, several cutting-edge stochastic parameterization methods that can potentially have high impact in filtering high-dimensional turbulent systems include: (i) the stochastic superparameterization [29]; (ii) the reducedorder modified quasilinear Gaussian algorithm [32]; (iii) the physics-constrained multilevel nonlinear regression model [18, 27]; (iv) a simple stochastic parameterization model that includes a linear damping and a combined, additive and multiplicative, stochastic forcing [7]; (v) and the Markov chain-type modeling; see, e.g., the stochastic multi-cloud model for convective parameterization [23].

References 1. Anderson, J.: An ensemble adjustment Kalman filter for data assimilation. Mon. Weather Rev. 129, 2884–2903 (2001) 2. Anderson, J.: An adaptive covariance inflation error correction algorithm for ensemble filters. Tellus A 59, 210–224 (2007) 3. Anderson, J., Anderson, S.: A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Mon. Weather Rev. 127, 2741–2758 (1999) 4. Bain, A., Crisan, D.: Fundamentals of Stochastic Filtering. Springer, New York (2009) 5. Belanger, P.: Estimation of noise covariance matrices for a linear time-varying stochastic process. Automatica 10(3), 267–275 (1974) 6. Bengtsson, T., Bickel, P., Li, B.: Curse of dimensionality revisited: collapse of the particle filter in very large scale systems. In: Nolan, D., Speed, T. (eds.) Probability and Statistics: Essays in Honor of David A. Freedman. IMS Lecture Notes – Monograph Series, vol. 2, pp. 316–334. Institute of Mathematical Sciences, Beachwood (2008) 7. Berry, T., Harlim, J.: Linear Theory for Filtering Nonlinear Multiscale Systems with Model Error Proc. R. Soc. A 2014 470, 20140168

8. Berry, T., Sauer, T.: Adaptive ensemble Kalman filtering of nonlinear systems. Tellus A 65, 20,331 (2013) 9. Bickel, P., Li, B., Bengtsson, T.: Sharp failure rates for the bootstrap filter in high dimensions. In: Essays in Honor of J.K. Gosh. IMS Lecture Notes – Monograph Series, vol. 3, pp. 318–329. Institute of Mathematical Sciences (2008) 10. Bishop, C., Etherton, B., Majumdar, S.: Adaptive sampling with the ensemble transform Kalman filter part I: the theoretical aspects. Mon. Weather Rev. 129, 420–436 (2001) 11. Branicki, M., Majda, A.: Dynamic stochastic superresolution of sparsely observed turbulent systems. J. Comput. Phys. 241(0), 333–363 (2013) 12. Clayton, A.M., Lorenc, A.C., Barker, D.M.: Operational implementation of a hybrid ensemble/4D-Var global data assimilation system at the Met Office. Quart. J. R. Meteorol. Soc. 139(675), 1445–1461 (2013) 13. Dee, D., da Silva, A.: Data assimilation in the presence of forecast bias. Quart. J. R. Meteorol. Soc. 124, 269–295 (1998) 14. Dimet, F.X.L., Talagrand, O.: Variational algorithm for analysis and assimilation of meteorological observations: theoretical aspects. Tellus A 38, 97–110 (1986) 15. Evensen, G.: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res. 99, 10,143–10,162 (1994) 16. Fisher, M., Tremolet, Y., Auvinen, H., Tan, D., Poli, P.: Weak-constraint and long window 4dvar. Technical report 655, ECMWF (2011) 17. Hamill, T., Whitaker, J.: Accounting for the error due to unresolved scales in ensemble data assimilation: a comparison of different approaches. Mon. Weather Rev. 133(11), 3132– 3147 (2005) 18. Harlim, J., Mahdi, A., Majda, A.: An ensemble kalman filter for statistical estimation of physics constrained nonlinear regression models. J. Comput. Phys. 257(Part A), 782–812 (2014) 19. Hunt, B., Kostelich, E., Szunyogh, I.: Efficient data assimilation for spatiotemporal chaos: a local ensemble transform Kalman filter. Physica D 230, 112–126 (2007) 20. Isaksen, L., Bonavita, M., Buizza, M., Fisher, M., Haseler, J., Leutbecher, M., Raynaud, L.: Ensemble of data assimilations at ECMWF. Technical report 636, ECMWF (2010) 21. Kalman, R., Bucy, R.: New results in linear filtering and prediction theory. Trans. AMSE J. Basic Eng. 83D, 95–108 (1961) 22. Keating, S.R., Majda, A.J., Smith, K.S.: New methods for estimating ocean eddy heat transport using satellite altimetry. Mon. Weather Rev. 140(5), 1703–1722 (2012) 23. Khouider, B., Biello, J.A., Majda, A.J.: A stochastic multicloud model for tropical convection. Commun. Math. Sci. 8, 187–216 (2010) 24. Kushner, H.: On the differential equations satisfied by conditional probablitity densities of markov processes, with applications. J. Soc. Indust. Appl. Math. Ser. A Control 2(1), 106–119 (1964) 25. Law, K.J.H., Stuart, A.M.: Evaluating data assimilation algorithms. Mon. Weather Rev. 140(11), 3757–3782 (2012)

Front Tracking 26. Majda, A., Harlim, J.: Filtering Complex Turbulent Systems. Cambridge University Press, Cambridge/New York (2012) 27. Majda, A., Harlim, J.: Physics constrained nonlinear regression models for time series. Nonlinearity 26, 201–217 (2013) 28. Majda, A., Harlim, J., Gershgorin, B.: Mathematical strategies for filtering turbulent dynamical systems. Discr. Contin. Dyn. Syst. A 27(2), 441–486 (2010) 29. Majda, A.J., Grooms, I.: New perspectives on superparameterization for geophysical turbulence. J. Comput. Phys. 271, 60–77 (2014) 30. Mehra, R.: On the identification of variances and adaptive kalman filtering. IEEE Trans. Autom. Control 15(2), 175– 184 (1970) 31. Moncrieff, M., Shapiro, M., Slingo, J., Molteni, F.: Collaborative research at the intersection of weather and climate. World Meteorol. Organ. Bull. 56(3), 1–9 (2007) 32. Sapsis, T., Majda, A.: Statistically accurate low-order models for uncertainty quantification in turbulent dynamical systems. Proc. Natl. Acad. Sci. 110(34), 13,705–13,710 (2013) 33. T´remolet, Y.: Accounting for an imperfect model in 4D-Var. Quart. J. R. Meteorol. Soc. 132(621), 2483–2504 (2006) 34. Wang, X., Parrish, D., Kleist, D., Whitaker, J.: GSI 3DVarbased ensemble-variational hybrid data assimilation for NCEP global forecast system: single resolution experiments. Mon. Weather Rev. 141(11), 4098–4117 (2013)

Front Tracking Nils Henrik Risebro Department of Mathematics, University of Oslo, Oslo, Norway

563

of u at a point .x; t/ is given by f .u.x; t//. Therefore, the function f is often called the flux function. Independently of the smoothness of the initial data u0 and of the flux function f , solutions to (1) will in general develop discontinuities, so by a solution we mean a solution in the weak sense, i.e., Z

1 0

Z

Z R

u't Cf .u/'x dxdt C

R

u0 .x/'.x; 0/ dx D 0; (2)

for all test functions ' 2 C01 .R  Œ0; 1//. Weak solutions are not unique, and to recover uniqueness one must usually impose extra conditions, often referred to as entropy conditions. Front tracking has been used both for theoretical purposes and as a practical numerical method. The existence of weak solutions for general systems of hyperbolic conservation laws was first established using the random choice method [9], but front tracking was used to prove the same result in [1] and [20]. Front tracking has been used as a numerical method both in one dimension, see, e.g., [13] for scalar equations, and for systems, see, e.g., [17, 19, 21]. In several space dimensions, front tracking has been proposed in conjunction with dimensional splitting, see, e.g., [11, 14]. It should also be mentioned that the name “front tracking” is also used for a related but different method where one tracks the discontinuities in u and uses a conventional method to compute u in the regions where u is continuous; see [10] for a description of this type of front tracking.

Weak Solutions and Entropy Conditions Front tracking is a method to compute approximate The weak formulation (2) implies that if u has a jump solutions to the Cauchy problem for hyperbolic con- discontinuity at some .x; t/, which moves with a speed servation laws, namely, s, then f .u.x; t// D su.x; t/; (3) ( ut C f .u/x D 0; t > 0; x 2 R; (1) where u.x; 0/ D u0 .x/: v.x; t/ D lim v.x C "; t/  v.x  "; t/: "#0 Here, the unknown u is a function of space (x) and time (t), and f is a given nonlinear function. In the above, (3) is called the Rankine–Hugoniot condition. Any space is one dimensional; hyperbolic conservation laws isolated discontinuity satisfying this will be a weak in several space dimensions are obtained by replacing solution near .x; t/. As an example consider Burgers’ the x derivative with a divergence. The unknown u can equation, namely, f .u/ D u2 =2, with initial data be a scalar or a vector, in which case (1) is called a ( system of conservation laws. The interpretation of (1) 1 x  0; u0 .x/ D is that it expresses conservation of u and that the flux 1 x > 0:

F

564

Front Tracking

(

In this case, both u.x; t/ D u0 .x/; 8 ˆ 0;

(6)

x  t=2;

where ul and ur are constants. This is called the Riemann problem for the conservation law. In the t=2 < x  t=2; scalar case, it turns out that the entropy solution to t=2  x; the Riemann problem can be constructed as follows: If ul < ur , let f^ .uI ul ; ur / denote the lower convex are weak solutions to ut C.u2 =2/x D 0 taking the same envelope of f in the interval Œu ; u , i.e., l r initial values. Hence weak solutions are not uniquely ˇ ˚ characterized by their initial data. f^ .uI ul ; ur / D sup g 2 C.Œul ; ur / ˇ g.u/  f .u/ The conservation law (1) is often obtained as a limit and g is convex in Œul ; ur g : of the parabolic equation (4) By construction f^0 .u/ is increasing, and thus we can  1 define its generalized inverse f^0 . The entropy as " # 0. If we multiply this with a 0 .u" /, where u 7! solution to (6) is .u/ is convex, we obtain  1 u.x; t/ D f^0 .x=t/: (7)  0  " "  " 2 " " 00 "  .u /t C q .u /x D "  ux ux x  " .u / ux ; If ul > ur , we repeat the above construction with the _ where q 0 D 0 f 0 . If u" ! u as " # 0, using the upper concave envelope f replacing f^ . If the flux function f is piecewise linear and continconvexity of , we get uous, also the envelopes f^ and f _ will be piecewise .u/t C q.u/x  0: (5) linear and continuous. This makes the construction of the entropy solution easy. As an example consider the The pair .; q/ is commonly referred to as an en- piecewise linear flux u"t C f .u" /x D "u"xx ;

tropy/entropy flux pair. For scalar conservation laws, f .u/ D ju C 1j C ju  1j  juj ; the most used entropy condition is that (5) should hold weakly for all convex functions . By a limiting and a density argument, it is sufficient to require and the Riemann problem that (5) holds weakly for the so-called Kruˇzkov en( tropy/entropy flux pairs, 2 x  0; u.x; 0/ D 2 x > 0: .u/ D ju  kj ; q.u/ D sign .u  k/ .f .u/  f .k// : By taking the convex envelope of the flux function We say that a function satisfying (2) and (5) is an en- between u D 2 and u D 2, we find that the entropy tropy solution. For scalar conservation laws, this gives solution is given by well posedness of (1): If u and v are two entropy so8 lutions, then ku.; t/  v.; t/kL1 .R/  ku0  v0 kL1 .R/ . ˆ 2 x  t; ˆ ˆ ˆ For an elaboration of this, as well as more accurate 0 since f ı is bounded. At .x0 ; t0 /, we solve the Riemann problem with ul D uı .x0 ; t0 / and ur D uı .x0 C; t0 /. This will again give a series of discontinuities, or fronts, emanating from .x0 ; t0 /. In this way we can continue the solution up to some t1  t0 . This process is called front tracking. This method was first considered in [7]. In [13] it is proved that for any fixed ı > 0, there are only a finite number of collisions for all t > 0, thus one can construct the entropy solution to the Cauchy problem (

  uıt C f uı x D 0; x 2 R; t > 0; uı .x; 0/ D uı0 .x/;

by a finite number of operations. The convergence of front tracking is shown by appealing to a general result regarding continuous dependence of entropy solutions to scalar conservation laws with respect to the flux function and the initial data. The relevant bound on the error then reads



ı

u .; t/  u.; t/ 1  uı  u0 1 0 L .R/ L .R/

ı

Ct ju0 jBV .R/ f  f Lip.M;M/ ;

(8)

where M is a bound on ju0 j and u is the exact entropy solution of (1). If f ..i C 1/ı/  f .i ı/ ; ı for u 2 Œi ı; .i C 1/ı/, i 2 Z,

f ı .u/ D f .i ı/ C .u  i ı/

then u0 can be chosen such that error is O.ı/. See Fig. 2 for a depiction of the fronts (discontinuities) of front tracking if f .u/ D u3 =3 and u0 .x/ D sin.2x/ and ı D 1=10. Front Tracking for Systems For general strictly hyperbolic systems, Lax, [18], proved that if ur is sufficiently close to ur , then the solution of the Riemann problem consists of n different waves, each wave can be a shock wave (discontinuity) or a rarefaction wave (meaning that the solution is continuous in x=t in some interval). In this case, we cannot make some approximation to f to get a simple solution of the Riemann problem. In order to define front tracking, one must approximate the solution itself. This is done by approximating the rarefaction parts of the solution with a step function in x=t; the step size is now determined by a parameter ı. This gives an approximate solution to the Riemann problem which is piecewise constant in x=t and hence

F

566

Front Tracking 1.5

1

t

0.5

0

−0.8

−0.6

−0.4

−0.2

0 x

0.2

0.4

0.6

0.8

1

Front Tracking, Fig. 2 The fronts in the .x; t / plane

can be used in the front tracking algorithm as in the scalar case. However, it is not clear that we are able to define the approximation uı .x; t/ for any t > 0. The reason is that too many fronts are defined, and explicit examples show that if one approximates all waves in the exact solution of the Riemann problem, then the front tracking algorithm will not define a solution for all positive t. This problem can be avoided by ignoring small fronts in the approximate solution of the Riemann problem. Using the Glimm interaction estimate, [9,23], one can show that when two fronts collide, the new waves created will be of a size bounded by the product of the strength of the colliding fronts. A careful analysis then shows front tracking to be well defined. See the books [2, 12] for precise statements and results. This analysis also establishes the existence of weak solutions by showing that the front tracking approximations converge and that their limits are weak solutions. The approximate solutions obtained by front tracking for systems have also been shown to be L1 stable with respect to the initial data, first for 2  2 systems [3], and then for general n  n systems with small initial data [4, 5]. Front tracking for systems of equations has also been used a practical numerical tool, see [17, 19, 21] and the references therein.

In Fig. 3 we show an example of a front tracking approximation to the solution of the initial value problems for the so-called p-system, namely, t C .u/x D 0  .u/t C u2 C p./ x D 0; 

where  models the density and u the velocity of a gas. To close this, the pressure p./ is specified as p./ D  ;  D

.  1/2 ;  D 1:4: 4

The computation in Fig. 3 uses the periodic initial values 0 .x/ D 0:1 C cos2 .x/; u0 .x/ D cos.x/: Several Space Dimensions and Other Equations In several space dimension, front tracking is more complicated. The discontinuities are no longer ordered, and their topology can be complicated. It is still possible to use front tracking as a building block for methods using dimensional splitting. For a conservation law in two space dimensions,

Front Tracking

567 log(ρ+1) 3.5 1.2 3 1 2.5 0.8 2

t 0.6

1.5

F

0.4

1

0.2

0.5

0

−0.8

−0.6

−0.4

−0.2

0 x

0.2

0.4

0.6

0.8

0

Front Tracking, Fig. 3 The logarithm of the density in the .x; t / plane

ut C f .u/x C g.u/y D 0;

Front tracking has also been used and shown to converge for Hamilton-Jacobi equations in one space this approach consists in letting un .x; y; t/ solve the dimension, see [6, 15], and has been used in conjuncone-dimensional conservation law in the time interval tion with dimensional splitting for Hamilton-Jacobi equations in several space dimensions [16]. .nt ; .n C 1/t/ (

@un @t n

C

@f .un / @x

D 0;

u .x; y; nt / given,

here y is a parameter.

Then ut .; ; .n C 1/t/ is used as initial value for the conservation law in the y-direction, namely, (

@vn @t n

C

@g.vn / @y

D 0;

nt < t < .n C 1/t

v .x; y; nt / D u .x; y; .n C 1/t/: n

This process is then repeated. To solve the onedimensional equations, front tracking is a viable tool, since it has no intrinsic CFL condition limiting the time step. For scalar equations, the convergence of front tracking/dimensional splitting approximations was proved in [11]. Dimensional splitting can also be viewed as a large time step Godunov method and has been used with some success to generate approximations to solutions of the Euler equations of gas dynamics in two and three dimensions, see [14].

References 1. Bressan, A.: Global solutions of systems of conservation laws by wave-front tracking. J. Math. Anal. Appl. 170(2), 414–432 (1992). 1 2. Bressan, A.: Hyperbolic Systems of Conservation Laws. Oxford Lecture Series in Mathematics and Its Applications, vol. 20. Oxford University Press, Oxford (2000). The onedimensional Cauchy problem. 1.3 3. Bressan, A., Colombo, R.M.: The semigroup generated by 2  2 conservation laws. Arch. Ration. Mech. Anal. 133(1), 1–75 (1995). 1.3 4. Bressan, A., Liu, T.-P., Yang, T.: L1 stability estimates for n  n conservation laws. Arch. Ration. Mech. Anal. 149(1), 1–22 (1999). 1.3 5. Bressan, A., Crasta, G., Piccoli, B.: Well-posedness of the Cauchy problem for n  n systems of conservation laws. Mem. Am. Math. Soc. 146(694), viii+134 (2000). 1.3 6. Coclite, G.M., Risebro, N.H.: Viscosity solutions of Hamilton-Jacobi equations with discontinuous coefficients. J. Hyperbolic Differ. Equ. 4(4), 771–795 (2007). 1.4 7. Dafermos, C.M.: Polygonal approximations of solutions of the initial value problem for a conservation law. J. Math. Anal. Appl. 38, 33–41 (1972). 1.2

568

Functional Equations: Computation

8. Dafermos, C.M.: Hyperbolic Conservation Laws in Continuum Physics. Grundlehren der Mathematischen Wissenschaften (Fundamental Principles of Mathematical Sciences), vol. 325, 3rd edn. Springer, Berlin (2010). 1.1 9. Glimm, J.: Solutions in the large for nonlinear hyperbolic systems of equations. Commun. Pure Appl. Math. 18, 697–715 (1965). 1, 1.3 10. Glimm, J., Grove, J.W., Li, X.L., Tan, D.C.: Robust computational algorithms for dynamic interface tracking in three dimensions. SIAM J. Sci. Comput. 21(6), 2240–2256 (electronic) (2000). 1 11. Holden, H., Risebro, N.H.: A method of fractional steps for scalar conservation laws without the CFL condition. Math. Comput. 60(201), 221–232 (1993). 1, 1.4 12. Holden, H., Risebro, N.H.: Front Tracking for Hyperbolic Conservation Laws. Applied Mathematical Sciences, vol. 152. Springer, New York (2011). First softcover corrected printing of the 2002 original. 1.1, 1.3 13. Holden, H., Holden, L., Høegh-Krohn, R.: A numerical method for first order nonlinear scalar conservation laws in one dimension. Comput. Math. Appl. 15(6–8), 595–602 (1988). Hyperbolic partial differential equations. V. 1, 1.2 14. Holden, H., Lie, K.-A., Risebro, N.H.: An unconditionally stable method for the Euler equations. J. Comput. Phys. 150(1), 76–96 (1999). 1, 1.4

Functional Equations: Computation Alfredo Bellen Department of Mathematics and Geosciences, University of Trieste, Trieste, Italy

Introduction

In the largest meaning, a functional equation is any equation where the unknown, to be solved for, is a function f belonging to a suitable function space F of one or more independent variables x which are supposed to vary in a fixed domain D. Such a broad definition includes a huge variety of equations. Among them, ordinary and partial differential equations, differential algebraic equations, and any other equation whose expressions describe the constraints the solution f and some derivatives must fulfil at any single fixed value x of the independent variables in the domain. Its general form is F .x; f .x// D 0, F being a suitable 15. Karlsen, K.H., Risebro, N.H.: A note on front tracking and algebraic or differential operator. equivalence between viscosity solutions of Hamilton-Jacobi Time evolution systems described by such equations equations and entropy solutions of scalar conservation laws. are often said to fulfil the principle of causality, that Nonlinear Anal. Ser. A Theory Method 50(4), 455–469 (2002). 1.4 is, the future state of the system depends solely by the 16. Karlsen, K.H., Risebro, N.H.: Unconditionally stable meth- present. Usually, in the literature these equations are ods for Hamilton-Jacobi equations. J. Comput. Phys. 180(2), not included in the class of functional equations which 710–735 (2002). 1.4 17. Langseth, J.O., Risebro, N.H., Tveito, A.: A conservative is instead reserved to equations where, for any fixed front tracking scheme for 1D hyperbolic conservation laws. x, the unknown function f is simultaneously involved In: Nonlinear Hyperbolic Problems: Theoretical, Applied, also at points other than x. A classical example is the and Computational Aspects (Taormina, 1992). Notes on Cauchy functional equation 18. 19.

20.

21.

22.

23.

Numerical Fluid Mechanics, vol. 43, pp. 385–392. Vieweg, Braunschweig (1993). 1, 1.3 Lax, P.D.: Hyperbolic systems of conservation laws. II. Commun. Pure Appl. Math. 10, 537–566 (1957). 1.3 Risebro, N.H., Tveito, A.: Front tracking applied to a nonstrictly hyperbolic system of conservation laws. SIAM J. Sci. Stat. Comput. 12(6), 1401–1419 (1991). 1, 1.3 Risebro, N.H.: A front-tracking alternative to the random choice method. Proc. Am. Math. Soc. 117(4), 1125–1139 (1993). 1 Risebro, N.H., Tveito, A.: A front tracking method for conservation laws in one dimension. J. Comput. Phys. 101(1), 130–139 (1992). 1, 1.3 Serre, D.: Systems of conservation laws. 1. Cambridge University Press, Cambridge (1999). Hyperbolicity, entropies, shock waves (trans: from the 1996 French original by Sneddon I.N.). 1.1 Yong, W.-A.: A simple approach to Glimm’s interaction estimates. Appl. Math. Lett. 12(2), 29–34 (1999). 1.3

f .x/ C f .y/ D f .x C y/;

.x; y/ 2 R2

(1)

whose solution is a scalar function f W R ! R that must fulfil the equation for any pair .x; y/ 2 R2 and the analogue system where f W Rn ! Rm , with x; y 2 Rn and .x; y/ 2 Rn Rn . Remark the difference between the domain of the equation, Rn  Rn , and the domain of the solution f , Rn . There exists a long list of functional equations, often associated with names of famous mathematicians and physicists such as Abel, Schroeder, d’Alambert, Jensen, Bellman, and many others, which come from different areas of theoretical and applied mathematics.

Functional Equations: Computation

Despite early examples, in a disguised formulation, go back to the thirteenth century, functional equations developed since the seventeenth century together with the concept of function whose deep understanding has been cause and effect of its development. A subclass of functional equations is determined by the case where the equation still depends on two or more specified values of the independent variable which are no longer independent of each other (as x and y were in the Cauchy equation (1)) but they all are functions of the same variable x. In other words, the domain of the equation is now included in the domain of the solution. To this subclass belong, for example, the Schroeder equation f .h.x// D cf .x/ and the Abel equation f .g.x// D f .x/ C 1 where h.x/ and g.x) are given functions that act as shifts (or deviating arguments) of x. More complicated dependency of the variables are found in the “composite equations” where the shift depends on the sought function itself, as in the Babbage equation f .f .x//  x D 0 and its generalization f .f .x// D h.x/, for a given h. The general form of such equations is F x; f .x/;

f h.x; f .x// D 0, where F W Rn  Rn  Rn ! Rn is an algebraic operator, h W Rn  Rn ! Rn is the shift, and the unknown f belongs to a suitable algebra of functions. For these equations, which have been analyzed and solved chiefly by analytical rather than numerical tools, we refer the interested reader to J. Aczel K and J. Dhombres [1] and A. D. Polyanin and A. I. Chernoutsan [20]. Since the pioneeristic papers by V. Volterra at the beginning of the last century (see the exhausting bibliography in H. Brunner [7]), the concept of functional equation extended to equations based on integral and integrodifferential operators suitable for modeling phenomena infringing the mentioned principle of causality. When the unknown function depends on a scalar independent time variable x D t, it is worth distinguishing among the cases where the deviating argument h.t/ attains values h.t/  t, h.t/  t or both. Usually, the three occurrences are referred to as retarded, advanced or mixedfunctional equations, respectively. Disregarding advanced and mixed functional equations, often related to the controversial principle of retro-causality (the present state of the system depends on the future), let us now consider the

569

class of retarded functional equations (RFE) that, from now on, will be written in the following form: F .t; yt / D 0;

(2)

or in the explicit form y.t/ D G.t; yt /;

(3)

where the unknown y is a Rd -valued function of one real variable, the functional G maps .R  X / into Rd , the state-space X being a suitable subspace of vectorvalued functions .1; 0 ! Rd , and, according to the Hale-Krasovski notation, the state yt 2 X (at the time t) is given by yt ./ D y.t C /;  2 .1; 0: The class of RFEs naturally extends to the class of retarded functional differential equations (RFDE) of the form y.t/ P D G.t; yt /

(4)

and neutral retarded functional differential equations (NRFDE) y.t/ P D G.t; yt ; yPt /

(5)

where the right-hand side functional G depends also on the derivative of the state yPt D y.t P C /, the sought function y.t/ is almost everywhere differentiable, and the state-space X is now included in LC.1; 0/, the set of locally Lipschitz-continuous functions. The class of RFDEs may be further extended to the implicit form M y.t/ P D G.t; yt /

(6)

where M is a possibly singular constant matrix. Besides (4), (6) incorporates retarded functional differential algebraic equations (RFDAE) 8

< y.t/ P D G t; yt ; zt

: 0 D H t; yt ; zt ;

F

570

Functional Equations: Computation

singularly perturbed problems

possibly unbounded, set of past points is involved. The latters are essentially integral equations that can 8

be further subdivided in the more restricted forms of < y.t/ P D G t; yt ; zt

Volterra integral equations (VIE) of the first and second : Pz.t/ D H t; yt ; zt ; kind  Rt and, in particular, NRFDE (5) that can be written as a 0 K.t; s; y.s//ds D g.t/ t  0 (10) (non-neutral) RFDAE of doubled dimension 2d for the g.0/ D 0; Rt  unknown .y.t/; z.t//T y.t/ D g.t/ C 0 K.t; s; y.s//ds t  0 (11) !    y.0/ D g.0/; z.t/

y.t/ P I 0 D G t; yt ; zt  z.t/ zP.t/ 00 Volterra integro differential equations (VIDE) (7)

In order to have well-posedness, the (2)–(6) are often endowed by suitable initial data y0 D y.0 C /;   0, which represents the initial state of the system at the initial time t D 0. After Volterra, various kinds of retarded functional and retarded functional differential equations (often comprehensively identified as Volterra functional equation) have been being used for modeling many phenomena in applied sciences and engineering where, case by case, they have been referred to as time-delay system, time-lag system, hereditary system, system with memory, system with after effect, etc. Being any Volterra functional equations (2)–(6) characterized by the action of the functionals G on the state yt , we can distinguish between discrete delay equations, usually called delay differential equations (DDE) and neutral delay differential equations (NDDE) 8 P D f .t; y.t/; y.t  1 /; : : : ; y.t  r // < y.t/ 0  t  tf ; DDE : y.t/ D .t/; t  0;

(

Rt t 0 y.t/ P D f t; y.t/; 0 K.t; s; y.s//ds y.0/ D y0 ; (12)

and equations in more general forms including deviating arguments in t such as, for example,

8 Rt ˆ P D f t; y.t/; y.t /; t  K.t; s; y.s//ds < y.t/ t 0 ˆ : y.t/ D .t/; t  0; (13) and the like, usually called Volterra delay integro differential equations (VDIDE) as well as other equations in nonstandard form (see H. Brunner this encyclopedia) which are referred to by the all-inclusive term Volterra functional equation.

The Numerics of Volterra Functional (8) Equations

For Volterra functional equations (2)–(5), a big deal of work has been done since the years 1970–1980 from the numerical point of view. The most natural tool for approximating the solution of a functional equation is global or piecewise polynomial collocation that, being a continuous method with dense output, provides an (9) approximation of the whole state yt , as required in the evaluation of G.t; yt / and G.t; yt ; yPt /, at any possible where, for any t, the state yt is involved with a finite value of the variable t. The strength of collocation lies in its adaptability or discrete (countable) set of past points t  i  t and distributed delay equations where a continuum, to any kind of equation where it provides accurate 8 y.t/ P D f .t; y.t/; y.t  1 /; : : : ; y.t  r /; ˆ ˆ < P  r // 0  t  tf ; y.t P  1 /; : : : ; y.t ˆ y.t/ D .t/; t  0; NDDE ˆ : P y.t/ P D .t/; t  0;

Functional Equations: Computation

solutions for stiff initial value problems, as well as problems with boundary, periodicity, and other conditions (see the exhaustive monograph by H. Brunner [7]). On the other hand, collocation is an intrinsically implicit method which requires the use of nonlinear solvers that, in many cases, represents the bottleneck of the overall procedure. Implicit and explicit methods for the numerical solution of VIEs (10), (11) and VDIEs (12), with no deviating arguments t  , have been developed by many authors and nowadays various well-established numerical methods are available. For the sake of brevity, we skip details and refer the interested reader to the encyclopedic monography by H. Brunner and P. van der Houwen [8] and the rich bibliography therein. Specific methods for the enlarged class of delay differential equations (8), (9) and delay integro differential equations (13) started developing in the 1970s by L. Tavernini [21] and C.W. Cryer and L. Tavernini [10] (see also C.W. Cryer [9]) and had a big impulse in the subsequent two decades, reported as state of the art by C. Baker [2]. The comprehensive book by A. Bellen and M. Zennaro [5] provides an up-todate overview of numerical methods for DDEs with particular attention to accuracy and stability analysis of methods based on continuous extensions of Runge– Kutta methods. More recently an original and unifying approach for the well-posedness and convergence analysis of most of the methods for RFDE in the form (5) appeared in A. Bellen, N. Guglielmi, S. Maset, and M. Zennaro [6]. Two specific chapters of this encyclopedia are concerned with Volterra functional equations. The first one, by N. Guglielmi, is focussed on stiff implicit problem (6) and neutral equations in the equivalent form (7) and faces some critical implementation issues of collocation methods. The second ones, by H. Brunner, provides an updated overview for more general Volterra functional equations, extended to boundary value problems for higher-order differential operators and partial Volterra and Fredholm integrodifferential equations in bounded or unbounded time-space domains. Other methods have been investigated for DDEs, such as waveform relaxation like iterations (see

571

Z. Jackiewicz, M. Kwapisz, and E. Lo [17] and B. Zubik-Kowal and S. Vandewalle [22]) and methods based on the reformulation of the equation as an abstract Cauchy problem in Banach space and subsequent discretization (see F. Kappel and W. Schappacher [18]). In particular, the semidiscretization of the equivalent hyperbolic PDEs based on the transversal method of lines allows to infringe the order barrier for the stability of Runge–Kutta methods and provides the sole to date known method, called abstract backward Euler, which is asymptotically stable for any asymptotically stable linear systems of DDEs (see A. Bellen and S. Maset [4]).

Continuous Runge–Kutta Methods In the construction of time-stepping methods for initial value problems (2)–(5), two approaches have been mainly pursued in literature. Both of them include implicit and explicit methods and have its core in the use of a continuous Runge–Kutta scheme .A./; b./; c/ where A./ D .ai;j .//si;j D1 is a polynomial matrix, b./ D .bi ./;    ; bs .//T is the polynomial vector of weights, and c D .c1 ; : : : ; cs /T is the vector of abscissas. Once a continuous piecewise approximation .t/ of the solution has been achieved until the nodal point tn , the approximate solution  is prolonged on the interval Œtn ; tn C hnC1  by the new piece defined as follows: .tn ChnC1 / D .tn /ChnC1

s X

bi ./Ki ; 0    1;

i D1

where the derivatives Ki 2 Rd , i D 1;    ; s are given by Ki D G.tn C ci hnC1 ; Ytin Cci hnC1 ; YPtin Cci hnC1 /

(14)

and the stage functions Ytin Cci hnC1 W .1; 0 ! Rd , i D 1;    ; s, denoted also by Ytin Cci hnC1 D Y i .tn C ci hnC1 /; 1 <   1, are defined in two different ways leading to two different methods. The first one, based on the continuous Runge-Kutta scheme .A; b./; c/ with constant matrix A, consists in setting

F

572

Functional Equations: Computation

Ps 8 < .tn / C hnC1 Pj D1 bj .ci /Kj ;  2 .0; 1/ Y i .tn C ci hnC1 / D .tn / C hnC1 sj D1 aij Kj ;  D 1 : .tn C ci hnC1 /   0:

and solving (14) and (15) for K D .K1 ;    ; Ks /T . Remark that, for every i , the stage functions Y i .tn C ci hnC1 / computed at  D 1 reduce to the traditional stage values Y i of the Runge-Kutta scheme. They enter into the formula only if the operator G takes the form G.t; y.t/; yt ; yPt /. The method, known as standard approach or interpolated continuous Runge–Kutta method for DDEs, has  Y i .tn C ci hnC1 / D

been extensively studied in the last 30 years and is exhaustively described from accuracy, stability, and implementative point of view in the comprehensive book [5]. The second method relies on the continuous RungeKutta scheme .A./; b./; c/, where A./ is a polynomial valued matrix, and the stage functions are given by

P .tn / C hnC1 sj D1 ai;j .ci /Kj ;  2 .0; 1 .tn C ci hnC1 /   0;

to be solved, along with (14), for K D .K1 ;    ; Ks /T . The method, properly called functional Runge-Kutta method, has been proposed in the 1970s and deeply investigated much later in S. Maset, L. Torelli, and R. Vermiglio [19] where new explicit schemes, minimizing the number of stages s, have been found up to the order 4. Both Runge-Kutta schemes .A; b./; c/ and .A./; b./; c/ can be implicit or explicit. If they are implicit, a full (implicit) system has to be solved for the derivatives Ki . In particular when they are the Runge– Kutta version of collocation at the abscissas ci , that is, ai;j D bj .ci / and ai;j ./ D bj ./ for i D 1;    ; s, the resulting standard approach (15) coincides with the functional Runge-Kutta method (16) for any functional equation. On the other hand, if .A; b./; c/ and .A./; b./; c/ are explicit and the methods are applied to functional equations where overlapping occurs, that is, where some stage function must be computed at points tn C ci hnC1 still lying inside the current integration interval, the system (14), (15) rising from the standard approach turns out to be implicit even if the underlying Runke-Kutta scheme was explicit. This makes the functional Runge-Kutta method a powerful competitor of the standard approach for non-stiff equations.

(15)

(16)

Delay and Neutral Delay Differential Equations: A Deeper Insight There are various significantly different kind of DDE and NDDE depending on the quality of the delays

i . Besides being nonnegative, each i may be just a constant delay, a time-dependent delay i D i .t/, and a state-dependent delay i D i .t; y.t// or even

i D i .t; yt /. In the class of time-dependent delays, a special role is played by the proportional delay t 

.t/ D qt, 0 < q < 1, characterizing the pantograph equation yP D ay.t/ C by.qt/, t  0, that exhibits two features: the initial data reduces to the initial value y.0/ D y0 and the deviated argument qt overlaps on any interval Œ0; tN; 8tN > 0. For initial value problems (8) and (9), a crucial issue, from both theoretical and numerical point of view, is the fulfilment of the following splicing condition at the initial point t D 0:  P  / D f 0; .0/; . 1/; : : : ; .0

 P P . r /; . 1 /; : : : ; . r/ ;

all i computed at t D 0. If the splicing condition is not fulfiled, the derivative y.t/ P has a jump discontinuity, called

Functional Equations: Computation

0-level discontinuity, at the initial point t D 0. Such a discontinuity reflects into a set of 1-level discontinuities at any subsequent point 1;i such that 1;i  .1;i ; y.1;i // D 0. Analogously, any s-level discontinuity point s;i gives rise to a set of (s+1)-level discontinuity points sC1;j such that sC1;j  .sC1;j ; y.sC1;j // D s;i and so forth for higher-level discontinuities. For non-neutral equations the solution gets smoother and smoother as the level rises and, in particular, at any s-level discontinuity point the solution is at least of class C s . On the contrary, for neutral equations the solution is not smoothed out and yP keeps being discontinuous at any level. Localization of such discontinuity points, which are often referred to as breaking points, is essential in the construction of any accurate numerical method. Remark that, for state-dependent delays, the breaking point cannot be located a priori and their accurate computation is a critical issue in the production of software for RFDE (see N. Guglielmi and E. Hairer [12, 13]). Tracking the breaking points is a particular need for NDDE with state-dependent delay because at any such point the solution could bifurcate or cease to exist even under the most favorable regularity assumptions about the function G and the initial data . For such equations the concept of generalized solution, continuing the classical solution beyond the breaking points, has been introduced as the solution of the more general functional equation yP 2 G.t; yt / inspired to the Filippov theory of discontinuous ordinary differential equations. Recently the problem has started being faced from the numerical point of view by A. Bellen and N. Guglielmi [3], G. Fusco and N. Guglielmi [11], and N. Guglielmi and E. Hairer [14, 15] where various regularizations have been proposed for (9) with one single state-dependent delay. In particular, reformulating the NDDE in the form (7) and solving the associated singularly perturbed equation 

y.t/ P D z.t/

Pz.t/ D G.t; yt ; zt /  z.t/;

leads to solutions y whose limit, as ! 1, converges either to the classic or to the generalized solution

573

and provides an important tool for analyzing the rich dynamics of the model. In this setting, the case of multiple delay is a real challenge recently faced by N. Guglielmi and E. Hairer [16] and still to be exhaustively explored.

References 1. Aczel, K J., Dhombres, J.: Functional Equations in Several Variables. Cambridge University Press, Cambridge (1989) 2. Baker, C.T.H.: Numerical analysis of Volterra functional and integral equations. In: Duff, I.S., Watson, G.A. (eds.) The State of the Art in Numerical Analysis. Clarendon, Oxford (1996) 3. Bellen, A., Guglielmi, N.: Solving neutral delay differential equations with state-dependent delays. J. Comput. Appl. Math. 229(2), 350–362 (2009) 4. Bellen, A., Maset, S.: Numerical solution of constant coefficient linear delay differential equations as abstract Cauchy problems. Numer. Math. 84(3), 351–374 (2000) 5. Bellen, A., Zennaro, M.: Runge Kutta Methods for Delay Differential Equations. Oxford University Press, Oxford (2003) 6. Bellen, A., Guglielmi, N., Maset, S., Zennaro, M.: Recent trends in the numerical solution of retarded functional differential equations. Acta Numer. 18, 1–110 (2009) 7. Brunner, H.: Collocation Methods for Volterra Integral and Related Functional Differential Equations. Cambridge University Press, Cambriedge (2004) 8. Brunner, H., van der Houwen, P.J.: The Numerical Solution of Volterra Equations. North Holland, Amsterdam (1986) 9. Cryer, C.W.: Numerical methods for functional differential equations. In: Schmitt, K. (ed.) Delay and Functional Differential Equations and Their Applications, pp. 17–101. Academic, New York (1972) 10. Cryer, C.W., Tavernini, L.: The numerical solution of Volterra functional differential equations by Euler’s method. SIAM J. Numer. Anal. 9, 105–129 (1972) 11. Fusco, G., Guglielmi, N.: A regularization for discontinuous differential equations with application to state-dependent delay differential equations of neutral type. J. Differ. Equ. 250, 3230–3279 (2011) 12. Guglielmi, N., Hairer, E.: Implementing Radau IIA methods for stiff delay differential equations. Computing 67(1), 1–12 (2001) 13. Guglielmi, N., Hairer, E.: Computing breaking points in implicit delay differential equations. Adv. Comput. Math. 29(3), 229–247 (2008) 14. Guglielmi, N., Hairer, E.: Recent approaches for statedependent neutral delay equations with discontinuities. Math. Comput. Simul. (2011, in press) 15. Guglielmi, N., Hairer, E.: Asymptotic expansion for regularized state-dependent neutral delay equations. SIAM J. Math. Anal. 44(4), 2428–2458 (2012) 16. Guglielmi, N., Hairer, E.: Regularization of neutral differential equations with several delays. J. Dyn. Differ. Equ. 25(1), 173–192 (2013)

F

574 17. Jackiewicz, Z., Kwapisz, M., Lo, E.: Waveform relaxation methods for functional-differential systems of neutral type. J. Math. Anal. Appl. 207(1), 255–285 (1997) 18. Kappel, F., Schappacher, W.: Nonlinear functionaldifferential equations and abstract integral equations. Proc. R. Soc. Edinb. Sect. A 84(1–2), 71–91 (1979) 19. Maset, S., Torelli, L., Vermiglio, R.: Runge–Kutta methods for retarded functional differential equations. Math. Models Methods Appl. Sci. 15(8), 1203–1251 (2005)

Functional Equations: Computation 20. Polyanin, A.D., Chernoutsan, A.I. (eds.): A Concise Handbook of Mathematics, Physics, and Engineering Sciences. Chapman & Hall/CRC, Boca Raton/London (2010) 21. Tavernini, L.: One-step methods for the numerical solution of Volterra functional differential equations. SIAM J. Numer. Anal. 4, 786–795 (1971) 22. Zubik-Kowal, B., Vandewalle, S.: Waveform relaxation for functional-differential equations. SIAM J. Sci. Comput. 21(1), 207–226 (1999)

G

Gabor Analysis and Algorithms Hans Georg Feichtinger1 and Franz Luef2 1 Institute of Mathematics, University of Vienna, Vienna, Austria 2 Department of Mathematics, University of California, Berkeley, CA, USA

Mathematics Subject Classification Primary 42C15; 42B35

Keywords and Phrases Gabor frames; Janssen representation; Time-frequency analysis

Motivation Abstract harmonic analysis explains how to describe the (global) Fourier transform (FT) of signals even over general LCA (locally compact Abelian) groups but typically requires square integrability or periodicity. For the analysis of time-variant signals, an alternative is needed, the so-called sliding window FT or the STFT, the short-time Fourier transform, defined over

FL was supported by an APART fellowship of the Austrian Academy of Sciences. HGFei was active as a member of the UnlocX EU network when writing this note.

phase space, the Cartesian, and the product of the time domain with the frequency domain. Starting from a signal f it is obtained by first localizing f in time using a (typically bump-like) window function g followed by a Fourier analysis of the localized part [1]. Another important application of time-frequency analysis is in wireless communication where it helps to design reliable mobile communication systems. This article presents the key ideas of Gabor analysis as a subfield of time-frequency analysis, as inaugurated by Denis Gabor’s work [21]. There are two equivalent views: either focus on redundancy reduction of the STFT by sampling it along some lattice (Gabor himself suggested to use the integer lattice in phase space) requiring stable linear reconstruction or to emphasize the representation of f as superposition of time frequency shifted atoms as building blocks. For realtime signal processing engineers also use the concept of filter banks to describe the situation. Here each frequency channel contains all the Gabor coefficients corresponding to a fixed frequency [4]. From a mathematical perspective Gabor analysis can be considered as a modern branch of harmonic analysis over the Heisenberg group. The most useful description of the Heisenberg group for time-frequency analysis is the one where Rd  Rd  R is endowed with the group law .x; !; s/ ˝ .y; ; t/ D .x C y; ! C ; s C t C y  !  x  /. The representation theory of the Heisenberg group constitutes the mathematical framework of time-frequency analysis [22]. Gabor analysis is a branch of time-frequency analysis, which has turned out to have applications in audio mining, music, wireless communication, pseudodifferential operators, function spaces, Schr¨odinger equations, noncommutative geometry, approximation

© Springer-Verlag Berlin Heidelberg 2015 B. Engquist (ed.), Encyclopedia of Applied and Computational Mathematics, DOI 10.1007/978-3-540-70529-1

576

Gabor Analysis and Algorithms

theory, or the Kadison-Singer conjecture [5,7,8,10,11, 23, 32–35]. Recent progress in wireless communication relies on modeling the transmission channels as pseudodifferential operators, whose symbol belongs to a Sj¨ostrand’s class [29, 43]. They possess a matrix representation with respect to Gabor frames with strong off-diagonal decay, which can be used to design transmission pulses and equalizers for mobile communication. Also multicarrier communication systems such as the OFDM (orthogonal frequency division multiplex systems) are naturally described by Gabor analysis. Recently, methods from algebraic geometry have been successfully invoked to address this circle of ideas [3, 20, 27, 37].

f D

X

a g:

(5)

2ƒ

His original suggestion (namely, to use the Gauss 2 function g0 .t/ WD e jt j and ƒ D Z2 ) turned out to be overoptimistic [2, 26]. The Balian-Low theorem implies that there is no smooth and well-localized atom such that the corresponding Gabor family is a frame for L2 .R/ (nor a Riesz basis). This is in sharp contrast to the wavelet case, where orthonormal bases can be found with compact support and arbitrary smoothness [28]. By now it is known that for ƒ D aZ  bZ with ab < 1 the associated Gabor system for the following Gabor atoms is a Gabor frame: the Gauss function [36, 40], the hyperbolic cosecant [31], and for many totally positive functions [25]. There are various equivalent ways to express the Gabor Analysis property of stable signal representation by a Gabor family. The concept of frames is the most popular ones: The STFT measures the time-variant frequency content G.g; ƒ/ constitutes a frame for L2 .Rd /, if there are of a distribution f using a well-localized and smooth constants A; B > 0 such that for all f 2 L2 .Rd / window g 2 L2 .Rd / centered at the origin of Rd . X In order to move it to some point z D .x; !/ 2 R2d jhf; gij2  Bkf k22 ; (6) Akf k22  on uses time-frequency shifts .z/, i.e., applying first 2ƒ the translation operator Tx g.t/ D g.t  x/ and then the modulation operator M! g.t/ D e 2 i !t g.t/; thus, or equivalently positive definiteness of the Gabor .z/ D M! Tx . Using notation from the more general frame operator Sf WD Sg;g f , given by X coorbit theory [12], the STFT can be expressed as Sf WD hf; gig for f 2 L2 .Rd /: (7) Z d 2ƒ Vg f .z/ D Vg f .x; !/ D f .t/g.t  x/e 2 i t ! R Such operators as well as the generalized frame operadt D hf; .x; !/gi D hf; .z/gi (1) tors X f 7! Sg;h f D hf; gih (8) and provides a description of f in which time and satisfy the important commutation relation S ı ./ D frequency play a symmetric role. The main reason for ./ ı S for all  2 ƒ. the rich structure of Gabor analysis is the noncomIt is useful to view them as a composition of two mutativity expressed by the following formulas, for bounded operators, first the (injective) coefficient mapx; ! 2 Rd ; z D .x; !/; z0 D .y; / 2 R2d : ping f 7! C f D .hf; gi/ from L2 .Rd / g

Tx M! D e 2 ix! M! Tx 0

.z/.z / D e

2 ix

0

.z C z /

.z/.z0 / D e 2 i.y!x/ .z0 /.z/

(2) (3) (4)

Following Gabor’s proposal in [21] one looks for atomic representations of functions (resp. distributions) f using building blocks from a so-called Gabor system G.g; ƒ/ WD fg WD ./gj 2 ƒg, involving a lattice ƒ in R2d and a window (Gabor atom) g:

2ƒ

to `2 .ƒ/ analyzing the time-frequency content of f with respect to the Gabor system G.g; ƒ/, and the (surjective, adjoint) synthesis operator .a / 7! Dh a D P 2 2 d 2ƒ a h , from ` .ƒ/ to L .R /. The factorization 1 1 gives different signal Id D S ı S D S ı S expansions, with gQ WD S 1 g and g t WD S 1=2 g, the canonical dual resp. tight Gabor atom. f D

X

hf; gQ  ig D

2ƒ

X 2ƒ

hf; gigQ  D

X

hf; gt igt

2ƒ

(9)

Gabor Analysis and Algorithms

577

The coefficients in the atomic decomposition of f are the samples of VgQ f over ƒ. The second version provides recovery from the samples of Vg f , using gQ as building block. The version involving gt is more symmetric and better suited for the definition of Gabor multipliers (also called time-variant filters), where the Gabor coefficients are multiplied with weights, because real-valued weights induce self-adjoint operators. Although the mapping f 7! Vg f is isometric from L2 .Rd / into L2 .R2d / for g 2 L2 .Rd / with kgk2 D 1 and Vg f is continuous and bounded for f 2 L2 , the boundedness of Cg resp. Dg is not granted for general g 2 L2 .Rd /. A universal sufficient condition [16] is membership of g in Feichtinger’s algebra S0 .Rd / introduced in [9]. f 2 S 0 .Rd / if for some Schwartz function 0 ¤ g 2 S.Rd / (e.g., the Gaussian) “ kf kS0 D

jVg f .x; !/jdxd! < 1:

If ƒ splits as ƒ D ƒ1  ƒ2 , then the same is true for ƒı , e.g., for ƒ D aZd  bZd , the adjoint lattice is ƒı D .1=b/Zd  .1=a/Zd . In this way the so-called Walnut representation for G.g; aZd  bZd / in [44] follows from the Janssen representation. An extensive discussion of the Walnut representation can be found in [22]. Corollary 1 For g 2 S 0 .Rd / Sg;g f D

X

Gn Tn=b f;

(12)

n2Zd

with bounded and continuous aperiodic function Gn and absolute convergence of the series in the operator norm sense on S 0 .Rd / resp. on L2 .Rd /.PHere Gn is the aperiodic function given by Gn .x/ D k2Z g.x  n b  ak/g.x  ak/. One reason for the usefulness of S 0 .Rd / is that for G.g; ƒ/ with g 2 S 0 .Rd / also the canonical dual and tight Gabor atoms gQ and g t are also in S 0 .Rd /, i.e., the building blocks in the discrete reconstruction formula have the same quality and (5) converges in S 0 .Rd / [24], and the frame operator is automatically invertible on S 0 .Rd /. The result in [45] is nowadays known as WexlerRaz biorthogonality relation and characterizes the class of all dual windows in terms of the Gabor system G.g; ƒı /.

R2d

Different windows define equivalent norms, and S.Rd /  S 0 .Rd /. One has isometric invariance of .S 0 .Rd /; jj  jjs0 /p under time-frequency shifts and the Fourier transform. Searching for a fast method to determine the coefficients in (5), the electrical engineers Raz and Wexler initiated a kind of duality theory [45], which was then fully established by three independent contributions [6, 30, 39] for the classical setting and in [13, 16] for general lattices [19]. The duality theory allows to Theorem 2 (Wexler-Raz) Let G.g; ƒ/ be a Gabor replace the questions of Gabor frame G.g; ƒ/ by an frame for L2 .Rd / with g 2 S 0 .Rd /. Then we have equivalent question involving the adjoint lattice of ƒ: Sg;h D I if and only if hh; .ı /gi D vol.ƒ/1 ıı ;0 for all ı 2 ƒı . ƒı D fz 2 R2d W ./.z/ D .z/./ for all  2 ƒg; Another cornerstone of duality theory of Gabor (10) analysis is the Ron-Shen duality principle [39], which since a Gabor frame-type operator Sg;h has a reprewas also obtained by Janssen in [30]. sentation in terms of time-frequency shifts f.ı / W ı 2 ƒı g, the so-called Janssen representation. In this Theorem 3 Let G.g; ƒ/ be a Gabor system. Then generality it was introduced in [13], but it was first G.g; ƒ/ is a Gabor frame for L2 .Rd / if and only if studied by Rieffel in [38] (see [34]). G.g; ƒı / is a Riesz basis for L2 .Rd /, i.e., if there exist positive constants C; D > 0 such that for all finite Theorem 1 For g; h 2 S0 .Rd / one has sequences .aı /ı 2ƒı we have that Sg;h D vol.ƒ/1

X

hh; .ı /gi .ı /;

(11)

ı 2ƒı

with absolute convergence in the operator norm of bounded operators on L2 .Rd /.

C

X

jaı j2  k

ı 2ƒı

X

ı 2ƒı

X

aı .ı /gk22  D

ı 2ƒı

jaı j2 :

(13)

G

578

Gabor Analysis and Algorithms

In fact, the corresponding condition numbers are equal, i.e., B=A D D=C . Feichtinger conjectured that any Gabor frame G.g; ƒ/ can be decomposed into a finite sum of Riesz bases. This conjecture has triggered a lot of interest, since its variant for general frames is equivalent to the Kadison-Singer conjecture [5]. The real-world applications of Gabor analysis has given a lot of impetus to implement the aforementioned results on a computer. The correct framework for these investigations are Gabor frames over finite Abelian groups [18, 19]. A natural quest is to investigate the relations between (continuous) Gabor frames for L2 .Rd / and the ones over finite Abelian groups [17, 32, 41]. There exists a large variety of MATLAB code concerning the computation of dual and tight windows and Gabor expansions in the finite setting. The corresponding algorithms make use of the positive definiteness of the Gabor frame matrix and the sparsity of the matrix, or suitable matrix factorizations. There is no space here to go into details. The best source to be referred here is the LTFAT toolbox [42] compiled and maintained by Peter Soendergaard.

11.

12.

13.

14.

15. 16.

17.

18.

19.

References 1. Allen, J.B., Rabiner, L.R.: A unified approach to shorttime Fourier analysis and synthesis. Proc. IEEE 65(11), 1558–1564 (1977) 2. Benedetto, J.J., Heil, C., Walnut, D.F.: Differentiation and the Balian-Low theorem. J. Fourier Anal. Appl. 1(4), 355–402 (1995) 3. Benedetto, J., Benedetto, R., Woodworth, J.: Optimal ambiguity functions and Weil’s exponential sum bound. J. Fourier Anal. Appl. 18(3), 471–487 (2012) 4. B¨olcskei, H., Hlawatsch, F.: Oversampled cosine modulated filter banks with perfect reconstruction. IEEE Trans. Circuits Syst. II 45(8), 1057–1071 (1998) 5. Casazza, P.G., Tremain, J.C.: The Kadison-Singer problem in mathematics and engineering. Proc. Nat. Acad. Sci. 103, 2032–2039 (2006) 6. Daubechies, I., Landau, H.J., Landau, Z.: Gabor timefrequency lattices and the Wexler-Raz identity. J. Fourier Anal. Appl. 1(4), 437–478 (1995) 7. Don, G., Muir, K., Volk, G., Walker, J.: Music: broken symmetry, geometry, and complexity. Not. Am. Math. Soc. 57(1), 30–49 (2010) 8. Evangelista, G., D¨orfler, M., Matusiak, E.: Phase vocoders with arbitrary frequency band selection. In: Proceedings of the 9th Sound and Music Computing Conference, Kopenhagen (2012) 9. Feichtinger, H.G.: On a new Segal algebra. Monatsh. Math. 92, 269–289 (1981) 10. Feichtinger, H.G.: Modulation spaces of locally compact Abelian groups. In: Radha, R., Krishna, M.,

20.

21. 22.

23.

24.

25. 26.

27.

28. 29.

Thangavelu, S. (eds.) Proceedings of the International Conference on Wavelets and Applications, Chennai, Jan 2002, pp. 1–56. Allied, New Delhi (2003) Feichtinger, H.G.: Modulation spaces: looking back and ahead. Sampl. Theory Signal Image Process. 5(2), 109–140 (2006) Feichtinger, H.G., Gr¨ochenig, K.: Banach spaces related to integrable group representations and their atomic decompositions, I. J. Funct. Anal. 86(2), 307–340 (1989) Feichtinger, H.G., Kozek, W.: Quantization of TF latticeinvariant operators on elementary LCA groups. In: Gabor Analysis and Algorithms. Theory and Applications, pp. 233–266. Birkh¨auser, Boston (1998) Feichtinger, H.G., Luef, F.: Wiener amalgam spaces for the fundamental identity of gabor analysis. Collect. Math. 57(2006), 233–253 (2006) Feichtinger, H.G., Strohmer, T.: Gabor Analysis and Algorithms. Theory and Applications. Birkh¨auser, Boston (1998) Feichtinger, H.G., Zimmermann, G.: A Banach space of test functions for Gabor analysis. In: Gabor Analysis and Algorithms. Theory and Applications, pp. 123–170. Birkh¨auser, Boston (1998) Feichtinger, H.G., Luef, F., Werther, T.: A guided tour from linear algebra to the foundations of Gabor analysis. In: Gabor and Wavelet Frames. Lecture Notes Series-Institute for Mathematical Sciences National University of Singapore, vol. 10, pp. 1–49. World Scientific, Hackensack (2007) Feichtinger, H.G., Hazewinkel, M., Kaiblinger, N., Matusiak, E., Neuhauser, M.: Metaplectic operators on Cn . Quart. J. Math. Oxf. Ser. 59(1), 15–28 (2008) Feichtinger, H.G., Kozek, W., Luef, F.: Gabor analysis over finite Abelian groups. Appl. Comput. Harmon. Anal. 26(2), 230–248 (2009) Fish, A., Gurevich, S., Hadani, R., Sayeed, A., Schwartz, O.: Delay-Doppler channel estimation with almost linear complexity. In: IEEE International Symposium on Information Theory, Cambridge (2012) Gabor, D.: Theory of communication. J. IEE 93(26), 429–457 (1946) Gr¨ochenig, K.: Foundations of time-frequency analysis. In: Applied and Numerical Harmonic Analysis. Birkh¨auser, Boston (2001) Gr¨ochenig, K., Heil, C.: Modulation spaces and pseudodifferential operators. Integr. Equ. Oper. Theory 34(4), 439–457 (1999) Gr¨ochenig, K., Leinert, M.: Wiener’s lemma for twisted convolution and Gabor frames. J. Am. Math. Soc. 17, 1–18 (2004) Gr¨ochenig, K., St¨ockler, J.: Gabor frames and totally positive functions. Duke Math. J. 162(6), 1003–1031 (2013) Gr¨ochenig, K., Han, D., Heil, C., Kutyniok, G.: The BalianLow theorem for symplectic lattices in higher dimensions. Appl. Comput. Harmon. Anal. 13(2), 169–176 (2002) Gurevich, S., Hadani, R., Sochen, N.: The finite harmonic oscillator and its applications to sequences, communication, and radar. IEEE Trans. Inf. Theory 54(9), 4239–4253 (2008) Heil, C.: History and evolution of the density theorem for Gabor frames. J. Fourier Anal. Appl. 13(2), 113–166 (2007) Hrycak, T., Das, S., Matz, G., Feichtinger, H.G.: Practical estimation of rapidly varying channels for OFDM systems. IEEE Trans. Commun. 59(11), 3040–3048 (2011)

Galerkin Methods 30. Janssen, A.J.E.M.: Duality and biorthogonality for Weyl-Heisenberg frames. J. Fourier Anal. Appl. 1(4), 403–436 (1995) 31. Janssen, A.J.E.M., Strohmer, T.: Hyperbolic secants yield Gabor frames. Appl. Comput. Harmon. Anal. 12(2), 259–267 (2002) 32. Kaiblinger. N.: Approximation of the Fourier transform and the dual Gabor window. J. Fourier Anal. Appl. 11(1), 25–42 (2005) 33. Laback, B., Balazs, P., Necciari, T., Savel, S., Ystad, S., Meunier, S., Kronland Martinet, R.: Additivity of auditory masking for short Gaussian-shaped sinusoids. J. Acoust. Soc. Am. 129, 888–897 (2012) 34. Luef, F.: Projective modules over noncommutative tori are multi-window Gabor frames for modulation spaces. J. Funct. Anal. 257(6), 1921–1946 (2009) 35. Luef, F., Manin, Y.I.: Quantum theta functions and Gabor frames for modulation spaces. Lett. Math. Phys. 88(1–3), 131–161 (2009) 36. Lyubarskii, Y.I.: Frames in the Bargmann space of entire functions. In: Entire and Subharmonic Functions. Volume 11 of Adv. Sov. Math., pp. 167–180. AMS, Philadelphia (1992) 37. Morgenshtern, V., Riegler, E., Yang, W., Durisi, G., Lin, S., Sturmfels, B., B¨olcskei, H.: Capacity pre-log of noncoherent SIMO channels via Hironaka’s theorem. IEEE Trans. Inf. Theory 59(7), 4213–4229 (2013) 38. Rieffel, M.A.: Projective modules over higher-dimensional noncommutative tori. Can. J. Math. 40(2), 257–338 (1988) 39. Ron, A., Shen, Z.: Weyl-Heisenberg frames and Riesz bases in L2 .Rd /. Duke Math. J. 89(2), 237–282 (1997) 40. Seip, K.: Density theorems for sampling and interpolation in the Bargmann-Fock space. I. J. Reine Angew. Math. 429, 91–106 (1992) 41. Soendergaard, P.L.: Gabor frames by sampling and periodization. Adv. Comput. Math. 27(4), 355–373 (2007) 42. Sondergaard, P.L., Torresani, B., Balazs, P.: The Linear Time Frequency Analysis Toolbox. Int. J. Wavelets Multires. Inf. Proc. 10(4), 1250032–58 (2012) 43. Strohmer, T.: Pseudodifferential operators and Banach algebras in mobile communications. Appl. Comput. Harmon. Anal. 20(2), 237–249 (2006) 44. Walnut, D.F.: Continuity properties of the Gabor frame operator. J. Math. Anal. Appl. 165(2), 479–504 (1992) 45. Wexler, J., Raz. S.: Discrete Gabor expansions. Signal Process. 21, 207–220 (1990)

Galerkin Methods Christian Wieners Karlsruhe Institute of Technology, Institute for Applied and Numerical Mathematics, Karlsruhe, Germany

579

Galerkin Methods for Elliptic Variational Problems The solution of a partial differential equation can be characterized by a variational problem. This allows for weak solutions in Hilbert or Banach spaces and for numerical approximations by Galerkin methods in subspaces of finite dimensions. The general procedure can be illustrated for a typical example, the Poisson problem: u D F in a domain   RD subject to the boundary values u D 0 on @. Integration by parts yields Z

Z ru  rv dx D 

F v dx : 

for all test functions v with vanishing boundary values. This weak variational problem is the basis for a Galerkin method, where the approximate solution and the test functions are chosen in the same discrete space. We now consider abstract variational problems. Let V be a Hilbert space with norm k  kV , and let aW V  V ! R be a bounded and elliptic bilinear form, i.e., positive constants C  ˛ > 0 exist such that ja.v; w/j  C kvkV kwkV and a.v; v/  ˛kvk2V . Then, for every functional f in the dual space V 0 of V , a unique solution u 2 V of the variational problem a.u; v/ D hf; vi for all v 2 V exists, satisfying kukV  ˛ 1 kf kV 0 . Let VN  V be a discrete subspace of finite dimension N . Then, a unique Galerkin approximation uN 2 VN of u exists, solving the discrete variational problem a.uN ; vN / D hf; vN i for all vN 2 VN and also satisfying kuN kV  ˛ 1 kf kV 0 , i.e., the discrete solution operator is stable independent of the choice of VN  V . Depending on a basis 1 ; : : : ; N of VN , N P an explicit representation uN D un n is obtained nD1

Mathematics Subject Classification

by solving the linear system A u D f in RN with   the positive definite matrix A D a.n ; m / m;nD1;:::;N   and the right-hand side f D hf; n i .

65N30; 65M60

The main property of the Galerkin approximation uN of u is the Galerkin orthogonality

nD1;:::;N

G

580

Galerkin Methods

a.u  uN ; vN / D 0 for all vN 2 VN : This yields directly ˛ ku  uN k2V  a .u  uN ; u  uN / D a .u  uN ; u/

The most important Galerkin method is the finite element method for elliptic boundary value problems, where a local basis associated to a triangulation is constructed. Then, the system matrix A is sparse with only O.N / nonzero entries.

D a .u  uN ; u  vN /  C ku  uN kV ku  vN kV ; and thus, ku  uN kV 

C ˛

inf ku  vN kV

vN 2VN

(Cea’s Lemma) :

Hence, up to a constant, the Galerkin approximation uN is the best possible approximation in VN of the solution u 2 V . As a consequence, the approximation error can be estimated using an interpolation operator IN W V ! VN , i.e., infvN 2VN kuvN kV  kuIN ukV , which in general can be bounded without solving a linear system but depending on the regularity of the solution. For a dense family .VN /N 2N of subspaces, Cea’s Lemma guarantees convergence to the solution for every f 2 V 0 . Convergence rates and uniform convergence of the solution operator require regularity and compactness properties. In addition, the Galerkin orthogonality provides a general approach to control the approximation error. Let  2 V 0 be any functional measuring a quantity of interest. Now, consider the solution z 2 V of the dual problem a.v; z/ D h; vi for all v 2 V and its Galerkin approximation zN 2 VN determined by a.vN ; zN / D h; vN i for all vN 2 VN . Then,

Nonconforming Galerkin Methods Often it is advantageous to use nonconforming approximations VN 6 V and discrete bilinear forms aN W VN  VN ! R. Then, the discrete solution uN 2 VN is defined by aN .uN ; vN / D hf; vN i for all vN 2 VN . Provided that the discrete bilinear forms are uniformly elliptic, again existence and stability of the discrete solution is guaranteed, and together with suitable consistency properties, convergence can be shown (extending Cea’s Lemma to the First and Second Strang Lemma). Nonconforming methods are more flexible in cases where the approximation of V requires high regularity (e.g., for the plate equation) or contains constraints (such as for many flow problems). In addition, they may improve the robustness of the method with respect to problem parameters, e.g., for nearly incompressible materials. Widely used nonconforming methods are discontinuous Galerkin finite elements for elliptic, parabolic, and hyperbolic problems using independent polynomial ansatz spaces in each element and suitable modifications of the bilinear form in order to ensure approximate continuity.

Ritz-Galerkin Methods h; u  uN i D a.u  uN ; z/ D a.u  uN ; z  zN / D a.u; z  zN / D hf; z  zN i

If the bilinear form is symmetric, the solution of the variational problem is also the minimizer of the functional J.v/ D 12 a.v; v/  hf; vi, and the Galerkin approximation is the minimizer of the restricted functional J jVN . More general, for a given functional J W V ! R, Ritz-Galerkin methods aim for minimizing the restriction J jVN . In the special case  that J is uniformly convex, i.e., J 12 .v C w/ C ˛8 kv C wk2V  12 J.v/ C 12 J.w/, a unique minimizer uN 2 VN exists, and we have

represents the error jh; u  uN ij in terms of the data f and the dual solutions. In combination with regularity properties and a priori bounds for the dual solution, this yields a priori estimates. For example, L2 error estimates for elliptic problems in H 1 are obtained by this method. Moreover, a posteriori error estimators can be constructed directly from the discrete dual solution zN , such as weighted residual estimates or more general goal  ˛ kuM  uN k2V 4 J.uM /C4 J.uN /  8 J 12 .uM C uN / : oriented error estimators.

Gas Dynamics Equations: Computation

For a dense family .VN /N 2N , the right-hand side vanishes in the limit, so that the discrete minimizers .uN /N 2N are a Cauchy sequence in V converging to u 2 V satisfying J.u/ D infv2V J.v/. Thus, the Galerkin approach is also a constructive method to prove the existence of solutions.

Generalizations and Limitations In principle, for any nonlinear function ˆW V ! V 0 , the solution u of the equation ˆ.u/ D 0 can be approximated by the discrete problem hˆ.uN /; vN i D 0 for vN 2 VN . A further application is Galerkin methods for time discretizations. Nevertheless, for nonsymmetric or indefinite problems, Galerkin methods may be not the optimal choice, and additional properties of the ansatz space VN are required to ensure stability. In many cases a different test space (which is the PetrovGalerkin method) or a least squares approach to minimize kˆ.uN /k in a suitable norm is more appropriate for non-elliptic problems.

581

main motivations for numerical computation for the gas dynamics equations. The fundamental equations governing the dynamics of gases are the compressible Euler equations, consisting of conservation laws of mass, momentum, and energy: 8 ˆ 1 and 0 > 0. These systems fit into the general form of hypertrical, radiation, or mechanical energy in a limited bolic conservation laws: space. Tracking these discontinuities and their interactions, especially when and where new waves arise and interact in the motion of gases, is one of the u 2 IR m ; x 2 IR d ; (4) @t u C r  f.u/ D 0;

G

582

where f W IR m ! .R I m /d is a nonlinear mapping. Besides (1) and (3), most of partial differential equations arising from physical or engineering science can be also formulated into form (4), or its variants, for example, with additional source terms or equations modeling physical effects such as dissipation, relaxation, memory, damping, dispersion, and magnetization. Hyperbolicity of system (4) requires that, for all  2 S d 1 , the matrix .  rf.u//mm have m real eigenvalues j .u; /; j D 1; 2;    ; m; and be diagonalizable. The main difficulty in calculating fluid flows with discontinuities is that it is very hard to predict, even in the process of a flow calculation, when and where new discontinuities arise and interact. Moreover, tracking the discontinuities, especially their interactions, is numerically burdensome (see [1, 6, 12, 16]). One of the efficient numerical approaches is shock capturing algorithms. Modern numerical ideas of shock capturing for computational fluid dynamics can date back to 1944 when von Neumann first proposed a new numerical method, a centered difference scheme, to treat the hydrodynamical shock problem, for which numerical calculations showed oscillations on mesh scale (see Lax [15]). von Neumann’s dream of capturing shocks was first realized when von Neumann and Richtmyer [27] in 1950 introduced the ingenious idea of adding a numerical viscous term of the same size as the truncation error into the hydrodynamic equations. Their numerical viscosity guarantees that the scheme is consistent with the Clausius inequality, i.e., the entropy inequality. The shock jump conditions, the Rankine-Hugoniot jump conditions, are satisfied, provided that the Euler equations of gas dynamics are discretized in conservation form. Then oscillations were eliminated by the judicious use of the artificial viscosity; solutions constructed by this method converge uniformly, except in a neighborhood of shocks where they remain bounded and are spread out over a few mesh intervals. Related analytical ideas of shock capturing, vanishing viscosity methods, are quite old. For example, there are some hints about the idea of regarding inviscid gases as viscous gases with vanishingly small viscosity in the seminal paper by Stokes [23], as well as the important contributions of Rankine [20], Hugoniot

Gas Dynamics Equations: Computation

[13], and Rayleigh [21]. See Dafermos [6] for the details. The main challenge in designing shock capturing numerical algorithms is that weak solutions are not unique; and the numerical schemes should be consistent with the Clausius inequality, the entropy inequality. Excellent numerical schemes should also be numerically simple, robust, fast, and low cost, and have sharp oscillation-free resolutions and high accuracy in domains where the solution is smooth. It is also desirable that the schemes capture vortex sheets, vorticity waves, and entropy waves, and are coordinate invariant, among others. For the one-dimensional case, examples of success include the Lax-Friedrichs scheme (1954), the Glimm scheme (1965), the Godunov scheme (1959) and related high order schemes; for example, van Leer’s MUSCL (1981), Colella-Wooward’s PPM (1984), Harten-Engquist-Osher-Chakravarthy’s ENO (1987), the more recent WENO (1994, 1996), and the Lax-Wendroff scheme (1960) and its two-step version, the Richtmyer scheme (1967) and the MacCormick scheme (1969). See [3, 4, 6, 8, 11, 17, 24, 25] and the references cited therein. For the multi-dimensional case, one direct approach is to generalize directly the one-dimensional methods to solve multi-dimensional problems; such an approach has led several useful numerical methods including semi-discrete methods and Strang’s dimensiondimension splitting methods. Observe that multi-dimensional effects do play a significant role in the behavior of the solution locally, and the approach that only solves one-dimensional Riemann problems in the coordinate directions clearly lacks the use of all the multi-dimensional information. The development of fully multidimensional methods requires a good mathematical theory to understand the multi-dimensional behavior of entropy solutions; current efforts in this direction include using more information about the multidimensional behavior of solutions, determining the direction of primary wave propagation and employing wave propagation in other directions, and using transport techniques, upwind techniques, finite volume techniques, relaxation techniques, and kinetic techniques from the microscopic level.

Gauss Methods

583

See [2, 14, 18, 24]. Also see [8, 10, 11, 17, 25] and the 16. Lax, P.D.: Mathematics and computing. In: Arnold, V.I., Atiyah, M., Lax, P. and Mazur, B. (ed.) Mathematics: references cited therein. Frontiers and Perspectives, pp. 417–432. American MathOther useful methods to calculate sharp fronts for ematical Society, Providence (2000) gas dynamics equations include front-tracking algo- 17. LeVeque, R.J.: Finite Volume Methods for Hyperbolic Problems. Cambridge University Press, Cambridge (2002) rithms [5, 9], level set methods [19, 22], among others.

References 1. Bressan, A., Chen, G.-Q., Lewicka, M., Wang, D.: Nonlinear Conservation Laws and Applications. IMA Volume 153 in Mathematics and Its Applications. Springer, New York (2011) 2. Chang, T., Chen, G.-Q., Yang, S.: On the Riemann problem for two-dimensional Euler equations I: interaction of shocks and rarefaction waves. Discret. Contin. Dyn. Syst. 1, 555–584 (1995) 3. Chen, G.-Q., Liu, J.-G.: Convergence of difference schemes with high resolution for conservation laws. Math. Comput. 66, 1027–1053 (1997) 4. Chen, G.-Q., Toro, E.F.: Centered difference schemes for nonlinear hyperbolic equations. J. Hyperbolic Differ. Equ. 1, 531–566 (2004) 5. Chern, I.-L., Glimm, J., McBryan, O., Plohr, B., Yaniv, S.: Front tracking for gas dynamics. J. Comput. Phys. 62, 83–110 (1986) 6. Dafermos, C.M.: Hyperbolic Conservation Laws in Continuum Physics, 3rd edn. Springer, Berlin/Heidelberg/New York (2010) 7. Ding, X., Chen, G.-Q., Luo, P.: Convergence of the fractional step Lax-Friedrichs scheme and Godunov scheme for isentropic gas dynamics. Commun. Math. Phys. 121, 63–84 (1989) 8. Fey, M., Jeltsch, R.: Hyperbolic Problems: Theory, Numerics, Applications, I, II. International Series of Numerical Mathematics, vol. 130. Birkh¨auser, Basel (1999) 9. Glimm, J., Klingenberg, C., McBryan, O., Plohr, B., Sharp, D., Yaniv, S.: Front tracking and two-dimensional Riemann problems. Adv. Appl. Math. 6, 259–290 (1985) 10. Glimm, J., Majda, A.: Multidimensional Hyperbolic Problems and Computations. IMA Volumes in Mathematics and Its Applications, vol. 29. Springer, New York (1991) 11. Godlewski, E., Raviart, P.: Numerical Approximation of Hyperbolic Systems of Conservation Laws. Springer, New York (1996) 12. Holden, H., Risebro, N.H.: Front Tracking for Hyperbolic Conservation Laws. Springer, New York (2002) 13. Hugoniot, H.: Sur la propagation du mouvement dans les corps et sp´ecialement dans les gaz parfaits, I;II. J. Ecole Polytechnique 57, 3–97 (1887); 58, 1–125 (1889) 14. Kurganov, A., Tadmor, E.: Solution of two-dimensional Riemann problems for gas dynamics without Riemann problem solvers. Numer. Methods Partial Diff. Equ. 18, 584–608 (2002) 15. Lax, P.D.: On dispersive difference schemes. Phys. D 18, 250–254 (1986)

18. Liu, X.D., Lax, P.D.: Positive schemes for solving multidimensional hyperbolic systems of conservation laws. J. Comput. Fluid Dyn. 5, 133–156 (1996) 19. Osher, S., Fedkiw, R.: Level Set Methods and Dynamic Implicit Surfaces. Springer, New York (2003) 20. Rankine, W.J.M.: On the thermodynamic theory of waves of finite longitudinal disturbance. Phil. Trans. Royal Soc. London, 160, 277–288 (1870) 21. Rayleigh, Lord J.W.S.: Aerial plane waves of finite amplitude. Proc. Royal Soc. London, 84A, 247–284 (1910) 22. Sethian, J.A.: Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational Geometry, Fluid Mechanics, Computer Vision, and Materials Science, 2nd edn. Cambridge University Press, Cambridge (1999) 23. Stokes, G.G.: On a difficulty in the theory of sound. Philos. Magazine, Ser. 3, 33, 349–356 (1948) 24. Tadmor, E., Liu, J.G., Tzavaras, A.: Hyperbolic Problems: Theory, Numerics and Applications, Parts I, II. American Mathematical Society, Providence (2009) 25. Toro, E.: Riemann Solvers and Numerical Methods for Fluid Dynamics: A Practical Introduction, 3rd edn. Springer, Berlin (2009) 26. von Neumann, J.: Proposal and analysis of a new numerical method in the treatment of hydrodynamical shock problem, vol. VI. In: Collected Works, pp. 361–379, Pergamon, London (1963) 27. von Neumann, J., Richtmyer, R.D.: A method for the numerical calculation of hydrodynamical shocks. J. Appl. Phys. 21, 380–385 (1950)

Gauss Methods John C. Butcher Department of Mathematics, University of Auckland, Auckland, New Zealand

Introduction Explicit s stage Runge–Kutta methods for the numerical solution of a differential equation system y 0 .x/ D f .x; y/

G

584

Gauss Methods

are characterized by a tableau 0 c2 c A D c3 :: bT :

a21 a31 :: :

the numerical solution produced is equivalent to the quadrature approximation Z

a32 :: : : : :

;

1 0

'.x/dx  '. 12 /:

This is the Gauss-Legendre quadrature formula of order 2 and it is natural to ask what happens if it is attempted to construct a two-stage method based on the where the strictly lower-triangular form of A indicates order 4 Gauss-Legendre formula on the interval Œ0; 1 . that the stages are evaluated in sequence using the This defines b > and c and it turns out that there is equations a unique choice of A which makes the Runge–Kutta method also of order 4. The tableau for this is i 1 X p p Yi D y0 C h aij f .x0 C hcj ; Yj /; 1 1 1  p63  63 2 4 4 p j D1 1 1 C 63 14 C 63 : (3) 2 4 i D 1; 2; : : : ; s; (1) b1 b2    bs

y1 D y0 C h

s X

1 2

bi f .x0 C hci ; Yi /:

1 2

(2)

This method was discovered by Hammer and Hollingsworth [7] and, at first sight, it might seem to be Pi 1 It is also possible that A is a full matrix, so that j D1 is a curiosity. However, it is simply the first instance, after P replaced by sj D1 . For example, the implicit midpoint the midpoint rule, of the family of implicit methods based on Gauss-Legendre quadrature [2, 9]. rule method These methods have a role in the numerical solution 1 1 2 2 of stiff problems but they have major disadvantages be1 cause of the possibility of order reduction and because they are expensive to implement. Both these issues has order 2. To actually evaluate the stage value Y1 , it will be discussed later. However, they have come into is necessary to solve the nonlinear equation prominence in recent years as symplectic integrators.   h h Y1 D y0 C f x0 C ; Y1 : 2 2 i D1

Existence of Methods

For a non-stiff problem, this nonlinear algebraic equation can be solved using functional iteration but, if the Jacobian matrix  @fi J D @yj

To see why there exists an s stage method based on Gaussian quadrature with order 2s and that this method is unique up to a permutation of the abscissae, we consider the set of order conditions and the socalled simplifying assumptions associated with these has eigenvalues with very large amplitudes, such as conditions. The conditions for a Runge–Kutta method arise in stiff problems, functional iteration would not to have order p are given by the equation converge, except with inappropriately small stepsizes. However, Newton’s method, or some variant of this, 1 ; ˚.t/ D can be used to calculate Y1 and the method becomes .t/ practical and efficient for many problems. If, instead of a genuine differential equation, the for every rooted tree t satisfying r.t/  p [1] ( Order method is used to solve the quadrature problem Conditions and Order Barriers). The statement that this y 0 .x/ D '.x/;

y.x0 / D 0;

equation holds will be denoted by G.p/. The simplifying assumptions relevant to this family of methods are

Gauss Methods

585

s X

1 bi cik1 D ; k iD1

B./ W

k D 1; 2; : : : ; ;

s X

1 aij cjk1 D cik ; i D 1; 2; : : : ; sI k D 1; 2; : : : ; k j D1

C./ W

D./ W

s X iD1

1 bi cik1 aij D bj .1  cjk /; k j D 1; 2; : : : ; sI k D 1; 2; : : : ;

E.; / W

s X

bi cik1 aij cjl1 D

i;j D1

1 ; .k C l/l

1 2 1 2



p

15 10

1 2p

C

5 36 15 5 10 36

p 15 5 15  15 36 p30 15 15 2 5 C p24 9p 36  24 5 C 3015 29 C 1515 36 5 4 5 18 9 18 5 36 p

2 9



p

:

It is known that B.s/ and C.s/ hold if and only if a Runge–Kutta method is, at the same time, a collocation method. Since these conditions hold in the case of Gauss methods, they are necessarily collocation methods [8].

kD 1; 2; : : : I l D 1; 2; : : : :

It is a classical result of numerical quadrature that B.2s/ implies that the members of abscissae vector c are the zeros of the shifted Legendre polynomial Ps .2x  1/ and this property, together with B.s/ implies B.2s/. In addition to this remark, we can collect together several other connections between B; C; D; E; and G as follows: Theorem 1 For any positive integer s, B.2s/ ^ C.s/ H) E.s; s/; B.2s/ ^ E.s; s/ H) C.s/; B.2s/ ^ D.s/ H) E.s; s/; B.2s/ ^ E.s; s/ H) D.s/; G.2s/ H) B.2s/; G.2s/ H) E.s; s/; B.2s/ ^ C.s/ ^ D.s/ H) G.2s/: The corollary of this is Theorem 2 For every positive integer s there exists a unique method of order 2s and this can be constructed by requiring B.2s/ and C.s/ to hold. In the Hammer and Hollingsworth method (3), the construction of the tableau starts with the shifted second degree Legendre polynomial 6x 2  6x C 1, with p 1 1 zeros 2 ˙ 6 3. This gives the c values and b1 ; b2 are then found from b1 C b2 D 1; b1 c1 C b2 c2 D 12 . Finally the rows of A are found from ai1 C ai 2 D ci ; ai1 c1 C ai 2 c2 D 12 ci2 ; i D 1; 2. The sixth order method with s D 3 is constructed in a similar way, starting with the third order shifted Legendre 20x 3  30xo2 C 12x  1, with n polynomial p p 1 1 15; 12 ; 12 C 10 15 : zeros 12  10

G Properties of Coefficients Existence of Gaussian Quadrature The question has been glossed over of the existence of b > and c satisfying B.2s/. The existence of the shifted Legendre polynomial of degree s is clear because the sequence can be constructed by the Gram-Schmidt process. Denote Ps .2x  1/ simply as P .x/. This polynomial has s distinct zeros in .0; 1/ because if it did not then there would exist a factorization P D QR such that deg.Q/ < s Rand R has a constant sign in 1 Œ0; 1 . By orthogonality 0 Q.x/2 R.x/dx D 0, which is impossible because the sign of the integrand does not change. Choose ci , i D 1; 2; : : : ; s as the zeros of P and, given a polynomial ' of degree 2s  1, divide by P and write the quotient and remainder as Q and R. Hence, Z

1

P .x/Q.x/dx  0

s X i D1

Z

1

C

R.x/dx  0

s X

 bi P .ci /Q.ci /  bi R.ci / D 0:

(4)

i D1

Choose bi , i D 1; 2; : : : ; s so that B.s/ holds. However, B.2s/ also holds because the first term of (4) is always zero. Location of ci It has already been noted that each c component lies between 0 and 1. As a convention, they will be written in increasing order 0 < c1 < c2 <    < cs < 1. They are also symmetrically placed in this interval.

586

Gauss Methods

Theorem 3 In a Gauss Runge–Kutta method ci D 1  csC1i ;

Gauss Methods, Table 1 Stability functions for Gauss methods

i D 1; 2; : : : ; s:

s

R.z/

R1 Ps 1 Proof In the formula 0 '.x/dx D i D1 bi '.ci / for deg.'/  s  1, replace '.x/ by '.1  x/ and a quadrature formula is found in which ci is replaced by 2 1  ci . Hence the uniqueness of the Gaussian quadrature formula gives the result. 3

1 C 12 z

Signs and Symmetry of bi Because the components of b > satisfy B.2s/, they also satisfy

1C

s X

Z

4

1  12 z 1 C 12 z C 1 1C 1 1

1 zC 2 1 zC 2 1 zC 2 1 zC 2 1 zC 2

1 2 z 12 1 2 z 12 1 2 z 10 1 2 z 10 3 2 z 28 3 2 z 28

C  C 

1 3 z 120 1 3 z 120 1 3 1 z C 1680 z4 84 1 3 1 z C 1680 z4 84

1

bi '.ci / D

'.x/dx;

deg.'/  2s  1:

Because the factor R.z/ determines the growth factor in each step of the numerical computation, its magnitude (5) determines the stability of the sequence yn , n D Theorem 4 The coefficients bi , i D 1; 2; : : : ; s sat- 0; 1; 2; : : :. The set of z values in the complex plane for isfy which jR.z/j  1 is defined to be the “stability region” for the method. Furthermore, (6) bi > 0; Definition 1 A Runge–Kutta method .A; b > ; c/ is “Abi D b1Csi : (7) stable” if R.z/, given by (9), satisfies jR.z/j  1, whenever Re.z/  0. P Proof In the proof of Theorem 3, siD1 bi '.ci / D P Ps A-stable methods have a central role in the numerb '.1ci / for any '. Therefore, siD1 bi '.ci / D PisD1 i ical solution of stiff problems and Gauss methods are (7) follows. To prove (6), i D1 b1Csi '.ci / and  2 likely candidates. The stability functions up to s D 4 substitute '.x/ D P .x/=.x  ci / into (5). This are shown in Table 1. gives Each of the stability functions displayed in this table Z 1 can be shown to be A-stable by a simple argument. P .x/ 2 dx > 0: bi D P 0 .ci /2 The poles are located in the right half-plane in each x  ci 0 case, so that R.z/ is analytic in the left half-plane and therefore jR.z/j is bounded by its maximum value on the imaginary axis, and this maximum value is 1. The Stability and Symplecticity general result suggested by these observations can be When the differential equation (the “linear test prob- stated here but its proof will be delayed. lem”) Theorem 5 For every s D 1; 2; : : :, the Gauss y.x0 / D y0 ; (8) y 0 D qy; method with s stages is A-stable. is solved using a Runge–Kutta method .A; b > ; c/ with stepsize h, the solution after a single step is written as AN-Stability R.hq/, where R.z/ satisfies Instead of the constant coefficient linear test problem (8), we consider the possibility that q is time depenY D y0 1 C hAqY D y0 1 C zAY; dent: > > R.z/y0 D y0 C hb qY D y0 C zb Y: y 0 D q.x/y; y.x0 / D y0 : (10) i D1

0

This gives R.z/ D 1 C zb > .I  zA/1 1:

If the real part of q.x/ is always nonpositive, then the exact solution is bounded and we consider the discrete (9) counterpart to this.

Gauss Methods

587

Definition 2 A Runge–Kutta method .A; b > ; c/ is “AN-stable” if y1 , the solution computed by this method after a single step satisfies jy1 j  jy0 j if Re.q.x//  0.

B-, BN- and Algebraic Stability The idea of nonlinear stability (G-stability) was introduced [5] in the context of linear multistep and oneleg methods. Subsequently B-stability (for autonomous problems) and BN-stability (for non-autonomous probAssuming that the ci are distinct, the criterion for lems) were introduced. In each case the linear test this property depends on a generalization of R: problem (8) or (10) is replaced by the nonlinear problem y 0 D f .y/ or y 0 D f .x; y/ on an inner product > 1 e R.Z/ D 1 C b Z.I  AZ/ 1; space where Z D diag.z1 ; z2 ; : : : ; zs /: (11) hY; f .Y /i  0 or hY; f .X; Y /i  0: e Theorem 6 If jR.Z/j  1 whenever z1 ; z2 ; : : : ; zs lie in the left half-plane, the method is AN-stable. In either the autonomous or the non-autonomous case, the exact solution is contractive; for example in the autonomous case An Identity on Method Coefficients For a Runge–Kutta method .A; b > ; c/, applied to a d 2 differential equation y 0 D f .x; y/ on an inner-product dx ky.x/k D 2hy.x/; f .y.x//i  0: space, we establish a relationship between values of hYi ; Fi i, where Fi D f .x0 C hci ; Yi /, and a specific For the numerical approximation, the corresponding matrix property is M D diag.b/A C A> diag.b/  bb > ;

kyn k  kyn1 k: (12)

For a Runge–Kutta method, this is referred to as B-stability (autonomous case) or BN-stability (non bi bj , i; j D autonomous case) and is related to the following property:

with elements mij D bi aij C bj aj i 1; 2; : : : ; s. Theorem 7

Definition 3 A Runge–Kutta method .A; b > ; c/ is “als s gebraically stable” if bi  0, i D 1; 2; : : : ; s and M X X ky1 k2 ky0 k2 D 2h bi hYi ; Fi i  h2 mij hFi ; Fj i given by (12) is positive semi-definite. i D1

i;j D1

The various stability concepts are closely related:

Proof Evaluate hy1 ; y1 i  hy0 ; y0 i using (2) and Theorem 8 For a Runge–Kutta method .A; b > ; c/ for bi .hYi ; Fi i C hFi ; Yi i/ using (1). Combining these which the ci are distinct, the following implications results we find hold ky1 k2  ky0 k2  2h

s X

bi hYi ; Fi i

i D1 s s s D X E DX E X D 2h y0 ; bi Fi C h2 bi Fi ; bi Fi i D1

 2h

D h2

i D1

s X

s D E X bi y0 C h aij Fj ; Fi

i D1

j D1

s X

.bi bj  bi aij  bj aj i /hFi ; Fj i

i;j D1

D h2

i D1

s X i;j D1

AN-stability ” BN-stability ” algebraic stability + + A-stability B-stability Proof Proofs will be given only for BN-stability ) AN-stability and AN-stability ) algebraic stability. BN-stability ) AN-stability: Consider the two-dimensional differential equation system   Re q.x/ y .x/ D   Im q.x/ "

0

 # Im q.x/   y.x/ Re q.x/

mij hFi ; Fj i: AN-stability ) algebraic stability:

G

588

Gauss Methods

To prove bi  0, choose Zi D t, where t is a small positive parameter and Zj D 0 (j ¤ i ). Substitute e into (11) and it is found that R.Z/ D 1  bi t C O.t 2 /. e Since jR.Z/j  1 for small t, it follows that bi  0. Let x denote a vector in Rs . To show that x > M x  0, substitute Z D tPı diag.x/ in (11). The result is

to any quadratic invariant and also to the symplectic property of Hamiltonian problems. Methods with the property that M D 0 are referred to as symplectic integrators and Gauss methods fit clearly into this category.

e R.Z/ D 1 C tPıb > diag.x/1  t 2 b > diag.x/Adiag.x/1 C O.t 3 /

Miscellaneous Questions

D 1 C tPıx > b  t 2 x > diag.b/Ax C O.t 3 /;

Even Order Expansions For a given autonomous problem and given Runge– 2 e so that jR.Z/j D 1 C t 2 C O.t 3 /, where Kutta method, let ˚h denote the operation of moving from one step value to the next, that is, yn D .˚h /n y0 .  D x > bb > x  x > diag.b/Ax  x > Adiag.b/x There is a special interest in methods, such as the Gauss > methods, for which ˚h ˚h D id because it will then D x M x follow that .˚h /n .˚h /n D id or .˚h /n D .˚h /n and this cannot be positive because 1 C t 2 cannot and the Taylor series expansion of the computed result, at a specific output point, will contain only even powers exceed 1 when t is small. of h. This observation makes it possible to speed up the use of extrapolation to increase the accuracy of Stability of Gauss Methods computed results. To apply the various stability requirements to the case of Gauss methods we first introduce the result: Theorem 9 All Gauss methods are algebraically sta- Implementation If the eigenvalues of A are real, it is possible to ble. incorporate transformations into the implementation Proof Each component of b > is positive from (6). process [3], and thus increase efficiency, at least for j 1 Let C denote the matrix with .i; j / element ci . large problems. However, for Gauss methods, it is Because the ci are distinct, C is non-singular. Hence, known that A has at most one real eigenvalue so that M will be positive semi-definite if and only if C > M C this technique cannot be applied in a straightforward has this same property. The .k; l/ element of C > M C manner. A similar difficulty exists with Radau IIA is found to be methods and a satisfactory solution to the implementation question is used in the code RADAU [6]. s X   l1 k1 bi aij C bj aj i  bi bj cj ci i;j D1 Order Reduction s s The order of a numerical method is not a complete   X X D cik1 bi . 1l cil / C bj k1 cjk cjl1  kl1 guide to its behaviour of either the error generated in i D1 j D1 a single step or the accumulated effect of these local errors. Asymptotically, that is for small values of h, 1 1 1 D l.kCl/ C k.kCl/  kl the local error is C hpC1 , where C depends on the D0 particular problem as well as the method. The value of p is thus a guide to how rapidly errors reduce as a Proof of Theorem 5 consequence of a reduction in h. However for many We can now complete the proof that Gauss methods are methods, including Gauss methods, this asymptotic A-stable by combining Theorem 9 with Theorem 8. behaviour is not observed for moderate ranges of the stepsize, such as those that might be used in practical Symplectic Integration computations. The “reduced order” for Gauss methods When M is the zero matrix and hY; f .X; Y /i D 0, we is typically more like s, rather than 2s. An analysis of see from Theorem 7 that yn is constant. This applies this phenomenon, based on test problems of the form

General Linear Methods

589

y 0 D L.y  g.x// C g 0 .x/, where g is a smooth f W Rm ! Rm , we consider a class of general linear methods (GLMs) defined by differentiable function and L 0, is given in [10]. 8 s r X X ˆ Œn1 ˆ Y ˆ D h a f .Y / C uij yj ; i ij j ˆ ˆ ˆ ˆ j D1 j D1 ˆ ˆ < i D 1; 2; : : : ; s;

References 1. Butcher, J.C.: Coefficients for the study of Runge–Kutta integration processes. J. Aust. Math. Soc. 3, 185–201 (1963) 2. Butcher, J.C.: Implicit Runge–Kutta processes. Math. Comput. 18, 50–64 (1964) 3. Butcher, J.C.: On the implementation of implicit Runge– Kutta methods. BIT 16, 237–240 (1976) 4. Butcher, J.C.: Numerical Methods for Ordinary Differential Equations, 2nd edn. Wiley, Chichester (2008) 5. Dahlquist, G.: Error analysis for a class of methods for stiff non–linear initial value problems. In: Watson, G.A. (ed.) Lecture Notes in Mathematics, vol. 506, pp. 60–72. Springer, Berlin/Heidelberg/New York (1976) 6. Hairer, E., Wanner, G.: Solving Ordinary Differential Equations II, Stiff Problems. Springer, Berlin/Heidelberg/New York (1996) 7. Hammer, P.C., Hollingsworth, J.W.: Trapezoidal methods of approximating solutions of differential equations. MTAC 9, 92–96 (1955) 8. Wright, K.: Some relationships between implicit RungeKutta, collocation and Lanczos methods, and their stability properties. BIT 10, 217–227 (1970) 9. Kuntzmann, J.: Neuere Entwicklungen der Methoden von Runge und Kutta. Z. Angew. Math. Mech. 41, T28–T31 (1961) 10. Prothero, A., Robinson, A.: On the stability and accuracy of one-step methods for solving stiff systems of ordinary differential equations. Math. Comput. 28, 145–162 (1974) 11. Sanz-Serna, J.M., Calvo, M.P.: Numerical Hamiltonian Problems. Chapman and Hall, London (1994)

General Linear Methods Zdzisław Jackiewicz Department of Mathematics and Statistics, Arizona State University, Tempe, AZ, USA

Synonyms General linear methods for ordinary differential equations; GLMs for ODEs

Introduction To approximate the solution y to an initial value problem for a system of ordinary differential equations (ODEs)   y 0 .t/ D f y.t/ ; t 2 Œt0 ; T ; y.t0 / D y0 ; (1)

s r ˆ X X ˆ Œn Œn1 ˆ ˆ yi D h bij f .Yj / C vij yj ; ˆ ˆ ˆ ˆ j D1 j D1 ˆ : i D 1; 2; : : : ; r;

(2)

n D 1; 2; : : : ; N . Here, N is a positive integer, h D .T t0 /=N is a fixed stepsize, Yi is an approximation of Œn stage order q to y.tn1 Cci h/, tn D t0 Cnh, and yi is an approximation of order p to the linear combination Œn of scaled derivatives of the solution y to (1), i.e., yi satisfy the relations yi D qi;0 y.tn / C qi;1 hy 0 .tn / C    C qi;p hp y .p/ .tn / Œn

CO.hpC1 /; i D 1; 2; : : : ; r, with some scalars qi;j . Such methods are characterized by the abscissa vector c D Œc1 ; : : : ; cs T , four coefficient matrices



 A D aij 2 Rss ; U D uij 2 Rsr ;



 B D bij 2 Rrs ; V D vij 2 Rrr ; the vectors q0 ; q1 ; : : : ; qp given by q0 D Œq1;0    qr;0 T ;

q1 D Œq1;1    qr;1 T ;

:::;

qp D Œq1;p    qr;p ; T

and four integers: p – the order, q – the stage order, r – the number of external approximations, and s – the number of stages or internal approximations. The GLMs are discussed in [1, 3, 11, 12, 14, 15]. They include as special cases many known methods for ODEs, e.g., Runge-Kutta (RK) methods, linear multistep and predictor-corrector methods in various implementation modes, one-leg methods, extended backward differentiation formulas, two-step RungeKutta (TSRK) methods, multistep Runge-Kutta methods, various classes of peer methods, and cyclic composite methods. The representation of some of these methods as GLMs (2) is discussed in [1,3,14,15].

G

590

General Linear Methods

The GLMs (2) are usually divided into four types depending on the structure of the coefficient matrix A, which determines the implementation costs of these methods. For type 1 or type 2 methods the matrix A is lower triangular with  D 0 or  > 0 on the diagonal, respectively. Such methods are appropriate for nonstiff or stiff differential systems in a sequential computing environment. For type 3 or type 4 methods the matrix A takes the form A D diag.; : : : ; /; with  D 0 or  > 0, respectively. Such methods are appropriate for nonstiff or stiff differential systems in a parallel computing environment. The coefficient matrix V determines the zerostability properties of GLMs (2) and its form is usually chosen in advance to guarantee that this property is automatically satisfied, i.e., that the matrix V is powerbounded. The specific choices of this matrix for some classes of GLMs are discussed below. We are mainly interested in methods for which the integers p, q, r, and s are close to each other. The choice p D q D r D s, U D I, and V D evT , where I is the identity matrix of dimension s, e D Œ1; : : : ; 1 T 2 Rs , v 2 Rs , and vT e D 1, leads to the class of so-called diagonally implicit multistage integration methods (DIMSIMs) which were first introduced in [2] and further investigated in [4, 5, 15]. The choice p D q D s, r D s C 1, q0 D e1 ,: : :,qs D esC1 , where e1 ,e2 ,: : :,esC1 is the canonical basis in Rr D RsC1 , and V D e1 vT , e1 ; v 2 RsC1 , v1 D 1, leads to the Nordsieck representation of DIMSIMs, which was introduced in [9]. In this representation, the components Œn yi of the vector of external approximations satisfy Œn yi D hi 1 y .i 1/ .tn / C O.hpC1 /, i D 1; 2; : : : ; r, i.e., they are approximations of order p at t D tn of the components of the Nordsieck vector z.t; h/ defined by

T z.t; h/ D y.t/T hy 0 .t/T    hp y .p/ .t/T :

The choice of p D q, r D s D p C 1, q0 D e1 , : : :, qp D epC1 , and some additional requirements on the coefficients of the methods which guarantee that the stability function has only one nonzero eigenvalue lead to the class of so-called GLMs with inherent Runge-Kutta stability (IRKS). This interesting class of methods, which was introduced in [8], will be discussed in more detail in section “GLMs with IRKS.”

Stage Order and Order Conditions In this section, we review the conditions on the abscissa vector c, the coefficient matrices A, U, B, V, and the vectors q0 ; q1 ; : : : ; qp which guarantee that the method (2) has stage order q D p and order p. To formulate Œn1 these conditions, we assume that the components yi of the input vector y Œn1 for the step from tn1 to tn satisfy the relations Œn1

yi

D

p X

qi k hk y .k/ .tn1 /

kD0

CO.hpC1 /;

i D 1; 2; : : : ; r:

(3)

Then the method (2) has stage order q D p and order p if Yi D y.tn1 Cci h/CO.hpC1 /; Œn

yi D

p X

i D 1; 2; : : : ; s; (4)

qi k hk y .k/ .tn /CO.hpC1 /;

i D 1; 2; : : : ; r;

kD0

(5) for the same scalars qi k . Define the vector w.z/ by Pp w.z/ D kD0 qk zk : We have the following theorem. Theorem 1 ([2]) The method (2) has stage order q D p and order p , i.e., the relation (3) implies (4) and (5), if and only if e cz D zAe cz C Uw.z/ C O.zpC1 /;

(6)

e z w.z/ D zBe cz C Vw.z/ C O.zpC1 /:

(7)

This theorem is very convenient in a symbolic manipulation environment. Comparing the free terms in (6) and (7) leads to the preconsistency conditions Uq0 D e, Vq0 D q0 , where e D Œ1; : : : ; 1 T 2 Rs . The vector q0 is called the preconsistency vector. Comparing terms of the first order in (6) and (7) leads to stage consistency and consistency conditions Ae C Uq1 D c, Be C Vq1 D q0 C q1 . The vector q1 is called the consistency vector. The stage order and order conditions (6) and (7) can be used to express the matrix U in terms of c and A and the matrix V in terms of c and B. Following [14] and [15], we will illustrate this for GLMs in Nordsieck form with p D q D r D s C 1. Put

General Linear Methods

 CD

cp c2  e c 2Š pŠ

591

2 Rp.pC1/ ;

and define the matrices K D Œkij 2 R.pC1/.pC1/ and E 2 R.pC1/.pC1/ by kij D 1 if j D i C 1, kij D 0 if j ¤ i C 1, and E D exp.K/. Then we have the following result about the representation formulas for the coefficients matrices U and V.

of GLMs. For the explicit methods (types 1 and 3), the coefficients of p.w; z/ are polynomials with respect to z while for the implicit formulas (types 2 and 4) the coefficients of p.w; z/ are rational functions with respect to z, which are more difficult to deal with than in the case of explicit methods. However, substituting z D zO=.1 C O C I into M.z/ we can work instead Oz/ and A D A O z/ defined by with a modified stability matrix M.O

Theorem 2 ([14,15]) Assume that p D q D r D sC1 O 1 O and that q0 D e1 , : : :, qs D esC1 . Then U D CA C K M.Oz/ WD M.z/ D M.Oz=.1 C Oz// D V C zOB.I  zOA/ and V D E  B C K: and the corresponding stability function

Linear Stability Theory of GLMs

O z// p.w; O zO/ WD p.w; z/ D det.wIM.z// D det.wIM.O

In this section, we investigate stability properties of whose coefficients are now polynomials with respect O is strictly lower triangular as for explicit to zO since A GLMs (2) with respect to the standard test equation methods. It can be verified that p.w; z/ corresponding y 0 D y; t  0; (8) to explicit methods (types 1 and 3) and p.w; O zO/ corresponding to implicit methods (types 2 and 4) where  2 C. Applying (2)–(8) we obtain the retake the form currence relation y Œn D M.z/y Œn1 ; z D h; n D 1; 2; : : : ; where the stability matrix M.z/ is defined by p.w; z/ D ws  p1 .z/ws1 C    C .1/s1 ps1 .z/w the relation M.z/ D V C zB.I  zA/1 U: We also C.1/s ps .z/; define the stability function p.w; z/ of GLMs (2) as the characteristic polynomial of M.z/, i.e., p.w; z/ D p.w; O zO/ D ws  pO1 .Oz/ws1 C    C .1/s1 pOs1 .Oz/w det.wI  M.z//; w 2 C. The GLM (2) is said to be C.1/s pOs .Oz/; absolutely stable for given z 2 C if for that z, all roots wi D wi .z/, i D 1; 2; : : : ; r, of p.w; z/ are inside the unit circle. The region A of absolute stability of (2) is where the set of all z 2 C such that the method is absolutely pk .z/ D pk;k1 zk1 C pk;k zk C    C pk;s zs ; stable, i.e., o n A D z 2 C W jwi .z/j < 1; i D 1; 2; : : : ; r : GLM (2) is said to be A-stable if its region of absolute stability contains a negative half-plane, i.e., fz 2 C W Re.z/ < 0g  A. GLM (2) is said to be L-stable if it is A-stable and, in addition, limz!1 .M.z// D 0, where .M.z// stands for the spectral radius of the stability matrix M.z/. We will now describe the construction of GLMs (2) with some desirable stability properties such as large regions of absolute stability for explicit methods and A- and L-stability for implicit methods. We illustrate this construction for the class of DIMSIMs with p D q D r D s, U D I, V D evT , vT e D 1, but similar approaches are also applicable to other classes

pOk .Oz/ D pOk;k1 zOk1 C pOk;k zOk C    C pOk;s zOs ; k D 2; 3; : : : ; s. We are mainly interested in construction of methods with Runge-Kutta stability, i.e., methods for which stability polynomials p.w; z/ or p.w; O zO/ have only one nonzero root. For the abscissa vector c fixed in advance, this leads to the systems of nonlinear equations pk;l D 0 or pOk;l D 0; l D k  1; k; : : : ; s;

k D 2; 3; : : : ; s; (9)

with respect to the components of the abscissas of the coefficient matrices A and V. The coefficients pk;l and pOk;l can be computed by a variant of the Fourier series method described in [5,14] which leads to the formulas

G

592

General Linear Methods

pk;l

N1 X N2 1 X D .1/ wks zl p.w ; z /; N1 N2 D1 D1   k

pOk;l D .1/k

N1 X N2 1 X wks zOl p.w O  ; zO /; N1 N2 D1 D1  

where w ,  D 1; 2; : : : ; N1 , and z , zO ,  D 1; 2; : : : ; N2 are complex numbers uniformly distributed on the unit circle and N1 and N2 are sufficiently large integers. These systems (9) were solved by least squares minimization in case of types 1 and 2 DIMSIMs and methods obtained in this way are listed in [14].

GLMs with IRKS We assume throughout this section that p D q, r D s D pC1, and that q0 D e1 , : : : , qp D epC1 . The GLM (2) satisfying the preconsistency condition Ve1 D e1 is said to have IRKS if BA XB, BU XV  VX, and det.wI  V/ D wp .w  1/, where X D X.˛; ˇ/ is a doubly companion matrix defined by X D X.˛; ˇ/ 2 ˛1 ˛2 6 1 0 6 6 :: : : : D6 6 : 6 4 0 0 0

3    ˛p ˛pC1  ˇpC1 7  0 ˇp 7 7 : : :: :: 7; : :: 7 7 :: 5 : 0 ˇ2  1 ˇ1

and the relation “ ” means that the matrices are identical except possibly their first rows. The significance of this property follows from a fundamental result discovered in [8, 15]. Theorem 3 Assume that GLM (2) has IRKS. Then the resulting method has Runge-Kutta stability, i.e., its stability function assumes the form p.w; z/ D wp .w  R.z// with R.z/ D

P .z/ D e z  EzpC1 C O.zpC1 /; .1  z/pC1

where P .z/ is a polynomial of degree p C 1 and E is the error constant of the method.

It was also discovered in [15] that the parameters ˇi , i D 1; 2; : : : ; p, appearing in X D X.˛; ˇ/ correspond to the errors of the vector of external approximations y Œn . Œn

Theorem 4 ([15]) The errors of yi are given by yi D hi 1 y .i 1/ .tn /  ˇpC2i hpC1 y .pC1/ .tn / Œn

CO.hpC2 /;

i D 2; 3; : : : ; p C 1:

As demonstrated in section “Linear Stability Theory of GLMs” the construction of GLMs (2) with RungeKutta stability is quite complicated and requires the solution of large systems of nonlinear equations (9) with respect to the unknown coefficients of the methods. This is usually accomplished by a least squares minimization starting with many random initial guesses. In contrast, as demonstrated in [8, 15] GLMs of any order with IRKS can be derived using only linear operations. This can be accomplished by the algorithm which was presented in [8, 15] (see also [14]). Special case of this algorithm adapted to the explicit methods was given in [7]. Implementation issues for GLMs such as a choice of appropriate starting procedures, a local error estimation for small and large stepsizes, construction of continuous interpolants, stepsize and order changing strategies, updating the vector of external approximations, and solving systems of nonlinear equations by simplified Newton iterations for implicit methods are discussed in [4, 6, 10, 13–15].

References 1. Butcher, J.C.: The Numerical Analysis of Ordinary Differential Equations. Runge-Kutta and General Linear Methods. Wiley, Chichester/New York (1987) 2. Butcher, J.C.: Diagonally-implicit multi-stage integration methods. Appl. Numer. Math. 11, 347–363 (1993) 3. Butcher, J.C.: Numerical Methods for Ordinary Differential Equations, 2nd edn. Wiley, Chichester (2008) 4. Butcher, J.C., Jackiewicz, Z.: Implementation of diagonally implicit multistage integration methods for ordinary differential equations. SIAM J. Numer. Anal. 34, 2119–2141 (1997) 5. Butcher, J.C., Jackiewicz, Z.: Construction of high order diagonally implicit multistage integration methods for ordinary differential equations. Appl. Numer. Math. 27, 1–12 (1998) 6. Butcher, J.C., Jackiewicz, Z.: A new approach to error estimation for general linear methods. Numer. Math. 95, 487–502 (2003)

Geometry Processing 7. Butcher, J.C., Jackiewicz, Z.: Construction of general linear methods with Runge-Kutta stability properties. Numer. Algorithms 36, 53–72 (2004) 8. Butcher, J.C., Wright, W.M.: The construction of practical general linear methods. BIT 43, 695–721 (2003) 9. Butcher, J.C., Chartier, P., Jackiewicz, Z.: Nordsieck representation of DIMSIMs. Numer. Algorithms 16, 209–230 (1997) 10. Butcher, J.C., Jackiewicz, Z., Wright, W.M.: Error propagation for general linear methods for ordinary differential equations. J. Complex. 23, 560–580 (2007) 11. Hairer, E., Wanner, G.: Solving Ordinary Differential Equations II. Stiff and Differential-Algebraic Problems. Springer, Berlin/Heidelberg/New York (1996) 12. Hairer, E., Nørsett, S.P., Wanner, G.: Solving Ordinary Differential Equations I. Nonstiff Problems. Springer, Berlin/Heidelberg/New York (1993) 13. Jackiewicz, Z.: Implementation of DIMSIMs for stiff differential systems. Appl. Numer. Math. 42, 251–267 (2002) 14. Jackiewicz, Z.: General Linear Methods for Ordinary Differential Equations. Wiley, Hoboken (2009) 15. Wright, W.M.: General linear methods with inherent RungeKutta stability. Ph.D. thesis, University of Auckland, New Zealand (2002)

Geometry Processing Kai Hormann Universit`a della Svizzera italiana, Lugano, Switzerland

Synonyms Geometry Processing The interdisciplinary research area of geometry processing combines concepts from computer science, applied mathematics, and engineering for the efficient acquisition, reconstruction, optimization, editing, and simulation of geometric objects. Applications of geometry processing algorithms can be found in a wide range of areas, including computer graphics, computeraided design, geography, and scientific computing. Moreover, this research field enjoys a significant economic impact as it delivers essential ingredients for the production of cars, airplanes, movies, and computer games, for example. In contrast to computeraided geometric design. In contrast to computer aided geometric design  B´ezier Curves and Surfaces, geometry processing focuses on polygonal meshes, and in

593

particular triangle meshes, for describing geometrical shapes rather than using piecewise polynomial representations.

Data Acquisition and Surface Reconstruction The first step of the geometry processing pipeline is to digitize real-world objects and to describe them in a format that can be handled by the computer. A common approach is to use a 3D scanner to acquire the coordinates of a number of sample points on the surface of the object. After merging the scans into a common coordinate system, the surface of the scanned object is reconstructed by approximating the sample points with a triangle mesh (see Fig. 1). 3D Scanning 3D scanning technology has advanced significantly over the last two decades and current devices are able to sample millions of points per second. Large objects, like buildings, are usually scanned with a time-of-flight scanner. It determines the distance to the object in a specific direction and hence the 3D coordinates of the corresponding surface point, by emitting a pulse of light, detecting the reflected signal, and measuring the time in between. The accuracy of this technique is on the order of millimeters. Higher accuracy can be obtained with handheld laser scanners. Such a scanner projects a laser line onto the object (see Fig. 1) and uses a camera to detect position and shape of the projected line. Knowing the positions of the laser emitter and the camera in the local coordinate frame of the scanner, the concept of triangulation can be used to compute the 3D coordinates of a set of surface sample points. This approach further requires to track position and direction of the scanner as it moves around the object, for example, by following a predefined scanning path or by mounting it onto a 3D measuring arm, so as to be able to transform all coordinates into a common global coordinate system. Handheld laser scanners are best suited for smaller objects and indoor scanning and provide an accuracy on the order of micrometers. A notable example of this scanning technique is the Digital Michelangelo Project [18], which digitized some of Michelangelo’s statues in Florence, including the David.

G

594

Geometry Processing

Geometry Processing, Fig. 1 3D scanning with a handheld laser scanner (a) gives a point cloud (b), and triangulating the points results in a piecewise linear approximation of the scanned object (c) m m An alternative with similar accuracy is provided by 1 X 1 X p and q N D qi ; p N D i structured-light scanners. Instead of a laser line, such m i D1 m i D1 scanners project specific light patterns onto the object, which are observed again by a camera. Due to the we consider the covariance matrix specific structure of the patterns, 3D coordinates of samples on the object’s surface can then be determined m X by triangulation. The most prominent member from M D .pi  p/.q N i  q/ N T this class of scanners is the Microsoft Kinect, which i D1 can be used to scan and reconstruct 3D objects within seconds [16]. and its singular value composition M D U˙V T . The solution of (1) is then given by

Registration In general, several 3D scans are needed to capture an object from all sides, and the resulting point clouds, which are represented in different local coordinate systems, need to be merged. This so-called registration problem can be solved with the iterative closest point (ICP) algorithm [2] or one of its many variants. To register two scans P and Q, this algorithm iteratively applies the following steps: 1. For each point in P , find the nearest neighbor in Q; 2. Find the optimal rigid transform for moving Q as close as possible to P ; 3. Apply this transformation to Q, until the best rigid transform is sufficiently close to the identity. As for the second step, let P D fp1 ; p2 ; : : : ; pm g and Q D fq1 ; q2 ; : : : ; qm g be the two point sets, such that qi 2 R3 has been identified as the nearest neighbor of pi 2 R3 for i D 1; : : : ; m. The task now is to solve the optimization problem

R D UVT

and

t D pN  Rq: N

Surface Reconstruction Once the surface samples are available in a common coordinate system, they need to be triangulated, so as to provide a piecewise linear approximation of the surface of the scanned object. This can be done by either interpolating or approximating the sample points, but due to inevitable measurement errors, approximation algorithms are usually preferred. The computational geometry community has developed many efficient algorithms for reconstructing triangle meshes from point clouds [8], using concepts like Voronoi diagrams and Delaunay triangulations, with the advantage of providing theoretical guarantees regarding the topological correctness of the result as long as certain sampling conditions are satisfied. Another approach is to define an implicit function F W R3 ! R, for example, by approximating the signed m distance to the samples [3], such that the iso-surface X min (1) S D fx 2 R3 W F .x/ D 0g approximates the kpi  .Rqi C t/k2 R;t i D1 sample points and hence the surface of the object. A popular algorithm from this class, which has been for some rotation R 2 R33 and some translation t 2 incorporated in a number of well-established geometry R3 . Denoting the barycenters of P and Q by processing libraries, is based on estimating an indicator

Geometry Processing

595

G

Geometry Processing, Fig. 2 The 15 local configurations considered in the marching cubes algorithm, based on the signs of the function values at the cell corners. All other configurations are equivalent to one of these 15 cases

function that separates the interior of the object from the surrounding space by efficiently solving a suitable Poisson problem [17]. The implicit function F is usually represented by storing the function values Fij k D F .xij k / at the nodes xij k of a regular or hierarchical grid, and the iso-surface S is extracted by the marching cubes algorithm [22] or one of its many variants. The key idea of this algorithm is to distinguish the 15 local configurations that can occur in each cell of the grid, based on the signs of Fij k at the cell corners, then to estimate points on S by linear interpolation of F along cell edges whose end points have function values with different signs, and finally to connect these points by triangles, with a predefined topology for each cell configuration (see Fig. 2). Iso-surface extraction with marching cubes can also be used to reconstruct surfaces from a volumetric function F that was generated by a computed tomography (CT) scan of the 3D object (see Fig. 3) or by magnetic resonance imaging (MRI).

or twice continuously differentiable, they need to be carried over with care to discrete surfaces (i.e., triangle meshes). The resulting discrete differential geometry should satisfy at least two main criteria. On the one hand, we require convergence, that is, continuous ideas need to be discretized such that a discrete property converges to the continuous property as the discrete surface converges to a smooth surface. On the other hand, we want structure preservation, that is, highlevel theorems like the Gauss–Bonnet theorem should hold in the discrete world. The most important concepts are surface normals and curvature, and we restrict our discussion to them. However, a comprehensive overview of the topic can be found in the SIGGRAPH Asia Course Notes by Desbrun et al. [5]. Normals For each triangle T D Œx; y; z of a triangle mesh with vertices x; y; z 2 R3 , the normal is easily defined as n.T / D

.y  x/  .z  x/ : k.y  x/  .z  x/k

Along each edge E, we can take the normal that is halfway between the normals of the two adjacent For many geometry processing tasks, it is impera- triangles T1 and T2 , tive to have available the concepts and tools from n.T1 / C n.T2 / differential geometry for working with surfaces. As ; (2) n.E/ D kn.T1 / C n.T2 /k those usually require the surface to be at least once

Discrete Differential Geometry

596

Geometry Processing

Geometry Processing, Fig. 3 Example of surface reconstruction via iso-surface extraction: original object (a), CT scan (b), and reconstructed triangle mesh (c)

and at a vertex V we usually average the normals of the n adjacent triangles T1 ; : : : ; Tn , Pn i n.Ti / : n.V / D PinD1 i n.Ti /

(3)

i D1

The weights i can either be constant, equal to the Geometry Processing, Fig. 4 A vertex V of a triangle mesh triangle area, or equal to the angle i of Ti at V (see with neighboring vertices V and adjacent triangles T . The angle i i Fig. 4). of Ti at V is denoted by i and the angles opposite the edge Ei by ˛i and ˇi . The dihedral angle E at a mesh edge E is the angle between the normals of the adjacent triangles

Gaussian Curvature It is well known that Gaussian curvature is zero for developable surfaces. Hence, it is reasonable to define the Gaussian curvature inside each mesh triangle to where these regions S.V / form a partition of the be zero, and likewise along the edges, because the surface of the entire mesh M . With this assumption, two adjacent triangles can be flattened isometrically the Gauss–Bonnet theorem is preserved: (i.e., without distortion) into the plane by simply rotating one triangle about the common edge into the Z X plane defined by the other. As a consequence, the KdA D K.V / D 2.M /; Gaussian curvature is concentrated at the vertices of M V 2M a triangle mesh and commonly defined as the angle defect where .M / is the Euler characteristic of the mesh M . n X As for the definition of S.V /, various approaches have

i ; K.V / D 2  been proposed, including the barycentric area, which is i D1 one-third of the area of each Ti , as well as the Voronoi area, which is the intersection of V ’s local Voronoi where i are the angles of the triangles Ti adjacent to the vertex V at V (see Fig. 4). Note that this cell (with respect to its neighbors Vi ) and the triangles value needs to be understood as the integral of Ti . the Gaussian curvature over a certain region S.V / around V : Mean Curvature Z Like Gaussian curvature, the mean curvature inside KdA; K.V / D each mesh triangle is zero, but it does not vanish at S.V /

Geometry Processing

597

the edges. In fact, the mean curvature associated with which differs slightly from the definition in (4) but an edge is often defined as converges to the same value as E approaches zero. Averaging H .E/ over the edges adjacent to a vertex V gives the discrete mean curvature vector associated (4) H.E/ D kEk E =2; to V , where E is the signed dihedral angle at E D ŒV; W , that is, the angle between the normals of the adjacent triangles (see Fig. 4), with positive or negative sign, depending on whether the local configuration is convex or concave. This formula can be understood by thinking of an edge as a cylindrical patch C.E/ with some small radius r that touches the planes defined by the adjacent triangles. As the mean curvature is 1=.2r/ at any point of C.E/ and the area of C.E/ is r kEk E , we get Z HdA; H.E/ D C.E/

independently of the radius r. The mean curvature at a vertex V is then defined by averaging the mean curvatures of its adjacent edges, 1X H.Ei /; 2 i D1 n

H.V / D

(5)

where the factor 1=2 is due to the fact that the mean curvature of an edge should distribute evenly to both end points. As for Gauss curvature, H.E/ and H.V / need to be understood as integral curvature values, associated to certain regions S.E/ and S.V / around E and V , respectively. Mean Curvature Vector Similarly, we can integrate the mean curvature vector H D H n, which is the surface normal vector scaled by the mean curvature, over the cylindrical patch C.E/ to derive the discrete mean curvature vector associated to the mesh edge E D ŒV; W ,

H .V / D

n n 1X 1X H .Ei / D .cot ˛i Ccot ˇi /.V Vi /; 2 i D1 4 i D1

where ˛i and ˇi are the angles opposite Ei in the adjacent triangles Ti 1 and Ti (see Fig. 4). Normalizing H .V / provides an alternative to the vertex normal in (3) and the length of H .V / gives an alternative to the vertex mean curvature in (5). Mesh Smoothing The aforementioned tools can be used for analyzing the quality of a surface, for example, by computing color-coded discrete curvature plots, but more importantly, they are essential for algorithms that improve the surface quality. Such denoising or smoothing algorithms remove the high-frequency noise, which may result from scanning inaccuracies, while maintaining the overall shape of the surface. The key idea behind these methods is to interpret the geometry of a triangle mesh (i.e., the vertex positions) as a function over the mesh itself. This function can then be smoothed by either using discrete diffusion flow [7] or by generalizing classical filter techniques from signal processing to triangle meshes [26].

Simplification

Modern scanning techniques deliver surface meshes with millions and even billions of triangles. As such highly detailed meshes are costly to process on the one hand and contain a lot of redundant geometric information on the other, they are usually simplified Z before further processing. A simplification algorithm 1 H dA D .V  W /  .n.T1 /  n.T2 //: reduces the number of triangles while preserving the H .E/ D 2 C.E/ overall shape or other properties of the given mesh. This strategy is also commonly used in computer While normalizing H .E/ results in the edge normal graphics to generate different versions of a 3D object vector in (2), the length of H .E/ gives the edge mean at various levels of detail (see Fig. 5), so as to increase curvature the rendering efficiency by adapting the object complexity to the current distance between the camera and H.E/ D kH .E/k D kEk sin. E =2/; object. Simplification algorithm can further be used to

G

598

Geometry Processing

Geometry Processing, Fig. 5 Neptune model at different levels of details with 4,000,000 (a), 400,000 (b), 40,000 (c), and 4,000 (d) triangles

a

b

c

Geometry Processing, Fig. 6 Local half-edge collapse (a), vertex removal (b), and edge collapse (c) operators for iterative mesh decimation

transfer and store geometric data progressively. For more information about mesh simplification and the related topic of mesh compression, we refer to the survey by Gotsman et al. [13].

Vertex Clustering The simplest mesh simplification technique subdivides the bounding box of a given mesh into a regular grid of cubic cells and replaces the vertices inside each cell by a unique representative vertex, for example, the cell center. Triangles with all vertices in the same cell degenerate during this clustering step and are removed by the subsequent cleanup phase. This approach is very efficient, and by appropriately choosing the cell size, it is easy to guarantee a given approximation tolerance between the original and the simplified mesh. However, it does not necessarily generate a mesh with the same topology as the original mesh. While this is problematic if, for example, manifoldness of the mesh is required by further processing tasks, it enables to simplify not only the geometry but also the topology of a mesh, which is advantageous for removing small topological holes resulting from scanning noise.

Mesh Decimation Mesh decimation algorithms iteratively remove one vertex and two triangles from the mesh, and the decimation order is based on some cost function. The three main decimation operators are shown in Fig. 6. The half-edge collapse operator moves a vertex p to the position of one of its neighbors q, so that the vertex itself and the two triangles adjacent to the connecting edge disappear. Note that collapsing p into q is different from collapsing q into p; hence, there are two possible collapse operations for each edge of the mesh. The decimation algorithm evaluates a cost function for each possible half-edge collapse, sorts the latter in a priority queue, and iteratively applies the simplification step with the currently smallest cost. As each half-edge collapse modifies the mesh in the local neighborhood, the costs for nearby half-edges may need to be recomputed and the priority queue updated accordingly. The standard cost function measures the distance between p and the simplified mesh after removing p, so that each decimation step increases the approximation error in the least possible way. However, depending on the application, it can be desirable to use other cost functions. For example, the cost function

Geometry Processing

can be based on the ratio of the circumcircle radius to the length of the shortest edge for the new triangles that are generated by a half-edge collapse, so as to compute simplified meshes with triangles that are close to equilateral. Or it can sum up the mean curvature associated with the new edges after removing p, so that simplification steps which smooth the mesh are preferred. Moreover, the distance function can be used in addition as a binary criterion to add only those collapses to the queue, which keep the simplified mesh within some approximation tolerance. The vertex removal operator deletes one vertex and retriangulates the resulting hole. A half-edge collapse can be seen as a special case of this operator, as it corresponds to a particular retriangulation of the hole. For vertices with six or more neighbors, there exist additional retriangulations; hence, the vertex removal operator offers more degrees of freedom than the halfedge collapse operator, which helps to improve the quality of the simplified mesh. Another generalization of the half-edge operator is the edge collapse operator. It joins two neighboring vertices p and q and moves them to a new position r, which can be different from q and is usually chosen to minimize some cost function. The most common approach is based on the accumulated quadric error metric [12]. For each triangle T D Œx; y; z of the initial mesh with normal n and distance d D nT x from the origin, the squared distance of a point v 2 R3 to the supporting plane P of T can be written as

599

itself. For each original mesh vertex p with adjacent triangles T1 ; : : : ; Tn , we define the error function Ep .v/ D

n X

dist.v; Pi / D vN

i D1

2

T

n X

! Qi vN D vN T Qp vN

i D1

as the sum of quadratic distances to the associated supporting planes P1 ; : : : ; Pn . This error function is a quadratic form with ellipsoidal iso-contours. Writing   Qp as A b Qp D bT c with A a symmetric 3  3 matrix and b 2 R3 , the position v 2 R3 which minimizes Ep .v/ can be found by solving the linear system Av D b. This, in turn, can be done robustly, even in the case of a rank-deficient matrix A, using the pseudoinverse of A. Now, whenever the edge between p and q is collapsed, the error function for the new point r is defined as Er D Ep C Eq with the associated matrix Qr D Qp C Qq , thus accumulating the squared distances to all supporting planes associated with p and q, and the position of r is the one that minimizes Er .

Parameterization

A parameterization of a surface is a bijective mapping from a suitable parameter domain to the surface. The basics of parametric surfaces were already developed about 200 years ago by Carl Friedrich Gauß. But dist.v; P /2 D vN T Qv; N only quite recently, the parameterization of triangle meshes has become a major research field in computerwhere vN D .v; 1/ 2 R4 are the homogeneous coordi- aided design and computer graphics, due to the many nates of v and the symmetric 4  4 matrix Q D nN nN T is applications ranging from texture mapping to remeshthe outer product of the vector nN D .n; d / 2 R4 with ing (see Figs. 7 and 8). These applications require

Geometry Processing, Fig. 7 The parameterization of a triangle mesh (a) over the plane can be used to map a picture (b) onto the surface of the mesh (c). This process is called texture mapping and is supported by the graphics hardware

G

600

Geometry Processing, Fig. 8 Applications of parameterizations: using the parameterization of the mesh over a rectangle to lift a regular grid from the parameter domain to the surface generates a regular quadrilateral remesh of the shape (a); fitting a B-

Geometry Processing

spline surface by minimizing the least squares distance between the mesh vertices and the surface at the corresponding parameter point results in a smooth approximation of the shape (b)

Geometry Processing, Fig. 9 Parameterization of a triangle mesh

parameterizations that minimize the inevitable metric distortion of the mapping. We restrict our discussion to methods which assume the surface to be topologically equivalent to a disk (i.e., it is a triangle mesh with exactly one boundary) and can thus be parameterized over a disk-like planar domain. A triangle mesh with arbitrary topology can either be split into several disklike patches, which are then parameterized individually, resulting in a texture atlas, or be handled with a global parameterization technique. For more details on mesh parameterization and its applications, we refer to the SIGGRAPH Asia Course Notes by Hormann et al. [15].

the parameterization of M , too. The mapping g is also piecewise linear and uniquely determined by the parameter points v D g.V / for each mesh vertex V . Hence, the task is to find parameter points v such that the resulting mappings g and f exhibit the least possible distortion with respect to some distortion measure. Distortion can be measured in various ways, resulting in different optimal parameterizations, but in general, the local distortion of a mapping f W R2 ! R3 is captured by the singular values 1  2  0 of the Jacobian Jf of f . In fact, considering the singular value decomposition of Jf , Jf D U

0 T 0 2 V ; 0 0

 1

Distortion of Mesh Parameterizations The parameterization of a mesh M is a continuous, preferably bijective mapping f W ˝ ! M between 33 and V 2 R22 are orthogonal some parameter domain ˝  R2 and M , such that where U 2 R each parameter triangle t is mapped linearly to the matrices, as well as the first-order Taylor expansion of corresponding mesh triangle T (see Fig. 9). Such a f about v, piecewise linear mapping f is usually specified by f .v C dv/  f .v/ C Jf dv; defining its inverse g D f 1 , which is often called

Geometry Processing

601

we can see that a disk with some small radius r around v is mapped to an ellipse with semiaxes of length r1 and r2 around the surface point f .v/ in the limit. If 1 D 2 D 1, then the mapping is called isometric and neither angles nor areas are distorted locally around v under the mapping f . A conformal mapping that preserves angles but distorts areas is characterized by 1 D 2 , and a mapping with 1 2 D 1 preserves areas at the cost of distorting angles and is called equiareal. In general, the metric distortion at v is then defined by E.1 ; 2 /, where the local distortion measure EW R2C ! RC is some nonnegative function which usually has a global minimum at .1; 1/ so as to favor isometry or with a minimum along the whole line .x; x/ for x 2 RC , if conformal mappings are preferred. As a mesh parameterization f is piecewise linear with constant Jacobian per triangle t, we can define the average distortion of f as N /D E.f

X

E.1t ; 2t /A.t/

t 2˝

.X

A.t/;

(6)

t 2˝

where 1t and 2t are the singular values of the Jacobian of the linear map f jt W t ! T and A.t/ denotes the area of t. Alternatively, we can also consider the average distortion of the inverse parameterization g D f 1 : N E.g/ D

X T 2M

E.1T ; 2T /A.T /

.X T 2M

A.T /;

(7)

and turns out to be quadratic in the parameter points. It can thus be minimized by solving a linear system. A potential disadvantage of harmonic maps is that they require to fix the boundary of the parameterization in advance. Otherwise, the parameterization degenerates, because ED takes its minimum for mappings with 1 D 2 D 0, so that an optimal parameterization is one that maps all surface triangles T to a single point. And even if the boundary is set up correctly, it may happen that some of the parameter triangles overlap each other, and so the parameterization is not bijective. Conformal Maps Another approach is to use the conformal energy [6,19] EC .1 ; 2 / D 12 .1  2 /2 as a local distortion measure in (7). This still yields a linear problem to solve, but only two of the boundary vertices need to be fixed in order to give a unique solution. Unfortunately, the resulting parameterization depends and can vary significantly on the choice of these two vertices. And even though it is possible to define and compute the best of all choices, the problem of potential non-bijectivity remains. Conformal and harmonic maps are closely related. Indeed, we first observe for the local distortion measures that ED .1 ; 2 /  EC .1 ; 2 / D 1 2

with the advantage that the sum of surface triangle and it is then straightforward to conclude that the areas in the denominator is constant and can thus be overall distortions defer by neglected upon minimization. Note that the singular .X X values of the linear map gjT are just the inverses of the A.˝/ N D .g/  EN C .g/ D E : A.t/ A.T / D T t T t linear map f jt , that is, 1 D 1=2 and 2 D 1=1 . A.M / t 2˝ T 2M In either case, the best parameterization with respect to the distortion measure E is then found by minimizing Therefore, if we take a conformal map, fix its boundary EN with respect to the unknown parameter points. and thus the area of the parameter domain ˝, and then compute the harmonic map with this boundary, then Harmonic Maps we get the same mapping, which illustrates the wellOne of the first parameterization methods that were known fact that any conformal mapping is harmonic, used in computer graphics considers the Dirichlet en- too. ergy [11, 24] of the inverse parameterization g. It is The conformal energy EC is clearly minimal for N given by E.g/ in (7) with the local distortion measure locally conformal mappings with 1 D 2 . However, it is not the only energy that favors conformality. The   ED .1 ; 2 / D 12 1 2 C 2 2 so-called MIPS energy [14]

G

602

Geometry Processing

1 2 1 2 C 2 2 C D 2 1 1 2

points are determined such that all the parameter triangles match up best in the least squares sense, which amounts to solving a sparse linear system. is also minimal if and only if 1 D 2 . An advantage This global step is followed by a local step where of this distortion measure is the symmetry with respect each parameter triangle is approximated by a rotated version of the rigidly mapped surface triangle, which to inversion, requires to determine optimal rotations, one for each triangle. Iterating both phases converges quickly EM .1T ; 2T / D EM .1t ; 2t /; toward a parameterization which tends to balance the so that it measures the distortion of both mappings deformation of angles and areas very well. f jt and gjT at the same time. The disadvantage is that minimizing either of the overall distortion energies in (6) and (7) is a nonlinear problem. However, EN M .f / is a quadratic rational function in the unknown paramAngle-Based Methods eter points and EN M .g/ is a sum of quadratic rational Instead of minimizing a deformation energy, it is also functions, and both can be minimized with standard possible to obtain parameterizations using different gradient-descent methods. Moreover, it is possible to concepts. For example, angle-based methods [25] aim guarantee the bijectivity of the resulting mapping. to find the 2D parameter triangulation such that the angles in each parameter triangle t are as close as Isometric Maps possible to the angles in the corresponding surface A local distortion measure that is minimal for locally triangle T . To ensure that all parameter triangles form isometric mappings is the Green–Lagrange deforma- a valid triangulation, a set of conditions on the angles tion tensor need to be satisfied, hence leading to a constrained optimization problem in the unknown angles, which 2 2 EG .1 ; 2 / D .1 2  1/ C .2 2  1/ ; can be solved using Lagrange multipliers. A simple post-processing step finally converts the solution into and the corresponding nonlinear optimization problem coordinates of the parameter points. By construction, can be solved efficiently with an iterative two-step this method tends to create parameterizations that are procedure [20]. An initial step maps all surface as conformal as possible in a certain sense and guarantriangles rigidly into the plane. Next, the parameter teed to be bijective. EM .1 ; 2 / D

Geometry Processing, Fig. 10 Subdividing an initial triangle mesh (a) gives a triangle mesh with four times as many triangles (b), and repeating the process (c) eventually results in a smooth limit surface (d)

Geometry Processing

Subdivision Surfaces Subdivision methods are essential in computer graphics as they provide a very efficient and intuitive way of designing curves and surfaces. The main idea of these methods is to iteratively refine a coarse control polygon or control mesh by adding new vertices, edges, and faces (see Fig. 10). This process generates a sequence of polygons or meshes with increasingly smaller edges which converges to a smooth curve or surface under certain conditions. In practice, few iterations of this refinement process suffice to generate curves and surfaces that appear smooth at screen resolution. We restrict our discussion to a brief overview of the most important surface subdivision methods. More details can be found in the SIGGRAPH Course Notes by Zorin and Schr¨oder [28] and in the books by Warren and Weimer [27], Peters and Reif [23], and Andersson and Stewart [1]. Triangle Meshes One of the simplest schemes for triangle meshes is the butterfly scheme [10]. This scheme adds new vertices, one for each edge of the mesh, and the positions of these new vertices are affine combinations of the nearby old vertices with weights shown in Fig. 11.

603

These weights stem from locally interpolating the eight old vertices by a bivariate cubic polynomial with respect to a uniform parameterization and evaluating it at the midpoint of the central edge. The new triangle mesh is then formed by connecting the new vertices as illustrated in Fig. 12, thus splitting each old triangle into four new triangles. The limit surfaces of the butterfly scheme are C 1 continuous except at the irregular initial vertices (those with other than six neighbors), but Zorin et al. [29] discuss a small modification that overcomes this drawback and yields limit surfaces that are C 1 -continuous everywhere. Another subdivision scheme for limit surfaces that are C 1 -continuous at the irregular initial vertices and even C 2 -continuous otherwise is the loop scheme [21]. This scheme has a simpler rule for the new edge vertices (see Fig. 13), but it also requires to compute a new position p k at subdivision level k 2 N for any old vertex p k1 at subdivision level k  1 by taking a convex combination of the neighboring old vertices p1k1 ; : : : ; pnk1 and the vertex itself (see Fig. 13), n X   p k D 1  nˇ.n/ p k1 C ˇ.n/ pik1 ; i D1

a

Geometry Processing, Fig. 11 Stencil for new vertices of the butterfly scheme

a

b

Geometry Processing, Fig. 13 Stencils for new vertices (a) and new positions of old vertices (b) for the loop scheme

b

Geometry Processing, Fig. 12 Topological refinement (1-to-4 split) of triangles (a) and quadrilaterals (b)

G

604

Geometry Processing

where the weight 1 5  ˇ.n/ D n 8



1 2 3 C cos 8 4 n

2 !

depends on the valency of p k1 . For example, the weight ˇ.6/ D 1=16 is used for regular vertices. Obviously, this scheme does not interpolate the initial values in general, but one can show that the limit position of the vertex p k is n X   pik p 1 D 1  n.n/ p k C .n/ i D1

with .n/ D

8ˇ.n/ : 3 C 8nˇ.n/

of the vertex stencil depend on the valency of the vertex. The limit surfaces of this scheme are C 2 continuous except at the irregular vertices, where they are only C 1 -continuous. Arbitrary Meshes The Doo–Sabin scheme [9] can be applied to meshes with arbitrary polygonal faces and uses a single subdivision rule for all new vertices. More precisely, for each polygonal face with n vertices, n new vertices are computed using the stencil in Fig. 15 with ˛i D

3 C 2 cos.2i =n/ ; 4n

i D 0; : : : ; n  1;

and they are connected to form the faces of the new mesh as shown in Fig. 15. This leads to a new mesh where each old vertex with valency n is replaced by an n-gon, each old edge by a quadrilateral, and each old face by a new face with the same number of vertices. Note that all vertices of the new mesh have valency four. The limit surface is C 1 -continuous and interpolates the barycenters of the initial faces.

For implementing the loop scheme, there basically are two choices. One way is to first compute the new positions of the old vertices without overwriting the old positions, followed by the creation of the new vertices and the refinement of the mesh, but this requires to reserve space for two sets of coordinates for each vertex. An alternative is to first compute the new vertices, then Useful Libraries to refine the mesh, and finally to update the positions of the old vertices, using the formula Implementing geometry processing algorithms from scratch can be a daunting task, but there exist a number n X   of excellent cross-platform C++ libraries for Windows, pNik p k D 1  n˛.n/ p k1 C ˛.n/ MacOS X, and Linux, which provide useful data struci D1 tures, algorithms, and tools. • The Computational Geometry Algorithms Library with ˛.n/ D 8=5  ˇ.n/, where pNik are the new CGAL is an open-source project with the goal to neighbors of the vertex p k1 after refinement. provide efficient and reliable algorithms for geometric computations in 2D and 3D. A special Quadrilateral Meshes feature of this library is that it can perform exact A similar subdivision scheme for quadrilateral meshes computations with guaranteed correctness. is the Catmull–Clark scheme [4]. Topologically, it • OpenFlipper is an open-source framework, which creates the subdivided mesh by inserting new vertices, provides a highly flexible interface for creating and one for each face and one for each edge of the old testing geometry processing algorithms, as well as mesh, and splitting each old quadrilateral into four as basic functionality like rendering, selection, and shown in Fig. 12. In addition, it also assigns a new user interaction. It is based on the OpenMesh data position to every old vertex. Figure 14 shows the rules structure and developed and maintained by the for computing all vertices, where the weights Computer Graphics Group at RWTH Aachen. • MeshLab is an extensible open-source system for 3 1 7 processing and editing unstructured 3D triangle ; ˇ.n/ D 2 ; .n/ D 2 ˛.n/ D 1  meshes and provides a set of tools for editing, 4n 2n 4n

Geometry Processing

a

605

b

c

Geometry Processing, Fig. 14 Stencils for new face vertices (a), new edge vertices (b), and new positions of old vertices (c) for the Catmull–Clark scheme

a

b

G

Geometry Processing, Fig. 15 Stencil for new vertices (a) and topological refinement (b) for the Doo–Sabin scheme

cleaning, healing, inspecting, rendering, and converting such meshes. It is based on the VCG library and developed and maintained by the Visual Computing Lab of CNR-ISTI in Pisa. • The libigl geometry processing library provides simple facet and edge-based topology data structures, mesh-viewing utilities for OpenGL and GLSL, and a wide range of functionality, including the construction of sparse discrete differential geometry operators. It is heavily based on the Eigen library and provides useful conversion tables for porting MATLAB code to libigl.

References 1. Andersson, L.-E., Stewart, N.F.: Introduction to the Mathematics of Subdivision Surfaces. SIAM, Philadelphia (2010) 2. Besl, P.J., McKay, N.D.: A method for registration of 3D shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992) 3. Carr, J.C., Beatson, R.K., Cherrie, J.B., Mitchell, T.J., Fright, W.R., McCallum, B.C., Evans, T.R.: Reconstruction and representation of 3D objects with radial basis functions. In: Proceedings of ACM SIGGRAPH 2001, Los Angeles, pp. 67–76 (2001) 4. Catmull, E., Clark, J.: Recursively generated B-spline surfaces on arbitrary topological surfaces. Comput. Aided Des. 10(6), 350–355 (1978)

5. Desbrun, M., Grinspun, E., Schr¨oder, P., Wardetzky, M.: Discrete Differential Geometry: An Applied Introduction. Number 14 in SIGGRAPH Asia 2008 Course Notes. ACM, New York (2008) 6. Desbrun, M., Meyer, M., Alliez, P.: Intrinsic parameterizations of surface meshes. Comput. Graph. Forum 21(3), 209–218 (2002). Proceedings of Eurographics 2002 7. Desbrun, M., Meyer, M., Schr¨oder, P., Barr, A.H.: Implicit fairing of irregular meshes using diffusion and curvature flow. In: Proceedings of ACM SIGGRAPH 1999, Los Angeles, pp. 317–324 (1999) 8. Dey, T.K.: Curve and Surface Reconstruction: Algorithms with Mathematical Analysis. Volume 23 of Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, New York (2007) 9. Doo, D., Sabin, M.: Behaviour of recursive division surfaces near extraordinary points. Comput. Aided Des. 10(6), 356–360 (1978) 10. Dyn, N., Levin, D., Gregory, J.A.: A butterfly subdivision scheme for surface interpolation with tension control. ACM Trans. Graph. 9(2), 160–169 (1990) 11. Eck, M., DeRose, T., Duchamp, T., Hoppe, H., Lounsbery, M., Stuetzle, W.: Multiresolution analysis of arbitrary meshes. In: Proceedings of ACM SIGGRAPH ’95, Los Angeles, pp. 173–182 (1995) 12. Garland, M., Heckbert, P.S.: Surface simplification using quadric error metrics. In: Proceedings of ACM SIGGRAPH 1997, Los Angeles, pp. 209–216 (1997) 13. Gotsman, C., Gumhold, S., Kobbelt, L.: Simplification and compression of 3D meshes. In: Iske, A., Quak, E., Floater, M.S. (eds.) Tutorials on Multiresolution in Geometric

Global Estimates for hp-Methods

606

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

25.

26.

27.

28.

29.

Modelling. Mathematics and Visualization, pp. 319–361. Springer, Berlin/Heidelberg (2002) Hormann, K., Greiner, G.: MIPS: an efficient global parametrization method. In: Laurent, P.-J., Sablonni`ere, P., Schumaker, L.L. (eds.) Curve and Surface Design: Saint-Malo 1999. Innovations in Applied Mathematics, pp. 153–162. Vanderbilt University Press, Nashville (2000) Hormann, K., Polthier, K., Sheffer, A.: Mesh Parameterization: Theory and Practice. Number 11 in SIGGRAPH Asia 2008 Course Notes. ACM, New York (2008) Izadi, S., Newcombe, R.A., Kim, D., Hilliges, O., Molyneaux, D., Hodges, S., Kohli, P., Shotton, J., Davison, A.J., Fitzgibbon, A.: KinectFusion: real-time dynamic 3D surface reconstruction and interaction. In: ACM SIGGRAPH 2011 Talks, Vancouver, pp. 23:1–23:1 (2001) Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Proceedings of the 4th Eurographics Symposium on Geometry Processing, Cagliari, pp. 61–70 (2006) Levoy, M., Pulli, K., Curless, B., Rusinkiewicz, S., Koller, D., Pereira, L., Ginzton, M., Anderson, S., Davis, J., Ginsberg, J., Shade, J., Fulk, D.: The digital Michelangelo project: 3D scanning of large statues. In: Proceedings of ACM SIGGRAPH 2000, New Orleans, pp. 131–144 (2000) L´evy, B., Petitjean, S., Ray, N., Maillot, J.: Least squares conformal maps for automatic texture atlas generation. ACM Trans. Graph. 21(3), 362–371 (2002). Proceedings of SIGGRAPH 2002 Liu, L., Zhang, L., Xu, Y., Gotsman, C., Gortler, S.J.: A local/global approach to mesh parameterization. Comput. Graph. Forum 27(5), 1495–1504 (2008). Proceedings of SGP Loop, C.T.: Smooth subdivision surfaces based on triangles. Master’s thesis, Department of Mathematics, The University of Utah, Aug 1987 Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3D surface construction algorithm. ACM SIGGRAPH Comput. Graph. 21(4), 163–169 (1987) Peters, J., Reif, U.: Subdivision Surfaces. Volume 3 of Geometry and Computing. Springer, Berlin/Heidelberg (2008) Pinkall, U., Polthier, K.: Computing discrete minimal surfaces and their conjugates. Exp. Math. 2(1), 15–36 (1993) Sheffer, A., de Sturler, E.: Parameterization of faceted surfaces for meshing using angle-based flattening. Eng. Comput. 17(3), 326–337 (2001) Vallet, B., L´evy, B.: Spectral geometry processing with manifold harmonics. Comput. Graph. Forum 27(2), 251–260 (2008). Proceedings of Eurographics 2008 Warren, J., Weimer, H.: Subdivision Methods for Geometric Design: A Constructive Approach. The Morgan Kaufmann Series in Computer Graphics and Geometric Modelling. Morgan Kaufmann, San Francisco (2001) Zorin, D., Schr¨oder, P.: Subdivision for Modeling and Animation. Number 23 in SIGGRAPH 2000 Course Notes. ACM (2000) Zorin, D., Schr¨oder, P., Sweldens, W.: Interpolating subdivision for meshes with arbitrary topology. In: Proceedings of ACM SIGGRAPH ’96, New Orleans, pp. 189–192 (1996)

Global Estimates for hp-Methods Leszek F. Demkowicz Institute for Computational Engineering and Sciences (ICES), The University of Texas at Austin, Austin, TX, USA

Mathematics Subject Classification 65N30; 35L15

Synonyms hp best approximation estimates; hp estimates

Short Definition Global hp estimates refer to the best approximation error estimates for hp finite element (FE) subspaces of H 1 ; H.curl/; H.div/, and L2 energy spaces. Introduction In hp finite elements, convergence is achieved by decreasing element size h and/or increasing the polynomial order p, i.e., h !0: p

(1)

Both element order p and element size h may vary locally. In other words, a FE mesh may employ elements of different p and h. This brings in concepts like hierarchical shape functions and hanging nodes (constrained approximation). We shall refer to meshes with variable h; p as hp meshes. Judiciously constructed (by various hp-adaptive algorithms) hp meshes deliver exponential rates of convergence, appropriately measured FE error as a function of the total number of unknowns N (degrees of freedom) (also CPU time and memory) decreases exponentially, error  C e ˇN ; r

C; ˇ; r > 0 :

(2)

Global Estimates for hp-Methods

607

id r r One arrives also frequently at hp meshes already in R I ! H 1 .˝/ ! H.curl; ˝/ ! H.div; ˝/ the process of generating an initial mesh. Handling r 0 complex geometries, avoiding volumetric locking and ! L2 .˝/ ! f0g : (3) locking for thin-walled structures, and controlling phase error in wave propagation are examples of If ˝ is simply connected, we obtain the the exact situations where an hp mesh may be employed from sequence, i.e., the range of each operator in the the very beginning. sequence coincides with the null space of the next operator. Polynomial and FE Exact Sequences The exact sequence structure is preserved by various Given ˝  R I 3 , we consider the classical differential finite element subspaces, complex:

G r

r

r

r

r

r

H .˝/ ! H.curl; ˝/ ! H.div; ˝/ ! L .˝/ [ [ [ [ 1

Wp

!

Qp

!

Vp

!

2

(4)

Yp

The diagram refers not to one but multiple scenarios. the possibility of anisotropy in polynomial order. For We start with ˝ coinciding with a master element. instance, for the cube, orders p; q; r may be different. Critical to the extension of the exact sequence The simplest construction, exploiting a tensor product 3 structure from a single element to a FE mesh is the structure, is offered on master cube ˝ D .0; 1/ , concept of a parametric element and the corresponding pullback maps known also as Piola transforms. Given Wp WD P p ˝ P q ˝ P r a pair of elements, master element KO and a physical element K, with the corresponding element map (a C 1 p1 q r Qp WD .P ˝P ˝P / diffeomorphism),  .P p ˝ P q1 ˝ P r /  .P p ˝ P q ˝ P r1 /

f W KO 3  ! x D f ./ 2 K; F WD r x; J WD det F ; (6)

Vp WD .P p ˝ P q1 ˝ P r1 /  .P p1 ˝ P q ˝ P r1 /  .P p1 ˝ P q1 ˝ P r / Yp WD P p1 ˝ P q1 ˝ P r1 :

(5)

the corresponding pullback maps are defined as follows: H 1 .K/ 3 w! w O WD w ı f

O 2 H 1 .K/;

O H.curl; K/ 3 q! qO WD .F T q/ ı f 2 H.curl; K/; The polynomial spaces correspond to N´ed´elec’s cube of the first type. Two analogous constructions are possible for simplices (triangle, tetrahedron) corresponding to N´ed´elec’s elements of the first and second types. 2D sequences for a triangle and the tensor product structure lead then to the exact sequences for the master prism. The most difficult case deals with the master pyramid – the spaces are no longer purely polynomial, they involve some non-polynomial functions as well. Tets and pyramids employ isotropic order of approximations, whereas the tensor product elements offer

O H.div; K/ 3 v! vO WD .J 1 F v/ ı f2 H.div; K/; L2 .K/ 3 y! yO WD .J 1 y/ ı f 2 L2 .K/: (7) The key to the understanding of the maps lies in the fact that they preserve the exact sequence structure. The first map corresponds to the concept of isoparametric finite elements and can be treated as a definition of w. O One computes then gradient @w=@ O i . If the exact sequence property is to be preserved, the H.curl/-conforming fields must transform the same

Global Estimates for hp-Methods

608

way as the gradient. Then, we compute the curl of such conforming fields, etc.; see [11], p. 34, for details. If the element map is an affine isomorphism, the pullback maps generate polynomial spaces; otherwise, they do not. The polynomial exact sequences can naturally be generalized to elements of variable order. Toward this goal, we associate with each edge e a separate edge order pe , and with each face a separate face order pf , isotropic for triangular and possibly anisotropic for a rectangular face. The edge and face orders must satisfy the minimum rule: the edge order must not exceed orders for adjacent faces, and the face order must not exceed orders for the adjacent elements. The corresponding variable order spaces are then identified as subspaces of the full-order polynomial spaces. The whole construction is easily understood with the use of hierarchical shape functions that provide the bases for the involved spaces which are naturally grouped into vertex shape functions (H 1 ), edge bubbles (H 1 ; H.curl/), face bubbles (H 1 ; H.curl/; H.div/), and element interior bubbles (all spaces). The number of shape functions corresponding to edge, face, and interior is a function of order pe ; pf :p, element shape, and a particular choice of N´ed´elec’s family of elements. For elements of variable order, one simply eliminates shape functions with order exceeding the prescribed order for edges and faces.

H 1 .˝/

r

Projection-Based Interpolation Given the conforming, piecewise polynomial subspaces of the energy spaces, we want to estimate the error for H 1 ; H.curl/; H.div/, and L2 projections onto the FE spaces Wp ; Qp ; Vp ; Yp . As for the h version of the FE method, this can be done by introducing appropriate interpolation operators and replacing the global projections with local interpolation operators. The corresponding interpolation errors should exhibit the same orders of convergence in terms of both h and p as the projection errors. Additionally, one strives to define those interpolation operators in such a way that they make the following diagram commute (de Rham diagram):

r

r

! H.curl; ˝/ ! H.div; ˝/ ! L2 .˝/

# ˘ grad Wp

Finally, the polynomial exact sequences can be generalized to piecewise polynomial spaces corresponding to FE meshes. Given a regular or irregular (with hanging nodes) FE mesh of affine finite elements, i.e., elements with affine maps, the pullback maps are used to transfer the polynomial spaces from the master elements to the physical space. The use of hierarchical shape functions and generalized connectivities corresponding to hanging nodes allows for the construction of general hybrid hp meshes consisting of elements of all shapes.

# ˘ curl r

!

Qp

Ideally, one would like to have such operators defined on the whole energy spaces. The projection-based (PB) interpolation is defined only on subspaces of the energy spaces incorporating additional regularity assumptions that secure the same operations as classical Lagrange, N´ed´elec and Raviart-Thomas interpolation operators for low-order FE spaces: evaluation of point values for H 1 -conforming elements, edge averages for tangential components of H.curl/-conforming functions, and face averages for H.div/-conforming func-

# ˘ div r

!

Vp

#P r

!

(8)

Yp

tions. Only the last operator P in the diagram (8) refers to L2 -projection that is defined on the whole L2 -space. The projection-based interpolation is done by performing a series of local evaluations at vertices and projections over element edges, faces, and interiors. It is through the projections that the optimal p-estimates are guaranteed. The procedure is local in the element sense – the element interpolant is determined by using values of the interpolated function (and its derivatives) from within the element only. We offer some intuition

Global Estimates for hp-Methods

609

by describing verbally the H 1 -conforming interpola- project the difference w  wf onto the edge bubbles to obtain edge contribution we . In the same way, for tion operator. Given a function P each face f , we project the difference w  wv  e we (9) onto the space of face w 2 H 1 .˝/ W wjK 2 H r .K/; r > 3=2 ; Pbubbles. PFinally, we project the difference w  wv  e we  f wf onto the element we construct its interpolant as a sum of vertex, edge, (interior) bubbles, to obtain the last contribution wint . The projections are done in norms dictated by the trace face, and element interior contributions: theorem: L2 -projections for edges, H 1=2 -seminorm X X grad we C wf C wint : (10) projections for faces, and H 1 -seminorm projection in ˘ w WD wv C the interior. e f Let ˝ be a master element with the correspondThe construction utilizes a natural decomposition of Wp into vertex shape functions, edge bubbles, face ing element polynomial exact sequence with (possibly bubbles, and element (interior) bubbles. We start with variable) order of approximation. The following pwf – the standard vertex interpolant: linear for test, estimates hold: trilinear for cubes, etc. Next, for each edge e, we kw  ˘ grad wkH 1 .˝/  C ln2 p p .r1/ kwkH r .˝/ r > 32 ; P p .˝/  Wp ; kq  ˘ curl qkH.curl;˝/  C ln p p r kqkH r .curl;˝/ kv  ˘ div vkH.div;˝/  C ln p p r kvkH r .div;˝/ Here C is a constant that is independent of both the functions and polynomial order p. The proof of the result is based on discrete Friedrichs’ inequalities and the existence of polynomialpreserving extension operators. To my best knowledge, such operators have been constructed so far only

kw  ˘ grad wkH 1 .˝/  C ln2 p kq  ˘ curl qkH.curl;˝/  C ln p kv  ˘ div vkH.div;˝/  C ln p

 r1 h p

 r h p

kvkH r .div;˝/ r > 0; .P p .˝//3  Vp ; P p .˝/  Yp :

approximation error

inf ku  whp k ; „ ƒ‚ …

whp

best approximation error

for cubes and simplices. For prisms and pyramids, therefore, the p-estimates above are still a conjecture only. Since the PB interpolation preserves the FE spaces, a version of the Bramble-Hilbert argument may be used to generalize the p-estimates to hp-estimates:

kqkH r .curl;˝/ r > 12 ; .P p .˝//3  Qp ; Vp ;

Applications First of all, the hp estimates help establish convergence of stable hp FE methods. If the method is stable, i.e., the actual FE approximation error is bounded by the best approximation error, C

r > 0; .P p .˝//3  Vp ; P p .˝/  Yp :

h p

 r

(11)

kwkH r .˝/ r > 32 ; P p .˝/  Wp ;

For details, see [9, 10] and the literature therein.

ku  uhp k „ ƒ‚ …

r > 12 ; .P p .˝//3  Qp ; Vp ;

(13)

(12)

with a mesh-independent stability constant C > 0, then the hp FE method will converge whenever h=p ! 0. The standard Bubnov-Galerkin method is naturally stable for elliptic (coercive) problems and mixed formulations where the stability is implied by the exact sequence. To this class belong, e.g., a mixed formulation for Darcy’s equation and standard variational formulations for the Maxwell equations. The hp estimates have been a crucial tool to prove discrete compactness for the hp methods; see [6] and the literature therein. The hp-estimates can also be

G

610

used to establish convergence of various stabilized formulations including discontinuous Galerkin (DG) methods. Finally, the hp estimates provide the backbone for the two-grid hp-adaptive strategy discussed in [8, 11]. Comments The hp FE method was created by Ivo Babuˇska and his school, and it originates from the p version of the FEM invented by Barna Szabo [18]. First, multidimensional results on the hp method were obtained by Babuˇska and Guo [12]. Over the last three decades, both Babuˇska and Guo published a large number of publications on the subject involving many collaborators. The concept of projectionbased interpolation discussed here derives from their work on the treatment of nonhomogeneous Dirichlet boundary conditions; see, e.g., [3]. Assessing the regularity of solutions to elliptical problems in standard Sobolev spaces (used in this entry) leads to suboptimal convergence results. This important technical issue led to the concept of countably normed Besov spaces developed in [1, 2]. The book by Christoph Schwab [16] remains the main source of technical results on exponential convergence. Over the years, the exponential convergence results have been established also for Stokes [17] and Maxwell equations [7] and more difficult problems involving stabilization; see, e.g., [14, 15]. The exponential convergence results have been extended to boundary elements (BE); see [4, 5, 13]. In particular, Heuer and Bespalov extended projection-based interpolation concepts to energy spaces relevant for the BE method. Finally, for information on three-dimensional hp codes, see Preface in [8] and the works of Wolfgang Bangerth, Krzysztof Fidkowski, Paul Houston, Paul Ledger, Spencer Sherwin, Joachim Schoeberl, Andreas Schroeder, and Tim Warburton.

References 1. Babuska, I., Guo, B.Q.: Direct and inverse approximation theorems of the p version of finite element method in the framework of weighted Besov spaces. Part 1: approximability of functions in weighted Besov spaces. SIAM J. Numer. Anal. 39, 1512–1538 (2002)

Global Estimates for hp-Methods 2. Babuska, I., Guo, B.Q.: Direct and inverse approximation theorems of the p version of finite element method in the framework of weighted Besov spaces. Part 2: optimal convergence of the p version finite element solutions. M3 AS 12, 689–719 (2002) 3. Babuˇska, I., Suri, M.: The p and hp-versions of the finite element method, basic principles and properties. SIAM Rev. 36(4), 578–632 (1994) 4. Bespalov A., Heuer, N.: The hp-BEM with quasi-uniform meshes for the electric field integral equation on polyhedral surfaces: a priori error analysis. Appl. Numer. Math. 60(7), 705–718 (2010) 5. Bespalov, A., Heuer, N.: The hp-version of the Boundary Element Method with quasi-uniform meshes for weakly singular operators on surfaces. IMA J. Numer. Anal. 30(2), 377–400 (2010) 6. Boffi, D., Costabel, M., Dauge, M., Demkowicz, L., Hiptmair, R.: Discrete compactness for the p-version of discrete differential forms. SIAM J. Numer. Anal. 49(1), 135–158 (2011) 7. Costabel, M., Dauge, M., Schwab, C.: Exponential convergence of hp-FEM for Maxwell’s equations with weighted regularization in polygonal domains. Math. Models Methods Appl. Sci. 15(4), 575–622 (2005) 8. Demkowicz, L.: Computing with hp Finite Elements. I. One- and Two-Dimensional Elliptic and Maxwell Problems. Chapman & Hall/CRC/Taylor and Francis, Boca Raton (2006) 9. Demkowicz, L.: Polynomial exact sequences and projection-based interpolation with applications to Maxwell equations. In: Boffi, D., Gastaldi, L. (eds.) Mixed Finite Elements, Compatibility Conditions and Applications. Volume 1939 of Lecture Notes in Mathematics, pp. 101– 158. Springer, Berlin (2008). See also ICES Report 06–12 10. Demkowicz, L., Buffa, A.: H 1 ; H.curl/ and H.div/conforming projection-based interpolation in three dimensions. Quasi-optimal p-interpolation estimates. Comput. Methods Appl. Mech. Eng. 194, 267–296 (2005) 11. Demkowicz, L., Kurtz, J., Pardo, D., Paszy´nski, M., Rachowicz, W., Zdunek, A.: Computing with hp Finite Elements. II. Frontiers: Three-Dimensional Elliptic and Maxwell Problems with Applications. Chapman & Hall/CRC, Boca Raton (2007) 12. Guo, G., Babuˇska, I.: The hp version of the finite element method. Part 1: the basic approximation results. Part 2: general results and applications. Comput. Mech. 1, 21–41, 203–220 (1986) 13. Heuer, N., Maischak, M., Stephan, E.P.: Exponential convergence of the hp-version for the boundary element method on open surfaces. Numer. Math. 83(4), 641–666 (1999) 14. Sch¨otzau, D., Schwab, Ch., Toselli, A.: Stabilized hpDGFEM for incompressible flow. Math. Models Methods Appl. Sci. 13(10), 1413–1436 (2003) 15. Sch¨otzau, D., Schwab, Ch., Wihler, T., Wirz, M.: Exponential convergence of hp-DGFEM for elliptic problems in polyhedral domains. Technical report 201240, Seminar for Applied Mathematics, ETH Z¨urich (2012)

Greedy Algorithms 16. Schwab, Ch.: p and hp-Finite Element Methods. Clarendon, Oxford (1998) 17. Schwab, Ch., Suri, M.: Mixed hp-FEM for Stokes and non-Newtonian flow. Comput. Methods Appl. Mech. Eng. 175(3–4), 217–241 (1999) 18. Szabo, B.A., Babuˇska, I.: Finite Element Analysis. Wiley, New York (1991)

611

also by using different ways of constructing (choosing coefficients of the linear combination) the m-term approximant from previously selected m elements of the dictionary.

Overview

Greedy Algorithms Vladimir Temlyakov Department of Mathematics, University of South Carolina, Columbia, SC, USA Steklov Institute of Mathematics, Moscow, Russia

Synonyms Greedy approximation; Matching pursuit; Projection pursuit

Definition Greedy algorithms provide sparse representation (approximation) of a given image/signal in terms of a given system of elements of the ambient space. In a mathematical setting image/signal is considered to be an element of a Banach space. For instance, a twodimensional signal can be viewed as a function of two variables belonging to a Hilbert space L2 or, more generally, to a Banach space Lp , 1  p  1. Usually, we assume that the system used for representation has some natural properties, and we call it a dictionary. For an element f from a Banach space X and a fixed m, we consider approximants which are linear combinations of m terms from a dictionary D. We call such an approximant an m-term approximant of f with respect to D. A greedy algorithm in sparse approximation is an algorithm that uses a greedy step in searching for a new element to be added to a given m-term approximant. By a greedy step, we mean one which maximizes a certain functional determined by information from the previous steps of the algorithm. We obtain different types of greedy algorithms by varying the abovementioned functional and

A classical problem of mathematical and numerical analysis is to approximately represent a given function. It goes back to the first results on Taylor’s and Fourier’s expansions of a function. The first step to solve the representation problem is to choose a representation system. Traditionally, a representation system has natural features such as minimality, orthogonality, simple structure, and nice computational characteristics. The most typical representation systems are the trigonometric system fe i kx g, the algebraic system fx k g, the spline system, the wavelet system, and their multivariate versions. In general we may speak of a basis  D f k g1 kD1 in a Banach space X . The second step to solve the representation problem is to choose the form of the approximant to be built from the chosen representation system  . In a classical way that was used for centuries, an approximant am is P a polynomial with respect to  : am WD m kD1 ck k . It was understood in numerical analysis and approximation theory that in many problems from signal/image processing it is more beneficial to use an m-term approximant with respect to  than a polynomial of order m. This means that for f 2 X we look for an P approximant of the form: am .f / WD k2.f / ck k , where .f / is a set of m indices which is determined by f . The third step to solve the representation problem is to choose a method of construction of the approximant. In linear theory, partial sums of the corresponding expansion of f with respect to the basis  is a standard method. It turns out that greedy approximants are natural substitutes for the partial sums in nonlinear theory. In many applications, we replace a basis by a more general system which may be a redundant system. This setting is much more complicated than the first one (bases case); however, there is a solid justification of importance of redundant systems in both theoretical questions and in practical applications (see, for instance, [5, 7, 12]).

G

612

Greedy Algorithms

Greedy Algorithms with Respect to Bases If  WD f n g1 nD1 is a Schauder basis for a Banach space X , then for any f 2 X there exists a unique P representation f D 1 nD1 cn .f / n that converges in X to f . Let  be normalized (k k k D 1). Consider the following reordering of the coefficients (greedy reordering) jcn1 .f /j  jcn2 .f /j  : : : : Then, the mth greedy approximant P of f with respect to  is defined as Gm .f / WD m j D1 cnj .f / nj : It is clear that Gm .f / is an m-term approximant of f . The above algorithm Gm ./ is a simple algorithm which describes a theoretical scheme to create an m-term approximation. We call this algorithm the Greedy Algorithm (GA). It is also called the Thresholding Greedy Algorithm (TGA). In order to understand the efficiency of this algorithm, we compare its accuracy with the bestpossible accuracy when an approximant is a linear combination of m terms from  . We define the best m-term approximation of f with regard to  is given by m .f / WD m .f;  / WD inf kf  ck ;

X

ck

k k;

k2

where the infimum is taken over coefficients ck and sets of indices  with cardinality jj D m. It is clear that for any f , we always have kf  Gm .f /k  m .f /. The best we can achieve with the greedy algorithm Gm is kf  Gm .f /k D m .f /; or the slightly weaker kf  Gm .f /k  C m .f /;

(1)

Greedy Algorithms with Respect to Redundant Systems Let H be a real Hilbert space with an inner product h; i and the norm kxk WD hx; xi1=2 . We say a set D of functions (elements) from H is a dictionary if each g 2 D has norm one .kgk D 1/ and the closure of span D is equal to H: We begin with a natural greedy algorithm the Pure Greedy Algorithm (PGA). This algorithm is defined inductively. To initialize we define f0 WD f , G0 .f / D 0. Each iteration of it consists of three steps. Then, for each m  1 we have the following inductive definition. The first step is a greedy step: 1. 'm 2 D is any element satisfying jhfm1 ; 'm ij D supg2D jhfm1 ; gij (we assume existence). 2. At the second step we update the residual fm WD fm1  hfm1 ; 'm i'm . 3. At the third step we update the approximant Gm .f / WD Gm1 .f / C hfm1 ; 'm i'm . The Pure Greedy Algorithm is also known as Matching Pursuit in signal processing. It is clear that the PGA provides an expansion of f into a series with respect to a dictionary D. This expansion is an example of a greedy expansion. One can visualize the PGA as a realization of a concrete strategy greedy orienteering in the game of orienteering. The rules of the game are the following: The goal is to reach the given point f starting at the origin 0. A dictionary D gives allowed directions of walk at each iteration. Then, PGA describes the strategy at which Gm .f / is the point closest to f that can be reached from the point Gm1 .f / by walking only in one of the directions listed in D. There are different modifications of PGA which have certain advantages over PGA itself. As it is clear from the greedy step of PGA, we need an existence assumption in order to run the algorithm. The Weak Greedy Algorithm (WGA) is a modification of PGA that does not need the existence assumption. In WGA we modify the greedy step of PGA so that

for all elements f 2 X , and with a constant C D C.X;  / independent of f and m. When X is a Hilbert space and  is an orthonormal basis, inequality (1) holds with C D 1. We call  a greedy basis if (1) holds for all f 2 X (see [9]). It is known (see [13]) that the univariate Haar basis and all reasonable univariate wavelet bases are greedy bases for Lp , 1 < p < 1. Greedy bases are well studied (see the book [16] and the survey papers (1w) 'm 2 D is any element satisfying [2, 10, 14, 15, 19]). jhfm1 ; 'm ij  tm supg2D jhfm1 ; gij; where ftk g, In addition to the concept of greedy basis, the tk 2 Œ0; 1/, is a given weakness sequence. following new concepts of bases were introduced in The Orthogonal Greedy Algorithm (OGA) and the a study of greedy approximation: quasi-greedy basis ([9]), democratic basis ([9]), almost greedy basis ([4]), Weak Orthogonal Greedy Algorithm (WOGA) are natpartially greedy basis ([4]), bidemocratic basis ([4]), ural modifications of PGA and WGA that are widely used in applications. The WOGA has the same greedy and semi-greedy basis ([3]).

Greedy Algorithms

step as the WGA and differs in the construction of a linear combination of '1 ; : : : ; 'm . In WOGA we do our best to construct an approximant out of Hm WD span.'1 ; : : : ; 'm ): we take an orthogonal projection onto Hm . Clearly, in this way we lose a property of WGA to build an expansion into a series in the case of the WOGA. However, this modification pays off in the sense of improving the convergence rate of approximation. An idea of relaxation proved to be useful in greedy approximation. This idea concerns construction of a greedy approximant. In WOGA, we build an approximant as an orthogonal projection on Hm which is a more difficult step than step (3) from PGA. The idea of relaxation suggests to build the mth greedy r approximant Gm .f / (r stands here for “relaxation”) r .f / as a linear combination of the approximant Gm1 from the previous iteration of the algorithm and an r obtained at the mth greedy step of the element 'm algorithm. There is vast literature on theoretical study and numerical applications of greedy algorithms in Hilbert spaces. See, for instance, [6, 8, 14, 16, 17]. For applications of greedy algorithms in learning theory see [16], Chap. 4. Greedy algorithms are also very useful in compressed sensing. A connection between results on the widths that were obtained in the 1970s and current results in compressed sensing is well known. The early theoretical results on the widths did not consider the question of practical recovery methods. The celebrated contribution of the work by Candes-Tao and Donoho was to show that the recovery can be done by the `1 minimization. While `1 minimization plays an important role in designing computationally tractable recovery methods, its complexity is still impractical for many applications. An attractive alternative to the `1 minimization is a family of greedy algorithms. They include the Orthogonal Greedy Algorithm (called the Orthogonal Matching Pursuit (OMP) in signal processing) discussed above, the Regularized Orthogonal Matching Pursuit (see [11]), and the Subspace Pursuit discussed in [1]. The reader can find further discussion of application of greedy algorithms in compressed sensing in [16], Chap. 5. It is known that in many numerical problems users are satisfied with a Hilbert space setting and do not consider a more general setting in a Banach space. However, application of Banach spaces is justified by two arguments. The first argument is a priori: The Lp

613

spaces are very natural and should be studied along with the L2 space. The second argument is a posteriori: The study of greedy approximation in Banach spaces has uncovered a very important characteristic of a Banach space X that governs the behavior of greedy approximation: the modulus of smoothness .u/ of X . It is known that all spaces Lp , 2  p < 1 have the modulus of smoothness of the same order u2 . Thus, many results which are known for the Hilbert space L2 and proven using the special structure of Hilbert spaces can be generalized to Banach spaces Lp , 2  p < 1. The new proofs use only the geometry of the unit sphere of the space expressed in the form .u/   u2 . The reader can find a systematic presentation of results on greedy approximation in Banach spaces in [16] and [15].

References 1. Dai, W., Milenkovich, O.: Subspace pursuit for compressive sensing signal reconstruction. IEEE Trans. Inf. Theory 55, 2230–2249 (2009) 2. DeVore, R.A.: Nonlinear approximation. Acta Numer. 51–150 (1998) 3. Dilworth, S.J., Kalton, N.J., Kutzarova, D.: On the existence of almost greedy bases in Banach spaces. Stud. Math. 158, 67–101 (2003) 4. Dilworth, S.J., Kalton, N.J., Kutzarova, D., Temlyakov, V.N.: The thresholding greedy algorithm, greedy bases, and duality. Constr. Approx. 19, 575–597 (2003) 5. Donoho, D.L.: Sparse components of images and optimal atomic decompositions. Constr. Approx. 17, 353–382 (2001) 6. Gilbert, A.C., Muthukrishnan, S., Strauss, M.J.: Approximation of functions over redundant dictionaries using coherence. In: The 14th Annual ACM-SIAM Symposium On Discrete Algorithms, Baltimore (2003) 7. Huber, P.J.: Projection pursuit. Ann. Stat. 13, 435–475 (1985) 8. Jones, L.: On a conjecture of Huber concerning the convergence of projection pursuit regression. Ann. Stat. 15, 880–882 (1987) 9. Konyagin, S.V., Temlyakov, V.N.: A remark on greedy approximation in Banach spaces. East J. Approx. 5, 365–379 (1999) 10. Konyagin, S.V., Temlyakov, V.N.: Greedy approximation with regard to bases and general minimal systems. Serdica Math. J. 28, 305–328 (2002) 11. Needell, D., Vershynin, R.: Uniform uncertainty principle and signal recovery via regularized orthogonal matching pursuit. Found. Comput. Math. 9, 317–334 (2009) 12. Schmidt, E.: Zur theorie der linearen und nichtlinearen Integralgleichungen. I. Math. Ann. 63, 433–476 (1906) 13. Temlyakov, V.N.: The best m-term approximation and greedy algorithms. Adv. Comp. Math. 8, 249–265 (1998)

G

614 14. Temlyakov, V.N.: Nonlinear methods of approximation. Found. Comput. Math. 3, 33–107 (2003) 15. Temlyakov, V.N.: Greedy approximation. Acta Numer. 17, 235–409 (2008) 16. Temlyakov, V.N.: Greedy Approximation. Cambridge University Press, Cambridge (2011) 17. Tropp, J.A.: Greed is good: algorithmic results for sparse approximation. IEEE Trans. Inf. Theory 50, 2231–2242 (2004) 18. Wojtaszczyk, P.: Greedy algorithms for general systems. J. Approx. Theory 107, 293–314 (2000) 19. Wojtaszczyk, P.: Greedy Type Bases in Banach Spaces. In: Constructive Function Theory, pp. 1–20. Darba, Sofia (2002)

Group Velocity Analysis Geir K. Pedersen Department of Mathematics, University of Oslo, Oslo, Norway

The dynamics of a wave train is governed by two different celerities, namely, the phase velocity and the group velocity. Each crest, or trough, moves with the phase velocity, while the energy moves with the group velocity. As a consequence, regions with high amplitudes in a wave train move with the group velocity, while individual crests, moving with the phase velocity, may move into the high-amplitude sequence and leave it again. In a nonuniform wave train, the group velocity also defines the speed at which a given wavelength is propagated. Hence, it is the differences in the group velocity which cause dispersion of waves. A harmonic wave mode may be given as    D A sin kx x C ky y C kz z  !.kx ; ky ; kz /t ; (1)

Group Velocity Analysis

cx.g/ D

@! ; @kx

cy.g/ D

@! ; @ky

cz.g/ D

@! : @kz

(3)

For isotropic media, where ! depends only on the norm, k, of the wave number vector and not the direction, the group velocity c.g/ is parallel to k. Moreover, we find the relation jc.g/ j D c .g/ D c C k

dc ; dk

which shows that for normal dispersion (phase velocity increases with wavelength) c .g/ < c, while c .g/ > c for abnormal dispersion. When c is constant, we have ! D ck and c.g/ D ck=k. Then, phase and group speeds equal the same constant and the waves are nondispersive. For periodic gravity surface waves in deep water, such p as swells, the dispersion relation becomes ! D have normal dispersion c D pgk. In this.g/case we 1 .g/ g=k and c D 2 c . For such waves the energy density is E D 12 gA2 , where A is the amplitude of the vertical surface excursion. For A D 3 m this yields an energy density of 44 KJ=m2 , which for a wave period of 20 s yields a flux density of cg E D 0:69 MW=m. Surface waves which are much shorter than 1:7 cm are dominated bypcapillary effects, and the phase speed becomes c D T k=, where T is the surface tension. In this case c decreases with wavelength and the group velocity, c .g/ D 32 c, is larger than the phase velocity. The group velocity is a crucial concept for description of nonuniform wave systems. It does appear in the asymptotic stationary-phase approximation, which goes back to the famous works by Cauchy and Poisson. A historic review is found in Craik [1], while recent descriptions are given in many textbooks, for instance, Mei et al. [2]. Combining harmonic modes such as (1) the solution for a wave system evolving from an initial surface elevation take on the form

where kx , ky , and kz are the components of the wave number vector, k, in the x, y, and z directions, respectively. The wave number vector is normal to the phase Z1 lines, such as the crests, and has a norm k D 2=, 1 .k/e i.kx!.k/t /d k: .x; t/ D where  is the wavelength. The frequency and the 2 i 1 wave number vector are related through the dispersion relation For large x and t, the exponent in the integral (2) ! D !.kx ; ky ; kz /: oscillates rapidly and the dominant contributions While the phase speed is c D !=k, the components of come from the vicinity of stationary points where group velocity are defined as the derivative of the exponent with respect to k

Group Velocity Analysis

is zero, which implies cg D x=t. This means that at a given location, the solution is dominated by the wavelength with a group velocity that conveys the energy to this location in the given time. A related theory is that of geometrical and physical optics. Rays are there defined as trajectories traversed by the group velocity, along which frequency and wavelength are preserved or are governed by simple evolution equations. Then the energy flux, or the wave action flux if a background current is present, is constant in ray tubes. Details are given by, for instance, Whitham [5] and Peregrine [3]. A celebrated example on the application of optics is the Kelvin ship-wave pattern [4]. For a point source generating gravity waves in deep water, the relation c .g/ D 12 c readily implies that the wave pattern is confined to a wedge of angle 38:94ı behind the source, as shown in Fig. 1. If the water is shallow, in the sense that generated waves are not short in comparision to the depth, this angle is larger. For a moving water beetle or a straw in a current, the waves generated are so short that they are dominated by capillary effects. In this case c .g/ > c and the waves are upstream of the disturbance but are rapidly damped by viscous effects due to their short wavelength. In narrow-band (in wavelength) approximations, the evolution is most transparently described in a coordinate system moving with a typical group velocity. Evolution equations for the amplitude may then be obtained by simplified equations, such as the cubic Schr¨odinger equation (see [2]).

615

ct

=

U

cos

qt

cg

t

Ut

Group Velocity Analysis, Fig. 1 A sketch of half the wave system generated by a point source in the surface moving from C to ı with speed U in time t . Since the wave system is stationary, c D U cos , where is angle between the direction of wave advance and the course of the source. The outer semicircle corresponds to locations reached by such phase speeds from C, while the waves actually may reach only the inner semicircle due to c .g/ D 12 c. The wedge angle then becomes arcsin.1=3/. Two types of waves, named transverse and diverging, may emerge as indicated by the fully drawn curves

References 1. Craik, A.D.: The origins of water wave theory. Annu. Rev. Fluid Mech. 36, 1–28 (2004) 2. Mei, C.C., Stiassnie, M., Yue, D.K.P.: Theory and Applications of Ocean Surface Waves. World Scientific, Singapore/Hackensack (2005) 3. Peregrine, D.H.: Interaction of water waves and currents. Adv. Appl. Mech. 16, 10–117 (1976) 4. Wehausen, J.V., Laitone, E.V.: Surface Waves in Fluid Dynamics III, vol. 9, pp. 446–778. Springer, Berlin (1960). Chapter 3 5. Whitham, G.B.: Linear and Nonlinear Waves. Pure & Applied Mathematics. Wiley, New York (1974)

G

H

Hamiltonian Systems J.M. Sanz-Serna Departamento de Matem´atica Aplicada, Universidad de Valladolid, Valladolid, Spain

It is sometimes useful to rewrite (1) in the compact form d y D J 1 rH.yI t/; (2) dt where y D .p; q/, rH D .@H=@p1 ; : : : ; @H=@pd I @H=@q1 ; : : : ; @H=@qd / and  0d d Id d : J D Id d 0d d 

Synonyms

(3)

Canonical systems

Origin of Hamiltonian Systems Definition Let I be an open interval of the real line R of the variable t (time) and ˝ a domain of the Euclidean space Rd  Rd of the variables .p; q/, p D .p1 ; : : : ; pd /, q D .q1 ; : : : ; qd /. If H.p; qI t/ is a real smooth function defined in ˝  I , the canonical or Hamiltonian system associated with H is the system of 2d scalar ordinary differential equations

mj rR j D Fj ;

j D 1; : : : ; N:

(4)

In the conservative case, where the force Fj is the gradient with respect to rj of a scalar potential V , that is,

@H d pi D  .p; qI t/; dt @qi d @H qi D C .p; qI t/; dt @pi

Newton’s Second Law in Hamiltonian Form Consider the motion of a system of N point masses in three-dimensional space (cases of interest range from stars or planets in celestial mechanics to atoms in molecular dynamics). If rj denotes the radius vector joining the origin to the j -th point, Newton’s equations of motion read

i D 1; : : : ; d: (1)

The function H is called the Hamiltonian, d is the number of degrees of freedom, and ˝ the phase space. Systems of the form (1) (which may be generalized in several ways, see below) are ubiquitous in the applications of mathematics; they appear whenever dissipation/friction is absent or negligible.

Fj D rrj V .r1 ; : : : ; rN I t/;

j D 1; : : : ; N; (5) the system (4) may be rewritten in the Hamiltonian form (1) with d D 3N by choosing, for j D 1; : : : ; N , .p3j 2 ; p3j 1 ; p3j / as the cartesian components of the momentum pj D mj rP j of the j -th mass, .q3j 2 ; q3j 1 ; q3j / as the cartesian components of rj , and setting

© Springer-Verlag Berlin Heidelberg 2015 B. Engquist (ed.), Encyclopedia of Applied and Computational Mathematics, DOI 10.1007/978-3-540-70529-1

618

Hamiltonian Systems

1X 1 2 1 T D p D p T M 1 p 2 j D1 mj j 2 N

H D T C V;

(6) (here M is the 3N  3N diagonal mass matrix diag.m1 ; m1 ; m1 I : : : I mN ; mN ; mN /). The Hamiltonian H coincides with the total, kinetic + potential, mechanical energy. Lagrangian Mechanics in Hamiltonian Form Conservative systems S more complicated than the one just described (e.g., systems including rigid bodies and/or constraints) are often treated within the Lagrangian formalism [1,3], where the configuration of S is (locally) described by d (independent) Lagrangian coordinates qi . For instance, the motion of a point on the surface of the Earth – with two degrees of freedom – may be described by the corresponding longitude and latitude, rather than by using the three (constrained) cartesian coordinates. The movements are then governed by the coupled second-order differential equations @L d @L  D 0; dt @qPi @qi

i D 1; : : : ; d;

The Hamiltonian Formalism in Quantum and Statistical Mechanics In the context of classical mechanics the transition from the Lagrangian format (7) to the Hamiltonian format (1) is mainly a matter of mathematical convenience as we shall discuss below. On the contrary, in other areas, including for example, quantum and statistical mechanics, the elements of the Hamiltonian formalism are essential parts of the physics of the situation. For instance, the statistical canonical ensemble associated with (4)–(5) possesses a density proportional to exp.ˇH /, where H is given by (6) and ˇ is a (7) constant.

where L D L.q; qI P t/ is the Lagrangian function of S. For each i D 1; : : : ; d , pi D @L=@qPi represents the generalized momentum associated with the coordinate qi . Under not very demanding hypotheses, the transformation .q; P q/ 7! .p; q/ may be inverted (i.e., it is possible to retrieve the value of the velocities qP from the knowledge of the values of the momenta p and coordinates q) and then (7) may be rewritten in the P t/, where, in form (1) with H.p; qI t/ D p T qP  L.q; qI the right-hand side, it is understood that the velocities have been expressed as functions of p and q (this is an instance of a Legendre’s transformation, see [1], Sect. 14). The function H often corresponds to the total mechanical energy in the system S. Calculus of Variations According to Hamilton’s variational principle of least action (see, e.g., Sect. 13 in [1] or Sects. 2-1–2-3 in [3]), the motions of the mechanical system S, we have just described, are extremals of the functional (action) Z

t1 t0

L.q.t/; q.t/I P t/ dtI

in fact (7) are just the Euler-Lagrange equations associated with (8). The evolutions of many (not necessarily mechanical) systems are governed by variational principles for functionals of the form (8). The corresponding Euler-Lagrange equations (7) may be recast in the first order format (1) by following the procedure we have just described ([2], Vol. I, Sects. IV and 9). In fact Hamilton first came across differential equations of the form (1) when studying Fermat’s variational principle in geometric optics.

(8)

First Integrals Assume that, for some index i0 , the Hamiltonian H is independent of the variable qi0 . It is then clear from (1) that, for any solution .p.t/; q.t// of (1) the value of pi0 remains constant; in other words the function pi0 is a first integral or conserved quantity of (1). In mechanics, this is expressed by saying that the momentum pi0 conjugate to the cyclic coordinate qi0 is a constant of motion; for instance, in the planar motion of a point mass in a central field, the polar angle is a cyclic coordinate and this implies the conservation of angular momentum (second Kepler’s law). Similarly qi0 is a first integral whenever H is independent of pi0 . In the autonomous case where the Hamiltonian does not depend explicitly on t, that is, H D H.p; q/ a trivial computation shows that for solutions of (1) .d=dt/H.p.t/; q.t// D 0, so that H is a constant of motion. In applications this often expresses the principle of conservation of energy.

Hamiltonian Systems

619

and the hyperbolic rotation p  D exp./p, q  D exp./q ( is an arbitrary constant).

opposite orientation. One may say that, when d D 1, a transformation is canonical if and only if it preserves oriented area. For d > 1 the situation is similar, if slightly more complicated to describe. It is necessary to consider two-dimensional bounded surfaces D  ˝ and orient them by choosing one of the two orientations of the boundary curve @D. The surface D is projected onto each of the d two-dimensional planes of the variables .pi ; qi / to obtain d two-dimensional domains ˘i .D/ with oriented boundaries; then we compute the number P S.D/ D i ˙Area.˘i .D//, where, when summing, a term is taken with the C (resp. with the ) sign if the orientation of the boundary of ˘i .D/ coincides with (resp. is opposite to) the standard orientation of the .pi ; qi / plane. Then a transformation  is canonical if and only if S.D/ D S. .D// for each D. In Euclidean geometry, a planar transformation that preserves distances automatically preserves areas. Similarly, it may be shown that the preservation of the sum S.D/ of oriented areas implies the preservation of similar sums of oriented 4–, 6–, . . . , 2d -dimensional measures (the so-called Poincar´e integral invariants). In particular a symplectic transformation preserves the orientation of the 2d -dimensional phase space (i.e., its Jacobian determinant  is > 0) and also preserves volume: for any bounded domain V  ˝, the volumes (ordinary Lebesgue measures) of V and  .V / coincide. The preceding considerations (and for that matter most results pertaining to the Hamiltonian formalism) are best expressed by using the language of differential forms. Lack of space makes it impossible to use that alternative language here and the reader is referred to [1], Chap. 8 (see also [6], Sect. 2.4).

Geometric Interpretation Consider first the case d D 1 where, in view of (10), canonicity means that the Jacobian determinant  D  det @.p  ; q  /[email protected]; q/ takes the constant value 1. The fact j  j D 1 entails that for any bounded domain D  ˝, the areas of D and  .D/ coincide. Furthermore  > 0 means that  is orientation preserving. Thus, the triangle with vertices A D .0; 0/, B D .1; 0/, C D .0; 1/ cannot be symplectically mapped onto the triangle with vertices A D .0; 0/, B  D .1; 0/, C  D .0; 1/ in spite of both having the same area, because the boundary path A ! B  ! C  ! A is oriented clockwise and A ! B ! C ! A has the

Changing Variables in a Hamiltonian System Assume that in (1) we perform an invertible change of variables y D .z/ where  is canonical. A straightforward application of the chain rule shows that the new system, that is, .d=dt/z D .0 .z//1 J 1 ry H..z/I t/, coincides with the Hamiltonian system .d=dt/z D J 1 rz K.zI t/ whose Hamiltonian function K.zI t/ D H..z/I t/ is obtained by expressing the old H in terms of the new variables. In fact, if one looks for a condition on y D .z/ that ensures that in the zvariables (1) becomes the Hamiltonian system with Hamiltonian H..z/I t/, one easily discovers the definition of canonicity in (9). The same exercise shows

Canonical Transformations The study of Hamiltonian systems depends in an essential way on that of canonical or symplectic transformations. Definition With the compact notation in (2), a differentiable transformation y  D .p  ; q  / D  .y/,  W ˝ ! Rd  Rd is called canonical (or symplectic) if its Jacobian matrix  0 .y/, with .i; j / entry @yi =@yj , satisfies, for each y D .p; q/ in ˝,  0 .y/T J  0 .y/ D J:

(9)

The composition of canonical transformations is canonical; the inverse of a canonical transformation is also canonical. By equating the entries of the matrices in (9) and taking into account the skew-symmetry, one sees that (9) amounts to d.2d  1/ independent scalar equations for the derivatives @yi =@yj . For instance, for d D 1, (9) is equivalent to the single relation @p  @q  @p  @q    1: @p @q @q @p

(10)

Simple examples of canonical transformations with d D 1 are the rotation p  D cos./psin./q;

q  D sin./pCcos./q (11)

H

620

that the matrix J in (9) and that appearing in (2) have to be inverses of one another. This most important result has of course its counterpart in Lagrangian mechanics or, more generally, in the calculus of variations (see [1], Sect. 12D or [2], Vol. I, Sects. IV and 8): to change variables in the EulerLagrange equations for (8) it is enough to first change variables in L and then form the Euler-Lagrange equations associated with the new Lagrangian function. However, in the Lagrangian case, only the change of the d coordinates q D .w/ is at our disposal; the choice of  determines the corresponding formulae P In the Hamiltonian for the velocities qP D  0 .w/w. case, the change y D .z/ couples the 2d -dimensional y D .p; q/ with the 2d -dimensional z and the class of possible transformations is, therefore, much wider. Jacobi’s method (see below) takes advantage of these considerations.

Hamiltonian Systems

Generating Function S1 Given a canonical transformation .p  ; q  / D  .p; q/, let us define locally a function S such that dS is given by (12) and assume that @.q  ; q/[email protected]; q/ is non-singular. Then, in lieu of .p; q/, we may take .q  ; q/ as independent variables and express S in terms of them to obtain a new function S1 .q  ; q/ D S.p.q  ; q/; q/, called the generating function (of the first kind) of the transformation. From (12) @S1 D pi ; @qi

@S1 D pi ; @qi

i D 1; : : : ; d I (13)

the relations in the first group of (13) provide d coupled equations to find the qi as functions of .p; q/ and those in the second group then allow the explicit computation of p  . For (11) the preceding construction yields S1 D .cot./=2/.q 2  2 sec./qq  C q 2 /, (provided that sin./ ¤ 0), an expression that, via (13) leads back to (11). Exact Symplectic Transformations Conversely, if S1 .q  ; q/ is any given function and A transformation .p  ; q  / D  .p; q/, .p; q/ 2 ˝ is the relations (13) define uniquely .p  ; q  / as functions said to be exact symplectic if of .p; q/, then .p; q/ 7! .p  ; q  / is a canonical   pdq  p dq transformation ([1], Sect. 47A). 0 1  d d  X X @qi @q  @pi dqi  pi D dpj C i dqj A Other Generating Functions @pj @qj The construction of S1 is only possible under the i D1 j D1  (12) assumption that .q ; q/ may be taken as independent variables. This assumption does not hold in many is the differential of a real-valued function S.p; q/ important cases, including that where  is the identity defined in ˝. transformation (with q  D q). It is therefore useful to For (12) to coincide with dS it is necessary but not introduce a new kind of generating function as follows: sufficient to impose the familiar d.2d  1/ relations If (12) is the differential of S.p; q/ (perhaps only arising from the equality of mixed second order deriva- locally), then d.p T q  C S / D q T dp  C p T dq. If tives of S . It is trivial to check that those relations @.p  ; q/[email protected]; q/ is non-singular, .p  ; q/ may play the coincide with the d.2d  1/ relations implicit in role of independent variables and if we set S2 .p  ; q/ D (9) and therefore exact symplectic transformations are p T q  .p  ; q/ C S.p.p  ; q/; q/ it follows that always symplectic. In a simply connected domain ˝, @S2 @S2  symplectic transformations are also exact symplectic; D pi ; i D 1; : : : ; d I (14)  D qi ; @q @p i i in a general ˝, a symplectic transformation is not necessarily exact symplectic and, when it is not, the here the first equations determine the p  as funci function S only exists locally. tions of .p; q/ and the second yield the q  explicitly. i

Generating Functions: Hamilton-Jacobi Theory Generating functions provide a convenient way of expressing canonical transformations.

The function S2 is called the generating function of the 2nd kind of  . The identity transformation is generated by S2 D p T q. For (11) with cos./ ¤ 0 (which ensures that .p  ; q/ are independent) we find: S2 D

 tan./  2 q C 2 csc./qp  C p 2 : 2

(15)

Hamiltonian Systems

621

Conversely if S2 .p  ; q/ is any given function and the relations (14) define uniquely .p  ; q  / as functions of .p; q/, then .p; q/ 7! .p  ; q  / is a canonical transformation ([1], Sect. 48B). Further kinds of generating functions exist ([3], Sect. 9-1, [1], Sect. 48). The Hamilton-Jacobi Equation In Jacobi’s method to integrate (1) (see [1], Sect. 47 and [3], Sect. 10-3) with time-independent H , a canonical transformation (14) is sought such that, in the new variables, the Hamiltonian K D H.p.p  ; q  /; q.p  ; q  // is a function K D K.p  / of the new momenta alone, that is, all the qi are cyclic. Then in the new variables – as pointed out above – all the pi are constants of motion and therefore the solutions of the canonical equations are given by pi .t/ D pi .0/, qi .t/ D qi .0/ C t.@K=@pi /p .0/ . Inverting the change of variables yields of course the solutions of (1) in the originally given variables .p; q/. According to (14) the required S2 .q  ; q/ has to satisfy the Hamilton-Jacobi equation  H

@S2 @S2 ; :::; ; q1 ; : : : ; qd @q1 @qd



D K.p1 ; : : : ; pd /:

This is a first-order partial differential equation ([2], Vol. II, Chap. II) for the unknown S2 called the characteristic function; the independent variables are .q1 ; : : : ; qd / and it is required to find a particular solution that includes d independent integration constants p1 , . . . , pd (a complete integral in classical terminology). Jacobi was able to identify, via separation of variables, a complete integral for several important problems unsolved in the Lagragian format. His approach may also be used with S1 and the other kinds of generating functions. Time-dependent Generating Functions So far we have considered time-independent canonical changes of variables. It is also possible to envisage changes .p  ; q  / D  .p; qI t/, where, for each fixed t,  is canonical. An example is afforded by (14) if the generating function includes t as a parameter: S2 D S2 .p  ; qI t/. In this case, the evolution of .p  ; q  / is governed by the Hamiltonian equations (1) associated with the Hamiltonian K D H C @S2 =@t, where in the right-hand side it is understood that the arguments .p; q/ of H and .p  ; q/ of S2 have been expressed as

functions of the new variables .p  ; q  / with the help of formulae (14). Note the contribution @S2 =@t that arises from the time-dependence of the change of variables. If S2 D S2 .p  ; qI t/ satisfies the Hamilton-Jacobi equation  H

 @S2 @S2 @S2 D 0; (16) ; :::; ; q1 ; : : : ; qd I t C @q1 @qd @t

then the new Hamiltonian K vanishes identically and all pi and qi remain constant; this trivially determines the solutions .p.t/; q.t// of (1). In (16) the independent variables are t and the qi and it is required to find a complete solution, that is, a solution S2 that includes d independent integration constants pi . It is easily checked that, conversely, (1) is the characteristic system for (16), so that it is possible to determine all solutions of (16) whenever (1) may be integrated explicitly ([2], Vol. II, Chap. II).

Hamiltonian Dynamics Symplecticness of the Solution Operator H the solution operator of (1) (t, t0 are We denote by ˚t;t 0 H is a real numbers in the interval I ). By definition, ˚t;t 0 0 0 transformation that maps the point .p ; q / in ˝ into the value at time t of the solution of (1) that satisfies the initial condition p.t0 / D p 0 , q.t0 / D q 0 . Thus, H .p 0 ; q 0 / we keep t0 , p 0 , and q 0 fixed and if in ˚t;t 0 let t vary, then we recover the solution of the initialvalue problem given by (1) in tandem with p.t0 / D p 0 , q.t0 / D q 0 . However, we shall be interested in seeing t and t0 as fixed parameters and .p 0 ; q 0 / as a variable H represents a transformation mapping the so that ˚t;t 0 H not to phase space into itself. (It is possible for ˚t;t 0 be defined in the whole of ˝; this happens when the solutions of the initial value problem do not exist up to time t.) Note that ˚tH2 ;t0 D ˚tH2 ;t1 ı ˚tH1 ;t0 for each t0 , t1 , t2 (the circle ı means composition of mappings). In H depends the autonomous case where H D H.y/, ˚t;t 0 only on the difference t  t0 and we write tHt0 instead H ; then the flow tH has the group property: of ˚t;t 0 H t Cs D tH ı sH , for each t and s. The key geometric property of Hamiltonian systems H is that ˚t;t is, for each fixed t0 and t, a canonical 0 transformation ([1], Sect. 44). In fact the canonicity of the solution operator is also sufficient for the system to be Hamiltonian (at least locally).

H

622

The simplest illustration is provided by the harmonic oscillator: d D 1, H D .1=2/.p 2 C q 2 /. The t-flow .p  ; q  / D t .p; q/ is of course given by (11) with  D t, a transformation that, as remarked earlier, is canonical. The group property of the flow is the statement that rotating through  radians and then through  0 radians coincides with a single rotation of amplitude  C  0 radians. The Generating Function of the Solution Operator Assume now that we subject (1) to the t-dependent .p; q/ canonical change of variables .p  ; q  / D tH 0 ;t with t0 fixed. The new variables .p  ; q  / remain con.p.t/; q.t// D tH ı stant: .p  .t/; q  .t// D tH 0 ;t 0 ;t H t;t0 .p.t0 /; q.t0 // D .p.t0 /; q.t0 //. Therefore, the new Hamiltonian K.p  ; q  I t/ must vanish identically and must satisfy (16). the generating function S2 of tH 0 ;t Now, as distinct from the situation in Jacobi’s method, we are interested in solving the initial value problem given by Hamilton-Jacobi equation (16) and the initial condition S.p  ; qI t0 / D p T q (for t D t0 the is the identity). transformation tH 0 ;t As an illustration, for the harmonic oscillator, as is given by (11) with  D t0  t; a noted above, tH 0 ;t simple computation shows that its generating function found in (15) satisfies the Hamilton-Jacobi equation .1=2/ .@S2 =@q/2 C q 2 C @S2 =@t D 0. Symplecticness Constrains the Dynamics H has a marked impact on the The canonicity of ˚t;t 0 long-time behavior of the solutions of (1). As a simple example, consider a system of two scalar differential equations pP D f .p; q/, qP D g.p; q/ and assume that .p 0 ; q 0 / is an equilibrium where f D g D 0. Generically, that is, in the “typical” situation, the equilibrium is hyperbolic: the real parts of the eigenvalues 1 and 2 of the Jacobian matrix @.f; g/[email protected]; q/ evaluated at .p 0 ; q 0 / have nonzero real part and the equilibrium is a sink ( m=2 C 1, U0 belongs to the Sobolev space W k;2 .Rm /, that is, its partial derivatives up to order k are square integrable over Rm , then there exists a unique continuously differentiable, classical solution U of (1) on Rm  Œ0; T / satisfying the initial condition U.x; 0/ D U0 .x/, for x in Rm . The time interval Œ0; T / of existence is maximal, in that either T D 1 or else T < 1 and maxx jrU.x; t/j ! 1, as t " T. The above result has a linear flavor as it rests on the linearized form of (1). The proof employs L2 bounds on U and its derivatives, up to order k, derived through “energy” type estimates induced by the extra conservation law (5). Existence theorems with similar flavor also apply to initial-boundary value problems for (1), on some domain  of Rm with boundary @. The initial data U0 , prescribed on , must be sufficiently smooth and compatible with the boundary conditions on @. Identifying the class of boundary conditions that render the initial-boundary value problem wellposed is a highly technical affair. The situation where the maximal time interval Œ0; T / of existence of a classical solution is finite is the rule rather than the exception. This is due to the wave breaking phenomenon, which may be seen, for instance, in the context of Burgers’ equation. Beyond the time waves break, one has to resort to weak solutions, namely, to bounded measurable functions U that satisfy (1) in the sense of distributions.

m X

˛ ŒŒG˛ .U / ;

(6)

˛D1

where the double bracket ŒŒ denotes the jump of the enclosed quantity across the shock, and the unit vector  in Rm and the scalar s are such that the vector .; s/ in RmC1 is normal to the shock. Thus, the shock may be realized as a wave, in the form of a surface in Rm , moving in the direction  with speed s. Shocks play a central role in hyperbolic conservation laws; see the entry on Riemann problems, in this Encyclopedia. The first major difficulty with weak solutions is loss of uniqueness of solutions for the Cauchy problem. This is easily seen in the context of the Riemann problem for Burgers’ equation. As a remedy, conditions have been proposed, with physical or mathematical provenance, that may weed out spurious weak solutions. Such a condition, with universal appeal, deems admissible solutions that satisfy the inequality

@t .U.x; t// C

m X

@˛ Q˛ .U.x; t//  0;

(7)

˛D1

in the sense of distributions. In particular, classical solutions are admissible as they satisfy (5). When (1) arises in physics, (7) expresses the Second Law of thermodynamics. In connection to the Cauchy problem, the above entropy admissibility condition anoints classical solutions: so long as it exists, any classical solution is unique and L2 -stable, not only in relation to other classical solutions, but even within the broader class of admissible weak solutions. On the other hand, in the absence of a classical solution, the entropy admissibility condition is not sufficiently discriminating to single out a unique admissible weak solution. This has been demonstrated in the context of the Euler equations. Consequently, the question of uniqueness of solutions to the Cauchy problem, at the level of generality discussed here .m > 1; n > 1/, is still open.

Hyperbolic Conservation Laws: Analytical Properties

661

The question of existence of weak solutions to the Cauchy problem is also open, and the situation looks even grimmer, as there are indications that this problem may be wellposed only for a special class of hyperbolic systems of conservation laws. Fortunately, this class includes the cases n D 1 and/or m D 1, for which much has been accomplished, as explained below. Let us first consider the case of a scalar conservation law, (1) with n D 1; m 1, so that U and the G˛ are scalar-valued. An important feature of this class is that any convex function .U / may serve as an entropy, with entropy flux Q.U / determined by integrating (4). Accordingly, a weak solution U is called admissible if it satisfies the inequality (7) for every convex function .U /. Because of this, very stringent, requirement, admissible weak solutions enjoy the following strong stability property. If U and V are any two admissible weak solutions to the Cauchy problem, with initial data U0 and V0 , then there is s > 0 such that, for any t > 0 and r > 0, Z

Z jU.x; t/V .x; t/jdx  jxj 0; U.; t/ has bounded variation and T V.1;1/ U.; t/  aT V.1;1/ U0 ./;

0 < t < 1: (12)

If U and V are admissible solutions with initial data U0 and V0 , Z

1

jU.x; t/  V .x; t/jdx

˛D1 1

and the kinetic formulation, which also yields valuable information on the regularity of weak solutions. The

Z

1

a 1

jU0 .x/  V0 .x/jdx;

0 < t < 1:

(13)

H

662

Hyperbolic Conservation Laws: Computation

When the initial data have large total variation, the total variation or the L1 norm of U.; t/ may blow up in finite time. Identifying the class of systems for which this catastrophe does not take place is currently a major open problem. Other features, such as the large time behavior of solutions to the Cauchy problem for (10), have also been investigated extensively. In the bibliography below, entries [1–5] are textbooks, listed in progressive order of technical complexity; [6] is an attempt for an encyclopedic presentation of the whole area, and it includes extensive bibliography; [7–10] are seminal papers.

describing the evolution of conserved quantities U 2 Rn according to flux function G W Rn ! Rn . (Herein, we only consider the 1D case for brevity.) Solutions of (1) admit various kinds of nonlinear and discontinuous waves. Numerical methods developed to accurately compute such waves have significantly influenced developments in modern computational science. Methods come in two forms: shock-fitting methods in which discontinuities are introduced explicitly in the solution and shock-capturing methods in which numerical dissipation is used to capture discontinuities within a few grid cells.

References

Classical Shock-Capturing Methods

1. Lax, P.D.: Hyperbolic Partial Differential Equations. American Mathematical Society, Providence (2006) 2. Smoller, J.A.: Shock Waves and Reaction-Diffusion Equations, 2nd edn. Springer, New York (1994) 3. Holden, H., Risebro, N.H.: Front Tracking for Hyperbolic Conservation Laws. Springer, New York (2002) 4. Bressan, A.: Hyperbolic Systems of Conservation Laws. Oxford University Press, Oxford (2000) 5. Serre, D.: Systems of Conservation Laws, vols. I, II. Cambridge University Press, Cambridge/New York (1999) 6. Dafermos, C.M.: Hyperbolic Conservation Laws in Continuum Physics, 3rd edn. Springer, Heidelberg/London (2010) 7. Lax, P.D.: Hyperbolic systems of conservation laws. Commun. Pure Appl. Math. 10, 537–566 (1957) 8. Glimm, J.: Solutions in the large for nonlinear hyperbolic systems of equations. Commun. Pure Appl. Math. 18, 697– 715 (1965) 9. Kruzkov, S.: First-order quasilinear equations with several space variables. Math Sbornik 123, 228–255 (1970) 10. Bianchini, S., Bressan, A.: Vanishing viscosity solutions of nonlinear hyperbolic systems. Ann. Math. 161, 223–342 (2005)

Equation 1 is not valid in the classical pointwise sense for discontinuous solutions. Instead, we will work with the integral form of (1). Introducing the sliding average R xCx=2 1 UN .x; t/ D x xx=2 U.; t/ d  gives the system of evolution equations

Hyperbolic Conservation Laws: Computation Knut-Andreas Lie Department of Applied Mathematics, SINTEF ICT, Oslo, Norway

Z t Ct 1 N N U .x; t C t/ D U .x; t/  x t   

 x x ; /  G U.x  ; / d : (2) G U.x C 2 2 Next, we partition the physical domain ˝ into a set of grid cells ˝i D Œxi 1=2 ; xi C1=2 and set t n D nt. This suggests a numerical scheme   UinC1 D Uin  r GinC1=2  Gin1=2 ;

(3)

where ri D t=x, Uin D UN i .xi ; t n / are unknown cell averages, and the numerical flux functions Gin˙1=2 are approximations to the average flux over each cell interface, Gin˙1=2

1 t

Z

t nC1

  G U.xi ˙1=2 ; / d :

(4)

tn

Because (1) has finite speed of propagation, the numerical fluxes are given in terms of neighboring cell n n A conservation law is a first-order system of PDEs in averages; that is, G n i C1=2 D G.Ui p ; : : : ; Ui Cq / D divergence form n G.U I i C 1=2/. Schemes on the form (3) are called conservative. If a @t U.x; t/ C @x G.U.x; t// D 0; t 0; x 2 R; (1) sequence of approximations computed by a consistent

Hyperbolic Conservation Laws: Computation

663

and conservative scheme converges to some limit, then this limit is a weak solution of the conservation law [4]. Centered Schemes Assume that U.x; t n / is piecewise constant and equals Uin inside ˝i . The integrand of (4) can then be approximated by 12 .G.Uin˙1 / C G.Uin //. This yields a centered scheme that unfortunately is notoriously unstable. 2 2 To stabilize, we add artificial diffusion, x t @x U discretized using standard centered differences and obtain the classical first-order Lax–Friedrichs scheme [3] UinC1 D

1 n Ui C1 C Uin1 2 i 1 h  r G.UinC1 /  G.Uin1 / 2

(5)

( V .x; 0/ D

Uin ;

x < xi C1=2 ;

UinC1 ;

x xi C1=2 ;

(7)

each of which admits a self-similar solution V .x=t/. Cell averages can now be correctly evolved a time step t by (3) if we use V .0/, or a good approximation thereof, to evaluate G in (4). The time step t is restricted by the time it takes for the fastest Riemann wave to cross a single cell, t max jj j  1; x j

(8)

where 1      n are the eigenvalues of the Jacobian matrix DG.U /. The inequality (8) is called the CFL condition, named after Courant, Friedrichs, and Lewy, who wrote one of the first papers on finite difference methods in 1928 [1]. If t satisfies (8), the numerical scheme (3) will be stable. An alternative interpretation of (8) is that the domain of dependence for the PDE should be contained within the domain of dependence for (3) so that all information that will influence UinC1 has time to travel into ˝i .

which is very robust and will always converge, although sometimes painstakingly slow. To see this, consider the trivial case of a stationary discontinuity satisfying @t U D 0. In this case, (5) will simply compute UinC1 as the arithmetic average of the cell averages in the two neighboring cells. The Lax– Friedrichs scheme can be written in conservative Example 1 Consider the advection of a scalar quanform (3) using the numerical flux tity in a periodic domain. Figure 1 shows the profile evolved for ten periods by the upwind, Lax–Friedrichs, and Lax–Wendroff schemes. The first-order schemes  1 n G.U n I i C 1=2/ D Ui  UinC1 smear the smooth and discontinuous parts of the ad2r vected profile. The second-order scheme preserves  1 C G.Uin / C G.UinC1 / : (6) the smooth profile quite well, but introduces spurious 2 oscillations around the two discontinuities. The second-order Lax–Wendroff scheme is obtained by using the midpoint rule to evaluate (4), with midHigh-Resolution Schemes point values predicted by (5) with grid spacing 12 x. Upwind and Godunov Schemes In the scalar case, we obtain a particularly simple twopoint scheme by using one-sided differences in the upwind direction from which the characteristics are pointing; that is, setting GinC1=2 D G.Uin / if G 0 .U / 0, or GinC1=2 D G.UinC1 / if G 0 .U / < 0. Upwind differencing is the design principle underlying Godunov schemes [2]. If U.x; t n / D Uin in each grid cell ˝i , the evolution of U can be decomposed into a set of local Riemann problems @t V C @x G.V / D 0;

High-resolution schemes are designed to have secondorder spatial accuracy or higher in smooth parts and high accuracy around shocks and other discontinuities (i.e., a small number of cells containing the wave). They use nonlinear dissipation mechanisms to provide solutions without spurious oscillations. Flux-Limiter Schemes Let GL .U n I i C 1=2/ be a low-order flux (e.g., (6)) and GH .U n I i C 1=2/ be a high-order flux (e.g., the Lax–Wendroff flux). Then, using the flux GinC1=2 D GL .U n I i C 1=2/

H

664

Hyperbolic Conservation Laws: Computation upwind

Lax–Friedrichs

Lax–Wendroff 1.2

1.2

1.2 1

1

1

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0

0

−0.2 0

0.2

0.4

0.6

0.8

1

−0.2 0

0 0.2

0.4

0.6

0.8

1

−0.2 0

0.2

0.4

0.6

0.8

1

Hyperbolic Conservation Laws: Computation, Fig. 1 Approximate solutions after ten periods of linear advection within a periodic domain

xi−1/2

xi+1/2

xi+3/2

xi−1/2

xi

xi+1/2

xi+1

xi+3/2

Hyperbolic Conservation Laws: Computation, Fig. 2 Computation of sliding average for upwind methods (left) and central methods (right)

Cin GH .U n I i C 1=2/  GL .U n I i C 1=2/

 (9)

.x  xi / ; x

x 2 ˝i ;

(10)

one can ensure that the resulting scheme is totalin (3) gives a high-resolution scheme for an appropriate variation diminishing (TVD) under certain assumplimiter function in D .U n I i / that is close to unity if tions on the nonlinear slope limiter ˚. Likewise, U is smooth and close to zero if U is discontinuous. higher-order reconstructions can be designed to satisfy an essentially non-oscillatory (ENO) property. For the averaging, there are two fundamentally difSlope-Limiter Schemes ferent choices, see Fig. 2. In upwind methods (x D xi ), Shock-capturing schemes can be constructed using the the temporal integrals in (2) are evaluated at points general REA algorithm: 1. Starting from known cell averages Uin , reconstruct xi ˙1=2 where UO .x; t/ is discontinuous. Hence, one a piecewise polynomial function UO .x; t n / defined cannot apply standard integration and extrapolation for all x. Constant reconstruction in each cell gives techniques. Instead, one must resolve the wave struca first-order scheme, linear gives second order, ture arising due to the discontinuity, solving a Riemann problem or generalizations thereof. For central methquadratic gives third order, etc. 2. Next, we evolve the differential equation, exactly or ods (x D xi C1=2 ), the sliding average is computed over a staggered grid cell Œxi ; xi C1 . Under a CFL condition approximately, using UO .x; t n / as initial data. nC1 O 3. Finally, we average the evolved solution U .x; t / of one half, the integrand will remain smooth so that onto the grid again to obtain new cell averages standard integration and extrapolation techniques can be applied. UinC1 . In the reconstruction, care must be taken to avoid Example 2 Figure 3 shows the advection problem introducing spurious oscillations. Using a linear recon- from Example 1 computed by a second-order nonstruction [9], oscillatory central scheme [6] with two different UO .x; t n / D Uin C ˚.Uin  Uin1 ; UinC1  Uin /

limiters. The dissipative minmod limiter always chooses the lesser slope and thus behaves more like

Hyperbolic Conservation Laws: Computation Hyperbolic Conservation Laws: Computation, Fig. 3 Linear advection problem computed by a second-order scheme with two different limiters

665 Minmod

Superbee 1.2

1.2 1

1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0

0

−0.2

−0.2

0

0.2

0.4

0.6

a first-order scheme. The compressive superbee limiter picks steeper slopes and flattens the top of the smooth wave. Computational Efficiency Explicit high-resolution schemes are essentially stencil computations that have an inherent parallelism that can be exploited to ensure computational efficiency. Moreover, high arithmetic intensity (i.e., large number of computations per data fetch) for high-order methods means that these methods can relatively easily exploit both message-passing systems and many-core hardware accelerators.

References ¨ 1. Courant, R., Friedrichs, K., Lewy, H.: Uber die partiellen differenzengleichungen der mathematischen Phys. Math. Ann. 100(1), 32–74 (1928) 2. Godunov, S.K.: A difference method for numerical calculation of discontinuous solutions of the equations of hydrodynamics. Mat. Sb. (NS) 47(89), 271–306 (1959)

0.8

1

0

0.2

0.4

0.6

0.8

1

3. Lax, P.D.: Weak solutions of nonlinear hyperbolic equations and their numerical computation. Commun. Pure Appl. Math. 7, 159–193 (1954) 4. Lax, P.D., Wendroff, B.: Systems of conservation laws. Commun. Pure Appl. Math. 13, 217–237 (1960) 5. LeVeque, R.J.: Finite Volume Methods for Hyperbolic Problems. Cambridge Texts in Applied Mathematics. Cambridge University Press, Cambridge (2002) 6. Nessyahu, H., Tadmor, E.: Nonoscillatory central differencing for hyperbolic conservation laws. J. Comput. Phys. 87(2), 408–463 (1990) 7. Toro, E.F.: Riemann Solvers and Numerical Methods for Fluid Dynamics, 3rd edn. Springer, Berlin, a practical introduction (2009) 8. Trangenstein, J.A.: Numerical Solution of Hyperbolic Partial Differential Equations. Cambridge University Press, Cambridge (2009) 9. van Leer, B.: Towards the ultimate conservative difference scheme, V. A second order sequel to Godunov’s method. J. Comput. Phys. 32, 101–136 (1979)

H

I

Immersed Interface/Boundary Method Kazufumi Ito and Zhilin Li Center for Research in Scientific Computation and Department of Mathematics, North Carolina State University, Raleigh, NC, USA

Introduction The immersed interface method (IIM) is a numerical method for solving interface problems or problems on irregular domains. Interface problems are considered as partial differential equations (PDEs) with discontinuous coefficients, multi-physics, and/or singular sources along a co-dimensional space. The IIM was originally introduced by LeVeque and Li [7] and Li [8] and further developed in [1, 11]. A monograph of IIM has been published by SIAM in 2006 [12]. The original motivation of the immersed interface method is to improve accuracy of Peskin’s immersed boundary (IB) method and to develop a higher-order method for PDEs with discontinuous coefficients. The IIM method is based on uniform or adaptive Cartesian/polar/spherical grids or triangulations. Standard finite difference or finite element methods are used away from interfaces or boundaries. A higher-order finite difference or finite element schemes are developed near or on the interfaces or boundaries according to the interface conditions, and it results in a higher accuracy in the entire domain. The method employs continuation of the solution from the one side to the other side of the domain separated by the interface. The continuation procedure uses the multivariable Taylor’s expansion of

the solution at selected interface points. The Taylor coefficients are then determined by incorporating the interface conditions and the equation. The necessary interface conditions are derived from the physical interface conditions. Since interfaces or irregular boundaries are one dimensional lower than the solution domain, the extra costs in dealing with interfaces or irregular boundaries are generally insignificant. Furthermore, many available software packages based on uniform Cartesian/polar/spherical grids, such as FFT and fast Poisson solvers, can be applied easily with the immersed interface method. Therefore, the immersed interface method is simple enough to be implemented by researchers and graduate students who have reasonable background in finite difference or finite element methods, but it is powerful enough to solve complicated problems with a high-order accuracy. Immersed Boundary Method and Interface Modeling The immersed boundary (IB) method was originally introduced by Peskin [22,23] for simulating flow patterns around heart valves and for studying blood flows in a heart [24]. First of all, the immersed boundary method is a mathematical model that describes elastic structures (or membranes) interacting with fluid flows. For instance, the blood flows in a heart can be considered as a Newtonian fluid governed by the Navier-Stokes equations   @u C .u  r/u C rp D u C F; (1)  @t with the incompressibility condition r u D 0, where  is fluid density, u fluid velocity, p pressure, and  fluid

© Springer-Verlag Berlin Heidelberg 2015 B. Engquist (ed.), Encyclopedia of Applied and Computational Mathematics, DOI 10.1007/978-3-540-70529-1

668

Immersed Interface/Boundary Method

viscosity. The geometry of the heart is complicated and is moving with time, so are the heart valves, which makes it difficult to simulate the flow patterns around the heart valves. In the immersed boundary model, the flow equations are extended to a rectangular box (domain) with a periodic boundary condition; the heart boundary and valves are modeled as elastic band that exerts force on the fluid. The immersed structure is typically represented by a collection of interacting particles Xk with a prescribed force law. Let ı.x/ be the Dirac delta function. In Peskin’s original immersed boundary model, the force is considered as source distribution along the boundary of the heart and thus can be written as

An important feature of the IB method is to use a discrete delta function ıh .x/ to approximate the Dirac delta function ı.x/. There are quite a few discrete delta functions ıh .x/ that have been developed in the literature. In three dimensions, often a discrete delta function ıh .x/ is a product of one-dimensional ones, ıh .x/ D ıh .x/ıh .y/ıh .z/:

(5)

A traditional form for ıh .x/ was introduced in [24]: 8 < 1 .1 C cos .x=2h// ; if jxj < 2h, ıh .x/ D 4h : 0; if jxj  2h.

Z

(6) Another commonly used one is the hat function: f.s; t/ı.x  X.s; t//d s; (2) F.x; t/ D .s;t / ( .h  jxj/= h2 ; if jxj < h, .x/ D ı where .s; t/ is the surface parameterized by s which h 0; if jxj  h. is one dimensional in 2D and two dimensional in 3D, (7) say a heart boundary, f.s; t/ is the force density. Since the boundary now is immersed in the entire domain, it With Peskin’s discrete delta function approach, one can is called the immersed boundary. The system is closed discretize a source distribution on a surface  as by requiring that the elastic immersed boundary moves Nb X at the local fluid velocity: Fij k D f .sl / ıh .xi X` / ıh .yj Y` / ıh .zk Z` /sl ; d X.s; t/ D u.X.s; t/; t/ dt Z D u.x; t/ ı .x X.s; t/; t/ d x;

lD1

(8)

where Nb is the number of discrete points f.X` ;Y` ;Z` /g on the surface .s; t/. In this way, the singular source (3) is distributed to the nearby grid points in a neighborhood of the immersed boundary .s; t/. The discrete here the integration is over the entire domain. delta function approach cannot achieve second-order For an elastic material, as first considered by Peskin, or higher accuracy except when the interface is aligned the force density is given by with a grid line. In the immersed boundary method, we also need to ˇ ˇ  ˇ @X ˇ @T interpolate the velocity at grid points to the immersed ˇ ˇ ; T.s; t/ D  ˇ ˇ  1 ; (4) f.s; t/ D @s @s boundary corresponding to (3). This is done again through the discrete delta function the unit tangent vector .s; t/ is given by .s; t/ D X @X=@s u.xi ; yj ; zk /ıh .xi  X / u.X; Y; Z/ D . The tension T assumes that elastic fiber j@X=@sj ij k band obeys a linear Hooke’s law with stiffness conıh .yj  Y /ıh .zk  Z/hx hy hz (9) stant . For different applications, the key of the immersed boundary method is to derive the force density. assume .X; Y; Z/ is a point on the immersed boundary In Peskin’s original IB method, the blood flow .s; t/, hx , hy , hz are mesh sizes in each coordinate in a heart is embedded in a rectangular box with a direction. Once the velocity is computed, the new periodic boundary condition. In numerical simulations, location of the immersed boundary is updated through a uniform Cartesian grid .xi ; yj ; zk / can be used. (3). Since the flow equation is defined on a rectangular

Immersed Interface/Boundary Method

669 ´

1 0.8

»

Ω+

0.6 0.4 0.2 0



Ω

−

−0.2 −0.4

Γ

−0.6 » = χ(´)

−0.8 −1 −1 −0.8−0.6−0.4−0.2

0

0.2 0.4 0.6 0.8

Immersed Interface/Boundary Method, Fig. 1 Left diagram: a rectangular domain  D C [ with an interface . The coefficients such as ˇ.x/ have a jump across the interface.

domain, standard numerical methods can be applied. For many application problems in mathematical biology, the projection method is used for small to modest Reynolds numbers. The immersed boundary method is simple and robust. It has been combined with and with adaptive mesh refinement [26, 27]. A few IB packages are available [3]. The IB method has been applied to many problems in mathematical biology and computational fluid mechanics. There are a few review articles on IB method given. Among them are the one given by Peskin in [25] and Mittal and Iaccarina [19] that highlighted the applications of IB method on computational fluid dynamics problems. The immersed boundary method is considered as a regularized method, and it is believed to be first-order accurate for the velocity, which has been confirmed by many numerical simulations and been partially proved [20].

1

Right diagram: the local coordinates in the normal and tangential directions, where is the angle between the x-axis and the normal direction

r  .ˇ.x/ru/  .x/ u D f .x/; ˇ ˇ ˇ ˇ Œˇun ˇ D v.s/ Œu ˇ D 0; 

(10)



where v.s/ 2 C 2 ./, f .x/ 2 C./,  is a smooth interface, and ˇ is a piecewise constant. Here un D @u D ru  n is the normal derivative, and n is the @n unit normal direction, and Œu is the difference of the limiting values from different side of the interface , so is Œun ; see Fig. 1 (Left diagram) for an illustration. Given a Cartesian mesh f.xi ; yj g; xi D i hx ; 0  i  M yj D j hy ; 0  j  N with the mesh size hx ; hy , the node .xi ; yj / is irregular if the central fivepoint finite difference stencil at .xi ; yj / has grid points from both side of the interface , otherwise is regular. The IIM uses the standard five-point finite difference scheme at regular grid: ˇi C 1 ;j ui C1;j C ˇi  1 ;j ui 1;j  .ˇi C 1 ;j C ˇi  1 ;j /uij 2

2

2

2

.hx /2 C

The Immersed Interface Method

x 2   ;

ˇi;j C 1 ui;j C1 C ˇi;j  1 ui;j 1  .ˇi;j C 1 C ˇi;j  1 /uij 2

uij D fij :

2

2

2

.hy /2 (11)

We describe the second immersed interface method The local truncation error at regular grid points is for a scalar elliptic equation in two-dimensional O.h2 /, where h D maxfhx ; hy g. domain, and we refer to [17] and references therein If .xi ; yj / is an irregular grid point, then the method for general equations, fourth-order method, and of undetermined coefficients the three-dimensional case. A simplified Peskin’s ns X model can be rewritten as a Poisson equation of the k Ui Cik ;j Cjk  ij Uij D fij C Cij (12) form: kD1

I

670

Immersed Interface/Boundary Method

is used to determine k ’s and Cij , where ns is the number of grid points in the finite difference stencil. We usually take ns D 9. We determine the coefficients in such a way that the local truncation error Tij D

ns X

C  Tij D a1 u C a2 uC C a3 u C a4 u C a5 u  C  Ca6 uC

C a7 u C a8 u C a9 u  C Ca10 uC

C a11 u C a12 u

u  f   Cij C O.max jk jh3 /;

  k u xi Cik ; yj Cjk  ij u.xi ; yj /

k

(17)

where h D maxfhx ; hy g. We drive additional interface (13) conditions [7,8,12] by taking the derivative of the jump conditions with respect to at D 0, and then we is as small as possible in the magnitude. can express the quantities from one side in terms of the We choose a projected point xij D .xi ; yj / on other side in the local coordinates . ; / as the interface  of irregular point .xi ; yj /. We use the v Taylor expansion at xij in the local coordinates . ; /   uC uC D u ; uC

D u ; D  u C C ; so that (12) matches (10) up to second derivatives at ˇ xij from a particular side of the interface, say the  00  00 C   uC side. This will guarantee the consistency of the finite D  u C  u C .  1/ u C u ; difference scheme. The local coordinates in the normal C   00 uC and tangential directions is

D u C .u  u /  ; kD1

f .xi ; yj /  Cij ;

D .x  x  / cos C .y  y  / sin ;

D .x  x  / sin C .y  y  / cos ;

(14)

  v0 C  00  uC D u  u  C  u C C ; ˇ

(18)

ˇ where is the angle between the x-axis and the normal where  D C . An alternative is to use a collocation ˇ direction, pointing to the direction of a specified side. In the neighborhood of .x  ; y  /, the interface  can be method, That is, we equate the interface conditions parameterized as @uC . k ; k / uC . k ; k / D u . k ; k /; ˇ C 0 D . /; with .0/ D 0;  .0/ D 0: (15) @ @u The interface conditions are given . k ; k / D v. k ; k /; ˇ  @ Œu.. /; / D 0; Œˇ.u .. /; /  0 . / u .. /; // where . k ; k / is the local coordinates of the three p closest projection points to .xi ; yj / along the equation D 1 C j0 . /j2 v. / at .xi ; yj /; Œˇ.u C u

/ D 0. In this way one can avoid the tangential derivative the data v, especially and the curvature of the interface at .x  ; y  / is 00 .0/. useful for the three-dimensional case. The Taylor expansion of each u.xi Cik ; yj Cjk / at xij can If we define the index sets K ˙ D fk W . k ; k / is on be written as the ˙ side of g; then a2j 1 terms are defined by ˙ u.xi Cik ; yj Cjk / D u. k ; k / D u˙ C k u˙ C k u

1 1 2 ˙ ˙ C k2 u˙ C k k u C k u

2 2 CO.h3 /;

(16)

where the C or  superscript depends on whether . k ; k / lies on the C or  side of . Therefore the local truncation error Tij can be expressed as a linear ˙ ˙ ˙ ˙ combination of the values u˙ ; u˙ , u , u , u , u

a1 D

X

k ;

a3 D

k2K 

a5 D

X

k k ;

k2K 

k k ;

a7 D

k2K 

1 X 2 a9 D

k k ; 2  k2K

X

1 X 2 k k ; 2  k2K

a11 D

X

k k k :

k2K 

(19)

Immersed Interface/Boundary Method

671

The a2j terms have the same expressions as a2j 1 except the summation is taken over K C . From (18) equat    ing the terms in (13) for .u ; u ; u ; u ; u

; u /, we obtain the linear system of equations for k ’s:

a3 C a4  a8

Once the k ’s are obtained, we set Cij D a12  1  a4 C .a8  a10 /00 v. C ˇ

.ik ; jk / ¤ .0; 0/;

k < 0

if

.ik ; jk / D .0; 0/:

if

.ik ; jk / ¤ .0; 0/I

k < 0; if

.ik ; jk / D .0; 0/;

gk D

v0 C ˇC

Enforcing the Maximum Principle Using an Optimization Approach The stability of the finite difference equations is guaranteed by enforcing the sign constraint of the discrete maximum principle; see, for example, Morton and Mayers [21]. The sign restriction on the coefficients k ’s in (12) are if

k  0;

subject to A D b;

(22)

(20)

Remark 1 • If Œˇ D 0, then the finite difference scheme is the standard one. Only correction terms need to be added at irregular grid points. The correction terms can be regarded as second-order accurate discrete delta functions. • If v  0, then the correction terms are zero. • If we use a six-point stencil and (20) has a solution, then this leads to the original IIM [7]. • For more general cases, say both  and f are discontinuous, we refer the reader to [7, 8, 12] for the derivation.

k  0

1 2 min k; gk2 ;  2

where  D Π1 ; 2 ;    ; ns T is the vector composed of the coefficients of the finite difference equation; A D b is the system of linear equations (20); and g 2 Rns has the following components: g 2 Rns ,

a1 C a2 D 0 Œˇ 00 Œˇ 00 C a D0 10 ˇC ˇC a5 C a6 C a12 .1  / 00 D 0 a7 C a8  D ˇ  a9 C a10 C a8 .  1/ D ˇ  a11 C a12  D 0:



(21)

We form the following constrained quadratic optimization problem whose solution is the coefficients of the finite difference equation at the irregular grid point xij :

ˇi Cik ;j Cjk ; if .ik ; jk / 2 f.1; 0/; .1; 0/; h2 .0; 1/; .0; 1/ g I

4ˇi;j ; if .ik ; jk / D .0; 0/I h2 gk D 0; otherwise: gk D 

(23)

With the maximum principle, the second-order convergence of the IIM has been proved in [11].

Augmented Immersed Interface Method The original idea of the augmented strategy for interface problems was proposed in [9] for elliptic interface problems with a piecewise constant but discontinuous coefficient. With a few modifications, the augmented method developed in [9] was applied to generalized Helmholtz equations including Poisson equations on irregular domains in [14]. The augmented approach for the incompressible Stokes equations with a piecewise constant but discontinuous viscosity was proposed in [18], for slip boundary condition to deal with pressure boundary condition in [17], and for the Navier-Stokes equations on irregular domains in [6]. There are at least two motivations to use augmented strategies. The first one is to get a faster algorithm compared to a direct discretization, particularly to take advantages of existing fast solvers. The second reason is that, for some interface problems, an augmented approach may be the only way to derive an accurate algorithm. This is illustrated in the augmented immersed interface method [18] for the incompressible Stokes equations with discontinuous viscosity in which the jump conditions for the pressure and the velocity are coupled together. The augmented techniques enable

I

672

Immersed Interface/Boundary Method

us to decouple the jump conditions so that the idea of the immersed interface method can be applied. While augmented methods have some similarities to boundary integral methods or the integral equation approach to find a source strength, the augmented methods have a few special features: (1) no Green function is needed, and therefore there is no need to evaluate singular integrals; (2) there is no need to set up the system of equations for the augmented variable explicitly; (3) they are applicable to general PDEs with or without source terms; and (4) the method can be applied to general boundary conditions. On the other hand, we may need estimate the condition number of the Schur complement system and develop preconditioning techniques. Procedure of the Augmented IIM We explain the procedure of the augmented IIM using the fast Poisson solver on an interior domain as an illustration. Assume we have linear partial differential equations with a linear interface or boundary condition. The the Poisson equation on an irregular domain , as an example, u D f .x/;

x 2 ;

q.u; un / D 0;

x 2 @; (24)

where q.u; un / D 0 is either a Dirichlet or Neumann boundary condition along the boundary @. To use an augmented approach, the domain  is embedded into a rectangle   R; the PDE and the source term are extended to the entire rectangle R: 8 Œu D g; on @; ˆ ˆ < f;if x 2 ; Œun D 0; on @; u D 0;if x 2 R n ; ˆ ˆ : u D 0; on @R: (25) and q.u; un / D 0 on @: (

On a Cartesian mesh .xi ; yj /, i D 0; 1;    ; M , j D 0; 1;    ; N , M  N , we use U and G to represent the discrete solution to (25). Note that the dimension of U is O.N 2 / while that of G is of O.N /. The augmented IIM can be written as

AB C D



U G



D

 F ; Q

(26)

where A is the matrix formed from the discrete fivepoint Laplacian; BG are correction terms due to the jump in u, and the boundary condition is discretized by an interpolation scheme C U C DG D Q, corresponding to the boundary condition q.u; un / D 0. The main reason to use an augmented approach is to take advantage of fast Poisson solvers. Eliminating U from (26) gives a linear system for G, the Schur complement system, .D  CA1 B/G D Q  CA1 F D F2 : def

(27)

This is an Nb Nb system for G, a much smaller linear system compared to the one for U , where Nb is the dimension of G. If we can solve the Schur complement system efficiently, then we obtain the solution of the original problem with one call to the fast Poisson solver. There are two approaches to solve the Schur complement system. One is the GMRES iterative method; the other one is a direct method such as the LU decomposition. In either of the cases, we need to know how to find the matrix vector multiplication without forming the sub-matrices A1 ; B; C; D explicitly. That is, first we set G D 0 and solve the first equation of the (26), to get U.0/ D A1 F . For a given G the residual vector of the boundary condition is then given by R.G/ D C.U.0/  U.G// C DG  Q:

Remark 2 For different applications, the augmented variable(s) can be chosen differently but the above procedure is the same. For some problems, if we need to use the same Schur complement at every The solution u to (25) is a functional u.g/ of g. time step, it is then more efficient to use the LU We determine g such that the solution u.g/ satisfies decomposition just once. If the Schur complement is the boundary condition q.u; un / D 0. Note that, varying or only used a few times, then the GMRES given g, we can solve (25) using the immersed in- iterative method may be a better option. One may terface method with a single call to a fast Poisson need to develop efficient preconditioners for the Schur complement. solver.

Immersed Interface/Boundary Method

673

Immersed Finite Element Method (IFEM) The IIM has also been developed using finite element formulation as well, which is preferred sometimes because there is rich theoretical foundation based on Sobolev space, and finite element approach may lead to a better conditioned system of equations. Finite element methods have less regularity requirements for the coefficients, the source term, and the solution than finite difference methods do. In fact, the weak form for one-dimensional elliptic interface problem .ˇu0 /0  u D f .x/ C vı.x  ˛/; 0 < x < 1 with homogeneous Dirichlet boundary condition is Z

1

 0 0  ˇu   uv dx D 

0

Z

1

f dx C v.˛/;

0

8 2 H01 .0; 1/:

8 0; 0  x < xj 1 ; ˆ ˆ ˆ ˆ ˆ x  xj 1 ˆ ˆ ; xj 1  x < xj ; ˆ ˆ h ˆ ˆ < xj  x C 1; xj  x < ˛; j .x/ D D ˆ ˆ ˆ ˆ  .xj C1  x/ ˆ ˆ ˆ ; ˛  x < xj C1 ; ˆ ˆ D ˆ ˆ : 0; xj C1  x  1; 8 0; 0  x < xj ; ˆ ˆ ˆ ˆ ˆ x  xj ˆ ˆ ; xj  x < ˛; ˆ ˆ D ˆ ˆ <  .x  xj C1 / j C1 .x/ D C 1; ˛  x < xj C1 ; ˆ D ˆ ˆ ˆ xj C2  x ˆ ˆ ˆ ; xj C1  x  xj C2 ; ˆ ˆ h ˆ ˆ : 0; xj C2  x  1:

(28) where

For two-dimensional elliptic interface problems (10), the weak form is

D

ˇ ; ˇC

D D h

ˇC  ˇ .xj C1  ˛/: ˇC

ZZ

ZZ

Using the modified basis function, it has been shown in [10] that the Galerkin method is second-order accu  Z rate in the maximum norm. For 1D interface problems, (29) the FD and FE methods discussed here are not very  vds; 8.x/ 2 H01 ./:  much different. The FE method likely perform better for self-adjoint problems, while the FD method is more Unless a body-fitted mesh is used, the solution flexible for general elliptic interface problems. obtained from the standard finite element method using the linear basis functions is only first-order accurate in Modified Basis Functions for Two-Dimensional the maximum norm. In [10], a new immersed finite eleProblems ment for the one-dimensional case is constructed using A similar idea above has been applied to twomodified basis functions that satisfy homogeneous dimensional problems with a uniform Cartesian jump conditions. The modified basis functions triangulation [15]. The piecewise linear basis function satisfy centered at a node is defined as: .ˇrur  uv/ d x D 

( i .xk / D

1; if k D i , 0; otherwise

f  dx

( and Œi D 0; Œˇi0 D 0: (30)

i .xj / D

1; if i D j

0; otherwise; ˇ

@i ˇˇ ˇ D 0; Œ u j D 0; @n ˇ

i j@ D 0:

(31) Obviously, if xj < ˛ < xj C1 , then only j and j C1 need to be changed to satisfy the second jump condition. Using the method of undetermined coefficients, We call the space formed by all the basis function i .x/ as the immersed finite element space (IFE). we can conclude that

I

674

a

Immersed Interface/Boundary Method

b

A (0; h)

−0.15 −0.2

T+

−0.25

¯+ E (0; y1) D

−0.3 (h − y2; y2)

−0.35

M

−0.4

¯− T−

(0; 0) B

−0.45 (h; 0)

−0.5 −0.5

C

c

−0.45

−0.4

−0.35

−0.3

−0.25

−0.2

−0.15

1 0.5 0

−0.5 −0.15 −0.2 −0.25 −0.3 −0.35 −0.4 −0.45 −0.5

−0.5

−0.45

−0.4

Immersed Interface/Boundary Method, Fig. 2 (a) A typical triangle element with an interface cutting through. The curve between D and E is part of the interface curve  which is approximated by the line segment DE. In this diagram, T is the triangle 4ABC , T C D 4ADE, T  D T  T C , and Tr is the

We consider a reference interface element T whose geometric configuration is given in Fig. 2a in which the curve between points D and E is a part of the interface. We assume that the coordinates at A, B, C , D, and E are .0; h/;

.0; 0/;

.h; 0/;

.0; y1 /;

.h  y2 ; y2 /; (32)

−0.35

−0.3

−0.25

−0.2

−0.15

region enclosed by the DE and the arc DME. (b) A standard domain of six triangles with an interface cutting through. (c) A global basis function on its support in the nonconforming immersed finite element space. The basis function has small jump across some edges

8 C u .x/ D a0 C a1 x C a2 .y  h/; ˆ ˆ ˆ < if x D .x; y/ 2 T C ; (33a) u.x/ D ˆ u .x/ D b0 C b1 x C b2 y; ˆ ˆ : if x D .x; y/ 2 T  ; uC .D/ D u .D/; uC .E/ D u .E/; ˇC

@u C @u  D ˇ ; @n @n

(33b) with the restriction 0  y1  h; 0  y2 < h. Once the values at vertices A, B, and C of the element T are specified, we construct the following where n is the unit normal direction of the line segment DE. This is a piecewise linear function in T that piecewise linear function:

Immersed Interface/Boundary Method

satisfies the natural jump conditions along DE. The existence and uniqueness of the basis functions and error estimates are given in [15]. It is easy to show that the linear basis function defined at a nodal point exists and it is unique. It has also been proved in [15] that for the solution of the interface problem (10), there is an interpolation function uI .x/ in the IFE space that approximates u.x/ to second-order accuracy in the maximum norm. However, as we can see from Fig. 2c, a linear basis function may be discontinuous along some edges. Therefore such IFE space is a nonconforming finite element space. Theoretically, it is easy to prove the corresponding Galerkin finite element method is at least first-order accurate; see [15]. In practice, its behaviors are much better than the standard finite element without any modifications. Numerically, the computed solution has super linear convergence. More theoretical analysis can be found in [2, 16]. The nonconforming immersed finite element space is also constructed for elasticity problems with interfaces in [4, 13, 29]. There are six coupled unknowns in one interface triangle for elasticity problems with interfaces. A conforming IFE space is also proposed in [15]. The basis functions are still piecewise linear. The idea is to extend the support of the basis function along interface to one more triangle to keep the continuity. The conforming immersed finite element method is indeed second-order accurate. The trade-off is the increased complexity of the implementation. We refer the readers to [15] for the details. The conforming immersed finite element space is also constructed for elasticity problems with interfaces in [4]. Finally, one can construct the quadratic nonconforming element using the quadratic Taylor expansion (16) at the midpoint of the interface. The relation of coefficients of both sides is determined by the interface conditions (17). Then the quadratic element on the triangle is uniquely by the values of the basis six points of the triangle.

675

where c D c.x/ > 0 is piecewise smooth. The second-order immersed interface method has been developed in [30]. We describe the higher-order method closely related to CIP methods [28]. CIP is one of the numerical methods that provides an accurate, lessdispersive and less-dissipative numerical solution. The method uses the exact integration in time by the characteristic method and uses the solution u and its derivative v D ux as unknowns. The piecewise cubic Hermite interpolation for each computational cell in each cell Œxj 1 ; xj based on solution values and its derivatives at two endpoints xj 1 ; xj . In this way the method allows us to take an arbitrary time step (no CFL limitation) without losing the stability and accuracy. That is, we use the exact simultaneous update formula for the solution u: u.xk ; t C t/ D

c.yk / u.yk ; t/ c.xk /

(35)

and for its derivative v:  c 0 .yk / c 0 .xk / c.yk /  u.yk ; t/ v.xk ; t C t/ D c.xk / c.xk / c.xk /   c.yk / 2 v.yk ; t/: (36) C c.xk / 

For the piecewise constant equation ut C c.x/ ux D 0, we use the piecewise cubic interpolation: F  .x/ in Œxj 1 ; ˛ and F C .x/ in Œ˛; xj of the form F ˙ .x/ D P3 ˙ k kD0 ak .x  ˛/ . The eight unknowns are uniquely determined via the interface relations and the interpolation conditions at the interface ˛ 2 .xj 1 ; xj /: Œu D 0;

Œcux D 0;

Œc 2 uxx D 0;

F  .xj 1 / D unj 1 ; F C .xj / D unj ;

Œc 3 uxxx D 0; (37)

Fx .xj 1 / D vjn1 ; FxC .xj / D vjn ;

(38)

Thus, we update solution .un ; v n / at node xj by

Hyperbolic Equations We consider an advection equation as a model equation

D F C .xj  c C t/; unC1 j

vjnC1 D FxC .xj  c C t/: (39)

Similarly, for (34) we have the method based on the ut C .c.x/u/x D 0; t > 0; x 2 R; u.0; x/ D u0 .x/; interface conditions Œcu D Œc 2 ux D Œc 3 uxx D (34) Œc 4 uxxx D 0 and the updates (35)–(36).

I

676

Index Concepts for Differential-Algebraic Equations

The d’Alembert-based method for the Maxwell 16. Li, Z., Lin, T., Lin, Y., Rogers, R.C.: Error estimates of an immersed finite element method for interface problems. equation that extends our characteristic-based method Numer. PDEs 12, 338–367 (2004) to Maxwell system is developed for the piecewise 17. Li, Z., Wan, X., Ito, K., Lubkin, S.: An augmented presconstant media and then applied to Maxwell system sure boundary condition for a Stokes flow with a non-slip boundary condition. Commun. Comput. Phys. 1, 874–885 with piecewise constant coefficients. Also, one can (2006) extend the exact time integration CIP method for 18. Li, Z., Ito, K., Lai, M.-C.: An augmented approach for 2 3 equations in discontinuous media in R and R and Stokes equations with a discontinuous viscosity and singular the Hamilton Jacobi equation [5]. forces. Comput. Fluids 36, 622–635 (2007)

References 1. Deng, S., Ito, K., Li, Z.: Three dimensional elliptic solvers for interface problems and applications. J. Comput. Phys. 184, 215–243 (2003) 2. Ewing, R., Li, Z., Lin, T., Lin, Y.: The immersed finite volume element method for the elliptic interface problems. Math. Comput. Simul. 50, 63–76 (1999) 3. Eyre, D., Fogelson, A.: IBIS: immersed boundary and interface software package. http://www.math.utah.edu/IBIS (1997) 4. Gong, Y.: Immersed-interface finite-element methods for elliptic and elasticity interface problems. North Carolina State University (2007) 5. Ito, K., Takeuchi, T.: Exact time integration CIP methods for scaler hyperbolic equations with variable and discontinuous coefficients. SIAM J. Numer. Anal. pp. 20 (2013) 6. Ito, K., Li, Z., Lai, M.-C.: An augmented method for the Navier-Stokes equations on irregular domains. J. Comput. Phys. 228, 2616–2628 (2009) 7. LeVeque, R.J., Li, Z.: The immersed interface method for elliptic equations with discontinuous coefficients and singular sources. SIAM J. Numer. Anal. 31, 1019–1044 (1994) 8. Li, Z.: The immersed interface method – a numerical approach for partial differential equations with interfaces. PhD thesis, University of Washington (1994) 9. Li, Z.: A fast iterative algorithm for elliptic interface problems. SIAM J. Numer. Anal. 35, 230–254 (1998) 10. Li, Z.: The immersed interface method using a finite element formulation. Appl. Numer. Math. 27, 253–267 (1998) 11. Li, Z., Ito, K.: Maximum principle preserving schemes for interface problems with discontinuous coefficients. SIAM J. Sci. Comput. 23, 1225–1242 (2001) 12. Li, Z., Ito, K.: The Immersed Interface Method: Numerical Solutions of PDEs Involving Interfaces and Irregular Domains. SIAM Frontier Series in Applied mathematics, FR33. Society for Industrial and Applied Mathematics, Philadelphia (2006) 13. Li, Z., Yang, X.: An immersed finite element method for elasticity equations with interfaces. In: Shi, Z.-C., et al. (eds.) Proceedings of Symposia in Applied Mathematics. AMS Comput. Phys. Commun. 12(2), 595–612 (2012) 14. Li, Z., Zhao, H., Gao, H.: A numerical study of electromigration voiding by evolving level set functions on a fixed cartesian grid. J. Comput. Phys. 152, 281–304 (1999) 15. Li, Z., Lin, T., Wu, X.: New Cartesian grid methods for interface problem using finite element formulation. Numer. Math. 96, 61–98 (2003)

19. Mittal, R., Iaccarino, G.: Immersed boundary methods. Annu. Rev. Fluid Mech. 37, 239–261 (2005) 20. Mori, Y.: Convergence proof of the velocity field for a Stokes flow immersed boundary method. Commun. Pure Appl. Math. 61, 1213–1263 (2008) 21. Morton, K.W., Mayers, D.F.: Numerical Solution of Partial Differential Equations. Cambridge University Press (1995) 22. Peskin, C.S.: Flow patterns around heart valves: a digital computer method for solving the equations of motion. PhD thesis, Physiology, Albert Einsten College of Medicine, University Microfilms 72–30 (1972) 23. Peskin, C.S.: Flow patterns around heart valves: a numerical method. J. Comput. Phys. 10, 252–271 (1972) 24. Peskin, C.S.: Numerical analysis of blood flow in the heart. J. Comput. Phys. 25, 220–252 (1977) 25. Peskin, C.S.: The immersed boundary method. Acta Numer. 11, 479–517 (2002) 26. Roma, A.: A multi-level self adaptive version of the immersed boundary method. PhD thesis, New York University (1996) 27. Roma, A., Peskin, C.S., Berger, M.: An adaptive version of the immersed boundary method. J. Comput. Phys. 153, 509–534 (1999) 28. Yabe, T., Aoki, T.: A universal solver for hyperbolic equations by cubic-polynomial interpolation. I. One-dimensional solver. Comput. Phys. Commun. 66, 219–232 (1991) 29. Yang, X., Li, B., Li, Z.: The immersed interface method for elasticity problems with interface. Dyn. Contin. Discret. Impuls. Syst. 10, 783–808 (2003) 30. Zhang, C., LeVeque, R.J.: The immersed interface method for acoustic wave equations with discontinuous coefficients. Wave Motion 25, 237–263 (1997)

Index Concepts for Differential-Algebraic Equations Volker Mehrmann Institut f¨ur Mathematik, MA 4-5 TU, Berlin, Germany

Introduction Differential-algebraic equations (DAEs) present today the state of the art in mathematical modeling of dynamical systems in almost all areas of science and engineering. Modeling is done in a modularized

Index Concepts for Differential-Algebraic Equations

677

way by combining standardized sub-models in a hierarchically built network. The topic is well studied from an analytical, numerical, and control theoretical point of view, and several monographs are available that cover different aspects of the subject [1, 2, 9, 14–16, 21, 28, 29, 34]. The mathematical model can usually be written in the form F .t; x; x/ P D 0; (1) where xP denotes the (typically time) derivative of x. Denoting by C k .I; Rn / the set of k times continuously differentiable functions from I D Œt ; t  R to Rn , one usually assumes that F 2 C 0 .I Dx DxP ; Rm / is sufficiently smooth and that Dx ; DxP Rn are open sets. The model equations are usually completed with initial conditions (2) x.t / D x:

give rise to distributional or other classes of solutions [8, 21, 27, 35] as well as multiple solutions [21]. Here we only discuss classical solutions, x 2 C 1 .I; Cn / that satisfy (1) pointwise. Different approaches of classifying the difficulties that arise in DAEs have led to different so-called index concepts, where the index is a “measure of difficulty” in the analytical or numerical treatment of the DAE. In this contribution, the major index concepts will be surveyed and put in perspective with each other as far as this is possible. For a detailed analysis and a comparison of various index concepts with the differentiation index (see [5, 12, 14, 21, 22, 24, 31]). Since most index concepts are only defined for uniquely solvable square systems with m D n, here only this case is studied (see [21] for the general case).

I

Linear DAEs

Index Concepts for DAEs E xP  Ax  f D 0;

(3)

with E; A 2 C 0 .I; Rm;n /, f 2 C 0 .I; Rm / often arise after linearization along trajectories (see [4]) with constant coefficients in the case of linearization around an equilibrium solution. DAE models are also studied in the case when x is infinite dimensional (see, e.g., [7, 37]), but here we only discuss the finite-dimensional case. Studying the literature for DAEs, one quickly realizes an almost Babylonian confusion in the notation, in the solution concepts, in the numerical simulation techniques, and in control and optimization methods. These differences partially result from the fact that the subject was developed by different groups in mathematics, computer science, and engineering. Another reason is that it is almost impossible to treat automatically generated DAE models directly with standard numerical methods, since the solution of a DAE may depend on derivatives of the model equations or input functions and since the algebraic equations restrict the dynamics of the system to certain manifolds, some of which are only implicitly contained in the model. This has the effect that numerical methods may have a loss in convergence order, are hard to initialize, or fail to preserve the underlying constraints and thus yield physically meaningless results (see, e.g., [2, 21] for illustrative examples). Furthermore, inconsistent initial conditions or violated smoothness requirements can

The starting point for all index concepts is the linear systems with constant coefficients. In this case, the smoothness requirements can be determined from the Kronecker canonical form [11] of the matrix pair .E; A/ under equivalence transformations E2 D PE1 Q, A2 D PA1 Q, with invertible matrices P; Q (see e.g., [21]). The size of the largest Kronecker block associated with an infinite eigenvalue of .E; A/ is called Kronecker index, and it defines the smoothness requirements for the inhomogeneity f . For the linear variable coefficient case, it was first tried to define a Kronecker index (see [13]). However, it was soon realized that this is not a reasonable concept [5, 17], since for the variable coefficient case, the equivalence transformation is P and it E2 D PE1 Q, A2 D PA1 Q  PE1 Q, locally does not reduce to the classical equivalence for matrix pencils. Canonical forms under this equivalence transformation have been derived in [17] and existence and uniqueness of solutions of DAEs has been characterized via global equivalence transformations and differentiations. Since the differentiation of computed quantities is usually difficult, it was suggested in [3] to differentiate first the original DAE (3) and then carry out equivalence transformations. For this, we gather the original equation and its derivatives up to order ` into a socalled derivative array:

678

Index Concepts for Differential-Algebraic Equations

2 6 6 F` .t; x; : : : ; x .`C1/ / D 6 4

F .t; x; x/ P d F .t; x; x/ P dt

:: : d ` / F .t; x; x/ P . dt

3 7 7 7: 5

The Strangeness Index An index concept that is closely related to the differen(4) tiation index and extends to over- and under determined systems is based on the following hypothesis.

We require solvability of (4) in an open set and define M` .t; x; x; P : : : ; x .`C1/ / D F`Ix;:::;x .`C1/ .t; x; x; P : : : ; x .`C1/ /; P P : : : ; x .`C1/ / D .F`Ix .t; x; x; P : : : ; x .`C1/ /; N` .t; x; x; 0; : : : ; 0/; g` .t / D F`It ;

where F`Iz denotes the Jacobian of F` with respect to the variables in z. The Differentiation Index The most common index definition is that of the differentiation index (see [5]).

Hypothesis 1 Consider the DAE (1) and suppose that there exist integers , a, and d such that the set L D fz 2 R.C2/nC1 j F .z/ D 0g associated with F is nonempty and such that for every point z0 D .C1/ / 2 L , there exists a (sufficiently .t0 ; x0 ; xP 0 ; : : : ; x0 small) neighborhood in which the following properties hold: 1. We have rank M .z/ D . C 1/n  a on L such that there exists a smooth matrix function Z2 of size . C 1/n a and pointwise maximal rank, satisfying Z2T M D 0 on L . 2. We have rank AO2 .z/ D a, where AO2 D Z2T N ŒIn 0    0 T such that there exists a smooth matrix function T2 of size n d , d D n  a, and pointwise maximal rank, satisfying AO2 T2 D 0. P 2 .z/ D d such that there 3. We have rank FxP .t; x; x/T exists a smooth matrix function Z1 of size n d and pointwise maximal rank, satisfying rank EO 1 T2 D d , where EO 1 D Z1T FxP .

Definition 1 Suppose that (1) is solvable. The smallest integer  (if it exists) such that the solution x is uniquely defined by F .t; x; x; P : : : ; x .C1/ / D 0 for all consistent initial values is called the differentiation index of (1). Definition 2 Given a DAE as in (1), the smallest value of  such that F satisfies Hypothesis 1 is called the Over the years, the definition of the differentiation instrangeness index of (1). dex has been slightly modified to adjust from the linear to the nonlinear case [3, 5, 6] and to deal with slightly It has been shown in [21] that if F as in (1) satisfies Hydifferent smoothness assumptions. In the linear case, it pothesis 1 with characteristic values , a, and d , then has been shown in [21] that the differentiation index  the set L R.C2/nC1 forms a (smooth) manifold of is invariant under (global) equivalence transformations, dimension n C 1. Setting and if it is well defined, then there exists a smooth, .C1/n;.C1/n / such pointwise nonsingular P D Z1T F .t; x; x/; P FO1 .t; x; x/  R 2 C.I; C

In 0 T . Then from the derivative array that RM D FO2 .t; x/ D Z2 F .t; x; zO/; 0 H M .t/Pz D N .t/z C g .t/, one obtains an ordinary differential equation (ODE): where zO D .x .1/ ; : : : ; x .C1/ /, and considering the reduced DAE

 I # " xP D ΠIn 0 R.t/M .t/Pz D ΠIn 0 R.t/N .t/ n x FO1 .t; x; x/ P 0 O D 0; (5) F .t; x; x/ P D FO2 .t; x/ CΠIn 0 R.t/g .t/; which is called underlying ODE. Any solution of the DAE is also a solution of this ODE. This motivates the interpretation that the differentiation index is the number of differentiations needed to transform the DAE into an ODE.

one has the following (local) relation between the solutions of (1) and (5). Theorem 1 ([19, 21]) Let F as in (1) satisfy Hypothesis 1 with values , a, and d . Then every sufficiently smooth solution of (1) also solves (5).

Index Concepts for Differential-Algebraic Equations

679

It also has been shown in [21] that if x  2 C 1 .I; Rn / kxO  xk  C.kxO  xk C max k t 2I is a sufficiently smooth solution of (1), then there exist an operator FO W D ! Y, D X open, given by holds. FO .x/.t/ D



 xP 1 .t/  L.t; x1 .t// ; x2 .t/  R.t; x1 .t//

Rt t

.s/ dsk1 / (9)

For the linear variable coefficient case, the following (6) relation holds.

Theorem 2 ([21]) Let the strangeness index  of (3) be well defined and let x be a solution of (3). Then the with X D fx 2 C.I; R / j x1 2 C .I; R /; x1 .t/ D 0g n  perturbation index  of (3) along x is well defined with and Y D C.I; R /. Then x is a regular solution of   D 0 if  D a D 0 and  D  C 1 otherwise. (6), i.e., there exist neighborhoods U X of x , and V Y of the origin such that for every b 2 V, the The reason for the two cases in the definition of the equation FO .x/ D b has a unique solution x 2 U that perturbation index is that in this way, the perturbadepends continuously on f . tion index equals the differentiation index if defined. The requirements of Hypothesis 1 and that of a well- Counting in the way of the strangeness index according defined differentiation index are equivalent up to some to the estimate (8), there would be no need in the (technical) smoothness requirements (see [18,21]). For extension (9). uniquely solvable systems, however, the differentiation It has been shown in [21] that the concept of the index aims at a reformulation of the given problem as perturbation index can also be extended to the nonan ODE, whereas Hypothesis 1 aims at a reformulation square case. as a DAE with two parts, one part which states all constraints and another part which describes the dynamical behavior. If the appropriate smoothness The Tractability Index conditions hold, then  D 0 if  D a D 0 and  D C1 A different index concept [14, 23, 24] is formulated in otherwise. its current form for DAEs with properly stated leading The Perturbation Index term: d Motivated by the desire to classify the difficulties (10) F .Dx/ D f .x; t/; t 2 I dt arising in the numerical solution of DAEs, the perturbation index introduced in [16] studies the effect of a with F 2 C.I; Rn;l /, D 2 C.I; Rl;n /, f 2 C.I perturbation in Dx ; Rn /, sufficiently smooth such that kernel F .t/ ˚ n

1

d

range D.t/ D Rl for all t 2 I and such that there (7) exists a projector R 2 C 1 .I; Rl;l / with range R.t/ D range D.t/ and kernel R.t/ D kernel F .t/ for all t 2 I. One introduces the chain of matrix functions: with sufficiently smooth and initial condition D x. O x.t/ O G0 D FD; G1 D G0 C B0 Q0 ; Gi C1 1 n Definition 3 If x 2 C .I; C / is a solution, then (1) D Gi C Bi Qi ; i D 1; 2; : : : ; (11) is said to have perturbation index  2 N along x, if  is the smallest number such that for all sufficiently where Qi is a projector onto Ni D kernel Gi , with smooth xO satisfying (7) the estimate (with appropriate Qi Qj D 0 for j D 0; : : : ; i 1, Pi D I Qi , B0 D fx , norms in the relevant spaces) and Bi D Bi 1 Pi 1 Gi D  dtd .DP1 : : : Pi D  /DPi 1 , where D  is the reflexive generalized inverse of D .1/ P 1 C  Ck k1 / satisfying .DD  / D R and .D  D/ D P0 . kxxk O  C.kxxkCk k O 1 Ck k (8) Definition 4 ([23]) A DAE of the form (10) with holds with a constant C independent of x, O provided properly stated leading term is said to be regular with that the expression on the right-hand side in (8) is tractability index  on the interval I, if there exist a sufficiently small. It is said to have perturbation index sequence of continuous matrix functions (11) such that  D 0 if the estimate PO D ; F .t; x; O x/

I

680

1. Gi is singular and has constant rank rNi on I for i D 0; : : : ;   1. 2. Qi is continuous and DP1 : : : Pi D  is continuously differentiable on I for i D 0; : : : ;   1. 3. Qi Qj D 0 holds on I for all i D 1; : : : ;   1 and j D 1; : : : ; i  1. 4. G is nonsingular on I.

Index Concepts for Differential-Algebraic Equations

and forms the desired sub-manifold of dimension d of Rn , where the differential equation evolves and contains the consistent initial values. The ODE case trivially is a differential equation on the manifold Rn . Except for differences in the smoothness requirements, the geometric index is equal to the differentiation index [5]. This then also defines the relationship to the other indices.

The chain of projectors and spaces allows to filter out an ODE for the differential part of the solution u D DP1 : : : P 1 D  Dx of the linear version of (10) with f .x; t/ D A.t/x.t/ C q.t/ (see [23]) which is The Structural Index given by A combinatorially oriented index was first defined for the linear constant coefficient case. Let .E.p/; A.p// d be the parameter dependent pencil that is obtained from  1  uP  .DP1 : : : P 1 D /u  DP1 : : : P 1 G AD .E; A/ by substituting the nonzero elements of E and dt 1 A by independent parameters pj . Then the unique inu D DP1 : : : P 1 G q: teger that equals the Kronecker index of .E.p/; A.p// Instead of using derivative arrays here, derivatives of for all p from some open and dense subset of the parameter set is called the structural index (see [25] projectors are used. The advantage is that the smoothand in a more general way [26]). For the nonlinear case, ness requirements for the inhomogeneity can be explicitly specified and in this form the tractability index can a local linearization is employed. Although it has been shown in [31] that the differbe extended to infinite-dimensional systems. However, entiation index and the structural index can be arbiif the projectors have to be computed numerically, trarily different, the algorithm of [25] to determine the then difficulties in obtaining the derivatives can be structural index is used heavily in applications (see, anticipated. e.g., [38]) by employing combinatorial information It is still a partially open problem to characterize the exact relationship between the tractability index and to analyze which equations should be differentiated the other indices. Partial results have been obtained and to introduce extra variables for index reduction in [5, 6, 22, 24], showing that (except again for differ- [36]. A sound analysis when this approach is fully ent smoothness requirements) the tractability index is justified has, however, only been given in special cases [10, 20, 36]. equal to the differentiation index and thus by setting  D 0 if  D a D 0 one has  D  C 1 if  > 0. The Geometric Index The geometric theory to study DAEs as differential equations on manifolds was developed first in [30, 32, 33]. One constructs a sequence of sub-manifolds and their parameterizations via local charts (corresponding to the different constraints on different levels of differentiation). The largest number of differentiations needed to identify the DAE as a differential equation on a manifold is then called the geometric index of the DAE. It has been shown in [21] that any solvable regular DAE with strangeness index  D 0 can be locally (near a given solution) rewritten as a differential equation on a manifold and vice versa. If one considers the reduced system (5), then starting with a solution x  2 C 1 .I; Rn / of (1), the set M D FO21 .f0g/ is nonempty

Conclusions Different index concepts for systems of differentialalgebraic equations have been discussed. Except for different technical smoothness assumptions (and in the case of the strangeness index, different counting) for regular and uniquely solvable systems, these concepts are essentially equivalent to the differentiation index. However, all have advantages and disadvantages when it comes to generalizations, numerical methods, or control techniques. The strangeness index and the perturbation index also extend to non-square systems, while the tractability index allows a direct generalization to infinite-dimensional systems.

Index Concepts for Differential-Algebraic Equations

References 1. Ascher, U.M., Petzold, L.R.: Computer Methods for Ordinary Differential and Differential-Algebraic Equations. SIAM Publications, Philadelphia (1998) 2. Brenan, K.E., Campbell, S.L., Petzold, L.R.: Numerical Solution of Initial-Value Problems in Differential Algebraic Equations, 2nd edn. SIAM Publications, Philadelphia (1996) 3. Campbell, S.L.: A general form for solvable linear time varying singular systems of differential equations. SIAM J. Math. Anal. 18, 1101–1115 (1987) 4. Campbell, S.L.: Linearization of DAE’s along trajectories. Z. Angew. Math. Phys. 46, 70–84 (1995) 5. Campbell, S.L., Gear, C.W.: The index of general nonlinear DAEs. Numer. Math. 72, 173–196 (1995) 6. Campbell, S.L., Griepentrog, E.: Solvability of general differential algebraic equations. SIAM J. Sci. Comput. 16, 257–270 (1995) 7. Campbell, S.L., Marszalek, W.: Index of infinite dimensional differential algebraic equations. Math. Comput. Model. Dyn. Syst. 5, 18–42 (1999) 8. Cobb, J.D.: On the solutions of linear differential equations with singular coefficients. J. Differ. Equ. 46, 310–323 (1982) 9. Eich-Soellner, E., F¨uhrer, C.: Numerical Methods in Multibody Systems. Teubner Verlag, Stuttgart (1998) 10. Est´evez-Schwarz, D., Tischendorf, C.: Structural analysis for electrical circuits and consequences for MNA. Int. J. Circuit Theor. Appl. 28, 131–162 (2000) 11. Gantmacher, F.R.: The Theory of Matrices I. Chelsea Publishing Company, New York (1959) 12. Gear, C.W.: Differential-algebraic equation index transformations. SIAM J. Sci. Stat. Comput. 9, 39–47 (1988) 13. Gear, C.W., Petzold, L.R.: Differential/algebraic systems and matrix pencils. In: K˚agstr¨om, B., Ruhe, A. (eds.) Matrix Pencils, pp. 75–89. Springer, Berlin (1983) 14. Griepentrog, E., M¨arz, R.: Differential-Algebraic Equations and Their Numerical Treatment. Teubner Verlag, Leipzig (1986) 15. Hairer, E., Wanner, G.: Solving Ordinary Differential Equations II: Stiff and Differential-Algebraic Problems, 2nd edn. Springer, Berlin (1996) 16. Hairer, E., Lubich, C., Roche, M.: The Numerical Solution of Differential-Algebraic Systems by Runge–Kutta Methods. Springer, Berlin (1989) 17. Kunkel, P., Mehrmann, V.: Canonical forms for linear differential-algebraic equations with variable coefficients. J. Comput. Appl. Math. 56, 225–259 (1994) 18. Kunkel, P., Mehrmann, V.: Local and global invariants of linear differential-algebraic equations and their relation. Electron. Trans. Numer. Anal. 4, 138–157 (1996) 19. Kunkel, P., Mehrmann, V.: A new class of discretization methods for the solution of linear differential algebraic equations with variable coefficients. SIAM J. Numer. Anal. 33, 1941–1961 (1996) 20. Kunkel, P., Mehrmann, V.: Index reduction for differentialalgebraic equations by minimal extension. Z. Angew. Math. Mech. 84, 579–597 (2004)

681 21. Kunkel, P., Mehrmann, V.: Differential-Algebraic Equations: Analysis and Numerical Solution. EMS Publishing House, Z¨urich (2006) 22. Lamour, R.: A projector based representation of the strangeness index concept. Preprint 07-03, Humboldt Universit¨at zu Berlin, Berlin, Germany, (2007) 23. M¨arz, R.: The index of linear differential algebraic equations with properly stated leading terms. Results Math. 42, 308–338 (2002) 24. M¨arz, R.: Characterizing differential algebraic equations without the use of derivative arrays. Comput. Math. Appl. 50, 1141–1156 (2005) 25. Pantelides, C.C.: The consistent initialization of differentialalgebraic systems. SIAM J. Sci. Stat. Comput. 9, 213–231 (1988) 26. Pryce, J.: A simple structural analysis method for DAEs. BIT 41, 364–394 (2001) 27. Rabier, P.J., Rheinboldt, W.C.: Classical and generalized solutions of time-dependent linear differentialalgebraic equations. Linear Algebra Appl. 245, 259–293 (1996) 28. Rabier, P.J., Rheinboldt, W.C.: Nonholonomic motion of rigid mechanical systems from a DAE viewpoint. SIAM Publications, Philadelphia (2000) 29. Rabier, P.J., Rheinboldt, W.C.: Theoretical and Numerical Analysis of Differential-Algebraic Equations. Handbook of Numerical Analysis, vol. VIII. Elsevier Publications, Amsterdam (2002) 30. Reich, S.: On a geometric interpretation of differentialalgebraic equations. Circuits Syst. Signal Process. 9, 367– 382 (1990) 31. Reißig, G., Martinson, W.S., Barton, P.I.: Differentialalgebraic equations of index 1 may have an arbitrarily high structural index. SIAM J. Sci. Comput. 21, 1987–1990 (2000) 32. Rheinboldt, W.C.: Differential-algebraic systems as differential equations on manifolds. Math. Comput. 43, 473–482 (1984) 33. Rheinboldt, W.C.: On the existence and uniqueness of solutions of nonlinear semi-implicit differential algebraic equations. Nonlinear Anal. 16, 647–661 (1991) 34. Riaza, R.: Differential-Algebraic Systems: Analytical Aspects and Circuit Applications. World Scientific Publishing Co. Pte. Ltd., Hackensack (2008) 35. Seiler, W.M.: Involution-the formal theory of differential equations and its applications in computer algebra and numerical analysis. Habilitation thesis, Fak. f. Mathematik, University of Mannheim, Mannheim, Germany (2002) 36. S¨oderlind, G.: Remarks on the stability of high-index DAE’s with respect to parametric perturbations. Computing 49, 303–314 (1992) 37. Tischendorf, C.: Coupled systems of differential algebraic and partial differential equations in circuit and device simulation. Habilitation thesis, Inst. f¨ur Math., Humboldt-Universit¨at zu Berlin, Berlin, Germany (2004) 38. Unger, J., Kr¨oner, A., Marquardt, W.: Structural analysis of differential-algebraic equation systems: theory and applications. Comput. Chem. Eng. 19, 867–882 (1995)

I

682

Information Theory for Climate Change and Prediction Michal Branicki School of Mathematics, The University of Edinburgh, Edinburgh, UK

Keywords Climate change; Information theory; Kulback-Leibler divergence; Relative entropy; Reduced-order predictions

Mathematics Subject Classification Primary: 94A15, 60H30, 35Q86, 35Q93; Secondary: 35Q94, 62B10, 35Q84, 60H15

Description The Earth’s climate is an extremely complex system coupling physical processes for the atmosphere, ocean, and land over a wide range of spatial and temporal scales (e.g., [5]). In contrast to predicting the smallscale, short-term behavior of the atmosphere (i.e., the “weather”), climate change science aims to predict the planetary-scale, long-time response in the “climate system” induced either by changes in external forcing or by internal variability such as the impact of increased greenhouse gases or massive volcanic eruptions [14]. Climate change predictions pose a formidable challenge for a number of intertwined reasons. First, while the dynamical equations for the actual climate system are unknown, one might reasonably assume that the dynamics are nonlinear and turbulent with, at best, intermittent energy fluxes from small scales to much larger and longer spatiotemporal scales. Moreover, all that is available from the true climate dynamics are coarse, empirical estimates of low-order statistics (e.g., mean and variance) of the large-scale horizontal winds, temperature, concentration of greenhouse gases, etc., obtained from sparse observations. Thus, a fundamental difficulty in estimating sensitivity of the climate system to perturbations lies in predicting the coarsegrained response of an extremely complex system from

Information Theory for Climate Change and Prediction

sparse observations of its past and present dynamics combined with a suite of imperfect, reduced-order models. For several decades, the weather forecasts and the climate change predictions have been carried out through comprehensive numerical models [5, 14]. However, such models contain various errors which are introduced through lack of resolution and a myriad of parameterizations which aim to compensate for the effects of the unresolved dynamical features such as clouds, ocean eddies, sea ice cover, etc. Due to the highly nonlinear, multi-scale nature of this extremely high-dimensional problem, it is quite clear that – despite the ever increasing computer power – no model of the climate system will be able to resolve all the dynamically important and interacting scales. Recently, a stochastic-statistical framework rooted in information theory was developed in [1, 10–12] for a systematic mitigation of error in reduced-order models and improvement of imperfect coarse-grained predictions. This newly emerging approach blends physics-constrained dynamical modeling, stochastic parameterization, and linear response theory, and it has at least two mathematically desirable features: (i) The approach is based on a skill measure given by the relative entropy which, unlike other metrics for uncertainty quantification in atmospheric sciences, is invariant under the general change of variables [9, 13]; this property is very important for unbiased model calibration especially in high-dimensional problems. (ii) Minimizing the loss of information in the imperfect predictions via the relative entropy implies simultaneous tuning of all considered statistical moments; this is particularly important for improving predictions of nonlinear, non-Gaussian dynamics where the statistical moments are interdependent.

Improving Imperfect Predictions Assume that the reduced-order model(s) used to approximate the truth resolve the dynamics within a finite-dimensional domain, , dim./< 1, of the full phase space. The variables, u 2 , resolved by the model can represent, for example, the first N Fourier modes of the velocity and temperature fields. We are interested in improving imperfect probabilistic predictions of the true dynamics on the resolved variables u 2  given the the time-dependent probability den-

Information Theory for Climate Change and Prediction

683

sity, tM .uu/, of the model for t 2 I which approximates the unknown truth density, , and it stems from the fact that (1) can be written as [13] the marginal probability density, t .uu /, of the truth. The lack of information in the probability density  relative to the density  M can be measured through the (4) P.;  M;L / D P.;  L / C P. L ;  M ;L /; relative entropy, P.;  M /, given by [8, 9] Z P.;  M / D

 ln 

 ; M

(1)

where we skipped the explicit dependence on time and space in the probability densities. The relative entropy P.;  M / originates from Shannon’s information theory (e.g., [3]), and it provides a useful measure of model error in imperfect probabilistic predictions (e.g., [10]) due to its two metric-like properties: (i) P.;  M / is nonnegative and zero only when  D  M , and (ii) P.;  M / is invariant under any invertible change of variables which follows from the independence of P of the dominating measure in  and  M . These properties can be easily understood in the Gaussian framework when  G D N .Nu ; R/ and  M ;G D N .Nu M ; R M /, and the relative entropy is simply expressed by P. G ;  M,G / D C

h

1 2

1 .Nu 2

h

 uN M /RM1 .Nu  uN M /

i

i trŒRRM1 ln detŒR RM1 dimŒ uN ; (2)

which also highlights the fact that minimizing P requires simultaneous tuning of both the model mean and covariance. Given a class M of reduced-order models for the resolved dynamics on u 2 , the best model MI 2 M for making predictions over the time interval, I  Œt t C T , is given by 

PI .;  M I / D min PI .;  M /; M 2M

Z PI .;  /  M

1 T

t

t CT

P.s ; sM /ds;

(3)

where PI .;  M / measures the total lack of information in  M relative to the truth density  within I; note that for T ! 0 the best model, MI 2 M, is simply the one minimizing the relative entropy (1) at time t. The utility of the relative entropy for quantifying the model error extends beyond the formal definition in (3) with

where  P   L D C 1 exp  iLD1 i Ei .uu/ ; Z

L  X  exp  i Ei .uu / ;

C D 

(5)

i D1

is the least-biased estimate of  based on L moment constraints Z

Z  .uu /Ei .uu /duu D L



.uu/Ei .uu/duu; 

i D 1; : : : ; L; (6)

for the set of functionals E  .E1 ; : : : ; E L / on the space  of the variables resolved by the imperfect models. Such densities were shown by Jaynes [7] to be least-biased in terms of information content and areR obtained by maximizing the Shannon entropy, S D   ln , subject to the constraints in (6). Here, we assume that the functionals E are given by tensor powers of the resolved variables, u 2 , so that Ei .uu / D u ˝ i and the expectations EN i yield the first L uncentered statistical moments of ; note that in this case,  L for L D 2 is a Gaussian density. In fact, the Gaussian framework when both the measurements of the truth dynamics and its model involve only the mean and covariance presents the most practical setup for utilizing the framework of information theory in climate change applications; note that considering only L D 2 in (5) does not imply assuming that the underlying dynamics is Gaussian but merely focuses on tuning to the available second-order statistics of the truth dynamics. In weather or climate change prediction, the complex numerical models for the climate system are calibrated (often in an ad hoc fashion) by comparing the spatiotemporal model statistics with the available coarse statistics obtained from various historical observations [5, 14]; we refer to this procedure as the calibration phase on the time interval Ic . The model optimization (3) carried out in the calibration phase can be represented, using the relationship (4), as

I

684

Information Theory for Climate Change and Prediction 

PIc .;  M Ic / D PIc .;  L / C min PIc . L ;  M ;L /; M 2M

(7) where MIc 2 M is the model with the smallest lack of information within Ic . The first term, PIc .;  L /, in (7) represents an intrinsic information barrier [1,10] which cannot be overcome unless more measurements L of the truth are incorporated. The second term in (7) can be minimized directly since the least-biased estimates,  L , of the truth which are known within Ic ;  note that if PIc . L ;  MIc / ¤ 0, the corresponding information barrier can be reduced by enlarging the class of models M. The utility of the information-theoretic optimization principle (7) for improving climate change projections is best illustrated by linking the statistical model fidelity on the unperturbed attractor/climate and improved probabilistic predictions of the perturbed dynamics. Assume that the truth dynamics are perturbed so that the corresponding least-biased density,  L ;ı , is perturbed smoothly to Z  L ;ı D  L C ı L ; ı L D 0; (8)  L

where  denotes the unperturbed least-biased density (5), and we skipped the explicit dependence on time and space. For stochastic dynamical systems with time-independent, invariant measure on the attractor, rigorous theorems guarantee this smooth dependence under minimal hypothesis [6]; for more general dynamics, this property remains as an empirical conjecture. Now, the lack of information in the perturbed least-biased model density,  M ;ı , relative to the perturbed least-biased estimate of the truth,  L ;ı , can be expressed as (see, e.g., [2, 10])  ı      P  L;ı ;  M ;ı D ln C M;ı =C ı C M ;ı  ı  EN ; (9) ı where EN D EN C ıEN denotes the vector of L statistical moments with respect to the perturbed truth density,  ı , and we suppressed the time dependence for simplicity. For smooth perturbations of the truth density ı L in (8), the moment perturbations ıEN remain small so that the leading-order Taylor expansion of (9) combined with the Cauchy-Schwarz inequality leads to the following link between the error in the perturbed and unperturbed truth and model densities:

1=2 ı 1=2 PI . L ;ı ;  M ;ı / 6 M  L2 .I/ EN L2 .I/   (10) C O .ıEN /2 ; where M , are the Lagrange multipliers of the unperturbed densities  L ,  M assumed in the form (5) and determined in the calibration phase Ic on the unperturbed attractor/climate. Thus, the result in (10) implies that optimizing the statistical model fidelity on the unperturbed attractor via (7) implies improved predictions of the perturbed dynamics. Illustration of the utility of the principle in (7) on a model of turbulent tracer dynamics, can be found in [11]. Multi-model Ensemble Predictions and Information Theory Multi-model ensemble (MME) predictions are a popular technique for improving predictions in weather forecasting and climate change science (e.g., [4]). The heuristic idea behind MME prediction framework is simple: given a collection of imperfect models, consider predictions obtained through the convex superposition of the individual forecasts in the hope of mitigating model error. However, it is not obvious which models, and with what weights, should be included in the MME forecast in order to achieve the best predictive performance. Consequently, virtually all existing operational MME prediction systems are based on equal-weight ensembles which are likely to be far from optimal [4]. The information-theoretic framework allows for deriving a sufficient condition which guarantees prediction improvement via the MME approach relative to the single model forecasts [2]. The probabilistic predictions of the multi-model ensemble are represented in the present framework by the mixture density u/  ˛MME ;t .u

X

˛i tM i .uu/;

u 2 ;

(11)

i

P where ˛i D 1, ˛i > 1, and tM i represent probability densities associated with the imperfect models Mi in the class M of available models. Given the MME density ˛MME ;t , the optimization principle (7) over the time interval I can be expressed in terms of the weight vector ˛ as

Information Theory for Climate Change and Prediction

685

    D min PI ; ˛MME : PI ; ˛MME 

MME in the training phase (12) that forL optimal-weight L M I /, where M  is the PI . ; ˛MME  / 6 PI . ;  I best single model within the training phase in terms Clearly, MME prediction with the ensemble of models of information content. However, MME predictions Mi 2 M is more skilful in terms of information content are inferior to the single model predictions when the than the single model prediction with M˘ when MME weights ˛ are such that the condition (14) with M˘ D MI is not satisfied. In summary, while the     (13) MME predictions can be superior to the single model PI ; ˛MME  PI ;  M ˘ < 0: predictions, the model ensemble has to be constructed It turns out [2] that by exploiting the convexity of with a sufficient care, and the information-theoretic the relative entropy in the second argument, i.e., framework provides means for accomplishing this task   (1) P  P Mi ˛ P ;  , it is posP ; i D1 ˛i  M i 6 i D1 i in a systematic fashion. sible to obtain a sufficient condition for improving imperfect predictions via the MME approach with ˛MME relative to the single model predictions with M˘ in the form ˛

I

References

  X   ˇi PI  L ;  M i ; PI  L ;  M ˘ > i ¤˘

ˇi D

˛i ; 1  ˛˘

X

ˇi D 1;

(14)

i ¤˘

where M˘ ; Mi 2 M, and  L is the least-biased density (5) based on L moment constraints which is practically measurable in the calibration phase. Further variants of this condition expressed via the statistical M moments EN , EN are discussed in [2]. Here, we only highlight one important fact concerning the improvement of climate change predictions via the MME approach; using analogous arguments to those leading to (10) in the single model framework and the convexity of the relative entropy, the following holds in the MME framework: 1=2   X ˛i Mi  2 PI  L ;ı ; ˛MME ;ı 6 i

  C O .ıEN /2 ;

L .I/

ı 1=2 EN 2

L .I/

(15)

where, for simplicity in exposition, the models Mi in MME are assumed to be in the least-biased form (5) and , M i are the Lagrange multipliers of the unperturbed truth and model densities determined in the calibration phase (see [2] for a general formulation). Thus, for sufficiently small perturbations, optimizing the weights ˛ in the density ˛MME on the unperturbed attractor via (12) implies improved MME predictions of the perturbed truth dynamics. The potential advantage of MME predictions lies in the fact [2]

1. Branicki, M., Majda, A.J.: Quantifying uncertainty for long range forecasting scenarios with model errors in non-Guassian models with intermittency. Nonlinearity 25, 2543–2578 (2012) 2. Branicki, M., Majda, A.J.: An information-theoretic framework for improving multi model ensemble forecasts. J. Nonlinear Sci. (2013, submitted) doi:10.1007/s00332-015-9233-1 3. Cover, T.A., Thomas, J.A.: Elements of Information Theory. Wiley-Interscience, Hoboken (2006) 4. Doblas-Reyes, F.J., Hagedorn, R., Palmer, T.N.: The rationale behind the success of multi-model ensembles in seasonal forecasting. II: calibration and combination. Tellus 57, 234–252 (2005) 5. Emanuel, K.A., Wyngaard, J.C., McWilliams, J.C., Randall, D.A., Yung, Y.L.: Improving the Scientific Foundation for Atmosphere-Land Ocean Simulations. National Academic Press, Washington, DC (2005) 6. Hairer, M., Majda, A.J.: A simple framework to justify linear response theory. Nonlinearity 12, 909–922 (2010) 7. Jaynes, E.T.: Information theory and statistical mechanics. Phys. Rev. 106(10), 620–630 (1957) 8. Kleeman, R.: Measuring dynamical prediction utility using relative entropy. J. Atmos. Sci. 59(13), 2057–2072 (2002) 9. Kullback, S., Leibler, R.: On information and sufficiency. Ann. Math. Stat. 22, 79–86 (1951) 10. Majda, A.J., Gershgorin, B.: Quantifying uncertainty in climage change science through empirical information theory. Proc. Natl. Acad. Sci. 107(34), 14958–14963 (2010) 11. Majda, A.J., Gershgorin, B.: Link between statistical equilibrium fidelity and forecasting skill for complex systems with model error. Proc. Natl. Acad. Sci. 108(31), 12599–12604 (2011) 12. Majda, A.J., Gershgorin, B.: Improving model fidelity and sensitivity for complex systems through empirical information theory. Proc. Natl. Acad. Sci. 108(31), 10044–10049 (2011) 13. Majda, A.J., Abramov, R.V., Grote, M.J.: Information Theory and Stochastics for Multiscale Nonlinear Systems. CRM Monograph Series, vol. 25. AMS, Providence (2005)

I

686 14. Randall, D.A.: Climate models and their evaluation. In: Climate Change 2007: The Physical Science Basis, Contribution of Working Group I to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change, pp. 589–662. Cambridge University Press, Cambridge/New York (2007)

Inhomogeneous Media Identification Fioralba Cakoni Department of Mathematics, Rutgers University, New Brunswick, NJ, USA

Definition Inhomogeneous media identification is the problem of determining the physical properties of an unknown inhomogeneity from its response to various interrogating modalities. This response, recorded in measured data, comes as a result of the interaction of the inhomogeneity with an exciting physical field. Inhomogeneous media identification is mathematically modeled as the problem of determining the coefficients of some partial differential equations with initial or boundary data from a knowledge of the solution on the measurement domain.

Inhomogeneous Media Identification

and hope that the reader can use these as starting point for further investigations. We begin by considering the propagation of sound waves of small amplitude in R3 viewed as a problem in fluid dynamics. Let p.x; t/ denote the pressure of the fluid which is a small perturbation of the static case, i.e., p.x; t/ D p0 C P1 .x; t/ C    where p0 > 0 is a constant. Assuming that p1 .x; t/ is time harmonic, ˚ p1 .x; t/ D < u.x/e i !t , we have that u satisfies (Colton-Kress 1998 [8]) u C

!2 uD0 c 2 .x/

where ! is the frequency and c.x/ is the sound speed. Equation (1) governs the propagation of time-harmonic acoustic waves of small amplitude in a slowly varying inhomogeneous medium. We still must prescribe how the wave motion is initiated and what is the boundary of the region contained in the fluid. We shall only consider the simplest case when the inhomogeneity is of compact support denoted by D, the region of consideration is all of R3 , and the wave motion is caused by an incident field ui satisfying the unperturbed linearized equations being scattered by the inhomogeneous medium. Assuming that c.x/ D c0 D constant for x 2 R3 n D, the total field u D ui C us satisfies u C k 2 n.x/u D 0

Formulation of the Problem This survey discusses only the problem of inhomogeneous media identification in inverse scattering theory. Scattering theory is concerned with the effects that inhomogeneities have on the propagation of waves and in particular time-harmonic waves. In the context of this presentation, scattering theory provides the mathematical tools for imaging of inhomogeneous media via acoustic, electromagnetic, or elastic waves with applications to such fields as radar, sonar, geophysics, medical imaging, and nondestructive testing. For reasons of brevity, we focus our attention on the case of acoustic waves and refer the reader to CakoniColton-Monk [5] for a comprehensive reading on media identification using electromagnetic waves. Since the literature in the area is enormous, we have only referenced a limited number of papers and monographs

(1)

in R3

(2)

and the scattered field us fulfills the Sommerfeld radiation condition 

@us  i kus lim jxj jxj!1 @jxj

 D0

(3)

which holds uniformly in all directions x=jxj where k D !=c0 is the wave number and n D c02 =c 2 is the refractive index in the case of non-absorbing media. An absorbing medium is modeled by adding an absorption term which leads to a refractive index with a positive imaginary part of the form n.x/ D

.x/ c02 Ci c 2 .x/ k

in terms of an absorption coefficient  > 0 in D. In the sequel, the refractive index n is assumed to be a piecewise continuous complex-valued function such

Inhomogeneous Media Identification

that n.x/ D 1 for x … D and 0 and =.n/  0. For a vector d 2 R3 , with jd j D 1, the function e i kxd satisfies the Helmholtz equations in R3 , and it is called a plane wave, since e i.kxd !t / is constant on the planes kx  d  !t Dconst. Summarizing, given the incident field ui and the physical properties of the inhomogeneity, the direct scattering problem is to find the scattered wave and in particular its behavior at large distances from the scattering object, i.e., its far-field behavior. The inverse scattering problem takes this answer to the direct scattering problem as its starting point and asks what is the nature of the scatterer that gave rise to such far-field behavior?

687

R3 was solved by Nachman [13], Novikov [14], and Ramm [16] who based their analysis on the fundamental work of Sylvester and Uhlmann [17]. Their uniqueness proof was considerably simplified by H¨ahner [9] (see [2, 13, 17] and the references in [8] and [10]). The uniqueness problem for an inhomogeneous media in R2 , which is a formerly determined problem, was recently solved by Bukhgeim [2]. In particular, under the assumptions on the refractive index stated in the Introduction, the following uniqueness result holds. Theorem 1 The refractive index n in (2) is uniquely determined from u1 .x; O d / for x; O d 2 S 2 and a fixed value of the wave number k.

Identification of Inhomogeneities from Far-Field Data

It is important to notice that owning to fact that u1 is real analytic in S 2 S 2 , for the uniqueness problem, it suffices to know u1 .x; O d / for x; O d on subsets of S 2 s It can be shown that radiating solutions u to the having an accumulation point. Helmholtz equation (i.e., solutions that satisfy the The identifiability problem for the matrix index of Sommerfeld radiation condition (3)) assume the refraction of an anisotropic media is more compliasymptotic behavior cated. In the mathematical model of the scattering by anisotropic media, Eq. (2) is replaced by   i kjxj  e 1 u1 .x/ ; jxj ! C1 O CO us .x/ D jxj jxj in R3 (5) r  Aru C k 2 n.x/u D 0 (4) uniformly for all directions xO where the function u1 defined on the unit sphere S 2 is known as the far-field where n satisfies the same assumptions as in Introducpattern of the scattered wave. For plane wave incidence tion and A is a 3 3 piecewise continuous matrixui .x; d / D e i kxd , we indicate the dependence of the valued function with a positive definite real part, i.e., far-field pattern on the incident direction d and the  ˛j j2 , ˛ > 0 in D, non-positive imaginary O d /. part, i.e.,  =.A/  0 in D and A D I in R3 n D. observation direction xO by writing u1 D u1 .x; The inverse scattering problem or in other words inho- In general, it is known that u .x; O d 2 S2 1 O d / for x; mogeneous media identification problem can now be does not uniquely determine the matrix A even it is formulated as the problem of determining the index known for all wave numbers k > 0, and hence without of refraction n (and hence also its support D) from a further a priori assumptions, the determination of D O d / for xO and is the most that can be hoped. To this end, H¨ahner knowledge of the far-field pattern u1 .x; d on the unit sphere S 2 (or a subset of S 2 ). All the (2000) proved that the support D of an anisotropic results presented here are valid in R2 as well. Also, inhomogeneity is uniquely determined from u .x; 1 O d/ it is possible to extend our discussion to the case of for x; O d 2 S 2 and a fixed value of the wave number k point source incidence and near-field measurements provided that either  ˇj j2 or  (see [3]). ˇj j2 for some constant ˇ > 1. Uniqueness The first question to approach the problem is whether the inhomogeneous media is identifiable from the exact data, which in mathematical terms is known as the uniqueness problem. The uniqueness problem for inverse scattering by an inhomogeneous medium in

Reconstruction Methods Recall the scattering problem described by (2)–(3) for the total field u D ui Cus with plane wave incident field ui WD e i kxd . The total field satisfies the LippmannSchwinger equation (see [8])

I

688

Inhomogeneous Media Identification

Z

the full nonlinear inverse medium problem. To write a nonlinear optimization setup, note that the inverse medium problem is equivalent to solving the system of equations composed by (6) and (7) for u and m where u1 is in practice the (noisy) measured data uı1 with Z ı > 0 being the noise level. Thus, a simple least square k2 O O d/ D  e i k xy m.y/u.y/ dy; x; O d 2 S 2: approach looks for minimizing the cost functional u1 .x; 4 R3 (7) kui C T mu  uk2L2 .BS 2 / Since the function m WD 1  n has support D, the .u; m/ WD kui k2L2 .BS 2 / integrals in (6) and (7) can in fact be written over a bounded domain containing D. The goal is to reconkuı1  F mu  uk2L2 .S 2 S 2 / struct m.x/ from a knowledge of (the measured) farC O d / based on (7). The dependence of field pattern u1 .x; kuı1 k2L2 .BS 2 / (7) on the unknown m is in a nonlinear fashion; thus, the inverse medium problem is genuinely a nonlinear for u and m over admissible sets, where T mu denotes problem. The reconstruction methods can, roughly the integral in (6) and F mu denotes the integral in speaking, be classified into three groups, Born or weak (7). The discrete versions of this optimization problem scattering approximation, nonlinear optimization tech- suffer from a large number of unknowns and thus niques, and qualitative methods (we remark that this is expensive. Regularization techniques are needed to classification is not inclusive). handle instability due to ill-posedness. A more rigorous mathematical approach to deal with nonlinearity in (6) and (7) is the Newton-type iterBorn Approximation Born approximation, known otherwise as weak scatter- ative method. To this end, it is possible to reformulate ing approximation, turns the inverse medium scattering the inverse medium problem as a nonlinear operator problem into a linear problem and therefore is often equation by introducing the operator F W m ! u1 that employed in practical applications. This process is maps m WD 1  n to the far-field pattern u1 .; d / for justified under restrictive assumption that the scattered plane incidence ui .x/ D e i kxd . In view of uniqueness field due to the inhomogeneous media is only a small theorem, F can be interpreted as an injective operator perturbation of incident field, which at a given fre- from B.B/ (the space of bounded functions defined quency is valid if either the corresponding contrast n1 on a ball B containing the support D of m) into is small or the support D is small. Hence, assuming L2 .S 2 S 2 / (the space of square integrable function that k 2 kmk1 is sufficiently small, one can replace u on S 2 S 2 ). From (7) we can write in (7) by the plane wave incident field e i kxd , thus Z k2 obtaining the linear integral equation for m O e i k xy m.y/u.y/ dy; (9) .F .m//.x; O d/ D  4 B Z k2 O /y x; O d 2 S2 O d/ D  e i k.xd m.y/ dy; x; O d 2 S 2: u1 .x; 4 R3 (8) where u.; d / is the unique solution of (6). Note that F Solving (8) for the unknown m corresponds to invert- is a compact operator, owning this to its analytic kernel; ing the Fourier transform of m restricted to the ball of thus, (9) is severely ill-posed. From the latter it can be radius 2k centered at the origin, i.e., only incomplete seen that the Fr´echet derivative vq of u with respect data is available. This causes uniqueness ambiguities to m (in direction q) satisfies the Lippmann-Schwinger and leads to severe ill-posedness of the inversion. For equation details we refer the reader to Langenberg [12]. Z i kjxyj e k2 vq .x; d / C Nonlinear Optimization Techniques 4 B jx  yj These methods avoid incorrect model assumptions in  m.y/vq .y; d / C q.y/u.y; d / dy; x 2 B herent in weak scattering approximation and consider e i kjxyj m.y/u.y/ dy; x 2 R3; R3 jx  yj (6) and the corresponding far-field pattern is given by

u.x/ D e i kxd 

k2 4

Inhomogeneous Media Identification

689

which implies the following expression for the Fr´echet methods seek to determine an approximation to the derivative of F support of the inhomogeneity by constructing a support indicator function and in some cases provide limited Z k2 information on material properties of inhomogeneous 0 i k.xd O /y O d/ D  e .F .m/q/.x; 4 B media. We provide here a brief expos´e of the linear   sampling method. To this end let us define the far-field m.y/vq .y; d / C q.y/u.y; d / dy; x; O d 2 S 2: operator F W L2 .S 2 / ! L2 .S 2 / by Z Observe that F 0 .m/q D vq;1 where vq;1 is the far2 u1 .xI O d; k/g.d /ds.d / (11) .F g/.x/ O WD field pattern of the radiating solution to v C k nv D S2 2 0 k uq. It can be shown that F .m/ is injective (see [8, 10]). With the help of Fr´echet derivative, it is now We note that by linearity .F g/.x/ O is the far-field patpossible to replace (7) by its linearized version tern corresponding to (1) where theZ incident field ui is a F .m/ C F 0 .m/q D u1

(10)

Herglotz wave function vg .x/ WD

e i kxd g.d /ds.d /. S2

For given k > 0 the far-field operator is injective which, given an initial guess m, it is solved for q to with dense range if and only2 if there does not2 exist a obtain an update mCq. Then as in the classical Newton nontrivial solution v; w 2 L .D/, v  w 2 H .D/ of iterations, this linearization procedure is iterated until the transmission eigenvalue problem some stopping criteria are satisfied. Of course the w C k 2 n.x/w D 0 and v C k 2 v D 0 in D (12) linearized equation inherits the ill-posedness of the @v nonlinear equation, and therefore regularization is rew D v and @w @D (13) @ D @ on ı quired. If u1 is again the noisy far-field measurements, Tikhonov regularization replaces (10) by such that v is a Herglotz wave function. Values of k > 0 for which (12)–(13) has nontrivial solutions ˚

˛q C ŒF 0 .m/  F 0 .m/q D ŒF 0 .m/  uı1  F .m/ are called transmission eigenvalues. If =.n/ D 0, there exits an infinite discrete set of transmission eigenwith some positive regularization parameter ˛ and values accumulating only at C1, [7]. Consider now O z; k/ where O D ˆ1 .x; the L2 adjoint ŒF 0 .m/  of F 0 .m/. Of course for the the far-field equation .F g/.x/ 1 i k xz e O (is the far-field pattern of Newton method to work, one needs to start with a good ˆ1 .x; z; k/ WD 4 initial guess incorporating available a priori informa- the fundamental solution ei kjxyj to the Helmholtz 4jxyj tion, but in principle the method can be formulated for equation). The far-field equation is severely ill-posed one or few incident directions. owing to the compactness of the far-field operator which is an integral operator with analytic kernel. Qualitative Methods Theorem 2 Assume that k is not a transmission In recent years alternative methods for imaging of eigenvalue. Then: (1) If z 2 D for given  > 0 inhomogeneous media have emerged which avoid inthere exists gz;;k 2 L2 .S 2 / such that kF gz;;k  correct model assumptions of weak approximations ˆ1 .; z; k/kL2 .S 2 / <  and the corresponding Herglotz but, as opposed to nonlinear optimization techniques, function satisfies lim kvgz;;k kL2 .D/ exists finitely, and require essentially no a priori information on the scat!0 lim kvgz;;k kL2 .D/ D C1. (2) tering media. Nevertheless, they seek limited informa- for a fixed  > 0, z!@D 2 2 tion about scattering object and need multistatic data, If z 2 R3 n D and  > 0, every g z;;k 2 L .S / i.e, several incident fields each measured at several satisfying kF g z;;k  ˆ1 .; z; k/kL2 .S 2 / <  is such observation directions. Such methods come under the that lim kv gz;;k kL2 .D/ D C1. !0 general title of qualitative methods in inverse scattering theory. Most popular examples of such approaches The linear sampling method is based on attempting to are linear sampling method (Cakoni-Colton [3]), fac- compute the function gz;;k in the above theorem by torization method (Kirsch-Grinberg [11]), and singu- using Tikhonov regularization as the unique minimizer lar sources method (Potthast [15]). Typically, these of the Tikhonov functional (see [8])

I

690

Inhomogeneous Media Identification

kF ı g  ˆ.; z/k2L2 ./ C ˛kgk2L2 .S 2 /

(14)

where the positive number ˛ WD ˛.ı/ is known as the Tikhonov regularization parameter and F ı g is the noisy far-field operator where u1 in (7) is replaced by the noisy far-field data uı1 with ı > 0 being the noise level (note that ˛ı ! 0 as ı ! 0). In particular, one expects that this regularized solution will be relatively smaller for z 2 D than z 2 R3 n D, and this behavior can be visualized by color coding the values of the regularized solution on a grid over some domain containing the support D of the inhomogeneity and thus providing a reconstruction of D. A precise mathematical statement on the described behavior of the regularized solution to the far-field equation is based on factorization method which instead of the farfield operator F considers .F  F /1=4 where F  is the L2 adjoint of F (see [11]). For numerical examples using linear sampling method, we refer the reader to [3]. Having reconstructed the support of the inhomogeneity D, we then obtain information on n.x/ for nonabsorbing media, i.e., if =.n/ D 0. Assume to this end that n.x/ > 1 (similar results hold for 0 < n.x/ < 1), fix a z 2 D, and consider a range of wave number k > 0. If gı;z;k is now the Tikhonov-regularized solution of the far-field equation (14), then we have that: (1) for k > 0 not a transmission eigenvalue lim kvgı;z;k kL2 .D/ exists finitely [1] and (2) for k > 0 ı!0

a transmission eigenvalue lim kvgı;z;k kL2 .D/ D C1 ı!0

0 < k1;D;n  k1;D;n.x/  k1;D;n

(16)

which is clearly seen to be isoperimetric for n.x/ equal to a constant. In particular, (16) shows that for n constant, the first transmission eigenvalue is monotonic decreasing function of n, and moreover, this dependence can be shown to be continuous and strictly monotonic. Using (16), for a measured first transmission eigenvalue k1;D;n.x/ , we can determine a unique constant n0 that satisfies 0 < n  n0  n , where this constant is such that k1;D;n0 D k1;D;n.x/ . This n0 is an integrated average of n.x/ over D. A more interesting question is what does the first transmission eigenvalue say about the matrix index of refraction A for the scattering problem for anisotropic media (5). Assuming n D 1,  j j2 and  =.A/ D 0 in (5), similar analysis for the corresponding transmission eigenvalue problem leads to the isoperimetric inequality 0 < k1;D;a  k1;D;A.x/  k1;D;a [6]. Hence, it is possible to compute a constant a0 such that k1;D;a0 equals the (measured) first transmission eigenvalue k1;D;A.x/ , and this constant satisfies 0 < a  a0  a , where a D inf a1 .x/, D

a D sup a3 .x/, and a1 .x/ and a3 .x/ are the smallest and the largest eigenvalues of the matrix A1 .x/, respectively. The latter inequality is of particular interest since A.x/ is not uniquely determined from the far field, and to our knowledge this is the only information obtainable to date about A.x/ that can be determined from far-field data (see [6] for numerical examples).

(for almost all z 2 D) [4]. In practice, this means if kgı;z;k kL2 .S 2 / is plotted against k, the transmission eigenvalues will appear as sharp picks and thus providing a way to compute transmission eigenvalues from far-field measured data. A detailed study of trans- Cross-References mission eigenvalue problem [7] reveals that the first transmission is related to the index of refraction n.  Optical Tomography: Applications More specifically, letting n D infD n.x/ and n D supD n.x/, the following Faber-Krahn type inequalities hold: 2 k1;n.x/;D 

1 .D/ n

References (15)

where k1;n.x/;D is the first transmission eigenvalue corresponding to d and n.x/ and 1 .D/ is the first Dirichlet eigenvalue for  in D, and

1. Arens, T.: Why linear sampling works. Inverse Probl. 20 (2004) 2. Bukhgeim, A.: J. Recovering a potential from Cauchy data in the two dimensional case. Inverse Ill-Posed Probl. 16 (2008) 3. Cakoni, F., Colton, D.: Qualitative Methods in Inverse Scattering Theory. Springer, Berlin (2006)

Initial Value Problems 4. Cakoni, F., Colton, D., Haddar, H.: On the determination of Dirichlet and transmission eigenvalues from far field data. Comptes Rendus Math. 348 (2010) 5. Cakoni, F., Colton, D., Monk, P.: The Linear Sampling Method in Inverse Electromagnetic Scattering CBMS-NSF, vol. 80. SIAM, Philadelphia (2011) 6. Cakoni, F., Colton, D., Monk, P., Sun, J.: The inverse electromagnetic scattering problem for anisotropic media. Inverse Probl. 26 (2010) 7. Cakoni, F., Gintides, D., Haddar, H.: The existence of an infinite discrete set of transmission eigenvalues. SIAM J. Math. Anal. 42 (2010) 8. Colton, D., Kress, R.: Inverse Acoustic and Electromagnetic Scattering Theory, 2nd edn. Springer, Berlin (1998) 9. H¨ahner, P.: A periodic Faddeev-type solution operator. J. Differ. Equ. 128, 300–308 (1996) 10. H¨ahner, P.: Electromagnetic wave scattering. In: Pike, R., Sabatier, P. (eds.) Scattering. Academic, New York (2002) 11. Kirsch, A., Grinberg, N.: The Factorization Method for Inverse Problems. Oxford University Press, Oxford (2008) 12. Langenberg, K.: Applied inverse problems for acoustic, electromagnetic and elastic wave scattering. In: Sabatier (ed.) Basic Methods of Tomography and Inverse Problems. Adam Hilger, Bristol/Philadelphia (1987) 13. Nachman, A.: Reconstructions from boundary measurements. Ann. Math. 128 (1988) 14. Novikov, R.: Multidimensional inverse spectral problems for the equation  C .v.x/  Eu.x// D 0. Transl. Funct. Anal. Appl. 22, 263–272 (1888) 15. Potthast, R.: Point Sourse and Multipoles in Inverse Scattering Theory. Research Notes in Mathematics, vol. 427. Chapman and Hall/CRC, Boca Raton (2001) 16. Ramm, A.G.: Recovery of the potential from fixed energy scattering data. Inverse Probl. 4, 877–886 (1988) 17. Sylvester, J., Uhlmann, G.: A global uniqueness theorem for an inverse boundary value problem. Ann. Math. 125 (1987)

Initial Value Problems Ernst Hairer and Gerhard Wanner Section de Math´ematiques, Universit´e de Gen`eve, Gen`eve, Switzerland

We describe initial value problems for ordinary differential equations and dynamical systems, which have a tremendous range of applications in all branches of science. We also explain differential equations on manifolds and systems with constraints.

691

Ordinary Differential Equations An ordinary differential equation is a formula yP D f .t; y/; which relates the time derivative of a function y.t/ to its function value. Any function y.t/ defined on an interval I  R and satisfying y.t/ P D f .t; y.t// for all t 2 I is called a solution of the differential equation. If the value y.t/ is prescribed at some point t0 , we call the problem yP D f .t; y/;

y.t0 / D y0

an initial value problem. In most situations of practical interest the function y.t/ is vector-valued, so that we are in fact concerned with a system yP1 D f1 .t; y1 ; : : : ; yn /; y1 .t0 / D y10 ; :: :: : : yPn D fn .t; y1 ; : : : ; yn /; yn .t0 / D yn0 : The differential equation is called autonomous if the vector field f does not explicitly depend on time t. An equation of the form P y/ y .k/ D f .t; y .k1/ ; : : : ; y; is a differential equation of order k. By introducing P : : : ; yk D y .k1/ , and the variables y1 D y, y2 D y, adding the equations yPj D yj C1 for j D 1; : : : ; k  1, such a problem is transformed into a system of firstorder equations. Example 1 The Van der Pol oscillator is an autonomous second-order differential equation. Written as a first-order system the equations are given by yP1 D y2 yP2 D  .1  y12 / y2  y1 : Since the problem is autonomous, solutions can conveniently be plotted as paths in the phase space .y1 ; y2 /. Several of them can be seen in Fig. 1 (left). Arrows indicate the direction of the flow. We observe that all solutions tend for a large time to a periodic solution (limit cycle).

I

692

Initial Value Problems

Initial Value Problems, Fig. 1 Solutions in the phase space of the Van der Pol oscillator for  D 0:4 (left); solutions of the linearized equation (right) −3

−2

2

2

1

1

−1

1

2

3

−2

−1

1

−1

−1

−2

−2

2

which leads, for 0 <  < 2, to complex p eigenvalues 1;2 D  ˙ i! with  D 2 and ! D 1   2 . The Systems of differential equations can be solved analyt- solutions are thus linear combinations of e t cos !t and ically only in very special situations. One of them are e t sin !t. Some of these outward spiraling solutions are displayed in Fig. 1 (right) and mimic those of the linear equations with constant coefficients, Van der Pol equation close to the origin.

Linear Systems with Constant Coefficients

yP D Ay;

y.0/ D y0 ;

where y.t/ 2 Rn , and A is a constant matrix of dimension n. A linear change of coordinates y D T z transforms the system into zP D z with  D T 1 AT . If T can be chosen such that  D diag .1 ; : : : ; n / is diagonal, we obtain zj .t/ D ej t cj , and the solution y.t/ via the relation y.t/ D T z.t/. The free parameters c1 ; : : : ; cn can be chosen to match the initial condition y.0/ D y0 . If the matrix A cannot be diagonalized, it can be transformed to upper triangular form (Schur or Jordan canonical form). Starting with zn .t/, the functions zj .t/ can be obtained successively by solving scalar, inhomogeneous linear equations with constant coefficients. An explicit formula for the solution of the linear system yP D Ay is obtained by using the matrix exponential

Existence, Uniqueness, and Differentiability of the Solutions Whenever it is not possible to find the solution of a differential equation in analytic form, it is still of interest to study its existence, uniqueness, and qualitative properties.

Existence and Uniqueness Consider a differential equation yP D f .t; y/ with a continuously differentiable function f W U ! Rn , where U  R Rn is an open set, and let .t0 ; y0 / 2 U . Then, there exists a unique function y W I ! Rn on a (maximal) open interval I D I.t0 ; y0 / such that • y.t/ P D f .t; y.t// for t 2 I and y.t0 / D y0 . 1 k X kt • .t; y.t// approaches the border of U whenever t : y.t/ D exp.At/ y0 ; exp.At/ D A kŠ tends to the left (or right) end of the interval I . kD0 • If z W J ! Rn is a solution of yP D f .t; y/ satisfying z.t0 / D y0 , then J  I and z.t/ D y.t/ Example 2 If we neglect in the Van der Pol equation, for t 2 J . for y1 small, the cubic term y12 y2 , we obtain the system The statement is still true if the differentiability assumption is weakened to a “local Lipschitz condition.” yP1 D y2 In this case the local existence and uniqueness result is known as the theorem of Picard–Lindel¨of. yP2 D  y2  y1 ;

Initial Value Problems

693

Variational Equation • If all eigenvalues of the matrix f 0 .y0 / satisfy If the dependence on the initial condition is of interest, 2n. J. Fourier Anal. Appl. 9, 1049–1056 (2003) 8. Brown, R., Uhlmann, G.: Uniqueness in the inverse conductivity problem with less regular conductivities in two dimensions. Commun. PDE 22, 1009–10027 (1997) 9. Bukhgeim, A.: Recovering the potential from Cauchy data in two dimensions. J. Inverse Ill-Posed Probl. 16, 19–34 (2008) 10. Bukhgeim, L., Uhlmann, G.: Recovering a potential from partial Cauchy data. Commun. PDE 27, 653–668 (2002) 11. Calder´on, A.P.: On an inverse boundary value problem. In: Seminar on Numerical Analysis and Its Applications to Continuum Physics, R´ıo de Janeiro, pp. 65–73. Sociedade Brasileira de Matematica, Rio de Janeiro (1980) 12. Caro, P., Ola, P., Salo, M.: Inverse boundary value problem for Maxwell equations with local data. Commun. PDE 34, 1425–1464 (2009) 13. Caro, P., Rogers, K.: Global uniqueness for the Calder´on problem with Lipschitz conductivities. arXiv: 1411.8001 14. Caro, P., Zhou, T.: On global uniqueness for an IBVP for the time-harmonic Maxwell equations. Analysis & PDE, 7(2), 375–405 (2014) 15. Cheney, M., Isaacson, D., Somersalo, E.: A linearized inverse boundary value problem for Maxwell’s equations. J. Comput. Appl. Math. 42, 123–136 (1992) 16. Colton, D., P¨aiv¨arinta, L.: The uniqueness of a solution to an inverse scattering problem for electromagnetic waves. Arch. Ration. Mech. Anal. 119, 59–70 (1992) 17. Dos Santos Ferreira, D., Kenig, C.E., Salo, M., Uhlmann, G.: Limiting Carleman weights and anisotropic inverse problems. Invent. Math. 178, 119–171 (2009) 18. Greenleaf, A., Lassas, M., Uhlmann, G.: The Calder´on problem for conormal potentials, I: global uniqueness and reconstruction. Commun. Pure Appl. Math. 56, 328–352 (2003) 19. Haberman, B.: Uniqueness in Calder´on’s problem for conductivities with unbounded gradient, arXiv:1410.2201 20. Haberman, B., Tataru, D.: Uniqueness in Calder´on’s problem with Lipschitz conductivities. Duke Math J., 162, 497– 516 (2013) 21. Heck, H., Wang, J.-N.: Stability estimates for the inverse boundary value problem by partial Cauchy data. Inverse Probl. 22, 1787–1796 (2006) 22. Holder, D.: Electrical Impedance Tomography. Institute of Physics Publishing, Bristol/Philadelphia (2005) 23. Ide, T., Isozaki, H., Nakata, S., Siltanen, S., Uhlmann, G.: Probing for electrical inclusions with complex spherical waves. Commun. Pure. Appl. Math. 60, 1415–1442 (2007) 24. Ikehata, M.: How to draw a picture of an unknown inclusion from boundary measurements: two mathematical inversion algorithms. J. Inverse Ill-Posed Probl. 7, 255–271 (1999) 25. Ikehata, M., Siltanen, S.: Numerical method for finding the convex hull of an inclusion in conductivity from boundary measurements. Inverse Probl. 16, 1043–1052 (2000)

Inverse Nodal Problems: 1-D 26. Imanuvilov, O., Uhlmann, G., Yamamoto, M.: The Calder´on problem with partial data in two dimensions. J. Am. Math. Soc. 23, 655–691 (2010) 27. Isakov, V.: On uniqueness in the inverse conductivity problem with local data. Inverse Probl. Imaging 1, 95–105 (2007) 28. Kenig, C., Salo, M., Uhlmann, G.: Inverse problem for the anisotropic Maxwell equations. Duke Math. J. 157(2), 369– 419 (2011) 29. Kenig, C., Sj¨ostrand, J., Uhlmann, G.: The Calder´on problem with partial data. Ann. Math. 165, 567–591 (2007) 30. Knudsen, K.: The Calder´on problem with partial data for less smooth conductivities. Commun. Partial Differ. Equ. 31, 57–71 (2006) 31. Kohn R., and Vogelius M.: Determining conductivity by boundary measurements II. Interior results. Comm. Pure Appl. Math. 38, 643–667 (1985) 32. Lassas, M., Uhlmann, G.: Determining a Riemannian man´ ifold from boundary measurements. Ann. Sci. Ecole Norm. Sup. 34, 771–787 (2001) 33. Lassas, M., Taylor, M., Uhlmann, G.: The Dirichlet-toNeumann map for complete Riemannian manifolds with boundary. Commun. Geom. Anal. 11, 207–222 (2003) 34. Lee, J., Uhlmann, G.: Determining anisotropic real-analytic conductivities by boundary measurements. Commun. Pure Appl. Math. 42, 1097–1112 (1989) 35. Melrose, R.B.: Geometric Scattering Theory. Cambridge University Press, Cambridge/New York (1995) 36. Nachman, A.: Reconstructions from boundary measurements. Ann. Math. 128, 531–576 (1988) 37. Nachman, A.: Global uniqueness for a two-dimensional inverse boundary value problem. Ann. Math. 143, 71–96 (1996) 38. Nachman, A., Street, B.: Reconstruction in the Calder´on problem with partial data. Commun. PDE 35, 375–390 (preprint) 39. Novikov, R.G.: Multidimensional inverse spectral problems for the equation  C .v.x/  Eu.x// D 0. Funktsionalny Analizi Ego Prilozheniya 22, 11–12, Translation in Funct. Anal. Appl. 22, 263–272 (1988) 40. Ola, P., Someralo, E.: Electromagnetic inverse problems and generalized Sommerfeld potential. SIAM J. Appl. Math. 56, 1129–1145 (1996) 41. Ola, P., P¨aiv¨arinta, L., Somersalo, E.: An inverse boundary value problem in electrodynamics. Duke Math. J. 70, 617– 653 (1993) 42. P¨aiv¨arinta, L., Panchenko, A., Uhlmann, G.: Complex geometrical optics for Lipschitz conductivities. Rev. Mat. Iberoam. 19, 57–72 (2003) 43. Ramm, A.: Recovery of the potential from fixed energy scattering data. Inverse Probl. 4, 877–886 (1988) 44. Sun, Z., Uhlmann, G.: Anisotropic inverse problems in two dimensions. Inverse Probl. 19, 1001–1010 (2003) 45. Sylvester, J.: An anisotropic inverse boundary value problem. Commun. Pure Appl. Math. 43, 201–232 (1990) 46. Sylvester, J., Uhlmann, G.: A uniqueness theorem for an inverse boundary value problem in electrical prospection. Commun. Pure Appl. Math. 39, 92–112 (1986) 47. Sylvester, J., Uhlmann, G.: A global uniqueness theorem for an inverse boundary value problem. Ann. Math. 125, 153–169 (1987)

725 48. Uhlmann, G.: Inverse boundary value problems and applications. Ast´erisque 207, 153–211 (1992) 49. Uhlmann, G., Vasy, A.: Low-energy inverse problems in three-body scattering. Inverse Probl. 18, 719–736 (2002) 50. Uhlmann, G., Wang, J.-N.: Complex spherical waves for the elasticity system and probing of inclusions. SIAM J. Math. Anal. 38, 1967–1980 (2007) 51. Uhlmann, G., Wang, J.-N.: Reconstruction of discontinuities using complex geometrical optics solutions. SIAM J. Appl. Math. 68, 1026–1044 (2008) 52. Zhdanov, M.S., Keller, G.V.: The Geoelectrical Methods in Geophysical Exploration. Methods in Geochemistry and Geophysics, vol. 31. Elsevier, Amsterdam/New York (1994) 53. Zhou, T.: Reconstructing electromagnetic obstacles by the enclosure method. Inverse Probl. Imaging 4, 547–569 (2010) 54. Zou, Y., Guo, Z.: A review of electrical impedance techniques for breast cancer detection. Med. Eng. Phys. 25, 79–90 (2003)

I

Inverse Nodal Problems: 1-D Chun-Kong Law Department of Applied Mathematics, National Sun Yat-sen University, Kaohsiung, Taiwan

Mathematics Subject Classification 34A55; 34B24

Synonyms Inverse Nodal Problems 2-D; Inverse Spectral Problems 1-D Theoretical Results; Inverse Spectral Problems 2-D: Theoretical Results; Inverse Spectral Problems 1-D Algorithms; Multidimensional Inverse Spectral Problems; Regularization of Inverse Problems

Glossary Nodal data A set of nodal points (zeros) of all eigenfunctions. Nodal length The distance between two consecutive nodal points of one eigenfunction.

726

Inverse Nodal Problems: 1-D

Quasinodal set A double sequence fxkn g that satisfies the asymptotic behavior as given in (2).

for a solution y, satisfying y.0/ D 0, y 0 .0/ D 1. After an iteration and some trigonometric calculations, when y.x/ D 0 and cos.sx/ is not close to 0, Z

1 tan.sx/ D 2s

Short Definition

x 0



1 .1  cos.2st//q.t/ dt C o 2 s

 :

This is the inverse problem of recovering parameters in estimates a Sturm-Liouville-type equation using the nodal data. From this, one can easily p derive asymptotic .n/ of the parameters sn D n and xk , by letting x D 1 .n/ and x D xk , respectively:

Description Consider the Sturm-Liouville operator H : Hy D y 00 C q.x/y ;

(1)

with boundary conditions 

y.0/ cos ˛ C y 0 .0/ sin ˛ D 0 : y.1/ cos ˇ C y 0 .1/ sin ˇ D 0

Here q 2 L1 .0; 1/ and ˛; ˇ 2 Œ0; /. Let  be the .n/ nth eigenvalue of the operator H and 0 < x1 < .n/ .n/ x2 <    < xn1 < 1 be the .n  1/ nodal points .n/ of the nth eigenfunction. The double sequence fxk g is called the nodal set associated with H . Also, let .n/ .n/ .n/ lk D xkC1  xk be the associated nodal length. We define the function jn .x/ on .0; 1/ by jn .x/ D maxfk W .n/ xk  xg. Hence, if x and n are fixed, then j D jn .x/ .n/ .n/ implies x 2 Œxj ; xj C1 /. In many applications, certain nodal set associated with a potential can be measured. Hence, it is desirable to recover the potential with this nodal set. The inverse nodal problem was first defined by McLaughlin [19]. She showed that knowledge of the nodal points alone can determine the potential function in L2 .0; 1/ up to a constant. Up till now, the issues of uniqueness, reconstruction, smoothness, and stability are all solved for q 2 L1 .0; 1/ [14, 16, 19, 25].

Z 1 1 sn D n C .1  cos.2sn t//q.t/dt 2sn 0   1 Co 2 sn Z x .n/ k k 1 .n/ xk D C 2 .1  cos.2sn t// q.t/ dt sn 2sn 0   1 (2) Co 3 : sn Hence, the nodal length is given by .n/ lk D

Z

 1 C sn 2sn2

.n/

xkC1 .n/

xk

 1 .1cos.2sn t//q.t/ dtCo 3 ; sn 

from which, one arrives at !

.n/

2sn2

sn ljn .x/

sn D 

1

 Z

.n/

xkC1 .n/

.1  cos.2sn t//q.t/ dt Co.1/

xk

! q.x/ where the convergence is pointwise a.e. as well as L1 . If we put in the asymptotic expression of sn , we obtain that pointwisely a.e. and in L1 [7, 16],

! Reconstruction Formula, Smoothness, and .n/ ljn .x/ Z 1 .n/ 2 2 Stability q : q.x/ D lim 2n  nljn .x/  1 C n!1 2n 2 0 For simplicity, we consider the Dirichlet boundary conditions ˛ D ˇ D 0. We can turn the SturmR 1(3) Thus, given the nodal set plus the constant 0 q, Liouville equation into the integral equation one can recover the potential function. Unlike most Z inverse spectral problems, the reconstruction formula sin sx 1 x y.x/ D sinŒs.x  t/ q.t/y.t/ dt; C here is direct and explicit. However, the problem s s 0

Inverse Nodal Problems: 1-D

727

is overdetermined, as the limit does not require starting terms. On the other hand, we define the difference quotient operator ı as follows:

D lim n  3

2

n!1

.n/

.n/

ı k ai

D

D

.n/

.n/ xi C1



D

.n/ xi

.n/ ı k1 ai C1

n!1

.n/

ai C1  ai



.n/ li

ai

; and

.n/ li

.n/ ı k1 ai

:



.n/

j 1 

X

.n/

.n/

lkC2  lk

D lim 2n 

2

Z 1  1 D 2n  q nC 2n 2 0    .n/ .n/ .n/ ljn .x/ C ıljn .x/  x  xjn .x/  1 : 

2

2

Furthermore, we let n o .n/ .n/ .n/ Fn.k/ .x/ D 2n3  2 ı k lj C ı kC1 lj  .x  xj / : With these definitions, one can show the following theorem [15, 16].





  .n/ .n/ lj  l1

D q.x/  q.0/; .n/

.n/

.n/

.n/

using the facts such as lk ılk D lkC1  lk ,

Hence, the above reconstruction formula, whose term   1 .n/ is a step function, can be linked up to a continuous .n/ lkC1 lk D o ; n3 function Fn.0/ .x/

.n/

ılk C ılkC1

i D1 3

n!1

.n/

lk

i D1

D lim n3  2 .n/ ıai

j 1 X



.n/ ılk

and

1 Do n2

 :

The rest of the proof is similar. Next, we would like to add that this inverse nodal problem is also stable [14]. Let X and X be the nodal sets associated with the potential function q and q respectively. Define Sn .X; X/ WD n2  2

n1 X

.n/

.n/

jlk  l k j

kD0

d0 .X; X/ WD limn!1 Sn .X; X/; d.X; X/ WD limn!1

and

Sn .X; X/ 1 C Sn .X; X/

:

Theorem 1 Suppose q is C N C1 on Œ0; 1 .N  1/. Then for each x 2 .0; 1/ and k D 0; : : : ; N , as n ! Note that it is easy to show that d.X; X/  d0 .X; X/. If d0 .X; X/ < 1, then 1,   1 : q .k/ .x/ D Fn.k/ .x/ C O d.X; X/ n : d0 .X; X/  1  d.X; X/ .k/ Conversely, if Fn is uniformly convergent on compact subsets of .0; 1/, for each k D 1; : : : ; N , then q is C N That means, d0 .X; X/ is close to 0 if and only if .k/ on .0; 1/, and Fn is uniformly convergent to q .k/ on d.X; X/ is close to 0. We shall state the following compact subsets of .0; 1/. theorem without proof: The proof depends on the definition of ı k ai . Let Theorem 2 kq  qkL1 D 2d0 .X; X/: .1/ G .1/ D limn!1 Fn . Then Z

Z

x

G .t/ dt D lim

x

.1/

0

Z

D lim 2n3  2 n!1

0

n!1 0 xh .n/ ıljn .t /

Fn.1/ .t/ dt i  .n/ C ı 2 ljn.t /  t  xjn .t / dt

 j 1

2 X 1  .n/ .n/ .n/ .n/ 2 .n/ x D lim 2n  lk ılk C x ı lk n!1 2 kC1 k 3

2

kD1

Numerical Aspects In [12], Hald and McLaughlin give two numerical algorithms for the reconstruction of q. One of the algorithms can be induced from (3) above, while the other needs the information about the eigenvalues as well. Some other algorithms are also given for the other coefficient functions such as elastic modulus and density function. In [9], a Tikhonov regularization

I

728

Inverse Nodal Problems: 1-D

approach is taken instead, on the foundation that the References problem is overdetermined R 1 and ill-posed. Let Q D fp 2 H 1 ..0; 1// W 0 p.x/dx D 0g. Also let 1. Browne, P.J., Sleeman, B.D.: Inverse nodal problems for Sturm-Liouville equations with eigenparameter dependent X.n/  RnC1 such that x 2 X.n/ implies x D boundary conditions. Inverse Probl. 12, 377–381 (1996) f0; x1 ; : : : ; xn1 ; 1/. Letting z.n; p/ be the zero set of 2. Buterin, S.A., Shieh, C.T.: Inverse nodal problems for difthe nth eigenfunction, we define a Tikhonov functional ferential pencils. Appl. Math. Lett. 22, 1240–1247 (2009) on X.n/ Q 3. Cheng, Y.H.: Reconstruction of the Sturm-Liouville operaZ

1

E.n; ; x; p/ D jx  z.n; p/j2 C 

p 0 .x/2 dx:

4.

0

Let p be the minimizer, which exists. When n is large enough and x D z.n; q/, then  kp  qkL2  C

1 C n5 n2

Z

1

q 02 :

5. 6. 7.

0

Further Remarks In fact, boundary data can also be reconstructed with nodal data [4]. Hill’s operator was also tackled with successfully [5]. In [10], some of the arguments above are refined and made more compact. C.L. Shen solved the inverse nodal problem for the density function [20, 21,23], while Hald and McLaughlin [13] weakened the condition to bounded variations. Shen, together with Shieh, further investigated the 2 2 vectorial SturmLiouville system with certain nodal sets [22]. Buterin and Shieh [2] gave some reconstruction formulas for the two coefficient functions p and q in the diffusion operator y 00 C .2p C q/y D 2 y. Law, Lian, and Wang [17] solved the inverse nodal problem for the one-dimensional p-Laplacian eigenvalue problem, which is a nonlinear analogue of the Sturm-Liouville operator. Finally, the problem for Dirac operators was also studied by C.F. Yang [27]. More studies on other equations or systems are encouraged. Right now, most methods here make use of asymptotics of eigenvalues and nodal points. It would be desirable to explore other methods that can avoid the overdetermination of data. We add here that X.F. Yang used the nodal data on a subinterval I D .0; b/, where b > 1=2, to determine the potential function in L1 .0; 1/ uniquely [6, 26]. Recently, it has been shown that an arbitrarily short interval I D .a1 ; a2 / containing the point 1/2 suffices to determine uniquely [1]. Furthermore, there is the issue of existence. Is there any condition, no matter how strong, that can guarantee that some sequence is the nodal set of some potential function?

8.

9.

10.

11.

12. 13.

14. 15.

16.

17.

18.

19.

20. 21.

tor on a p-star graph with nodal data. Rocky Mt. J. Math. 42, 1431–1446 (2011) Cheng, Y.H., Law, C.K.: On the quasinodal map for the Sturm-Liouville problem. Proc. R. Soc. Edinb. 136A, 71–86 (2006) Cheng, Y.H., Law, C.K.: The inverse nodal problem for Hill’s equation. Inverse Probl. 22, 891–901 (2006) Cheng, Y.H., Law, C.K., Tsay, J.: Remarks on a new inverse nodal problem. J. Math. Anal. Appl. 248, 145–155 (2000) Chen, Y.T., Cheng, Y.H., Law, C.K., Tsay, J.: L1 convergence of the reconstruction formula for the potential function. Proc. Am. Math. Soc. 130, 2319–2324 (2002) Cheng, Y.H., Shieh, C.T., Law, C.K.: A vectorial inverse nodal problem. Proc. Am. Math. Soc. 133(5), 1475–1484 (2005) Chen, X., Cheng, Y.H., Law, C.K.: Reconstructing potentials from zeros of one eigenfunction. Trans. Am. Math. Soc. 363, 4831–4851 (2011) Currie, S., Watson, B.A.: Inverse nodal problems for SturmLiouville equations on graphs. Inverse Probl. 23, 2029–2040 (2007) Guo, Y., Wei, G.: Inverse problems: Dense nodal subset on an interior subinterval, J. Differ. Equ. 255, 2002–2017 (2013) Hald, O.H., McLaughlin, J.R.: Solutions of inverse nodal problems. Inverse Probl. 5, 307–347 (1989) Hald, O.H., Mclaughlin, J.R.: Inverse problems: recovery of BV coefficients from nodes. Inverse Probl. 14, 245–273 (1998) Law, C.K., Tsay, J.: On the well-posedness of the inverse nodal problem. Inverse Probl. 17, 1493–1512 (2001) Law, C.K., Yang, C.F.: Reconstructing the potential function and its derivatives using nodal data. Inverse Probl. 14, 299– 312 (1998) Law, C.K., Shen, C.L., Yang, C.F.: The inverse nodal problem on the smoothness of the potential function. Inverse Probl. 15, 253–263 (1999); Erratum 17, 361–364 (2001) Law, C.K., Lian, W.C., Wang, W.C.: Inverse nodal problem and Ambarzumyan problem for the p-Laplacian. Proc. R. Soc. Edinb. 139A, 1261–1273 (2009) Lee, C.J., McLaughlin, J.R.: Finding the density for a membrane from nodal lines. In: Chavent, G., et al. (eds.) Inverse Problems in Wave Propagation, pp. 325–345. Springer, New York (1997) McLaughlin, J.R.: Inverse spectral theory using nodal points as data-a uniqueness result. J. Differ. Equ. 73, 354–362 (1988) Shen, C.L.: On the nodal sets of the eigenfunctions of the string equation. SIAM J. Math. Anal. 6, 1419–1424 (1988) Shen, C.L.: On the nodal sets of the eigenfunctions of certain homogeneous and nonhomgeneous membranes. SIAM J. Math. Anal. 24, 1277–1282 (1993)

Inverse Optical Design

729

Electromagnetic design, however, remains primarily restricted to heuristic methods in which scientists intuit structures that (hopefully) have the characteristics they desire. Inverse design promises to overtake such methods and provide an efficient approach for achieving nonintuitive, superior designs. The inverse design problem cannot be solved by simply choosing a desired electric field and numerically computing the dielectric structure. It is generally unknown whether such a field can exist and, if so, whether the dielectric structure producing it has a simple physical realization. Instead, the inverse problem needs to be approached through iteration: given an initial structure, how should one iterate such that the final structure most closely achieves the desired functionality? From this viewpoint, it is clear that inverse design problems can be treated as optimization problems, in which the “merit function” to be optimized represents Inverse Optical Design the desired functionality. The merit function is subject to the constraint that all fields, frequencies, etc., must Owen D. Miller and Eli Yablonovitch be solutions of Maxwell’s equations; consequently, Department of Electrical Engineering and Computer inverse design is also sometimes referred to as PDESciences, University of California, Berkeley, CA, constrained optimization. USA This entry primarily focuses on the methodology for finding the optimal design of an electromagnetic structure. We describe the physical mechanism underpinning adjoint-based optimization, in which two simSynonyms ulations for each iteration provide information about Electromagnetic shape optimization; Electromagnetic how to update the structure. We then discuss some topology optimization; Inverse electromagnetic design applications of the method and key research results in the literature.

22. Shen, C.L., Shieh, C.T.: An inverse nodal problem for vectorial Sturm-Liouville equations. Inverse Probl. 16, 349–356 (2000) 23. Shen, C.L., Tsai, T.M.: On a uniform approximation of the density function of a string equation using eigenvalues and nodal points and some related inverse nodal problems. Inverse Probl. 11, 1113–1123 (1995) 24. Shieh, C.T., Yurko, V.A.: Inverse nodal and inverse spectral problems for discontinuous boundary value problems. J. Math. Anal. Appl. 374, 266–272 (2008) 25. Yang, X.F.: A solution of the inverse nodal problem. Inverse Probl. 13, 203–213 (1997) 26. Yang, X.F.: A new inverse nodal problem. J. Differ. Equ. 169, 633–653 (2001) 27. Yang, C.F., Huang, Z.Y.: Reconstruction of the Dirac operator from nodal data. Integral Equ. Oper. Theory 66, 539–551 (2010)

Definition Electromagnetic Optimization Inverse optical design requires finding a dielectric structure, if it exists, that produces a desired optical response. Such a problem is the inverse of the more common problem of finding the optical response for a given dielectric structure.

Overview Inverse design represents an important new paradigm in electromagnetics. Over the past few decades, substantial progress has been made in computing the electromagnetic response of a given structure with sources, to the point where several commercial programs provide computational tools for a wide array of problems.

As previously discussed, successful inverse design finds a structure through iterative optimization: an initial design is created, computations are done to find a new design, the design is updated, and the loop continues. Whether an optimization is successful is defined by the efficiency and effectiveness of the computations for finding a new design. In some fields of optimization, stochastic methods such as genetic algorithms or simulated annealing provide the computations. The inefficiency of completing the many electromagnetics simulations required, however, renders such methods generally ineffective in the optical regime. Instead, the so-called “adjoint” approach provides quicker computations while

I

730

Inverse Optical Design

exploiting the fact that the fields must be solutions of Maxwell’s equations. Adjoint-based optimization is well known in mathematics and engineering. As applied to PDE-constrained problems, [4] provides a general introduction while [1] and [10] work out the optimization equations for elasticity and electromagnetic systems, respectively. Instead of taking a purely equation-based approach, however, we will present the adjoint-based optimization technique from a more intuitive viewpoint, to understand the physical origins of adjoint fields [9,11]. For concreteness, we will consider a simplified problem in two dimensions with a specific merit function. The dimensionality will allow us to treat the electric field as a scalar field. The picture will be clearer with these simplifications, and generalizing to three dimensions and a larger group of merit functions does not change the underlying optimization mechanism. The crux of the optimization routine is the decision of how to update the structure from one iteration to the next. Consider, for example, a problem in which the electric field intensity at a single point, x0 , is to be maximized. The merit function J would take the form J D

1 jE.x0 /j2 2

 p D 40

2  1 2 C 21

 a3 Einc

(3)

where Einc is the value of the incident field at the location of the dielectric. Although Eq. 3 assumes a three-dimensional sphere, the two-dimensional case differs only by numerical pre-factors. The addition of dielectric at a point x 0 , then, is equivalent to the addition of an electric dipole driven by Einc D E.x 0 /. The change in field at x0 can be expressed as ıE.x0 / D E.x 0 /Edipole;x 0 .x0 /, where Edipole;x 0 .x0 / is the normalized electric field at x0 from a dipole at x 0 and E.x 0 / (1) provides the driving term. ıJ can be rewritten:

If the change in structure is small between the two iterations, the change in the fields is also relatively small. The change in merit function can then be approximated as  1  E .x0 /ıE.x0 / C E.x0 /ıE  .x0 / 2   D Re E  .x0 /ıE.x0 /

would take thousands or millions of simulations per iteration and is clearly unfeasible. The adjoint method, however, gleans the same information from only two simulations. This is accomplished by exploiting symmetry properties. The first step is to recognize that a small piece of dielectric acts like an electric dipole. If a small sphere of radius a and dielectric constant 2 is added to a background with dielectric 1 , the scattering from the sphere will be approximately equivalent to the fields radiated by an electric dipole with dipole moment [5]:

  ıJ D Re E  .x0 /ıE.x0 /   D Re E  .x0 /E.x 0 /Edipole;x 0 .x0 /   D Re E  .x0 /Edipole;x0 .x 0 /E.x 0 /   D Re W .x 0 /E.x 0 /

(4)

ıJ 

The first step in Eq. 4 is the replacement of ıE.x0 /. (2) The next step is the realization that placing a dipole at x 0 and measuring at x0 is equivalent to placing a Eq. 2 is very important: it states that the change in dipole at x0 and measuring at x 0 . This can be proved merit function is simply the product of the (conjugated) by the symmetry of the Green’s function or by the original field E.x0 / with the change in field incurred by recognition that the optical paths are identical and the the change in geometry, ıE.x0 /. The question becomes fields must therefore be equivalent. The final step is to whether a change in geometry can be chosen to ensure define the adjoint field W .x 0 / D E  .x0 /Edipole;x0 .x 0 /. that ıJ > 0 (or ıJ < 0), so that the merit function By analogy with the definition of ıE, it is clear that W .x 0 / is the field of a dipole at x0 with Einc D increases (decreases) each iteration. The simplest method for choosing a new structure E  .x0 /. Figure 1 motivates this particular sequence of would be brute force. One could add or subtract a operations. With the original form of ıJ , as in Eq. 2, one would small piece of dielectric at every allowable point in the domain, run a simulation to check whether the merit need a simulation to find E.x0 / and then countless function has increased, and then choose the structure simulations to find ıE.x0 / for every possible x 0 at that most increased the merit function. However, this which to add dielectric, as the dipole location would

Inverse Optical Design

a

Inverse Optical Design, Fig. 1 Illustration of how adjointbased electromagnetic optimization exploits symmetry properties. (a) shows a simple but inefficient method for testing whether to add dielectric at x 0 . A first simulation (black) finds E.x0 / and E.x 0 /. Then a second simulation  (red) is run with an electric  dipole at x 0 . Finally, ıJ D Re E  .x0 /E.x 0 /Edipole;x0 .x0 / . In order to decide the location of optimal ıJ , many simulations

change every simulation. By switching the measurement and dipole locations, however, the countless simulations have been reduced to a single one. Placing the dipole at x0 and measuring the resulting field at x 0 , one can calculate W .x 0 / everywhere with a single simulation. This is why adjoint optimization requires only two simulations. Importantly, if the merit function had been the sum or integral of the field intensities on a larger set of points, still only two simulations would be required. In that case, dipoles would be simultaneously placed at each of the points. For a mathematically rigorous derivation of the adjoint field in a more general electromagnetics setting, consult [10].

731

b

with dipoles at each different x 0 would have to be run. In (b) the equivalent calculation is more efficiently completed. The second simulation (red) is of an electric dipole at x0 , instead of x 0 , and the fields are multiplied at x 0 . In this way, only a single extra simulation is required, with a dipole at x0 , and ıJ is known for all possible values of x 0

to design a high-bandwidth T-junction and an efficient 90ı waveguide bend, respectively, in photonic crystal platforms. Plasmonics represents another field in which optimal design may prove particularly useful. In a recent paper, Andkjær et al. [2] designed a grating coupler to efficiently couple surface plasmons to incoming and outgoing waves. The coupler was superior to previous designs achieved by other methods and demonstrates how one might couple into or out of future plasmonbased technologies. Although the examples above and the previous discussion focused on optimizing merit functions in which the fields are the primary variables, the technique extends to eigenfrequencies and other variables. Bandgap optimization, in which the gap between two eigenfrequencies is maximized, is actually a self-adjoint problem for which only a single simulation per iteration is Applications required. Optimal structures with large bandgaps were Adjoint-based optimization has been used as a de- designed in [3] and [8]. sign tool in numerous electromagnetics applications. The authors of [12], for example, designed scattering cylinders such that the radiation from a terminated Discussion waveguide was highly directional and asymmetric. Their design was nonintuitive and would have been Inverse design has been an invaluable tool in fields such almost impossible to achieve through heuristic meth- as aerodynamic design and mechanical optimization. ods. Although nominally designed in the rf regime, the It seems clear that it can provide the same function design would work at optical frequencies in a scaled- in optical design, especially as computational power continues to improve. Through a more intuitive down configuration. Similar research has shown other ways in which understanding of the optimization mechanism, the optimized designs can mold the flow of light in de- technique may become more accessible to a wider sirable ways. Using photonic crystals for waveguiding audience of researchers. By treating dielectrics through and routing is a promising technology, but achieving their dipole moments and iterating through small optimal designs is difficult in practice. In [6] and [7], changes in structure, simple initial structures can the authors used adjoint-based optimization techniques morph into nonintuitive, superior designs. Whereas

I

732

Inverse Problems: Numerical Methods

the current forefront of electromagnetic computation is Inverse Problems: Numerical Methods the quick solution of the response to a given structure, the inverse problem of computing the structure for a Martin Burger given response may prove much more powerful in the Institute for Computational and Applied Mathematics, future. Westf¨alische Wilhelms-Universit¨at (WWU) M¨unster, M¨unster, Germany

Cross-References  Adjoint Methods as Applied to Inverse Problems

Synonyms Discretization; Ill-posed problems; Inverse problems; Numerical methods; Regularization

References 1. Allaire, G., De Gournay, F., Jouve, F., Toader, A.: Structural optimization using topological and shape sensitivity via a level set method. Control Cybern. 34(1), 59 (2005) 2. Andkjær, J., Nishiwaki, S., Nomura, T., Sigmund, O.: Topology optimization of grating couplers for the efficient excitation of surface plasmons. J. Opt. Soc. Am. B 27(9), 1828–1832 (2010) 3. Cox, S., Dobson, D.: Band structure optimization of twodimensional photonic crystals in H-polarization. J. Comput. Phys. 158(2), 214–224 (2000) 4. Giles, M., Pierce, N.: An introduction to the adjoint approach to design. Flow Turbul. Combust. 65, 393–415 (2000) 5. Jackson, J.: Classical Electrodynamics, 3rd edn. Wiley, New York (1999) 6. Jensen, J., Sigmund, O.: Topology optimization of photonic crystal structures: a high-bandwidth low-loss Tjunction waveguide. J. Opt. Soc. Am. B 22(6), 1191–1198 (2005) 7. Jensen, J., Sigmund, O., Frandsen, L., Borel, P., Harpoth, A., Kristensen, M.: Topology design and fabrication of an efficient double 90 degree photonic crystal waveguide bend. IEEE Photon. Technol. Lett. 17(6), 1202 (2005) 8. Kao, C., Osher, S., Yablonovitch, E.: Maximizing band gaps in two-dimensional photonic crystals by using level set methods. Appl. Phys. B Lasers Opt. 81(2), 235–244 (2005) 9. Lalau-Keraly, C.M., Bhargava, S., Miller, O.D., Yablonovitch, E.: Adjoint shape optimization applied to electromagnetic design. Opt. Exp. 21(18), 21,693–21,701 (2013) 10. Masmoudi, M., Pommier, J., Samet, B.: The topological asymptotic expansion for the Maxwell equations and some applications. Inverse Probl. 21, 547 (2005) 11. Miller, O.D.: Photonic design: from fundamental solar cell physics to computational inverse design. Phd thesis, University of California, Berkeley (2012). http://arxiv.org/abs/ 1308.0212 12. Seliger, P., Mahvash, M., Wang, C., Levi, A.: Optimization of aperiodic dielectric structures. J. Appl. Phys. 100, 34,310 (2006)

Definition Inverse problems are problems where one looks for a cause of an observed or desired effect via mathematical model, usually by inverting a forward problem. The forward problem such as solving partial differential equations is usually well posed in the sense of Hadamard, whereas the inverse problem such as determining unknown parameter functions or initial values is ill posed in most cases and requires special care in numerical approaches.

Introduction Inverse problem approaches (often called inverse modeling in engineering) have become a key technique to recover quantitative information in many branches of science. Prominent examples include medical image reconstruction, nondestructive material testing, seismic imaging, and remote sensing. The common abstract approach to inverse problems is to use a forward model that links the unknown u to the available data f , which very often comes as a system of partial differential equations (or integral formulas derived from partial differential equations, cf., e.g., [5, 13]). Solving the forward model given u is translated to evaluating a (possibly nonlinear) operator F . The inverse problem then amounts to solving the operator equation F .u/ D f:

(1)

Particular complications arise due to the fact that in typical situations the solution of (1) is not well

Inverse Problems: Numerical Methods

posed, in particular u does not depend continuously on the data and the fact that practical data always contain measurement and modeling errors. Therefore any computational approach is based on a regularization method, which is a well-posed approximation to (1) parameterized by a regularization parameter ˛, which tunes the degree of approximation. In usual convention, the original problem is recovered in the limit ˛ ! 0 and no noise, and in presence of noise, ˛ needs to be tuned to obtain optimal reconstructions. We will here take a deterministic perspective and denote by ı the noise level, i.e., the maximal norm difference between f and the exact data Ku (u being the unknown exact solution). The need for regularization makes numerical methods such as discretization and iterative schemes quite peculiar in the case of inverse problems; they are always interwoven with the regularization approach. There are two possible issues appearing: • The numerical methods can serve themselves as regularizations, in which case classical questions of numerical analysis have to be reconsidered. For Example, if regularization is achieved by discretization, it is not the key question how to obtain a high order of convergence as the discretization fineness decreases to zero, but it is at least equally important how much robustness is achieved with respect to the noise and how the discretization fineness is chosen optimally in dependence of the noise level. • The regularization is carried out by a different approach, e.g., a variational penalty. In this case one has to considered numerical methods for a parametric problem, and robustness is desirable in the case of small regularization parameter and decreasing noise level, which indeed yields similarities to the first case, often also to singular perturbation problems in differential equations.

733

and the Levenberg-Marquardt method for nonlinear problems ukC1 D uk  .F 0 .uk / F 0 .uk / C ˇk I /1 F 0 .uk / .F .uk /  f /:

(4)

In this case the regularization is the maximal number of iterations k , which are carried out, i.e., ˛ D k1 . Instead of convergence, one speaks of semiconvergence in this respect (cf., [9]): • In the case of exact data f D Ku , one seeks classical convergence uk ! u . • In the case of noisy data f D Ku C nı with knı k  ı, one seeks to choose a maximal number of iterates k .ı/, such that uık.ı/ converges to u as ı ! 0, where uık denotes the sequence of iterates obtained with data f D Ku C nı . Note that in this case the convergence concerns the stopped iterates of different iteration sequences. Major recent challenges are iterative methods in reflexive and nonreflexive Banach spaces (cf., [8, 14]).

Regularization by Discretization Discretization of infinite-dimensional inverse problems needs to be understood as well as a regularization technique, and thus again convergence as discretization fineness tends to zero differs from classical aspects of numerical analysis. If the data are taken from an m-dimensional subspace and the unknown is approximated in an n-dimensional subspace, one usually ends up with a problem of the form Qm F .Pn u/ D f;

(5)

typically with Qm and Pn being projection operators. Again a semiconvergence behavior appears, where the Iterative Regularization Methods regularization parameter is related to n1 respectively and m1 . Iterative methods, which we generally write as In the case of linear inverse problems, it is well understood that the discretization leads to an illukC1 D G.uk ; F .uk /  f; ˇk /; (2) conditioned linear system, and the choice of basis functions is crucial for the conditioning. In particular yield a first instance of numerical methods for regular- Galerkin-type discretization with basis functions in ization. Simple examples are the Landweber iteration the range of the adjoint operator are efficient ways to discretize inverse problems (cf., [5, 12]). The role ukC1 D uk  ˇk F 0 .uk / .F .uk /  f / (3) of adaptivity in inverse problems has been explored

I

734

recently (cf., [1]), again with necessary modifications compared to the case of well-posed problems due to the fact that a posteriori error estimates without assumptions on the solution are impossible.

Variational Regularization Methods

Inverse Problems: Numerical Methods

(cf., e.g., [2]). In particular in the twenty-first century, nonsmooth regularization functionals such as total variation and `1 -type norms became more and more popular, since they can introduce prior knowledge more effectively. A variety of numerical optimization methods has been proposed in such cases, in particular Augmented Lagrangian methods have become popular (cf., [6]).

The most frequently used approach for the stable solution of inverse problems are variational regularization techniques, which consist in minimizing a functional Bayesian Inversion of the form (cf., [3, 5]) The use of Bayesian approaches for inverse problems E˛ .u/ D D.F .u/; f / C ˛J.u/; (6) has received growing attention in the recent years (cf., e.g., [7]) due to frequent availability of prior where J is a regularization functional and D is an knowledge as well as increasing detail in the statistical appropriate distance measure, frequently a square norm characterization of noise and other uncertainties in in a Hilbert space (related to classical least-squares inverse problems, which can be handled naturally. The basis of Bayesian inversion in a finite-dimensional methods) setup is Bayes’ formula for the posterior probability density 1 (7) D.F .u/; f / D kF .u/  f k2 : p.f ju/ p.u/ 2 : (8) p.ujf / D p.f / The role of the regularization functional from a the- Here p.f ju/ is the data likelihood, into which the oretical point of view is to enforce well posedness forward model and the noise are incorporated, and p.u/ in the minimization of E˛ , typically by enforcing respectively p.f / are a priori probability densities for compactness of sublevel sets of E˛ in an appropriate the unknown and the data, respectively. Since p.f / topology. Having in mind the Banach-Alaoglu theo- is just a scaling factor when f is fixed, it is usually rem, it is not surprisingly that the most frequent choice neglected. Most effort is used to model the prior probof regularization functionals are powers of norms in ap- ability density, which is often related to regularization propriate Banach spaces, whose boundedness implies functionals in variational methods via weak compactness. From a practical point of view, the role of the regularization functional is to introduce a p.u/  e ˛J.u/ : (9) priori knowledge by highly penalizing unexpected or unfavorable solutions. In particular in underdetermined A standard approach to compute estimates is maxicases, the minimization of J needs to determine appro- mum a posteriori probability (MAP) estimation, which priate solutions, a paradigm which is heavily used in amounts to maximize p.ujf / subject to u. By the the adjacent field of compressed sensing (cf., [4]). equivalent minimization of the negative log likelihood, Besides discretization issues as mentioned above, MAP estimation can be translated into variational rega key challenge is the construction of efficient ularization; the role of the statistical approach boils optimization methods to minimize E˛ . In the past down to selecting appropriate regularization functionsquared norms or seminorms in Hilbert spaces (e.g., als and data terms based on noise models. In order in L2 or H 1 ) have been used frequently, so that rather to quantify uncertainty, also conditional mean (CM) standard algorithms for differentiable optimization estimates Z have been used. The main challenge when using uO D u p.ujf / d u; (10) Newton-type methods is efficient solution of the arising large linear systems; several preconditioning variances, and other quantifying numbers of the approaches have been proposed, some at the interface posterior distribution are used. These are all based to optimal control and PDE-constrained optimization on integration of the posterior in very high dimensions,

Inverse Spectral Problems: 1-D, Algorithms

since clearly the infinite-dimensional limit should be approximated. The vast majority of approaches is based on Markov chain Monte Carlo (MCMC) methods (cf., e.g., [7]); see also [15] for a deterministic approach. The construction of efficient sampling schemes for posterior distribution with complicated priors is a future computational challenge of central importance. A program related to classical numerical analysis is the convergence of posteriors and different estimates as the dimension of the space for the unknown (possibly also for the data) tends to infinity, an issue that has been investigated under the keyword of discretization invariance in several instances recently (cf., [10, 11]).

735 15. Schwab, C., Stuart, A.M.: Sparse deterministic approximation of Bayesian inverse problems. Inverse Probl. to appear. (2012)

Inverse Spectral Problems: 1-D, Algorithms Paul E. Sacks Department of Mathematics, Iowa State University, Ames, IA, USA

Synonyms References 1. Benameur, H., Kaltenbacher, B.: Regularization of parameter estimation by adaptive discretization using refinement and coarsening indicators. J. Inverse Ill-Posed Probl. 10, 561–584 (2002) 2. Biegler, L., Ghattas, O., Heinkenschloss, M., van Bloemen Waanders, V. (eds.) Large-Scale PDE-Constrained Optimization, Springer, New York (2003) 3. Burger, M., Osher, S.: Convergence rates of convex variational regularization. Inverse Probl. 20, 1411–1421 (2004) 4. Donoho, D.L.: Compressed sensing. IEEE Trans. Inform. Theory 52, 1289–1306 (2006) 5. Engl, H., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Kluwer, Dordrecht (1996) 6. Goldstein, T., Osher, S.: The split Bregman method for L1regularized problems. SIAM J. Imagin. Sci. 2, 323–343 (2009) 7. Kaipio, J., Somersalo, A.: Statistical and Computational Inverse Problems. Springer, Heidelberg (2005) 8. Kaltenbacher, B., Schoepfer, F., Schuster, T.: Convergence of some iterative methods for the regularization of nonlinear ill-posed problems in Banach spaces. Inverse Probl. 25, 065003 (2009) 9. Kaltenbacher, B., Neubauer, A., Scherzer, O.: Iterative Regularization Methods for Nonlinear Ill-Posed Problems. De Gruyter, Berlin (2008) 10. Lassas, M., Saksman, S., Siltanen, S.: Discretization invariant Bayesian inversion and Besov space priors. Inverse Probl. Imaging 3, 87–122 (2009) 11. Lehtinen, M.S., P¨aiv¨arinta, L., Somersalo, E.: Linear inverse problems for generalised random variables. Inverse Probl. 5, 599–612 (1989) 12. Natterer, F.: Numerical methods in tomography. Acta Numer. 8, 107–141 (1999) 13. Natterer, F.: Imaging and inverse problems of partial differential equations. Jahresber. Dtsch. Math. Ver. 109, 31–48 (2007) 14. Osher, S., Burger, M., Goldfarb, D., Xu, J., Yin, W.: An iterative regularization method for total variation based image restoration. Multiscale Model. Simul. 4, 460–489 (2005)

Inverse eigenvalue problems; Inverse Sturm-Liouville problems; Numerical methods

Introduction In this entry we will describe techniques which have been developed for numerical solution of inverse spectral problems for differential operators in one space dimension, for which the model is the inverse SturmLiouville problem. Let V D V .x/ be a given real valued potential on the interval Œ0; 1 and consider the eigenvalue problem  00 C.V .x// D 0 0 < x < 1 .0/ D .1/ D 0 (1) As is well known, there exists an infinite sequence of real eigenvalues 1 < 2 < : : : n ! C1

(2)

We will always assume at least that V 2 L2 .0; 1/, although much of what is said below is valid in larger spaces. The inverse spectral problem of interest is to recover V .x/ from spectral data – there are many different versions of this, depending on exactly what is meant by “spectral data.” In the simplest case this would simply mean the eigenvalues, but one quickly sees that this is not enough information, unless the class of V ’s is considerably restricted. We therefore define some additional quantities. Let n .x/ be an eigenfunction corresponding to n normalized by jjn jjL2 .0;1/ D 1, and set

I

736

Inverse Spectral Problems: 1-D, Algorithms

n D

1 n0 .0/2

reduction Rto the case when the mean value of the 1 potential 0 V .s/ ds is zero. This may be done by R1 first making an estimate of 0 V .s/ ds based on the (4) asymptotic behavior of the eigenvalues, such as (3)

n D log .jn0 .1/j=jn0 .0/j/

Z 1 Also, let n denote the nth eigenvalue of (1) when the 0 V .s/ ds D lim n  .n/2 (9) boundary condition at x D 1 is replaced by  .1/ C n!1 0 H.1/ D 0 for some fixed H 2 R. The following asymptotic expressions are known. which follows from (5), and R 1 then taking into account the obvious fact that n  0 V .s/ ds is the nth eigenR1 Z 1 1 X value for the shifted potential V .x/  0 V .s/ ds. n D .n/2 C V .s/ ds C an an2 < 1 (5) 0

1 n D 2.n/2

nD1

  bn 1C n

cn n D n

1 X

1 X

Computational Methods bn2

N , which are the exact values these quantities would have when

Inverse Spectral Problems: 1-D, Algorithms

737

V D 0. The integral equation (11) may be numerically solved, for example, by a collocation method or by seeking the solution as a linear combination of suitable basis functions. Method of Overdetermined Hyperbolic Problems The next method was introduced in [14], in which the inverse spectral problem was shown to be equivalent to a certain overdetermined boundary value problem for a hyperbolic partial differential equation, which may be solved by an iteration technique. The method is easily adaptable to any of Problems 1–3, as well as many other variants – we will focus on Problem 2 for definiteness. The kernel K.x; t/ appearing in (11) is known to have a number of other interesting properties (see [4, 9]), the first of which we need is that it serves as the kernel of an integral operator which transforms a solution of  00 C  D 0

.0/ D 0

(12)

into a corresponding solution of  00 C .  V .x// D 0

.0/ D 0

(13)

Specifically, if we denote by .x; p / the solution of (13) normalized by 0 .0; / D , then .x; / D sin

Z p x C

x

K.x; t/ sin

p t dt

(14)

0

The key point here is that K does not depend on . The second property of K we will use here is that if defined for t < 0 by odd extension, it satisfies the Goursat problem Ktt  Kxx C V .x/K D 0 1 K.x; ˙x/ D ˙ 2

Z

0 < jtj < x < 1 (15)

1 0

1

Kx .1; t/ sin

p p p n t dt D n ..1/n e n cos n /

0

(18)

With spectral data n ; n known, these systems of equations can be shown to uniquely determine (since K.1; t/; Kx .1; t/ are both odd) K.1; t/ WD G0 .t/

Kx .1; t/ WD G1 .t/

V .s/ ds

0 a > rs , where r# D rs3 =rc . As any realistic source can only produce

I

756

Invisibility Cloaking

bounded power, it follows that k must go to zero as

! 0. But then V ! 0 outside D. The dipole source will become essentially cloaked: the energy flowing from it is all channeled to the annulus and virtually does not escape outside the radius r# . For this reason the annulus r# > jxj > rs is called the cloaking region. In fact any finite collection of dipole sources located at fixed positions in the cloaking region which produce bounded power must all become cloaked as

! 0 [33]. If these dipole sources are not active sources, but rather polarizable dipoles whose strength is proportional to the field acting on them, then these must also become cloaked [5, 35]. Somehow the field in D must adjust itself so that the field acting on each polarizable dipole in the collection is almost zero. It is still not exactly clear what is and is not cloaked by the annulus. Although some progress has recently been made: see, for example, [2]. Numerical evidence suggests that dielectric disks within the cloaking region are only partially cloaked [4]. Cloaking also extends to polarizable dipoles near two or three dimensional superlenses. There is also numerical evidence [24] to suggest that an object near a superlens can be cloaked at a fixed frequency if the appropriate “antiobject” is embedded in the superlens (Fig. 3).

Active Exterior Cloaking

it reduces to finding a polynomial which is approximately 0 within one disk in C and approximately 1 within a second disjoint disk [17]. To see this, let Br . /  R2 denote the disk of radius r centered at x D . ; 0/. Suppose we are given a potential V .x/ which, for simplicity, is harmonic in R2 . The desired cloaking device, located at the origin, produces a potential Vd .x/ which is harmonic in R2 n f0g with Vd .x/ almost zero outside a sufficiently large ball B .0/ so that the cloaking device is hard to detect outside the radius  . At the same time we desire that the total potential V .x/CVd .x/ (and its gradient) be almost zero in a ball B˛ .ı/  B .0/ not containing the origin, which is the cloaking region: a (non-resonant) object can be placed there with little disturbance to the surrounding fields because the field acting on it is very small. After applying the inverse transformation z D 1=.x1 Cix2 / and introducing harmonic conjugate potentials to obtain the analytic extensions v and vd of V and Vd , the problem becomes: find vd .z/ analytic in C such that vd  0 in B1= .0/ and vd  v in B˛ .ı /, where B˛ .ı / is the image of B˛ .ı/ under the inverse transformation. Since the product of two analytic functions is again analytic, this can be reformulated: find w.z/ analytic in C such that w  0 in B1= .0/ and w  1 in B˛ .ı /. To recover vd one needs to multiply w by a polynomial which approximates v in B˛ .ı /. When 1= and ˛ are small enough, one can take w.z/ to be the Hermite interpolation polynomial of degree 2n1 1 satisfying

Active cloaking has the advantage of being broadband, but may require advance knowledge of the probing fields. Miller [28] found that active controls rather than passive materials could be used to achieve interior cloaking. Active exterior cloaking is easiest to see in the context of two-dimensional electrostatics, where

w.0/ D 0;

wj .0/ D wj .ı / D 0

4

4

2

2

0

0

−2

−2

−4

w.ı / D 1; for j D 1; 2; : : : ; n  1 (11)

−4 −5 −2.5

0

2.5

5

7.5

10

Invisibility Cloaking, Fig. 3 Left: Equipotentials for the real part of the potential with one fixed dipole source on the right and a neighboring polarizable dipole on the left, outside the cylinder with "s D 1 C 1012 i . Right: The equipotentials when the

−5

− 2.5

0

2.5

5

7.5

10

cylinder is moved to the right so it cloaks the polarizable dipole, leaving the exterior field close to that of the fixed dipole in free space (Taken from [33])

Invisibility Cloaking Invisibility Cloaking, Fig. 4 Left: Scattering of waves by a kite-shaped object, with the three active cloaking devices turned off. Right: The wave pattern with the devices turned on, showing almost no scattering

757 20

20

0

0

−20 −20

where wj .z/ is the j th derivative of w.z/. As n ! 1, one can show that w.z/ converges to 0 (and to 1) in the side of the figure eight jz2  ı zj < ı2 =4 containing the origin (and containing ı , respectively). Outside this figure eight, and excluding the boundary, w.z/ diverges to infinity. In practice the cloaking device cannot be a point, but should rather be an extended device encompassing the origin, and the device should produce the required potential Vd . Choosing the boundary of this device to be where Vd is not too large forces the device to partially wrap around the cloaking region, leaving a “throat” connecting the cloaking region to the outside. The width of the throat goes to zero as n ! 1, but it appears to go to zero slowly. Thus one can get good cloaking with throat sizes that are not too small. This active exterior cloaking extends to the Helmholtz equation (see Fig. 4) and in that context works over a broad range of frequencies [18]. Numerical results show that an object can be effectively cloaked from an incoming pulse with a device having throats that are reasonably large.

References 1. Alu, A., Engheta, N.: Achieving transparency with plasmonic and metamaterial coatings. Phys. Rev. E 72, 016623 (2005) 2. Ammari, H., Ciraolo, G., Kang, H., Lee, H., Milton, G.W.: Spectral theory of a Neumann–Poincar´e-type operator and analysis of cloaking due to anomalous localized resonance. Arch. Ration. Mech. Anal. 208(2), 667–692 (2013) 3. Benveniste, Y., Miloh, T.: Neutral inhomogeneities in conduction phenomenon. J. Mech. Phys. Solids 47, 1873 (1999) 4. Bruno, O.P., Lintner, S.: Superlens-cloaking of small dielectric bodies in the quasistatic regime. J. Appl. Phys. 102, 124502 (2007)

0

20

−20 −20

0

20

5. Bouchitt´e, G., Schweizer, B.: Cloaking of small objects by anomalous localized resonance. Q. J. Mech. Appl. Math. 63, 437–463 (2010) 6. Cai, W., Chettiar, U., Kildishev, A., Milton, G., Shalaev, V.: Non-magnetic cloak with minimized scattering. Appl. Phys. Lett. 91, 111105 (2007) 7. Calder´on, A.P.: On an inverse boundary value problem. Seminar on Numerical Analysis and its Applications to Continuum Physics (Rio de Janeiro, 1980), pp. 65–73, Soc. Brasil. Mat., R´ıo de Janeiro (1980) 8. Chen, H., Chan, C.T.: Acoustic cloaking in three dimensions using acoustic metamaterials. Appl. Phys. Lett. 91, 183518 (2007) 9. Dolin, L.S.: To the possibility of comparison of threedimensional electromagnetic systems with nonuniform anisotropic filling. Izv. Vyssh. Uchebn. Zaved. Radiofizika 4(5), 964–967 (1961) 10. Eleftheriades, G., Balmain, K. (eds.): Negative-Refraction Metamaterials. IEEE/Wiley, Hoboken (2005) 11. Greenleaf, A., Kurylev, Y., Lassas, M., Uhlmann, G.: Fullwave invisibility of active devices at all frequencies. Commun. Math. Phys. 275, 749–789 (2007) 12. Greenleaf, A., Kurylev, Y., Lassas, M., Uhlmann, G.: Electromagnetic wormholes and virtual magnetic monopoles from metamaterials. Phys. Rev. Lett. 99, 183901 (2007) 13. Greenleaf, A., Kurylev, Y., Lassas, M., Uhlmann, G.: Electromagnetic wormholes via handlebody constructions. Commun. Math. Phys. 281, 369–385 (2008) 14. Greenleaf, A., Kurylev, Y., Lassas, M., Uhlmann, G.: Approximate quantum and acoustic cloaking. J. Spectr. Theory 1, 27–80 (2011). doi:10.4171/JST/2. arXiv:0812.1706v1 15. Greenleaf, A., Lassas, M., Uhlmann, G.: Anisotropic conductivities that cannot detected in EIT. Physiolog. Meas. (special issue on Impedance Tomography) 24, 413–420 (2003) 16. Greenleaf, A., Lassas, M., Uhlmann, G.: On nonuniqueness for Calder´on’s inverse problem. Math. Res. Lett. 10(5–6), 685–693 (2003) 17. Guevara Vasquez, F., Milton, G.W., Onofrei, D.: Active exterior cloaking. Phys. Rev. Lett 103, 073901 (2009) 18. Guevara Vasquez, F., Milton, G.W., Onofrei, D.: Broadband exterior cloaking. Opt. Express 17, 14800–14805 (2009)

I

758 19. Kerker, M.: Invisible bodies. J. Opt. Soc. Am. 65, 376–379 (1975) 20. Kohn, R., Shen, H., Vogelius, M., Weinstein, M.: Cloaking via change of variables in electrical impedance tomography. Inver. Prob. 24, 015016 (2008) 21. Kohn, R., Onofrei, D., Vogelius, M., Weinstein, M.: Cloaking via change of variables for the Helmholtz Equation. Commun. Pure Appl. Math. 63, 1525–1531 (2010) 22. Kohn, R., Vogelius, M.: Identification of an unknown conductivity by means of measurements at the boundary. In: McLaughlin, D. (ed.) Inverse Problems. SIAM-AMS Proceedings vol. 14, pp. 113–123. American Mathematical Society, Providence (1984). ISBN 0-8218-1334-X 23. Lee, J., Uhlmann, G.: Determining anisotropic real-analytic conductivities by boundary measurements. Commun. Pure Appl. Math. 42, 1097–1112 (1989) 24. Lai, Y., Chen, H., Zhang, Z.-Q., Chan, C.T.: Complementary media invisibility cloak that cloaks objects at a distance outside the cloaking shell. Phys. Rev. Lett. 102, 093901 (2009) 25. Leonhardt, U.: Optical conformal mapping. Science 312, 1777–1780 (2006) 26. Leonhardt, U., Philbin, T.: General relativity in electrical engineering. New J. Phys. 8, 247 (2006) 27. Leonhardt, U., Tyc, T.: Broadband invisibility by noneuclidean cloaking. Science 323, 110–112 (2009) 28. Miller, D.A.B.: On perfect cloaking. Opt. Express 14, 12457–12466 (2006) 29. Milton, G.: The Theory of Composites. Cambridge University Press, Cambridge/New York (2002) 30. Milton, G.: New metamaterials with macroscopic behavior outside that of continuum elastodynamics. New J. Phys. 9, 359 (2007)

Invisibility Cloaking 31. Milton, G., Briane, M., Willis, J.: On cloaking for elasticity and physical equations with a transformation invariant form. New J. Phys. 8, 248 (2006) 32. Milton, G.W., Nicorovici, N.-A.P., McPhedran, R.C., Podolskiy, V.A.: A proof of superlensing in the quasistatic regime, and limitations of superlenses in this regime due to anomalous localized resonance. Proc. R. Soc. A 461, 3999–4034 (2005) 33. Milton, G., Nicorovici, N.-A.: On the cloaking effects associated with anomalous localized resonance. Proc. R. Soc. A 462, 3027–3059 (2006) 34. Nicorovici, N.-A.P., McPhedran, R.C., Milton, G.W.: Optical and dielectric properties of partially resonant composites. Phys. Rev. B 49, 8479–8482 (1994) 35. Nicorovici, N.-A.P., Milton, G.W., McPhedran, R.C., Botten, L.C.: Quasistatic cloaking of two-dimensional polarizable discrete systems by anomalous resonance. Opt. Express 15, 6314–6323 (2007) 36. Pendry, J.B.: Negative refraction makes a perfect lens. Phys. Rev. Lett. 85, 3966–3969 (2000) 37. Pendry, J.B., Schurig, D., Smith, D.R.: Controlling electromagnetic fields. Science 312, 1780–1782 (2006) 38. Pendry, J.B., Schurig, D., Smith, D.R.: Calculation of material properties and ray tracing in transformation media. Opt. Express 14, 9794 (2006) 39. Schurig, D., Mock, J., Justice, B., Cummer, S., Pendry, J., Starr, A., Smith, D.: Metamaterial electromagnetic cloak at microwave frequencies. Science 314, 977–980 (2006) 40. Sylvester, J., Uhlmann, G.: A global uniqueness theorem for an inverse boundary value problem. Ann. Math. 125, 153–169 (1987) 41. Ward, A., Pendry, J.: Refraction and geometry in Maxwell’s equations. J. Modern Opt. 43, 773–793 (1996)

K

Kinetic Equations: Computation Lorenzo Pareschi Department of Mathematics, University of Ferrara, Ferrara, Italy

Mathematics Subject Classification 65D32; 65M70; 65L04; 68Q25; 82C40

Synonyms Boltzmann equations; Collisional equations; Transport equations

Short Definition Kinetic equations bridge the gap between a microscopic description and a macroscopic description of the physical reality. Due to the high dimensionality, the construction of numerical methods represents a challenge and requires a careful balance between accuracy and computational complexity.

Description Kinetic Equations Particle systems can be described at the microscopic level by systems of differential equations describing the individual motions of the particles. However, they

are extremely costly from a numerical point of view and bring little intuition on how a large particle system behaves. Therefore, one is led to seek reduced descriptions of particle systems which still preserve an accurate description of the physical phenomena. Kinetic models intend to describe particle systems by means of a distribution function f .x; v; t/. This object represents a number density in phase space, i.e., f dx dv is the number of particles in a small volume dx dv in position-velocity space about the point .x; v/ of this space. In this short entry, we will focus on computational methods for the interacting particle case described by the Boltzmann equation. This is motivated by its relevance for applications and by the fact that it contains all major difficulties present in other kinetic equations. From a numerical perspective, most of the difficulties are due to the multidimensional structure of the distribution function. In particular the approximation of the collisional integral is a real challenge for numerical methods, since the integration runs on a highlydimensional manifold and is at the basis of the macroscopic properties of the equation. Further difficulties are represented by the presence of fluid-kinetic interfaces and multiple scales where most numerical methods loose their efficiency because they are forced to operate on a very short time scale. Although here we review briefly only deterministic numerical methods, let us mention that several realistic numerical simulations are based on MonteCarlo techniques [1, 13, 19]. In the next paragraphs, we summarize the main ideas at the basis of two of the most popular way to approximate the distribution function in the velocity space, namely, the discretevelocity method [3, 4, 15, 20] and the spectral method

© Springer-Verlag Berlin Heidelberg 2015 B. Engquist (ed.), Encyclopedia of Applied and Computational Mathematics, DOI 10.1007/978-3-540-70529-1

760

Kinetic Equations: Computation

[2, 9, 11, 12, 16–18]. Finally, we shortly introduce the basic principles for the construction of schemes which are robust in fluid regions [6–8, 10].

Z D T D

v2Rd

1 d

f .v/dv;

Z

uD

1 

Z v2Rd

vf .v/dv;

ju  vj2 f .v/dv:

(5) Boltzmann Equation Taking into account only binary interactions, the be- Discrete-Velocity Methods havior of a dilute gas of particles is described by the Historically this was the first method for discretizing Boltzmann equation [5, 21] the Boltzmann equation in velocity space. The discretization is built starting from physical rather than @f C v  rx f D Q.f; f / (1) numerical considerations. We assume the gas particles @t can attain only a finite set of velocities where f .t; x; v/, x; v 2 Rd (d  2) is the timedependent particle distribution function in the phase space and the collision operator Q is defined by Z

Z Q.f; f /.v/ D 

v 2Rd

 2Sd 1

B.cos ; jv  v j/

 f0 f 0  f f d dv :

(2)

v2Rd

VN D fv1 ; v2 ; v3 ; : : : ; vN g;

vi 2 Rd

and denote by fj .x; t/ D f .vj ; x; t/, j D 1; : : : ; N . The collision pair .vi ; vj / $ .vk ; vl / is admissible if vi ; vj ; vk ; vl 2 VN and preserves momentum and energy:

jvi j2 C jvj j2 D jvk j2 C jvl j2 : vi C vj D vk C vl ; Time and position act only as parameters in Q and therefore will be omitted in its description. In (2) we The set of admissible output pairs .vk ; vl / correspond0 used the shorthands f D f .v/, f D f .v /, f D ing to a given input pair .vi ; vj / will be denoted by 0 0 f .v 0 /, and f D f .v /. The velocities of the colliding C ij . pairs .v; v / and .v 0 ; v0 / are related by The discrete collision operator is obtained as a quadrature formula based on the weights aijkl related jv  v j v C v to the collision .vi ; vj / $ .vk ; vl / which must satisfy C ; v0 D 2 2 the relations v C v  jv  v j v0 D  : N 2 2 X aijkl  0; aijkl D 1; 8i; j D 1; : : : ; N : The collision kernel B is a nonnegative function which k;lD1 only depends on jv  v j and cos  D ..v  v /=jv  v j/  . Boltzmann’s collision operator has the funda- Next, we introduce the transition rates Akl ij D S jvi  mental properties of conserving mass, momentum, and vj jakl , where S is the cross-sectional area of particles, ij energy: and write the discrete Boltzmann equation as Z @fi Q.f; f / .v/ dv D 0: .v/ D 1; v; jvj2 C vi  rx fi D Qi .f; f /; v2Rd @t (3) Moreover, any equilibrium distribution function M such that Q.M; M / D 0 has the form of a locally Maxwellian distribution M.; u; T /.v/ D

   ju  vj2 ; (4) exp  .2T /d=2 2T

with Qi .f; f / D

N X

Akl ij .fk fl  fi fj /:

j;k;lD1 k;l2Cij

The discretized Boltzmann equation has the nice propwhere ; u; T are the density, mean velocity, and erty of preserving the essential physical features (conservations, H-theorem, equilibrium states). However, temperature of the gas:

Kinetic Equations: Computation

761

from a computational point of view the discrete and Boltzmann equation presents two main drawbacks.   First, the computational cost is larger than O.N 2 /, and x  .x C y/ d 1 Q B  ; jx C yj B.x; y/ D 2 second the accuracy is rather poor, typically less than jxjjx C yj first-order (see [15] for example). .d 2/ : jx C yj Spectral Methods Spectral methods have been constructed recently with the goal to compensate the drawbacks of discretevelocity approximation. For the sake of simplicity, we summarize their derivation in the case of the space homogeneous Boltzmann equations, although the schemes can be effectively used to compute the collision integral in a general setting. Related approaches have been presented in [2, 11]. The approximate function fN is represented as the truncated Fourier series:

fN .v/ D

N X

fOk e i kv ; fOk D

kDN

1 .2/d

Z

f .v/e i kv dv:

D

As shown in [12] when B satisfies the decoupling Q assumption B.x; y/ D a.jxj/ b.jyj/, it is possible to O m/ by a sum approximate each ˇ.l;

ˇ.l; m/ '

A X

˛p .l/˛p0 .m/:

(7)

pD1

This gives a sum of A discrete convolutions, with A  N , and by standard FFT techniques a computational cost of O.A N d log2 N /. Denoting by N D .2N C 1/d the total number of grid points, this is equivalent to O.A N log2 N / instead of O.N 2 /. Moreover, one gets the following consistency result of spectral accuracy [12]

k The spectral equation is the projection of the collision Theorem 1 For all k > d  1 such that f 2 Hp integral QR .f; f /, truncated over the ball of radius R centered in the origin, in PN , the .2N C 1/d Rk kfN k2H k p R R;M dimensional vector space of trigonometric polynomials kQ .f; f /PN Q .fN ; fN /kL2  C1 Mk of degree at most N , i.e.,  C2  C k kf kHpk C kQR .fN ; fN /kHpk : N @fN D PN QR .fN ; fN / @t Asymptotic-Preserving Methods Let us now consider the time discretization of the where PN denotes the orthogonal projection on PN in scaled Boltzmann equation L2 .D /. A straightforward computation leads to the following set of ordinary differential equations: @f 1 C v  rx f D Q.f; f / (8) @t " N X d fOk .t/ O m/ fOl fOm ; k D N; : : : ; N ˇ.l; D where " > 0 is the Knudsen number. For small dt l;mDN value of ", we have a stiff problem, and standard time lCmDk discretization methods are forced to operate on a very (6) O m/ are the kernel modes, given by small time scale. On the other hand, in such regime where ˇ.l; formally Q.f; f /  0 and the distribution function O ˇ.l; m/ D ˇ.l; m/  ˇ.m; m/ with is close to a local Maxwellian. Thus, the moments of the Boltzmann equation are well-approximated by the Z Z solution to the Euler equations of fluid-dynamics Q ˇ.l; m/ D B.x; y/ ı.x  y/ x2BR

e

i lx

e

y2BR

i my

dx dy;

@t u C rx  F .u/ D 0;

(9)

K

762

Kinetic Equations: Computation

with

then essential ingredients of computational methods for kinetic equations.

u D .; w; E/T ;

F .u/ D .w; %w ˝ .w C pI /;

Ew C pw/ ; p D T; T

where I is the identity matrix and ˝ denotes the tensor product. We say that a time discretization method for (8) of stepsize t is asymptotic preserving (AP) if, independently of the stepsize t, in the limit " ! 0 becomes a consistent time discretization method for the reduced system (9). When "  1 the problem is stiff, and we must resort on implicit integrator to avoid small time step restriction. This however requires the inversion of the collision integral Q.f; f / which is prohibitively expensive from the computational viewpoint. On the other hand, when f  M Œf  we know that the collision operator Q.f; f / is well-approximated by its linear counterpart Q.M; f / or by a simple relaxation operator .M  f /. If we denote by L.f / the selected linear operator, we can rewrite the equation introducing a penalization term as 1 1 @f C v  rx f D .Q.f; f /  L.f // C L.f /: @t " " The idea now is to be implicit (or exact) in the linear part L.f / and explicit in the deviations from equilibrium Q.f; f /  L.f /. This approach has been successfully introduced in [7, 8] using implicit-explicit integrators and in [6,10] by means of exponential techniques. We refer also to [14] for analogous techniques.

Conclusions Computational methods for kinetic equations represent an emerging field in scientific computing. This is testified by the large amount of scientific papers which has been produced on the subject in recent years. We do not seek to review all of them here and focused our attention to the challenging case of the Boltzmann equation of rarefied gas dynamic. The major difficulties in this case are represented by the discretization of the multidimensional integral describing the collision process and by the presence of multiple time scales. Fast algorithms and robust stiff solvers are

References 1. Bird, G.: Molecular Gas Dynamics and Direct Simulation of Gas Flows. Clarendon, Oxford (1994) 2. Bobylev, A., Rjasanow, S.: Difference scheme for the Boltzmann equation based on the Fast Fourier Transform. Eur. J. Mech. B 16, 293–306 (1997) 3. Bobylev, A., Palczewski, A., Schneider, J.: On approximation of the Boltzmann equation by discrete velocity models. C. R. Acad. Sci. Paris S´er. I. Math. 320, 639–644 (1995) 4. Buet, C.: A discrete velocity scheme for the Boltzmann operator of rarefied gas dynamics. Transp. Theory Stat. Phys. 25, 33–60 (1996) 5. Cercignani, C., Illner, R., Pulvirenti, M.: The mathematical theory of dilute gases. Appl. Math. Sci. 106 (1994) 6. Dimarco, G., Pareschi, L.: Exponential Runge-Kutta methods for stiff kinetic equations. SIAM. J. Num. Anal. 49, 2057–2077 (2011) 7. Dimarco G., Pareschi, L.: Asymptotic preserving Implicit Explicit Runge-Kutta methods for nonlinear kinetic equations. SIAM J. Num. Anal. 51, 1064–1087 (2013) 8. Filbet, F., Jin, S.: A class of asymptotic-preserving schemes for kinetic equations and related problems with stiff sources. J. Comput. Phys. 229, 7625–7648 (2010) 9. Filbet, F., Mouhot, C.: Analysis of spectral methods for the homogeneous Boltzmann equation. Trans. Am. Math. Soc. 363, 1947–1980 (2011) 10. Gabetta, E., Pareschi, L., Toscani, G.: Relaxation schemes for nonlinear kinetic equations. SIAM. J. Numer. Anal. 34, 2168–2194 (1997) 11. Gamba, I., Tharkabhushaman, S.: Spectral - Lagrangian based methods applied to computation of Non-Equilibrium Statistical States. J. Comp. Phys. 228, 2012–2036 (2009) 12. Mouhot, C., Pareschi, L.: Fast algorithms for computing the Boltzmann collision operator. Math. Comput. 75(256), 1833–1852 (2006) (electronic) 13. Nanbu, K.: Direct simulation scheme derived from the Boltzmann equation I. Monocomponent gases. J. Phys. Soc. Jpn 49, 2042–2049 (1980) 14. Lemou, M., Mieussens, L.: A new asymptotic preserving scheme based on micro-macro formulation for linear kinetic equations in the diffusion limit. SIAM J. Sci. Comput. 31, 334–368 (2008) 15. Panferov, V., Heintz, A.: A new consistent discrete-velocity model for the Boltzmann equation. Math. Methods Appl. Sci. 25, 571–593 (2002) 16. Pareschi, L., Perthame, B.: A spectral method for the homogeneous Boltzmann equation. Transp. Theory Stat. Phys. 25, 369–383 (1996) 17. Pareschi, L., Russo, G.: Numerical solution of the Boltzmann equation I. Spectrally accurate approximation of the collision operator. SIAM. J. Numer. Anal. 37, 1217–1245 (2000) 18. Pareschi, L., Toscani, G., Villani, C.: Spectral methods for the non cut-off Boltzmann equation and numerical grazing collision limit. Numer. Math. 93, 527–548 (2003)

Korteweg-de Vries Equation 19. Rjasanow, S., Wagner, W.: A stochastic weighted particle method for the Boltzmann equation. J. Comput. Phys. 124, 243–253 (1996) 20. Rogier, F., Schneider, J.: A direct method for solving the Boltzmann equation. Transp. Theory Stat. Phys. 23, 313–338 (1994) 21. Villani, C.: A Survey of mathematical topics in kinetic theory. In: Friedlander, S., Serre, D. (eds.) Handbook of Fluid Mechanics. Elsevier, Amsterdam (2002)

Korteweg-de Vries Equation Alper Korkmaz Department of Mathematics, C ¸ ankiri Karatekin University, C ¸ ankiri, Turkey

763

properties, these waves preserve their original sizes, shapes, and velocities after interacting with other solitary waves; therefore, they are named as solitons. In contrast with soliton solutions of Schr¨odinger equation, the velocity of the KdV equation depends on the magnitude of the soliton. Since the KdV equation is an integrable Hamiltonian system, it has infinitely many conserved quantities. Miura [6] showed infinitely many conserved quantities for the KdV equation by discovering a special transformation mapping solutions of one equation to solutions of a second one. This transformation was generalized by Gardner et al. [7]. The lowest four conserved quantities for the KdV equation (1) are the following: Z1 C1 D

Synonyms

Udx

(2)

U 2 dx

(3)

1

Z1

KdV equation; Korteweg-de Vries equation; Soliton

C2 D 1

In the second quarter of the nineteenth century, Z1  ˇ 2 3 J.S. Russell observed the motion of a solitary wave C3 D U  3 Ux dx (4) ˛ preserving its shape, magnitude, and velocity for 1 kilometers along a channel. Following the publication #   Z1 " of his findings [1], a scientific discussion on the ˇ 36 ˇ 2 4 2 2 .Uxx / dx U  12 U.Ux / C existence of such a conserved wave traveling long C4 D ˛ 5 ˛ 1 distances started. Even though Airy [2], one of the de(5) velopers of linear wave theory, claimed that the energy concentrated in the middle of the wave will deform Analytic single soliton solution of magnitude 3v: and destroy the solitary during its propagation, Boussi r r nesq [3] and Rayleigh [4] derived approximations to 1 ˛v ˛v 2 1 x ˛vt C x0 / nonlinearly modeled solitary waves together with some U.x; t/ D 3v sech . 2 ˇ 2 ˇ perturbation analysis of a nonlinear model. A special type of nonlinear solitary waves containing quadratic propagates to the right at a velocity ˛v: Interaction of nonlinear term and cubic dispersion term is named as two soliton solution for the KdV equation (1): the Korteweg-de Vries (KdV) equation of the form: U.x; t/ D 12ˇ.log /xx (6) @u.x; t/ @3 u.x; t/ @u.x; t/ C ˛u.x; t/ Cˇ D 0 (1) where @t @x @x 3 in which ˛ and ˇ are positive real parameters and x and t denote space and time variables, respectively. It was first introduced by Korteweg and de Vries [5]. The KdV equation plays a very prominent role in the study of nonlinear dispersive waves. The balance between the nonlinear and dispersive terms enables solitary wave solutions. Due to their particle-like

D 1 C e ı1 C e ı2 C e ı1 Cı2 ıi D i x  i3 t C &i ; i D 1; 2   r ci 1  2 2 ; i D 1; 2; 1 D  0:481 ; i D

D 1 C 2 ˇ 2 D 1:072

K

764

Korteweg-de Vries Equation

Here ci ; i D 1; 2 denote the magnitudes of initially well-separated two solitons. The split of an arbitrary function into solitons given by [8], as:    jxj  25 U.x; 0/ D 0:5 1  tanh 5 The triple soliton splitting case has the initial condition [9]: ! x1 2 2 U.x; 0/ D sech p 3 108ˇ as the Maxwellian initial condition [10] U.x; 0/ D exp.x2 /

(7)

generates waves from single solitary wave. A well-known behavior of KdV equation with the Maxwellian initial condition cited above depends on whether ˇ < ˇc or ˇ > ˇc , where ˇc is critical parameter [11]. Berezin and Karpman [12] proved that the critical value for the Maxwellian IC used in some simulations is ˇc D 0:0625: The KdV equation is a model for many physical phenomena such as ion acoustic waves, long waves in shallow water, bubble-liquid mixtures, wave phenomena in enharmonic crystals, and geophysical fluid dynamics. Moreover, the KdV equation charms numerical analysts as it has an analytical solution. So far, many schemes and algorithms have been developed to simulate the solutions of the KdV equation of which differential quadrature methods [13], finite elements [8], radial basis functions method [14], TaylorGalerkin method [15], and Chebyshev spectral method [16] can be listed as some.

References 1. Russell, J.S.: Report on Waves. Report of the 14th Meeting of the British Association for the Advancement of Science. John Murray, London, pp. 311–390 (1844)

2. Airy, G.B.: Tides and waves. Encyc. Metrop. Art. 192, 241–396 (1845) 3. Boussinesq, J.V.: Essai sur la theorie des eaux courantes, Memoires presentes par divers savants ‘l’. Acad. des Sci. Inst. Nat. France. XXIII 23, 1–680 (1877) 4. Rayleigh, L.: On waves. Phil. Mag. S.5, 1257–1279 (1876) 5. Korteweg, D.J., deVries, G.: On the change of form of long waves advancing in a rectangular canal, and on a new type of long stationary waves. Philos. Mag. 39, 422–443 (1895) 6. Miura, R.M.: Korteweg-de Vries equation and generalizations. I. A remarkable explicit nonlinear transformation. J. Math. Phys. 9, 1202–1204 (1968) 7. Gardner, C.S., Greene, J.M., Kruskal, M.D., Miura, R.M.: Korteweg-de Vries equation and generalizations. IV. Methods for exact solution. Commun. Pure. Appl. Math. 27, 97–133 (1974) 8. Gardner, L.R.T., Gardner, G.A., Ali, A.H.A.: A finite element solution for the Korteweg-de Vries equation using cubic B-splines, U. C. N. W. Maths. Preprint 89.01, (1989) 9. Debussche, A., Printems, J.: Numerical simulation of the stochastic Korteweg-de Vries equation. Phys. D 134, 200–226 (1999) 10. Gardner, L.R.T., Gardner, G.A., Ali, A.H.A.: Simulations of solutions using quadratic spline finite elements. Comput. Methods Appl. Mech. Eng. 92, 231 (1991) 11. Jeffrey, A., Kakutani, T.: Weak non-linear dispersive waves: a discussion centered around the KdV equation. SIAM Rev. 14, 522 (1972) 12. Berezin, Y.A., Karpman, V.A.: Nonlinear evolution of distubances in plasmas abd other dispersive media. Sov. Phys. JETP 24, 1049 (1967) 13. Korkmaz, A.: Numerical algorithms for solutions of Korteweg-de Vries equation. Numer. Methods Partial Differ. Equ. 26(6), 1504–1521 (2010) 14. Dag, ˙I., Dereli, Y.: Numerical solutions of KdV equation using radial basis functions. Appl. Math. Model. 32(4), 535–546 (2008) 15. Canivar, A., Sari, M., Dag, ˙I.: A Taylor-Galerkin finite element method for the KdV equation using cubic Bsplines. Phys. B Condens. Matter 405(16), 3376–3383 (2010) 16. Helal, M.A.: A Chebyshev spectral method for solving Korteweg-de Vries equation with hydrodynamical application. Chaos Solitons Fract. 12(5), 943–950 (2001)

L

Large-Scale Computing for Molecular Dynamics Simulation Aiichiro Nakano, Rajiv K. Kalia, Ken-ichi Nomura, and Priya Vashishta Department of Computer Science, Department of Physics and Astronomy, and Department of Chemical Engineering and Materials Science, University of Southern California, Los Angeles, CA, USA

Mathematics Subject Classification 65Y05; 70F10; 81V55

Synonyms High performance computing for atomistic simulation

To understand atomistic mechanisms that govern macroscopic material behavior, large-scale molecular dynamics (MD) simulations [1] involving multibillion atoms are performed on parallel supercomputers consisting of over 105 processors [2]. In addition, special-purpose computers are built to enable longtime MD simulations extending millisecond time scales (or 1012 time steps using a time discretization unit of 1015 s) [3] (for extending the time scale, see also  Transition Pathways, Rare Events and Related Questions). Key enabling technologies for such large spatiotemporal-scale MD simulations are efficient algorithms to reduce the computational complexity and parallel-computing techniques to map these algorithms onto parallel computers.

Linear-Scaling Molecular-Dynamics Simulation Algorithms

The MD approach (see also  Applications to Real Size Biological Systems) follows the time evolution of the positions, rN = fri ji = 1,. . . ,N g, of N atoms by Large-scale computing for molecular dynamics sim- solving coupled ordinary differential equations [1]: ulation combines advanced computing hardware and efficient algorithms for atomistic simulation to study d2 @  N material properties and processes encompassing large mi 2 ri D  (1) E r ; dt @ri spatiotemporal scales.

Short Definition

where t is the time, and ri and mi are the position and mass of the i -th atom, respectively. Atomic force law is mathematically encoded in the interatomic Material properties and processes are often dictated potential energy E(rN /, and key to large-scale MD by complex dynamics of a large number of atoms. simulations is, foremost, linear-scaling algorithms that

Description

© Springer-Verlag Berlin Heidelberg 2015 B. Engquist (ed.), Encyclopedia of Applied and Computational Mathematics, DOI 10.1007/978-3-540-70529-1

766

Large-Scale Computing for Molecular Dynamics Simulation

Large-Scale Computing for Molecular Dynamics Simulation, Fig. 1 Schematic of an embedded divide-and-conquer algorithm [2]. (Left) The physical space is subdivided into spatially localized cells, with local atoms constituting subproblems (bottom), which are embedded in a global field (shaded) solved with a tree-based algorithm. (Right) To solve the subproblem in domain ˛ in the divide-and-conquer density functional

theory algorithm, coarse multigrids (gray) are used to accelerate iterative solutions on the original real-space grid (corresponding to the grid refinement level, l = 3). The bottom panel shows fine grids adaptively generated near the atoms (spheres) to accurately operate the ionic pseudopotentials on the electronic wave functions

compute E(rN / in O.N / time. This algorithmic and mathematical challenge is often addressed based on data-locality principles. An example is embedded divide-and-conquer (EDC) algorithms, in which the physical system is divided into spatially localized computational cells and these cells are embedded in a global mean field that is computed efficiently with tree-based algorithms (Fig. 1) [2]. There exist a hierarchy of MD simulation methods with varying accuracy and computational complexity. In classical MD simulation, E(rN / is often an analytic function E MD (frij g,frij k g,frij kl g) of atomic pair, rij , triplet, rij k , and quadruplet, rij kl , positions, where the hardest computation is the evaluation of the long-range electrostatic interaction between all atomic pairs. The fast multipole method (FMM) algorithm reduces the O.N 2 / computational complexity of the resulting N -body problem to O.N / [4]. In the FMM, the physical system is recursively divided into subsystems to form an octree data structure, and the electrostatic field is computed recursively on the octree with O.N / operations, while maintaining spatial locality at each recursion level. In addition to computing the electrostatic potential and forces, the FMM can be used to compute atomistic stress tensor components based on a complex charge method [5]. Furthermore, a space-time multiresolution MD approach [2] utilizes temporal locality through multiple time stepping, which uses different force-update schedules for different force components [6, 7]. Specifically, forces

from neighbor atoms are computed at every MD step, whereas forces from farther atoms are updated less frequently. To simulate the breakage and formation of chemical bonds with moderate computational costs, various reactive molecular dynamics (RMD) simulation methods have been developed [2]. In RMD, the interatomic potential energy E RMD (rN ,fqi g,fBij g) typically depends on the atomic charges fqi j i = 1,. . . ,N g and the chemical bond orders B ij between atomic pairs (i , j /, which change dynamically adapting to the local environment to describe chemical reactions. To describe charge transfer, RMD uses a charge equilibration scheme, in which atomic charges are determined at every MD step to minimize the electrostatic energy with the charge-neutrality constraint. This variable N -charge problem amounts to solving a dense linear system of equations, which requires O.N 3 / operations. A fast RMD algorithm uses FMM to perform the required matrix-vector multiplications with O.N / operations [2]. It further utilizes the temporal locality of the solutions to reduce the amortized computational cost averaged over simulation steps to O.N /. To accelerate the convergence, a multilevel preconditioned conjugate-gradient (MPCG) method splits the Coulomb-interaction matrix into short- and long-range parts and uses the sparse short-range matrix as a preconditioner [8]. The extensive use of the sparse preconditioner enhances the data locality and thereby improves the computational efficiency.

Large-Scale Computing for Molecular Dynamics Simulation

In quantum molecular dynamics (QMD) simulation, the interatomic potential energy is computed quantum mechanically [9]. One approach to approximately solve the resulting exponentially complex quantum N -body problem is density functional theory (DFT, see  Density Functional Theory), which reduces the complexity to O.N 3 / by solving M one-electron problems self-consistently instead of one M -electron problem (the number of electrons M is on the order of N /. The DFT problem can be formulated as a minimization of the energy functional E QMD (rN , M / with respect to electronic wave functions (or KohnSham orbitals), M (r) = f n (r) j n = 1,. . . ,M g subject to orthonormality constraints (see  Fast Methods for Large Eigenvalues Problems for Chemistry and  Numerical Analysis of Eigenproblems for Electronic Structure Calculations). Various linear-scaling DFT algorithms have been proposed [10,11] based on a data locality principle called quantum nearsightedness [12] (see  Linear Scaling Methods). Among them, divideand-conquer density functional theory (DC-DFT) [13] is highly scalable beyond 105 processors [2]. In the DC-DFT algorithm, the physical space is a union of overlapping domains,  D †˛ ˛ (Fig. 1), and physical properties are computed as linear combinations of domain properties that in turn are computed from local electronic wave functions. For DFT calculation within each domain, one implementation uses a real-space approach based on adaptive multigrids [2] (see  Finite Difference Methods). Similar datalocality and divide-and-conquer concepts have been applied to design O.N / algorithms for high-accuracy QM methods [14], including the fragment molecular orbital method [15]. A major advantage of the EDC simulation algorithms is the ease of codifying error management. The EDC algorithms often have a well-defined set of localization parameters, with which the computational cost and the accuracy are controlled. For example, the total energy computed with the DC-DFT algorithm converges rapidly as a function of its localization parameter (i.e., the depth of the buffer layer to augment each domain for avoiding artificial boundary effects). The DC-DFT-based QMD algorithm has also overcome the energy drift problem, which plagues most O.N / DFT-based QMD algorithms, especially with large basis sets (>104 unknowns per electron, necessary for the transferability of accuracy) [2].

767

Scalable Parallel Computing To perform large-scale MD simulations, it is necessary to decompose the computation in the O.N / MD algorithms to subtasks and map them onto parallel computers [1]. A parallel computer in general consists of a number of compute nodes interconnected via a communication network [16]. Within each node, multicore processors, each consisting of simpler processors called cores, share common memory [17]. There are several schemes for mapping MD algorithms onto parallel computers [1]. For large granularity (i.e., the number of atoms per processor, N /P > 102 /, spatial decomposition is optimal, where each processor is assigned a spatial subsystem and is responsible for the computation of the forces on the atoms within its spatial subsystem. For finer granularity (N=P  1), on the other hand, force decomposition (i.e., force computations are divided among processors) and other hybrid decomposition schemes become more efficient [18–20]. Parallelization schemes also include loadbalancing capability [21]. For irregular data structures, the number of atoms assigned to each processor varies significantly, and this load imbalance degrades the parallel efficiency. Load balancing can be stated as an optimization problem, in which we minimize the loadimbalance cost as well as the size and the number of messages. Parallel efficiency is defined as the speedup achieved using P processors over one processor, divided by P . Parallel efficiency over 0.9 has been achieved on a cluster of multicore compute nodes with P > 105 combining a hierarchy of parallelization schemes [22], including: 1. Internode parallelization based on message passing [23], in which independent processes (i.e., running programs) on different nodes exchange messages over a network. 2. Intra-node (inter-core), multithreading parallelization [24] on multicore central processing units (CPUs) as well as on hardware accelerators such as graphics processing units (GPUs) [25], in which multiple threads (i.e., processes sharing certain hardware resources such as memory) run concurrently on multiple cores within each compute node. 3. Intra-core, single-instruction multiple data (SIMD) parallelization [16,26], in which a single instruction

L

768

Large-Scale Computing for Molecular Dynamics Simulation, Fig. 2 Close-ups of fracture simulations for nanocrystalline nickel without and with amorphous sulfide grain-boundary phases, where red, blue and yellow colors represent nickel atoms inside grains (>0.5 nm from grain

executes on multiple operands concurrently in a vector processing unit within each core. A number of software packages have been developed for parallel MD simulations. Widely available packages for MD include Amber (http://ambermd. org), Desmond (http://www.schrodinger.com/products/ 14/3), DL POLY (http://www.cse.scitech.ac.uk/ccg/ software/DL POLY), Gromacs (http://www.gromacs. org), and NAMD (http://www.ks.uiuc.edu/Research/ namd). Parallel implementations of MD and RMD are found in LAMMPS (http://lammps.sandia.gov). DFT-based QMD packages include CP2K (http:// cp2k.berlios.de), Quantum ESPRESSO (http://www. quantum-espresso.org), SIESTA (http://www.icmab. es/siesta), and VASP (http://cms.mpi.univie.ac.at/ vasp), along with those specialized on linear-scaling DFT approaches such as Conquest (http://hamlin. phys.ucl.ac.uk/NewCQWeb/bin/view), ONETEP (http://www.tcm.phy.cam.ac.uk/onetep), and OpenMX (http://www.openmx-square.org). Finally, quantumchemical approaches to QMD are implemented in, e.g., GAMESS (http://www.msg.ameslab.gov/gamess), Gaussian (http://www.gaussian.com), and NWChem (http://www.nwchem-sw.org).

Large-Scale Computing for Molecular Dynamics Simulation

boundaries), nickel atoms within 0.5 nm from grain boundaries, and sulfur atoms, respectively. The figure shows a transition from ductile, transgranular tearing (left) to brittle, intergranular cleavage (right). White arrows point to transgranular fracture surfaces

billion-to-trillion atoms on massively parallel supercomputers consisting of over 105 processors to study various material processes such as instability at fluid interfaces and shock-wave propagation [27, 28]. The largest RMD simulations include 48 millionatom simulation of solute segregation-induced embrittlement of metal [29]. This simulation answers a fundamental question encompassing chemistry, mechanics, and materials science: How a minute amount of impurities segregated to grain boundaries of a material essentially alters its fracture behavior. A prime example of such grain-boundary mechano-chemistry is sulfur segregation-induced embrittlement of nickel, which is an important problem for the design of the next-generation nuclear reactors to address the global energy problem. Experiments have demonstrated an essential role of sulfur segregation-induced grain boundary amorphization on the embrittlement, but the central question remains unsolved: Why does amorphization cause embrittlement? The RMD simulation (Fig. 2) establishes the missing link between sulfur-induced intergranular amorphization and embrittlement [29]. The simulation results reveal that an order-of-magnitude reduction of grain-boundary shear strength due to amorphization, combined with tensile-strength reduction, allows the crack tip to always find an easy propagation path. This mechanism explains all experimental observations and elucidates the experimentally found link Large-Scale Molecular Dynamics between grain-boundary amorphization and embrittleApplications ment. While large-scale electronic structure calcuUsing scalable parallel MD algorithms, computational scientists have performed MD simulations involving lations involving over 104 atoms have been re-

Large-Scale Computing for Molecular Dynamics Simulation

769

Large-Scale Computing for Molecular Dynamics Simulation, Fig. 3 Snapshots of the atomic configuration during DC-DFT-based QMD simulation of thermite reaction, where green, red, and gray spheres show the positions of Fe, O and Al atoms, respectively. Yellow meshes at time 0 ps show the nonoverlapping cores used by the DC-DFT algorithm

ported (see  Large-Scale Electronic Structure and Nanoscience Calculations), QMD simulations extending a long trajectory are usually limited to thousands of atoms. Examples of systems studied by large QMD simulations include metals under extreme conditions [30], reaction of nanoenergetic materials [31], and ionic conductivity in batteries [32]. Chemical reactions in energetic materials with nanometer-scale microstructures (or nanoenergetic materials) are very different from those in conventional energetic materials. For example, in conventional thermite materials made of aluminum and iron oxide, the combustion front propagates at a speed of cm/s. In nanothermites of aluminum nanoparticles embedded in iron oxide, the combustion speed is accelerated to km/s. Such rapid reactions cannot be explained by conventional diffusion-based mechanisms. DC-DFT-based QMD simulation has been performed to study electronic processes during thermite reaction [31]. Here, the reactants are Al and Fe2 O3 , and the products are Al2 O3 and Fe (Fig. 3). The simulation results reveal a concerted metal-oxygen flip mechanism that enhances mass diffusion and reaction rate at the metal/oxide interface. This mechanism leads to novel two-stage reactions, which explain experimental observation in thermite nanowire arrays.

Conclusions Large-scale MD simulations to encompass large spatiotemporal scales are enabled with scalable al-

gorithmic and parallel-computing techniques based on spatiotemporal data-locality principles. The spatiotemporal scale covered by MD simulation on a sustained petaflops computer (which can operate 1015 floatingpoint operations per second) per day is estimated as N T  2 (e.g., N D 2 billion atoms for T D 1 ns) [22], which continues to increase on emerging computing architectures.

References 1. Rapaport, D.C.: The art of molecular dynamics simulation. 2nd edn. Cambridge University Press, Cambridge (2004) 2. Nakano, A., et al.: De novo ultrascale atomistic simulations on high-end parallel supercomputers. Int. J. High Perform. Comput. Appl. 22, 113 (2008) 3. Shaw, D.E., et al.: Anton, a special-purpose machine for molecular dynamics simulation. Commun. ACM. 51, 91 (2008) 4. Greengard, L., Rokhlin, V.: A fast algorithm for particle simulations. J. Comput. Phys. 73, 325 (1987) 5. Ogata S., et al.: Scalable and portable implementation of the fast multipole method on parallel computers. Comput. Phys. Commun. 153, 445 (2003) 6. Martyna, G.J., et al.: Explicit reversible integrators for extended systems dynamics. Mol. Phys. 87, 1117 (1996) 7. Schlick, T., et al.: Algorithmic challenges in computational molecular biophysics. J. Comput. Phys. 151, 9 (1999) 8. Nakano, A.: Parallel multilevel preconditioned conjugategradient approach to variable-charge molecular dynamics. Comput. Phys. Commun. 104, 59 (1997) 9. Car, R., Parrinello, M.: Unified approach for molecular dynamics and density functional theory. Phys. Rev. Lett. 55, 2471 (1985) 10. Goedecker, S.: Linear scaling electronic structure methods. Rev. Mod. Phys. 71, 1085 (1999)

L

770 11. Bowler, D.R., et al.: Introductory remarks: Linear scaling methods - Preface. J. Phys. Condens. Matter 20, 290301 (2008) 12. Kohn, W.: Density functional and density matrix method scaling linearly with the number of atoms. Phys. Rev. Lett. 76, 3168 (1996) 13. Yang, W.: Direct calculation of electron-density in densityfunctional theory. Phys. Rev. Lett. 66, 1438 (1991) 14. Goedecker, S., Scuseria, G.E.: Linear scaling electronic structure methods in chemistry and physics. Comput. Sci. Eng. 5, 14 (2003) 15. Kitaura, K., et al.: Fragment molecular orbital method: an approximate computational method for large molecules. Chem. Phys. Lett. 313, 701 (1999) 16. Grama, A., et al.: Introduction to parallel computing. 2nd edn. Addison Wesley, Harlow (2003) 17. Asanovic, K., et al.: The landscape of parallel computing research: A view from Berkeley. University of California, Berkeley (2006) 18. Plimpton, S.J.: Fast parallel algorithms for short-range molecular dynamics. J. Comput. Phys. 117, 1 (1995) 19. Kale, L., et al.: NAMD2: greater scalability for parallel molecular dynamics. J. Comput. Phys. 151, 283 (1999) 20. Shaw, D.E.: A fast, scalable method for the parallel evaluation of distance-limited pairwise particle interactions. J. Comput. Chem. 26, 1318 (2005) 21. Devine, K.D., et al.: New challenges in dynamic load balancing. Appl. Num. Math. 52, 133 (2005) 22. Nomura, K., et al.: A metascalable computing framework for large spatiotemporal-scale atomistic simulations. Proceedings of International Parallel and Distributed Processing Symposium IPDPS 2009, IEEE, Rome (2009) 23. Gropp, W., Lusk, E., Skjellum, A.: Using MPI. 2nd edn. MIT, Cambridge (1999) 24. Chapman, B., Jost, G., van der Pas, R.: Using OpenMP. MIT, Cambrige (2007) 25. Phillips, J.C., Stone, J.E.: Probing biomolecular machines with graphics processors. Commun. ACM. 52, 34 (2009) 26. Peng, L., et al.: Exploiting hierarchical parallelisms for molecular dynamics simulation on multicore clusters. J. Supercomput. 57, 20 (2011) 27. Glosli, J.N., et al.: Extending stability beyond CPU millennium: a micron-scale atomistic simulation of KelvinHelmholtz instability. Proceedings of Supercomputing (SC07), ACM, New York (2007) 28. Germann, T.C., Kadau, K.: Trillion-atom molecular dynamics becomes a reality. Int. J. Mod. Phys. C. 19, 1315 (2008) 29. Chen, H.P., et al.: Embrittlement of metal by solute segregation-induced amorphization. Phys. Rev. Lett. 104, 155502 (2010) 30. Gygi, F., et al.: Large-scale first-principles molecular dynamics simulations on the BlueGene/L platform using the Qbox code. Proceedings of Supercomputing 2005 (SC05), ACM, Washington, DC (2005) 31. Shimojo, F., et al.: Enhanced reactivity of nanoenergetic materials: A first-principles molecular dynamics study based on divide-and-conquer density functional theory. Appl. Phys. Lett. 95, 043114 (2009) 32. Ikeshoji, T., et al.: Fast-ionic conductivity of LiC in LiBH4 . Phys. Rev. B. 83, 144301 (2011)

Large-Scale Electronic Structure and Nanoscience Calculations

Large-Scale Electronic Structure and Nanoscience Calculations Juan C. Meza1 and Chao Yang2 1 School of Natural Sciences, University of California, Merced, CA, USA 2 Computational Research Division, MS-50F, Lawrence Berkeley National Laboratory, Berkeley, CA, USA

Synonyms Electronic structure; Nanoscience

Kohn-Sham

equations;

Definition The electronic structure of an atomic or molecular system can yield insights into many of the electrical, optical, and mechanical properties of materials. Realworld problems, such as nanostructures, are difficult to study, however, as many algorithms do not scale well with system size requiring new techniques better suited to large systems.

Overview The electronic structure of a system can be described by the solution of a quantum many-body problem described by the Schr¨odinger equation: H  D E; where H is a many-body Hamiltonian operator that describes the kinetic energy and the Coulomb interaction between electron–electron and electron–nucleus pairs,  is a many-body wavefunction, and E is the total energy level of the system. One popular approach for solving these types of problems relies on reformulating the original problem in terms of a different basic variable, the charge density, and using single-particle wavefunctions to replace the many-body wavefunctions. This approach is known as Kohn-Sham density functional theory (DFT) and can be viewed as a search for the minimizer of a certain functional of the charge density.

Large-Scale Electronic Structure and Nanoscience Calculations

771

From a mathematical viewpoint, it can be shown that the first order necessary optimality condition (Euler-Lagrange equation) for minimizing the KohnSham energy yields the following set of nonlinear eigenvalue equations (known as the Kohn-Sham equations): H./ i D i i ; i D 1; 2; : : : ; ne ; where H./ D  C Vion C VH ./ C Vxc ./, Vion , VH , and Vxc are the ionic, electron–electron (Hartree), and exchange-correlation potentials, and ne is the number of electrons. Here P .r/ is the electron charge density e defined by .r/ D ni D1 j i .r/j2 : Although these equations contain far fewer degrees of freedom compared to the many-body Schr¨odinger equation, they are more difficult in terms of their mathematical structures. The most popular method to solve the Kohn-Sham equations is the Self-Consistent Field (SCF) iteration. The computational complexity of most of the existing algorithms is O.n3e /, which can limit their applicability to large nanoscience problems. We will describe briefly some of the general strategies one may use to reduce the overall complexity of these algorithms and where the challenges lie in doing this.

Equation 1 defines a self-consistent map from  to itself. This map is sometimes referred to as the Kohn-Sham map. Because the Jacobian of this map is difficult to compute or invert, a practical approach for finding the fixed point of the Kohn-Sham map is to apply a Broyden type Quasi-Newton algorithm to solve (1) iteratively. This is generally known as a SCF iteration. The convergence of a SCF iteration depends largely on the choice of an effective Broyden updating scheme for approximating the Jacobian at each iteration. Such a scheme is known as charge mixing in the physics literature. The dominant cost of a SCF iteration is the evaluation of the Kohn-Sham map, that is, the right hand side of (1). The most widely used technique for performing such an evaluation is to partially diagonalize H./ and compute its ne smallest eigenvalues and the corresponding eigenvectors. For large-scale problems, the eigenvalue problem is often solved by an iterative method such as a Lanczos or Davidson algorithm. An alternative approach is to treat the eigenvalue problem as a constrained minimization problem and apply an iterative minimization algorithm such as the locally optimal block preconditioned conjugate gradient (LOBPCG) algorithm [6] to minimize the trace of X  HX subject to the orthonormality constraint X  X D I. Because an effective preconditioner can be used in this approach, it is often more efficient than a Lanczos-based algorithm. Both the Lanczos and the LOBPCG algorithms require performing orthogonalization among at least ne basis vectors, which for large ne incurs a cost of O.n3e /. To reduce the frequency of orthogonalization, one may apply a simple subspace iteration to p .k/ .H /, where p .k/ . / is a polynomial constructed at the kth SCF iteration to amplify the spectral components associated with the desired eigenvalues of H while filtering out the unwanted components. Although this algorithm may use approximately the same number of matrix-vector multiplications as that used in a Lanczos, Davidson, or LOBPCG algorithm, the basis orthogonalization cost is much lower (but not completely eliminated) for large ne , as is shown in [20]. A recently developed method [9] for evaluating the Kohn-Sham map without resorting to performing a spectral decomposition of H relies on using a rational approximation to gˇ .  / to compute the diagonal

The Kohn-Sham Map and the SCF Iteration A useful concept for analyzing algorithms applied to large-scale Kohn-Sham problems is the following alternative definition of the charge density: h i   O ˇ .O  /XO  D diag gˇ .H./  / ;  D diag Xg (1) where XO 2 Cnn contains the full set of eigenvectors of a discretized Kohn-Sham Hamiltonian, O is a diagonal matrix containing the corresponding eigenvalues of the Hamiltonian, gˇ . / is the Fermi-Dirac function: gˇ . ; / D

ˇ  2 D 1tanh . / ; 1 C exp.ˇ.  // 2 (2)

where ˇ is a parameter chosen in advance and proportional to the inverse of the temperature, and  is the chemical potential, which is chosen so that  trace gˇ .H./  I / D ne . At zero temperature, ˇ D 1 and (2) reduces to a step function that drops from 1 to 0 at .

L

772

Large-Scale Electronic Structure and Nanoscience Calculations

entries of gˇ .H  I / directly. The rational approxi- in [17, 18]. In this algorithm, the search direction and the step length are determined simultaneously from a mation to gˇ .  / has the form: subspace that consists of the existing wave functions  np X .i / , the gradient of the Lagrangian, and the search X !j gˇ .  /  Im ; direction produced in the previous iteration. A special  zj j D1 strategy is employed to minimize the total energy within the search space, while maintaining the orwhere zj and !j are carefully chosen poles and weight- thonormality constrained required for X .i C1/ . Solving ing factors that minimize the approximation error. The the subspace minimization problem is equivalent to number of poles required (np ) is typically less than solving a nonlinear eigenvalue problem of a much a hundred. Although computing gˇ .H  I / would smaller dimension. require us to compute .H  zi I /1 , which is likely to be completely dense, for np complex poles zi, a significant amount of savings can be achieved if we Linearly Scaling Algorithms only need the diagonal elements of gˇ .H  I /. Instead of computing the entire matrix .H  zi I /1 , Most of the algorithms discussed above can be one only needs to compute its diagonal. This task implemented efficiently on modern high-performance can be accomplished by using a special algorithm parallel computers. However, for large nanoscience which we refer to as selected inversion [10, 11]. The problems that consist of more than tens of thousands complexity of selected inversion is O.ne / for quasi- of atoms, many of these existing algorithms are still 3=2 1D problems (e.g., nanotubes and nanowires), O.ne / quite demanding in terms of computational resources. for quasi-2D problems (e.g., graphene), and O.n2e / for In recent years, there has been a growing level general 3D problems. of interest in developing linearly scaling methods

Solving the Kohn-Sham Problem by Constrained Minimization The Kohn-Sham problem can also be solved by minimizing the Kohn-Sham total energy directly. In this case, we seek to find min

X  X DIne

1 Etot .X /  traceŒX  . L C VOion /X

2 1 C T L  C T xc ./; (3) 2

where L 2 Rnn and Vion 2 Rnn are matrix representations of finite dimensional approximations to the Laplacian and the ionic potential operator respectively. The matrix L is either the inverse or the pseudoinverse of L depending on the boundary condition imposed in the continuous model, and X 2 Cnne contains approximate single-particle wavefunctions as its columns. This approach has been attempted by several researchers [8, 14]. Most of the proposed methods treat the minimization of the total energy and constraint satisfaction separately. A more efficient direct constrained minimization (DCM) algorithm was proposed

[1, 2, 4, 5, 12, 13, 16, 19] for electronic structure calculations. For insulators and semiconductors, the computational complexity of these algorithms indeed scales linearly with respect to ne or the number of atoms. However, it is rather challenging to develop a linearly scaling algorithm for metallic systems for reasons that we will give below. In general, a linear scaling algorithm should meet the following criteria: • The complexity for evaluating the Kohn-Sham map must be O.ne /. • The total number of SCF iterations must be relatively small compared to ne . While most of the existing research efforts focus exclusively on the first criterion, we believe the second criterion is equally important. All existing linearly scaling algorithms exploit the locality property of the single-particle wavefunctions (orbitals) or density matrices to reduce the complexity of the charge density (Kohn-Sham map) evaluation. The locality property has its roots in the “nearsightedness” principle first suggested by Kohn [7] and further investigated in [15]. In mathematical terms, the locality property implies that the invariant subspace spanned by the smallest ne eigenvectors can be represented by a set of basis vectors that have local nonzero support (i.e., each basis vector has a relatively small number

Large-Scale Electronic Structure and Nanoscience Calculations

773

of nonzero elements.), or the density matrix D D gˇ .H./  I / is diagonally dominant, and the offdiagonal entries of the matrix decay rapidly to zero away from the diagonal. As a result, there are three main classes of linearly scaling methods. In the first class of methods, one relaxes the orthonormality constraint of the single-particle wavefunctions but requires them to have localized nonzero support. As a result, the Kohn-Sham map can be evaluated by solving a sparse generalized eigenvalue problem. An iterative method such as the localized subspace iterations (LSI) [3] can be used to compute the desired invariant subspace. Because each basis vector of the invariant subspace is forced to be sparse, the matrixvector multiplication used in such an algorithm can be evaluated efficiently with a complexity of O.ne /. More importantly, because such an algorithm does not perform basis reorthogonalization, it does not incur the O.n3e / cost of conventional eigensolvers. The second class of methods employs a divideand-conquer principle originally suggested in [19] to divide the problem into several subproblems defined on smaller subregions of the material domain. From a mathematical viewpoint, these are domain decomposition methods. A similar approach is used in the recently developed linear-scaling three-dimensional fragment (LS3DF) method [16]. These methods require local solutions to be patched together in a nontrivial way to preserve the total charge and to eliminate charge transfer between different regions. The third class of linearly scaling methods relies on using either polynomial or rational approximations of D D gˇ .H  I / and truncation techniques that ignore small off-diagonal entries in D to reduce the complexity of the Kohn-Sham map evaluation to O.ne /. It is important to note that the number of terms used in the polynomial or rational approximation to gˇ .H  I / must be small enough in order to achieve linear scaling. For insulators and semiconductors in which the gap between the occupied and unoccupied states is relatively large, this is generally not difficult to achieve. For metallic systems that have no band gap, one may need a polynomial of very high degree to approximate gˇ .H  I / with sufficient accuracy. It is possible to accurately approximate gˇ .H  I / using recently developed pole expansion techniques [9] with less than 100 terms even when the band gap is very small. However, since the off-diagonal elements of D decay slowly to zero for metallic systems, the

evaluation of the Kohn-Sham map cannot be performed in O.ne / without losing accuracy at low temperature. Linearly scaling algorithms can also be designed to minimize the total energy directly. To achieve linear scaling, the total energy minimization problem is reformulated as an unconstrained minimization problem. Instead of imposing the orthonormality constraint of the single-particle wavefunctions, we require them to have localized support. Such localized orbitals allow the objective and gradient calculations to be performed with O.ne / complexity. The original version of orbital minimization methods uses direct truncations of the orbitals. They are known to suffer from the possibility of being trapped at a local minimizer [4]. The presence of a large number of local minimizers in this approach is partially due to the fact that direct truncation tends to destroy the invariance property inherent in the KohnSham DFT model, and introduces many local minima in the Kohn-Sham energy landscape. This problem can be fixed by applying a localization procedure prior to truncation.

L Cross-References  Density Functional Theory  Hartree–Fock Type Methods  Schr¨odinger Equation for Chemistry  Self-Consistent Field (SCF) Algorithms

References 1. Barrault, M., Cances, E., Hager, W.W., Bris, C.L.: Multilevel domain decomposition for electronic structure calcluations. J. Comput. Phys. 222, 86–109 (2006) 2. Galli, G.: Linear scaling methods for electronic structure calculations and quantum molecular dynamics simulations. Curr. Opin. Solid State Mater. Sci. 1, 864–874 (1996) 3. Garcia-Cervera, C., Lu, J., Xuan, Y., Weinan, E.: A linear scaling subspace iteration algorithm with optimally localized non-orthogonal wave functions for Kohn-Sham density functional theory. Phys. Rev. B 79(11), 115110 (2009) 4. Goedecker, S.: Linear scaling electronic structure methods. Rev. Mod. Phys. 71(4), 1085–1123 (1999) 5. Kim, J., Mauri, F., Galli, G.: Total energy global optimizations using non-orthogonal localized orbitals. Phys. Rev. B 52(3), 1640–1648 (1995) 6. Knyazev, A.: Toward the optimal preconditioned eigensolver: locally optimal block preconditioned conjugate gradient method. SIAM J. Sci. Comput. 22(2), 517–541 (2001) 7. Kohn, W.: Density functional and density matrix method scaling linearly with the number of atoms. Phys. Rev. Lett. 76(17), 3168–3171 (1996)

774 8. Kresse, G., Furthm¨uller, J.: Efficiency of ab initio total energy calculations for metals and semiconductors using a plane-wave basis set. Comput. Mater. Sci. 6, 15–50 (1996) 9. Lin, L., Lu, J., Ying, L., Weinan, E.: Pole-based approximation of the Fermi-Dirac function. Chin. Ann. Math. 30B, 729 (2009) 10. Lin, L., Yang, C., Lu, J., Ying, L., Weinan, E.: A fast parallel algorithm for selected inversion of structured sparse matrices with application to 2D electronic structure calculations. SIAM J. Sci. Comput. 33, 1329–1351 (2011) 11. Lin, L., Yang, C., Meza, J.C., Lu, J., Ying, L., Weinan, E.: Selinv – an algorithm for selected inversion of a sparse symmetric matrix. ACM Trans. Math. Softw. 37, 40:1–40:19 (2011) 12. Mauri, F., Galli, G.: Electronic-structure calculation and molecular dynamics simulations with linear system-size scaling. Phys. Rev. B 50(7), 4316–4326 (1994) 13. Ordej´on, P., Drabold, D.A., Grumbach, M.P., Martin, R.M.: Unconstrained minimization approach for electronic computations that scales linearly with system size. Phys. Rev. B 48(19), 14646–14649 (1993) 14. Payne, M.C., Teter, M.P., Allen, D.C., Arias, T.A., Joannopoulos, J.D.: Iterative minimization techniques for ab initio total energy calculation: molecular dynamics and conjugate gradients. Rev. Mod. Phys. 64(4), 1045–1097 (1992) 15. Prodan, E., Kohn, W.: Nearsightedness of electronic matter. PNAS 102(33), 11635–11638 (2005) 16. Wang, L.W., Zhao, Z., Meza, J.: Linear-scaling threedimensional fragment method for large-scale electronic structure calculations. Phys. Rev. B 29, 165113–165117 (2008) 17. Yang, C., Meza, J.C., Wang, L.W.: A constrained optimization algorithm for total energy minimization in electronic structure calculation. J. Comput. Phys. 217, 709–721 (2006) 18. Yang, C., Meza, J.C., Wang, L.W.: A trust region direct constrained minimization algorithm for the Kohn-Sham equation. SIAM J. Sci. Comput. 29(5), 1854–1875 (2007) 19. Yang, W.: A local projection method for the linear combination of atomic orbital implementation of density-functional theory. J. Chem. Phys. 94(2), 1208–1214 (1991) 20. Zhou, Y., Saad, Y., Tiago, M.L., Chelikowsky, J.R.: Selfconsistent field calculations using Chebyshev-filtered subspace iteration. J. Comput. Phys. 219, 172–184 (2006)

Lattice Boltzmann Methods Paul Dellar1 and Li-Shi Luo2;3 1 OCIAM, Mathematical Institute, Oxford, UK 2 Department of Mathematics and Statistics, Old Dominion University, Norfolk, VA, USA 3 Beijing Computational Science Research Center, Beijing, China

Mathematics Subject Classification 82B40; 76D05; 76M99; 35Q20; 35Q30; 35Q82

Lattice Boltzmann Methods

Synonyms Lattice Boltzmann Method (LBM)

Short Definition The lattice Boltzmann method (LBM) is a family of methods derived from kinetic equations for computational fluid dynamics, chiefly used for nearincompressible flows of Newtonian fluids.

Description The primary focus of computational fluid dynamics (CFD) is the solution of the nonlinear Navier–Stokes– Fourier (NSF) equations that describe mass, momentum, and energy transport in a fluid. For the most common case of incompressible flow, these reduce to the Navier–Stokes (NS) equations for momentum transport alone, as supplemented by an elliptic equation to determine the pressure. The NSF equations may be derived from the Boltzmann equation of kinetic theory, with transport coefficients calculated from the underlying interatomic interactions. The lattice Boltzmann method (LBM) is distinguished by being a discretization of the Boltzmann equation, rather than a direct discretization of the NS equations.

Kinetic Theory and the Boltzmann Equation Kinetic theory describes a dilute monatomic gas through a distribution function f .x; ; t/ for the number density of particles at position x moving with velocity  at time t. The distribution function evolves according to the Boltzmann equation [2, 6] @t f C  rf D CŒf; f :

(1)

The quadratic integral operator CŒf; f represents binary collisions between pairs of particles. The first few moments of f with respect to particle velocity  give hydrodynamic quantities: the fluid density , velocity u, momentum flux …, and energy flux Q,

Lattice Boltzmann Methods

775

Z

Z

convenient to impose a constant temperature 0 when evaluating f .0/ . This takes the place of an independent Z Z energy evolution equation, and the last of (5) then e on the … D  f d; Q D  f d; (2) holds with … rather than the traceless part … right-hand side. A temperature evolution equation may in convenient units with the particle mass scaled to be reintroduced under the Boussinesq approximation unity. Collisions conserve mass, momentum, and en- using a second distribution function [5, 10]. ergy, while relaxing f towards a Maxwell–Boltzmann distribution D

f

.0/

f d;

D .2 /

3=2

u D

 f d;

  exp ku  k2 =.2 / :

Derivation of the Hydrodynamic (3) Equations

These together imply conservation of the temperature 2

, given p by Tr … D 3 C kuk in energy units for which is the Newtonian or isothermal sound speed. Hydrodynamics describes near-equilibrium solutions, f  f .0/ , for which a linearized collision operator is sufficient. A popular model is the Bhatnagar–Gross–Krook (BGK) form [1] @t f C  rf D 

 1 f  f .0/ 

(4)

The NSF equations describe solutions of the Boltzmann equation that vary slowly on macroscopic timescales 0  , where 0 may be a fluid eddy turnover time. The ratio  D =0 may be identified with the Knudsen number Kn. The modern Chapman– Enskog expansion [2] seeks solutions of (1) or (4) through a multiple-scale expansion of both the distribution function and the time derivative: f D

1 X nD0

 n f .n/ ;

@t D

1 X nD0

 n @tn :

(6)

that relaxes f towards an equilibrium distribution f .0/ with the same , u, as f . This satisfies all the require- This expansion of f implies corresponding expansions ments necessary for deriving the NSF equations, but of the moments: the Prandtl number is fixed at unity. The more general Z Z Gross–Jackson model [7] allows the specification of .n/ D f .n/ d; u.n/ D f .n/ d; any finite number of relaxation times in place of the Z Z above single relaxation time . .n/ .n/ .n/ … D f d; Q D f .n/ d: (7) Moments of the Boltzmann equation (1) give an infinite hierarchy of evolution equations for the moments of f . The first few are The expansion of @t prevents the overall expansion from becoming disordered after long times t  0 =, @t  C r .u/ D 0; @t .u/ C r … D 0; but requires additional solvability conditions, namely, that .n/ D 0, u.n/ D 0 for n  1. Equivalently, 1  e e .0/  (5) one may expand the non-conserved moments … D @t … C r Q D  … … :  ….0/ C ….1/ C    , Q D Q.0/ C Q.1/ C    , while Each evolution equation involves the divergence of the leaving the conserved moments  and u unexpanded. next higher moment. The first two right-hand sides vanEvaluating (5) at leading order gives the compressish because collisions conserve microscopic mass and ible Euler equations momentum. The right-hand side of the third equation e of the momentum flux arises from the traceless part … @0  C r .u/ D 0; @0 .u/ C r ….0/ D 0: (8) being an eigenfunction of the BGK collision operator and an eigenfunction of the linearized Boltzmann The inviscid momentum flux ….0/ D I C uu, with collision operator for Maxwell molecules. The latter I the identity tensor, is given by the second moment of property holds to a good approximation for other in- f .0/ . Evaluating the last of (5) at leading order gives teratomic potentials [2]. 1 Temperature fluctuations p are O.Ma2 / when the @0 ….0/ C r Q.0/ D  ….1/ ; (9) Mach number Ma D kuk= is small. It is then 0

L

776

Lattice Boltzmann Methods

The aim now is to choose the velocity set f i g, .0/ the equilibria fj .; u/, and the collision matrix ij so that the moment equations obtained from (11) coincide with the system (5) obtained previously from (1). The continuous Maxwell–Boltzmann equilibrium f .0/ emerged from properties of Boltzmann’s collision .0/ operator C Œf; f , but the fj .; u/ in (11) must be supplied explicitly. The discrete moments ….0/ and Q.0/ should remain unchanged from continuous kinetic theory, at least to O.Ma2 /. The discrete collision operator should conserve mass and momentum, and … should be an eigenfunction. The simplest choice ij D  1 ıij gives Discrete Kinetic Theory the BGK collision operator (4), but more general Discrete kinetic theory preserves the above structure choices improve numerical stability [3] and treatment that leads to the NS equations, but restricts the particle of boundary conditions. The most common equilibria velocity to a finite set,  2 f 0 ; : : : ;  N 1 g. The are the quadratic polynomials [9, 14] 

previous integral moments become sums over a finite 9 3 .0/ fj .; u/ D wj  1 C 3 u   j C .u   j /2  kuk2 ; set fi .x; t/, one for each  i : 2 2 (12) P P  D i fi ; u D i  i fi ; with weights w0 D 4=9, w1;:::;4 D 1=9, and w6;:::;9 D P P … D i  i  i fi ; Q D i  i  i  i fi : (10) 1=36 for the D2Q9 lattice shown in Fig. 1. The particle velocities  i are scaled so that ix ; iy 2 f1; 0; 1g and

D 1=3. The  i thus form an integer lattice. The above The discrete analogue of the linearized Boltzmann .0/ fj may be derived from a low Mach number expanequation is sion of the Maxwell–Boltzmann distribution, or as a moment expansion in the first few of Grad’s [6] tensor   P .0/ @t fi C  i rfi D  j ij fj  fj ; (11) Hermite polynomials 1,  ,    I. The w and  are i i i i i the weights and quadrature nodes for a Gauss–Hermite where ij is a constant N N matrix giving a quadrature that holds exactly for polynomials of degree general linear collision operator. This linear, constant- 5 or less. The , u, ….0/ moments of the discrete and coefficient hyperbolic system is readily discretized, as continuous equilibria thus coincide exactly [9], while described below. the Q.0/ moment differs by an O.Ma3 / term uuu.

where Q.0/ is known from f .0/ , and we evaluate (8). After @0 ….0/ using the Euler equations  some ma nipulation, ….1/ D  .r u/ C .r u/T D  S becomes the NS viscous stress for an isothermal fluid with dynamic viscosity  D  . The multiple-scale expansion may be avoided by taking Ma D O./. This so-called diffusive scaling removes the separation of timescales by bringing the viscous term u r u into balance with .=/r 2 u, pushing the @0 ….0/ term in (9) to higher order [11].

J+1

6

2

5

3

0

1

7

4

8

J

J−1 I−1

I

I+1

Lattice Boltzmann Methods, Fig. 1 D2Q9 and D3Q19 lattices. The velocities  i are scaled so that i˛ 2 f1; 0; 1g for ˛ 2 fx; y; zg

Lattice Boltzmann Methods

777

Space–Time Discretization For each i , we may write the left-hand side of (11) as a total derivative dfi =ds along the characteristic .x; t/ D .x 0 C  i s; t C s/ parametrized by s. Integrating (11) along this characteristic for a timestep t gives [10] fi .x C  i t; t C t/  fi .x; t/ D h i X .0/  2 0t ij fj  fj .x C  i s; t C s/ ds: j (13) Approximating the remaining integral by the trapezoidal rule gives fi .x C  i t; t C t/  fi .x; t/ D X 1  t ij fj .x C  i t; t C t/ C fj .x; t/ 2 j

  .0/ .0/ fj .x C  i t; t C t/  fj .x; t/ C O .t=/3 : (14)

requires t  to justify neglecting the error in (14). The same restriction is needed in the reverse derivation of partial differential equations from (16) using Taylor expansions in t [11]. However, the algorithm (16) successfully captures slowly varying hydrodynamic behavior on macroscopic timescales 0  t even when t  . The ratio t= may be identified with the grid-scale Reynolds number Regrid D kukx=, with x D t in standard LB units. Stability for Regrid  1 is essential for applying the LBM to turbulent flows. Stable 2D simulations have been demonstrated [3] with Regrid & 100 and a collision matrix ij that suppresses the oscillations with period 2t that arise in the nonconserved moments when Regrid > 1. These successes do not imply that the LBE correctly captures arbitrary solutions of the discrete Boltzmann equation evolving on the collisional timescale , such as kinetic initial and boundary (Knudsen) layers [2, 6]. The LBE reproduces just enough of the true Boltzmann equation to capture the isothermal NS equations. It does not capture Burnett and higher order corrections relevant for rarefied flows at finite Knudsen numbers, and it does not capture Knudsen boundary layers.

Neglecting the error term, and collecting all terms evaluated at t C t to define   1 X .0/ ; f i .x; t/ D fi .x; t/ C t ij fj  fj 2 j (15) leads to an explicit scheme, the lattice Boltzmann equation (LBE), for the f i : f i .x C  i t; t C t/ D   X .0/ f i .x; t/  t ij f j .x; t/  fj .x; t/ ; (16) j

 1 with discrete collision matrix  D I C 12 t . When  D  1 I this transformation reduces to replacing  with  C t=2. Taking moments of (15) gives the P P conserved moments  D i f i and u D i  i f i , unaffected by the collision term that distinguishes f i .0/ from fi . We may thus evaluate the fi in (16). However, non-conserved moments such as … must be found by inverting (15) for the fi . The errors involving t from the space–time discretization of (11) are in principle entirely independent of the O. 2 / error in the derivation of the NS equations. However, the above usage of the trapezoidal rule

Wider Applications The core lattice Boltzmann algorithm described above has been extended into many wider applications: large eddy simulations of turbulent flows, multiphase flows, and soft condensed matter systems such as colloids, suspensions, gels, and polymer solutions [4, 12]. The LBM is commonly characterized as a second-order accurate scheme at fixed Mach number. However, the spatial derivatives on the left-hand side of (11) are treated exactly in deriving (13). The only approximation lies in the treatment of the collision integral. Comparisons with pseudo-spectral simulations for the statistics of turbulent flows show comparable accuracy when the LBM grid is roughly twice as fine as the pseudo-spectral collocation grid [13]. The nonequilibrium momentum flux ….1/ is proportional to the local strain rate S D .r u/ C .r u/T under the Chapman–Enskog expansion, so S may be computed locally from .…  ….0/ / at each grid point with no spatial differentiation [15]. Adjusting the local collision time  to depend on S extends the LBM to large eddy simulations using the Smagorinsky turbulence model, with an effective eddy viscosity turb / jjSjj,

L

778

and to further generalized Newtonian fluids whose viscosities are functions of jjSjj. The straightforward implementation of boundary conditions by reflecting particles from solid boundaries makes the LBM attractive for simulating pore-scale flows in porous media and particle-scale flows of suspensions. The Brownian thermal fluctuations omitted in the Boltzmann equation, but relevant for colloids, may be restored by adding random noise to the nonconserved moments during collisions [4]. There are many LB formulations for multiphase and multicomponent flows [8]. They are essentially diffuse interface capturing schemes that use interactions between neighboring grid points to mimic the inter-particle interactions responsible for interfacial phenomena.

Least Squares Calculations 13. Peng, Y., Liao, W., Luo, L.S., Wang, L.P.: Comparison of the lattice Boltzmann and pseudo-spectral methods for decaying turbulence: low-order statistics. Comput. Fluids 39, 568–591 (2010) 14. Qian, Y.H., d’Humi`eres, D., Lallemand, P.: Lattice BGK models for the Navier–Stokes equation. Europhys. Lett. 17, 479–484 (1992) 15. Somers, J.A.: Direct simulation of fluid flow with cellular automata and the lattice-Boltzmann equation. Appl. Sci. Res. 51, 127–133 (1993)

Least Squares Calculations ˚ Bj¨orck Ake Department of Mathematics, Link¨oping University, Link¨oping, Sweden

Introduction References 1. Bhatnagar, P.L., Gross, E.P., Krook, M.: A model for collision processes in gases. I. Small amplitude processes in charged and neutral one-component system. Phys. Rev. 94, 511–525 (1954) 2. Cercignani, C.: The Boltzmann Equation and its Applications. Springer, New York (1988) 3. Dellar, P.J.: Incompressible limits of lattice Boltzmann equations using multiple relaxation times. J. Comput. Phys. 190, 351–370 (2003) 4. D¨unweg, B., Ladd, A.J.C.: Lattice Boltzmann simulations of soft matter systems. Adv. Polym. Sci. 221, 1–78 (2009) 5. Eggels, J.G.M., Somers, J.A.: Numerical-simulation of free convective flow using the lattice-Boltzmann scheme. Int. J. Heat Fluid Flow 16, 357–364 (1995) 6. Grad, H.: Principles of the kinetic theory of gases. In: Fl¨ugge, S. (ed.) Thermodynamik der Gase. Handbuch der Physik, vol. 12, pp. 205–294. Springer, Berlin (1958) 7. Gross, E.P., Jackson, E.A.: Kinetic models and the linearized Boltzmann equation. Phys. Fluids 2, 432–441 (1959) 8. Gunstensen, A.K., Rothman, D.H., Zaleski, S., Zanetti, G.: Lattice Boltzmann model of immiscible fluids. Phys. Rev. A 43, 4320–4327 (1991) 9. He, X., Luo, L.S.: Theory of the lattice Boltzmann method: from the Boltzmann equation to the lattice Boltzmann equation. Phys. Rev. E 56, 6811–6817 (1997) 10. He, X., Chen, S., Doolen, G.D.: A novel thermal model of the lattice Boltzmann method in incompressible limit. J. Comput. Phys. 146, 282–300 (1998) 11. Junk, M., Klar, A., Luo, L.S.: Asymptotic analysis of the lattice Boltzmann equation. J. Comput. Phys. 210, 676–704 (2005) 12. Ladd, A.J.C.: Numerical simulations of particulate suspensions via a discretized Boltzmann equation. Part 1. Theoretical foundation. J. Fluid Mech. 271, 285–309 (1994)

A computational problem of primary importance in science and engineering is to fit a mathematical model to given observations. The influence of errors in the observations can be reduced by using a greater number of measurements than the number of unknown parameters. Least squares estimation was first used by Gauss in astronomical calculations more than two centuries ago. It has since been a standard approach in applications areas that include geodetic surveys, photogrammetry, signal processing, system identification, and control theory. Recent technological developments have made it possible to generate and treat problems involving very large data sets. As an example, consider a model described by a scalar function f .x; t/, where x 2 Rn is an unknown parameter vector to be determined from measurements bi D f .x; ti / C ei ; i D 1; : : : ; m.m > n/, where ei are errors. In the simplest case f .x; ti / is linear in x: f .x; t/ D

n X

xj j .t/;

(1)

j D1

where j .t/ are known basis functions. Then the measurements form an overdetermined system of linear equations Ax D b, where A 2 Rmn is a matrix with elements aij D j .ti /. It is important that the basis function j .t/ are chosen carefully. Suppose that f .x; t/ is to be modeled by a polynomial of degree n. If the basis functions

Least Squares Calculations

779

are chosen as the monomials t j , then A will be a The covariance matrix of the least squares estimate xO O D  2 .AT V 1 A/1 and an unbiased estimate Vandermonde matrix. Such matrices are notoriously ill is V.x/ 2 conditioned and this can lead to an inaccurate solution. of  is given by s 2 D rO T V 1 rO =.m  n/. In the special case of weighted least squares, the covariance matrix is V D D 2 , D D diag.d1 ; : : : ; dm /. After a diagonal The Least Squares Principle scaling this is equivalent to the scaled standard problem In the standard Gauss–Markov linear model, it is as- minx kDb  .DA/x/k2 . sumed that a linear relation Ax D y holds, where A 2 Rmn is a known matrix of full column rank, x is a parameter vector to be determined, and y 2 Rm a Calculating Least Squares Estimates constant but unknown vector. The vector b D f C e is a vector of observations and e a random error vector. It Comprehensive discussions of methods for solving is assumed that e has zero mean and covariance matrix least squares problems are found in [6] and [1]. In the  2 I , where  2 is an unknown constant. following we write the algebraic linear least squares Theorem 1 (The Gauss–Markov Theorem) In the problems in the form minx kAx  bk2 . The singular value decomposition (SVD) is a powlinear Gauss–Markov model, the best linear unbiased erful tool both for analyzing and solving the linear estimator of x is the least square estimate xO that least squares problem. The SVD of A 2 Rmn of minimizes the sum of squares rank.A/ D n is S.x/ D

kr.x/k22

D

m X

A D U˙V

ri2 ;

i D1

where r.x/ D b  Ax is the residual vector. A necessary condition for a minimum is that the gradient vector @S=@x is zero. This condition gives AT .b  Ax/ D 0, i.e., r.x/ ? R.A/, the range of A. It follows that xO satisfies the normal equations ATAx D AT b. The best linear unbiased estimator of any linear O functional c T x is c T x. The covariance matrix of the estimate xO is V.x/ O D  2 .ATA/1 . The residual vector rO D b  AxO is uncorrelated with xO and an unbiased estimate of  2 is given by s 2 D kOr k22 =.m  n/. In the complex case A 2 Cmn , b 2 Cm , the complex scalar product has to be used in Gauss– Markov theorem. The least squares estimate minimizes krk22 D r H r, where r H denotes the complex conjugate transpose of r. The normal equations are AH Ax D AH b. This has applications, e.g., in complex stochastic processes. It is easy to generalize the Gauss–Markov theorem to the case where the error e has a symmetric positive definite covariance matrix  2 V . The least squares estimate then satisfies the generalized normal equations AT V 1 Ax D AT V 1 b:

(2)

T

  ˙1 V T D U1 ˙1 V T ; (3) D U1 U2 0 

where ˙1 D diag.1 ; 2 ; : : : ; n /. Here 1  2     ;  n  0 are the singular values of A and the matrices U D .u1 ; u2 ; : : : ; um / and V D .v1 ; v2 ; : : : ; vn / are square orthogonal matrices, whose columns are the left and right singular vectors of A. If n > 0 the least squares solution equals x D V ˙11 .U1T b/ D

n X ci vi ;  i D1 i

ci D uTi b

(4)

If A has small singular values, then small perturbations in b can give rise to large perturbations in x. The ratio .A/ D 1 =n is the condition number of A. The condition number of the least squares solution x can be shown to depend also on

the ratio krk2 =n kxk2 and krk2 . The second equals [1] .x/ D .A/ 1 C n kxk2 term will dominate if krk2 > n kxk2 . Because of the high cost of computing and modifying the SVD, using the expansion (4) is not always justified. Simpler and cheaper alternative methods are available. The Method of Normal Equations If A 2 Rmn has full column rank, the solution can be obtained from the normal equations. The symmetric

L

780

Least Squares Calculations

matrix ATA 2 Rnn is first formed. Then the Cholesky factorization ATA D RT R is computed, where R is an upper triangular matrix with positive diagonal elements. These operations require mn2 Cn3 =3 floating point operations (flops). For a right-hand side b, the least squares solution is obtained by computing d D AT b 2 Rn and solving two triangular systems RT z D d and Rx D z. The residual matrix is r D b  Ax. This requires 2n.2m C n/ flops. The estimated covariance matrix of x is Vx D s 2 .RT R/1 D s 2 S S T ;

s 2 D r T r=.m  n/;

S D R1 :

(5)

The estimated variance of any linear functional  D f T x is V D s 2 f T S S T f D s 2 v T v;

RT v D f:

(6)

and can be computed without forming Vx . Setting f D ei gives the variance of the component xi . The com1 ponents of the normalized residual rQ D diag.Vx /1 rO s should be uniformly distributed random variables. This can be used to detect and identify bad observations.

 c Q b D Pn    P2 P1 b D 1 ; c2

 0 : r D P1 P2    Pn c2 T

Rx D c1 ; (8)

Using orthogonality it follows that krk2 D kc2 k2 . If the diagonal elements in the triangular factor R are chosen to be positive, then R is uniquely determined and mathematically (not numerically) the same as the Cholesky factor from the normal equations. Thus, the expression (5) for the estimated covariance matrix is valid. It is recommended that column pivoting is performed in the QR factorization. This will yield a QR factorization of A˘ for some permutation matrix ˘ . The standard strategy is to choose at each step k D 1; : : : ; n, the column that maximizes the diagonal element rkk in R. Then the sequence r11  r22      rnn > 0 is nonincreasing, and the ratio r11 =rnn is often used as a rough approximation of .A/. A rectangular matrix A 2 Rmn , m > n can be transformed further to lower (or upper) bidiagonal form by a sequence of two-sided orthogonal transformations 1 ˛1 C Bˇ2 ˛2 C B C B :: C : B DB ˇ 3 C B C B : :: ˛ A @ n ˇnC1 0

QR Factorizations and Bidiagonal Decomposition The method of normal equations is efficient and sufficiently accurate for many problems. However, forming the normal equations squares the condition number of the problem. This can be seen by using the SVD to show that ATA D V ˙U T U˙V T D V ˙12 V T and hence .ATA/ D  2 .A/. Methods using orthogonal transformations preserve the condition number and should be preferred unless the problem is known to be well conditioned. The QR factorization of the matrix A 2 Rmn of full column rank is

ADQ

 R D Q1 R; 0

U T AV D

 B ; 0

(9)

where U D .u1 ; u2 ; : : : ; um /, V D .v1 ; v2 ; : : : ; vn /. This orthogonal decomposition requires 4.mn2 n3 =3/ flops, which is twice as much as the QR factorization. It is essentially unique once the first column u1 D Ue1 has been chosen. It is convenient to take u1 D b=ˇ1 , ˇ1 D kbk2 . Then U T b D ˇ1 e1 and setting x D Vy, (7) we have

  where Q D Q1 Q2 2 Rmm is orthogonal and R 2 Rnn upper triangular. It can be computed in 2.mn2  n3 =3/ flops using Householder transformations. The matrix Q is then implicitly represented as Q D P1 P2    Pn where Pi D I  2vi viT , kvi k2 D 1. Only the Householder vectors vi need to be stored and saved. The least squares solution and the residual vector are then obtained in about 8mn3n2 flops from

U T .b  Ax/ D



ˇ1 e1  By : 0

The least squares solution can be computed in O.n/ flops by solving the bidiagonal least squares problem miny kBy  ˇ1 e1 k2 . The upper bidiagonal form makes the algorithm closely related to the iterative LSQR algorithm in [7]. Also, with this choice of u1

Least Squares Calculations

781

the decomposition will terminate early with a core U1 and V2 approximate the range and null space of A, subproblem if an entry ˛i or ˇi is zero ([8]]. respectively. Rank-Deficient Problems Rank deficiency in least squares problems can arise in different ways. In statistics one often has a large set of variables, called the factors, that are used to control, explain, or predict other variables. The set of factors correspond to the columns of a matrix A D .a1 ; a2 ; : : : ; an /. If these are highly collinear, then the approximate rank of A is less than n and the least squares solution is not unique. Often the rank of A is not known in advance, but needs to be determined as part of the solution process. In the rank-deficient case one can seek the least squares solution of minimum norm, i.e., solve the problem

Large-Scale Problems Many applications lead to least squares problems where A is large and sparse or structured. In the QR factorization of a sparse matrix, the factor Q will often be almost full. This is related to the fact that Q D AR1 and even if R is sparse R1 will have no zero elements. Therefore, computing the factor Q explicitly for a sparse matrix should be avoided. A QR algorithm for banded matrices which processes rows or block of rows sequentially is given in [6, Chap. 27]. An excellent source book on factorization of matrices with more irregular sparsity is [3]. An efficient iterative method for solving large sparse least squares problems is the Krylov subspace method LSQR (see [7]). It uses a Lanczos process to generate n the vectors vi , ui C1 , i D 1; 2; : : : and the columns of min kxk2 ; S D fx 2 R j kb  Axk2 D ming: (10) x2S the matrix B in (9). LSQR only requires one matrixvector product with A and AT per iteration step. If A This problem covers as special cases both overdeter- is rank deficient, LSQR converges to the pseudoinverse mined and underdetermined linear systems. The so- solution. lution is always unique and called the pseudoinverse solution. It is characterized by x 2 R.AT / and can be obtained from the SVD of A as follows. If rank.A/ D Regularization of Least Squares Problems r < n, then j D 0, j > r, and x D A b D V1 ˙11 .U1T b/ D

r X ci vi ;  i D1 i

ci D uTi b; (11)

i.e., it is obtained simply by excluding terms corresponding to zero singular values in the expansion (4). The matrix A D V1 ˙11 U1T is called the pseudoinverse of A. In some applications, e.g., in signal processing, one has to solve a sequence of problems where the rank may change. For such problems methods that use a pivoted QR factorization have the advantage over the SVD in that these factorizations can be efficiently updated; see [4]. One useful variant is the URV decomposition, which has the form

In discrete approximations to inverse problems, the singular values i of A cluster at zero. If the exact righthand side b is contaminated by white noise, this will affect all coefficients ci in the SVD expansion (4) more or less equally. Any attempt to solve such a problem without restriction on x will lead to a meaningless solution.

Truncated SVD and Partial Least Squares If the SVD of A is available, then regularization can be achieved simply by including in the SVD expansion only terms for which 1 > tol, for some tolerance tol only. An often more efficient alternative is to use partial least squares (PLS). Like truncated SVD it computes a sequence of approximate least squares solutions by orthogonal projections onto lower dimen T

 R11 R12  V1 T : (12) sional subspaces. PLS can be implemented through A˘ D URV D U1 U2 0 R22 V2T a partial reduction of A to lower bidiagonal form. It is used extensively in chemometrics, where it was Here R11 is upper triangular and the entries of R12 and introduced in [10]. The connection to the bidiagonal R22 have small magnitudes. The orthogonal matrices decomposition is exhibited in [2].

L

782

Tikhonov Regularization Tikhonov regularization is another much used method. In this a penalty is imposed on the 2-norm of kxk2 of the solution. GivenhA 2 Rmn a regularized least i 2 2 2 squares problem minx kAxbk2 C kxk2 is solved, where the parameter  governs the balance between a small residual and a smooth solution. In statistics Tikhonov regularization is known as “ridge regression.” The solution x./ D .ATA C 2 I /1 AT b can be computed by Cholesky factorization. In terms of the n X ci i SVD expansion, it is x./ D vi . Methods 2  C 2 i D1 i using QR factorization, which avoid forming the crossproduct matrix ATA, can also be used [1]. The optimal value of  depends on the noise level in the data. The choice of  is often a major difficulty in the solution process and often an ad hoc method is used; see [5]. In the LASSO (Least Absolute Shrinkage and Selection) method a constraint involving the one norm kxk1 is used instead. The resulting problem can be solved using convex optimization methods. LASSO tends to give solutions with fewer nonzero coefficients than Tikhonov regularization; see [9]. This property is fundamental for its use in compressed sensing.

References ˚ Numerical Methods For Least Squares Prob1. Bj¨orck, A.: lems, pp. xvii+408. SIAM, Philadelphia (1996). ISBN 089871-360-9 2. Bro, R., Eld´en, L.: PLS works. Chemometrics 23, 69–71 (2009) 3. Davis, T.A.: Direct Methods for Sparse Linear Systems, Fundamental of Algorithms, vol. 2. SIAM, Philadelphia (2006) 4. Fierro, R.D., Hansen, P.C., Hansen, P.S.K.: UTV tools: Matlab templates for rank-revealing UTV decompositions. Numer. Algorithms 20, 165–194 (1999) 5. Hansen, P.C.: Rank-Deficient and Discrete Ill-Posed Problems. In: Numerical Aspects of Linear Inversion, pp. x+224. SIAM, Philadelphia (1998). ISBN 978-0-898716-26-9 6. Lawson, C.L., Hanson, R.J.: Solving Least Squares Problems, pp. xii+337. Prentice-Hall, Englewood Cliffs (1974). Revised republication by SIAM, Philadelphia (1995). ISBN 0-89871-356-0 7. Paige, C.C., Saunders, M.A.: LSQR. An algorithm for sparse linear equations and sparse least squares. ACM Trans. Math. Softw. 8, 43–71 (1982) 8. Paige, C.C., Strakoˇs, Z.: Core problems in linear algebraic systems. SIAM J. Matrix Anal. Appl. 27(2), 861–875 (2006) 9. Tibshirani, R.: Regression shrinkage and selection via the LASSO. R. Stat. Soc. B. 58(1), 267–288 (1996)

Least Squares Finite Element Methods 10. Wold, S., Ruhe, A., Wold, H., Dunn, W.J.: The collinearity problem in linear regression, the partial least squares (pls) approach to generalized inverses. SIAM J. Sci. Stat. Comput. 5, 735–743 (1984)

Least Squares Finite Element Methods Pavel Bochev1 and Max Gunzburger2 1 Computational Mathematics, Sandia National Laboratories, Albuquerque, NM, USA 2 Department of Scientific Computing, Florida State University, Tallahassee, FL, USA

The root cause for the remarkable success of early finite element methods (FEMs) is their intrinsic connection with Rayleigh-Ritz principles. Yet, many partial differential equations (PDEs) are not associated with unconstrained minimization principles and give rise to less favorable settings for FEMs. Accordingly, there have been many efforts to develop FEMs for such PDEs that share some, if not all, of the attractive mathematical and algorithmic properties of the Rayleigh-Ritz setting. Least-squares principles achieve this by abandoning the naturally occurring variational principle in favor of an artificial, external energy-type principle. Residual minimization in suitable Hilbert spaces defines this principle. The resulting least-squares finite element methods (LSFEMs) consistently recover almost all of the advantages of the Rayleigh-Ritz setting over a wide range of problems, and with some additional effort, they can often create a completely analogous variational environment for FEMs. A more detailed presentation of least-squares finite element methods is given in [1]. Abstract LSFEM theory Consider the abstract PDE problem find u 2 X

such that

Lu D f

in Y;

(1)

where X and Y are Hilbert spaces, L W X 7! Y is a bounded linear operator, and f 2 Y is given data. Sandia National Laboratories is a multiprogram laboratory operated by the Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the US Department of Energy’s National Nuclear Sec- urity Administration under contract DE-AC04-94AL85000.

Least Squares Finite Element Methods

783

  Assume (1) to be well posed so that there exist positive for the unknown vector uE h , where Qhij D Ljh ; Lih Y   constants ˛ and ˇ such that and fEih D Li ; f Y . (2) Theorem 1 Assume that (2), or equivalently, (4), holds and that X h X . Then: The energy balance (2) is the starting point in the – The bilinear form Q.; / is continuous, symmetric, and strongly coercive. development of LSFEMs. It gives rise to the uncon– The linear functional F ./ is continuous. strained minimization problem, i.e., the least-squares – The problem (5) has a unique solution u 2 X that is principle (LSP): also the unique solution of (3).

– The problem (7) has a unique solution uh 2 X h that fJ; X g ! min J.uI f /; J.uI f / D kLu  f k2Y ; is also the unique solution of (6). u2X – The LSFEM approximation uh is optimally accurate (3) with respect to solution norm k  kX . for which (1) is where J.u; f / is the residual energy functional. well posed, i.e., for some constant C > 0 From (2), it follows that J.I / is norm equivalent: ˇkukX  kLukY  ˛kukX

8 u 2 X:

ˇ 2 kuk2X  J.uI 0/  ˛ 2 kuk2X

8 u 2 X:

(4)

Norm equivalence (4) and the Lax-Milgram Lemma imply that the Euler-Lagrange equation of (3)

ku  uh kX  C inf ku  v h kX vh 2X h

(9)

– The matrix Qh of (8) is symmetric and positive definite. 

Theorem 1 only assumes that (1) is well posed   and that X h is conforming. It does not require L to find u 2 X such that Lv; Lu Y  Q.u; w/ be positive self-adjoint as it would have to be in the   D F .w/  Lv; f Y 8 w 2 X (5) Rayleigh-Ritz setting, nor does it impose any compatibility conditions on X h that are typical of other is well posed because Q.u; w/ is an equivalent inner FEMs. Despite the generality allowed for in (1), the product on X X . The unique solution of (5), resp. (3), LSFEM based on (6) recovers all the desirable features coincides with the solution of (1). possessed by finite element methods in the RayleighWe define an LSFEM by restricting (3) to a family Ritz setting. This is what makes LSFEMs intriguing of finite element subspaces X h X , h ! 0. The and attractive. LSFEM approximation uh 2 X h to the solution u 2 X of (1) or (3) is the solution of the unconstrained Practical LSFEM Intuitively, a “practical” LSFEM minimization problem has coding complexity and conditioning comparable to that of other FEMs for the same PDE. The LSP fJ; X g

in (3) recreates a true Rayleigh-Ritz setting for (1), fJ; X h g ! min J.uh I f /; J.uI f /DkLuh f k2Y : h h yet the LSFEM fJ; X h g in (6) may be impractical. u 2X (6) Thus, sometimes it is necessary to replace fJ; X g by a To compute uh , we solve the Euler-Lagrange equation practical discrete alternative fJ h ; X h g. Two opposing corresponding to (6): forces affect the construction of fJ h ; X h g: a desire to keep the resulting LSFEM simple, efficient, and practical and a desire to recreate the true Rayleigh-Ritz find uh 2 X h such that Q.uh ; wh / setting. The latter requires J h to be as close as possible D F .wh /8 wh 2 X h : (7) to the “ideal” norm-equivalent setting in (3). The transformation of J.; / into a discrete funch Let fjh gN so that uh D tional J h .; / illustrates the interplay between these j D1 denote a basis for X PN h h j D1 uj j . Then, problem (7) is equivalent to the issues. To this end, it is illuminating to write the energy linear system of algebraic equations balance (2) in the form Qh uE h D fEh

(8)

C1 kSX uk0  kSY ı Luk0  C2 kSX uk0 ;

(10)

L

784

where SX ; SY are norm-generating operators for X; Y , respectively, with L2 ./ acting as a pivot space. At the least, practicality requires that the basis of X h can be constructed with no more difficulty than for Galerkin FEM for the same PDE. To secure this property, we ask that the domain D.SX / of SX contains “practical” discrete subspaces. Transformation of (1) into an equivalent first-order system PDE achieves this. Then, practicality of the “ideal” LSFEM (6) depends solely on the effort required to compute SY ı Luh . If this effort is deemed reasonable, the original energy norm jjjujjj D kSY ı Luk0 can be retained and the transition process is complete. Otherwise, we proceed to replace the composite operator SY ıL by a computable discrete approximation SYh ı Lh . We may need a projection operator h that maps the data f to the domain of SYh . The conversion process and the key properties of the resulting LSFEM can be encoded by the transition diagram

Least Squares Finite Element Methods

attractive properties of the Rayleigh-Ritz setting, including identical convergence rates and matrix condition numbers. A mesh-dependent normequivalence (12) distinguishes the quasi-normequivalent class, which admits the broadest range of LSFEMs, but can give problems with higher condition numbers. Examples We use the Poisson equation for which L D  to illustrate different classes of LSFEMs. One energy balance (2) for this equation corresponds to X D H 2 ./ \ H01 ./ and Y D L2 ./: ˛kuk2  kuk0  ˇkuk2 : The associated LSP fJ; X g !

min J.uI f /; J.uI f / D ku  f k20 u2X

J.uI f / D kSY ı .Lu  f /k20 ! jjjujjj leads to impractical LSFEMs because finite element subspaces of H 2 ./ are not easy to construct. # # # Transformation of u D f into the equivalent J h .uh I f / D kSYh ı.Lh uh  h f /k0 ! jjjuh jjjh (11) first-order system and the companion norm-equivalence diagram r  q D f and ru C q D 0 (13) C1 kukX  jjjujjj  C2 kukX (12) can solve this problem. The spaces X D H 1 ./ # # # 0 C1 .h/kuh kX  jjjuh jjjh  C2 .h/kuh kX : ŒL2 ./ d , Y D H 1 ./ ŒL2 ./ d have practical finite element subspaces and provide the energy balance Because L defines the problem being solved, the choice of Lh governs the accuracy of the LSFEM. The goal ˛.kuk1 C kqk0 /  kr  qk1 C kru C qk0 here is to make J h as close as possible to J for the exact solution of (1). On the other hand, SY defines the  ˇ.kuk1 C kqk0 /: energy balance of (1), i.e., the proper scaling between data and solution. As a result, the main objective in the This energy balance gives rise to the minus-one norm choice of SYh is to ensure that the scaling induced by J h LSP is as close as possible to (2), i.e., to “bind” the LSFEM ( to the energy balance of the PDE. fJ; X g ! min J.u; qI f /; J.u; qI f / .u;q/2X Taxonomy of LSFEMs Assuming that X h is ) practical, restriction of fJ; X g to X h transforms (3) 2 2 D kr  q  f k1 C kru C qk0 : into the compliant LSFEM fJ; X h g in (6). Apart from this “ideal” LSFEM which reproduces the classical Rayleigh-Ritz principle, there are two other (14) kinds of LSFEMs that gradually drift away from this setting, primarily by simplifying the approxi- However, (14) is still impractical because the normmations of the norm-generating operator SY . Mesh- generating operator SH 1 D ./1=2 is not independent C1 .h/ and C2 .h/ in (12) characterize the computable in general. The simple approximation h norm-equivalent class, which retains virtually all SH 1 D hI yields the weighted LSFEM

Levin Quadrature

785

( fJ ; X g ! h

)

h

min

J .u ; q I f /; J .u ; q I f / D h kr  q  f h

.uh ;qh /2X h

h

h

h

h

h

2

h

k20

C kru C h

qh k20

(15)

which is quasi-norm equivalent. The more accurate spectrally equivalent preconditioner for  gives the h h 1=2 approximation SH , where Kh is a discrete minus-one norm LSFEM 1 D hI C K

)

( fJ ; X g ! h

h

min

J .u ; q I f /; J .u ; q I f / D kr  q  f h

.uh ;qh /2X h

h

h

h

which is norm equivalent. The first-order system (13) also has the energy balance

h

h

h

k2h

C kru C h

qh k20

(16)

Levin Quadrature Sheehan Olver School of Mathematics and Statistics, The University of Sydney, Sydney, NSW, Australia

˛.kuk1 C kqkdiv /  kr  qk0 C kru C qk0  ˇ.kuk1 C kqkdiv /

Mathematics Subject Classification which corresponds to X D H01 ./ H.div; / and Y D L2 ./ ŒL2 ./ d . The associated LSP 65D30; 41A60 ( fJ; X g !

Synonyms

min J.u; qI f /; J.u; qI f /

.u;q/2X

) D kr  q  f

k20

C kru C

qk20

Levin rule; Levin-type method (17)

Short Definition is practical. Approximation of the scalar u by standard Levin quadrature is a method for computing highly nodal elements and of the vector q by div-conforming oscillatory integrals that does not use moments. elements, such as Raviart-Thomas, BDM, or BDFM, yields a compliant LSFEM which under some conditions has the exact same local conservation property as Description the mixed Galerkin method for (13). Levin quadrature is a method for calculating integrals of the form

Reference 1. Bochev, P., Gunzburger, M.: Least Squares Finite Element Methods. Springer, Berlin (2009)

Z

b

I Œf D

f .x/e i !g.x/ dx; a

L

786

Levin Quadrature

wherepf and g are suitably smooth functions, where ˝ Rd , x 2 Rd and f; g W Rd ! R. On i D 1, and ! is a large real number. rectangular domains ˝ D Œa; b Œc; d , this consists If u satisfies the differential equation of solving the PDE [1] u0 .x/ C i !g 0 .x/u.x/ D f .x/;

(1)

then

uxy C i !gy ux C i !gx uy C .i !gxy  ! 2 gx gy /u D f using collocation and approximating

I Œf D u.b/e i !g.b/  u.a/e i !g.a/: I Œf  u.b; d /e i !g.b;d /  u.a; d /e i !g.a;d /

In Levin quadrature we represent u

n X

u.b; c/e i !g.b;c/ C u.a; c/e i !g.a;c/ : ck

k .x/

For other domains, the dimension of the integral can be reduced by solving the PDE

kD1

for some basis 1 .x/; : : : ; n .x/, typically a polyr  u C i !rg  u D f; nomial basis such as monomials k .x/ D x k1 or Chebyshev polynomials k .x/ D Tk1 .x/. The coefficients c1 ; : : : ; cn are determined by solving (1) using a where u W Cd ! Cd , so that collocation method: for a sequence of points x1 ; : : : ; xn Z (such as Chebyshev points), solve the linear system e i !g u  d s: I Œf D @˝ n X

ck .

0 k .x1 /

C i !g 0 .x1 /

k .x1 //

D f .x1 /; : : : ;

ck .

0 k .xn /

C i !g 0 .xn /

k .xn //

D f .xn /:

Iterating the procedure reduces the integral to a univariate integral, at which point standard Levin quadrature is applicable [5]. Levin quadrature can be generalized to other oscillators which satisfy a linear differential equation, such as Bessel functions or Airy functions. We refer the reader to [2, 3, 6, 7].

kD1 n X kD1

We then have the approximation I Œf  QŒf D

n X

ck Œ

k .b/e

i !g.b/



k .a/e

i !g.a/

:

kD1

When g 0 .x/ ¤ 0 for x 2 .a; b/, a and b are including as collocation points and f is differentiable with bounded variation, then the error of approximating I Œf by QŒf decays like O.! 2 /: If f is m C 1 times differentiable and m collocation points are clustered like O.! 1 / near each endpoint, or if m derivatives at the endpoints are used in the collocation system, then the error decay improves to O.! m2 / [4]. The approach can be generalized to multivariate oscillatory integrals Z I Œf D

f .x/e i !g.x/d x; ˝

References 1. Levin, D.: Procedure for computing one- and twodimensional integrals of functions with rapid irregular oscillations. Math. Comp. 38, 531–538 (1982) 2. Levin, D.: Fast integration of rapidly oscillatory functions. J. Comput. Appl. Math. 67, 95–101 (1996) 3. Levin, D.: Analysis of a collocation method for integrating rapidly oscillatory functions. J. Comput. Appl. Math. 78, 131–138 (1997) 4. Olver, S.: Moment-free numerical integration of highly oscillatory functions. IMA J. Num. Anal. 26, 213–227 (2006) 5. Olver, S.: On the quadrature of multivariate highly oscillatory integrals over non-polytope domains. Numer. Math. 103, 643–665 (2006) 6. Olver, S.: Numerical approximation of vector-valued highly oscillatory integrals. BIT 47, 637–655 (2007) 7. Xiang, S.: Numerical analysis of a fast integration method for highly oscillatory functions. BIT 47, 469–482 (2007)

Lie Group Integrators

787

Lie Group Integrators Hans Z. Munthe-Kaas Department of Mathematics, University of Bergen, Bergen, Norway

Synopsis Lie group integrators (LGIs) are numerical time integration methods for differential equations evolving on smooth manifolds, where the time-stepping is computed from a Lie group acting on the domain. LGIs are constructed from basic mathematical operations in Lie algebras, Lie groups, and group actions. An extensive survey is found in [12]. Classical integrators (Runge-Kutta and multistep methods) can be understood as special cases of Lie group integrators, where the Euclidean space Rn acts upon itself by translation; thus in each time step, the solution is updated by adding an update vector, e.g., Euler method for y.t/ P D f .y.t//, for y; f .y/ 2 Rn steps forwards from t to t C h as

In the cases where the Lie groups are matrix groups, LGIs are numerical integrators based on matrix commutators and matrix exponentials and are thus related to exponential integrators. The general framework of LGI may also be applied in very general situations where Lie group actions are given in terms of differential equations. The performance of LGIs depends on how efficiently the basic operations can be computed and how well the Lie group action approximates the dynamics of the system to be solved. In many cases, a good choice of action leads to small local errors, and a higher cost per step can be compensated by the possibility of taking longer time steps, compared to classical integrators. Lie group methods are by construction preserving the structure of the underlying manifold M . Since all operations are intrinsic, it is not possible to drift off M . Furthermore, these methods are equivariant with respect to the group action, e.g., in the example of the sphere, the methods will not impose any particular coordinate system or orientation on the domain, and all points in the domain are treated equivalently.

Building Blocks

ynC1 D yn C hf .yn /:

Consider instead a differential equation evolving on the Applications of LGI generally involve the following surface of a sphere, zP.t/ D v.z/ z.t/; where z; v 2 R3 steps: and denotes the vector product. Let vO denote the hat 1. Choose a Lie group and Lie group action which can be computed fast and which captures some essential map, a skew-symmetric matrix given as features of the problem to be solved. This is similar 0 1 to the task of finding a preconditioner in iterative 0 v.3/ v.2/ solution of linear algebraic equations. @ A vO WD v.3/ 0 v.1/ ; (1) 2. Identify the Lie algebra, commutator, and exponenv.2/ v.1/ 0 tial map of the Lie group action. 3. Write the differential equation in terms of the inwe can write the equation as zP.t/ D v .z/z.t/. By finitesimal Lie algebra action, as in (2) below. freezing vO at zn , we obtain a step of the exponential 4. Choose a Lie group integrator, plug in all building Euler method as blocks, and solve the problem. We briefly review the definition of these objects and O n //zn : znC1 D exp.hv.z illustrate by examples below. A group is a set G with an identity element e 2 G and associative group product Here exp.hv.z O n // is the matrix exponential of a skew- a; b 7! ab such that every a 2 G has a multiplicative symmetric matrix. This is an orthogonal matrix which inverse a1 a D aa1 D e. A left group action of G on acts on the vector zn as a rotation, and hence znC1 sits a set M is a map W G M ! M such that e  p D p exactly on the sphere. This is the simplest (nonclassi- and .ab/  p D a  .b  p/ for all a; b 2 G and p 2 M . cal) example of a Lie group integrator. A Lie group is a group G which also has the structure

b

L

788

Lie Group Integrators

of a smooth differentiable manifold such that the map a; b 7! a1 b is smooth. If M also is a manifold, then a smooth group action is called a Lie group action. The Lie algebra g of a Lie group G is the tangent space of G at the identity e, i.e., g is the vector space obtained by taking the derivative at t D 0 of all smooth curves .t/ 2 G such that .0/ D e:

y.t/ P D f .t; y/  y;

y.0/ D y0 ;

(2)

for a given function f W R M ! g. A Lie group integrator is a numerical time-stepping procedure for (2) which is based on intrinsic Lie group operations, such as exponentials, commutators, and the group action on M .

g D fV D P .0/ W .t/ 2 G; .0/ D eg  Te G: By differentiation, we define the infinitesimal Lie al- Methods (Examples) gebra action W g M ! TM which for V 2 g and Lie Euler: ynC1 D exp.hf .tn ; yn //  yn . p 2 M produces a tangent V  p 2 Tp M as Lie midpoint: ˇ @ ˇˇ V p D ..t/  p/ 2 Tp M; where V D P .0/. K D hf .tn C h=2; exp .K=2/  yn / @t ˇt D0 ynC1 D exp.K/  yn The exponential map expW g ! G is the t D 1 flow of the infinitesimal action; more precisely, we define exp.V / 2 G as exp.V / WD y.1/, where y.t/ 2 G is Lie RK4: There are several similar ways of turning the classical RK4 method into a 4 order Lie group the solution of the initial value problem integrator [16,18]. The following version requires only two commutators: y.t/ P D V  y.t/; y.0/ D e: The final operation we need in order to define a Lie group method is the commutator or Lie bracket, a bilinear map Œ;  W g g ! g defined for V; W 2 g as ˇ @2 ˇˇ ŒV; W D exp.sV / exp.tW / exp.sV /: @s@t ˇsDt D0 The commutator measures infinitesimally the extent to which two flows exp.sV / and exp.tW / fail to commute. We denote adV the linear operator W 7! ŒV; W W g ! g. In the important case where G is a matrix Lie group, the exponential is the matrix exponential and the commutator is the matrix commutator ŒV; W D V W  W V . If G acts on a vector space M by matrix multiplication a  p D ap, then also the infinitesimal Lie algebra action V  p D Vp is given by matrix multiplication.

K1 D hf .tn ; yn / K2 D hf .tn =2; exp.K1 =2/  yn / K3 D hf .tn C h=2; exp.K2 =2  ŒK1 ; K2 =8/  yn / K4 D hf .tn C h=2; exp.K3 /  yn / ynC1 D exp .K1 =6 C K2 =3 C K3 =3 C K4 =6 ŒK1 ; K2 =3  ŒK1 ; K4 =12/  yn RKMK methods: This is a general procedure to turn any classical Runge-Kutta method into a Lie group integrator of the same order. Given the coefficients aj;` ; bj ; cj of an s-stage and pth order RK method, a single step y.tn /  yn 7! ynC1  y.tn C h/ is given as Uj D

s X

9 > > > > > =

aj;` K`

`D1

Definition Given a smooth manifold M and a Lie group G with Lie algebra g acting on M . Consider a differential equation for y.t/ 2 M written in terms of the infinitesimal action as

Fj D hf .tn C cj h; exp.Uj /  yn / Kj D d exp1 Uj .Fj / ynC1 D exp

s X `D1

!

b` K`  yn ;

> > > > > ;

j D 1; : : : ; s

Lie Group Integrators 1 1 where d exp1 Uj .Fj /DFj  2 ŒUj ; Fj C 12 ŒUj ; ŒUj ; Fj

P Bj p j 1 ad4Uj Fj C    D  720 j D0 j Š adUj Fj is the inverse of the Darboux derivative of the exponential map, truncated to the order of the method and Bj are the Bernoulli numbers [12, 17].

Crouch-Grossman and commutator-free methods: Commutators pose a problem in the application of Lie group integrators to stiff equations, since the commutator often increases the stiffness of the equations dramatically. Crouch-Grossman [6, 19] and more generally commutator-free methods [5] avoid commutators by doing basic time-stepping using a composition of exponentials. An example of such a method is CF4 [5]: K1 D hf .tn ; yn / K2 D hf .tn =2; exp.K1 =2/  yn / K3 D hf .tn C h=2; exp.K2 =2/  yn / K4 D hf .tn C h=2; exp.K1 =2/  exp.K3  K1 =2/  yn / ynC1 D exp .K1 =4 C K2 =6 C K3 =6  K4 =12/  exp .K2 =6 C K3 =6 C K4 =4  K1 =12/  yn Magnus methods: In the case where f .t; y/ D f .t/ is a function of time alone, then (2) is called an equation of Lie type. Specialized numerical methods have been developed for such problems [1, 10]. Explicit Magnus methods can achieve order 2p using only p function evaluations, and they are also easily designed to be time symmetric.

789

b

in terms of the infinitesimal Lie algebra action, the differential equation becomes yP D v.y/y, and we may apply any Lie group integrator. Note that for lowdimensional rotational problems, all basic operations can be computed fast using Rodrigues-type formulas [12]. Isospectral action: Isospectral differential equations are matrix-valued equations where the eigenvalues are first integrals (invariants of motion). Consider M D Rnn and the action of G D SO.n/ on M by similarity transforms, i.e., for a 2 G and y 2 M , we define a  y D ayaT . By differentiation, of the action we find the infinitesimal action for V 2 g D so.n/ as V  y D Vy  yV ; thus for this action, (2) becomes y.t/ P D f .t; y/  y D f .t; y/y  yf .t; y/; where f W R M ! g. See [2, 12] for more details. Affine action: Let G D Gl.n/ Ì Rn be the affine linear group, consisting of all pairs a; b where a 2 Rnn is an invertible matrix and b 2 Rn is a vector. The affine action of G on M D Rn is .a; b/  y D ay C b. The Lie algebra of G is g D gl.n/ Ì Rn , i.e., g consists of all pairs .V; b/ where V 2 Rnn and b 2 Rn . The infinitesimal action is given as .V; b/  y D Vy C b. This action is useful for differential equations of the form y.t/ P D L.t/y CN.y/, where L.t/ is a stiff linear part and N is a nonstiff nonlinear part. Such equations are cast in the form (2) by choosing f .t; y/ D .L.t/; N.y//. Applications of Lie group integrators to such problems are closely related to exponential integrators. For stiff equations it is important to use a commutator-free Lie group method.

Coadjoint action: Many problems of computational mechanics are naturally formulated as Lie-Poisson Lie Group Actions (Examples) systems, evolving on coadjoint orbits of the dual of a Rotational problems: Consider a differential equation Lie algebra [14]. Lie group integrators based on the y.t/ P D v.y.t// y.t/, where y; v 2 R2 and jjy.0/jj D coadjoint action of a Lie group on the dual of its Lie 1. Since jjy.t/jj D 1 for all t, we can take M to be algebra are discussed in [7]. the surface of the unit sphere. Let G D SO.3/ be the Classical integrators as Lie group integrators: The special orthogonal group, consisting of all orthogonal simplest of all group actions is when G D M D Rn , matrices with determinant 1. Let .t/ 2 G be a curve with vector addition as group operation and group such that .0/ D e. By differentiating .t/T .t/ D e, action. From the definitions, we find that in this case we find that P .0/T C .0/ D 0, thus g D so.3/, the set g D Rn , the commutator is 0, and the exponential map of all skew-symmetric 3 3 matrices. The infinitesimal is the identity map from Rn to itself. The infinitesimal Lie algebra action is left multiplication with a skew Lie algebra action becomes V  y D V ; thus, (2) P D f .t; y/, where f .t; y/ 2 Rn . We see matrix, the commutator is the matrix commutator, and reduces to y.t/ the exponential map is the matrix exponential. Written that classical integration methods are special cases of

L

790

Linear Elastostatics

Lie group integrators, and all the examples of methods 14. Marsden, J., Rat¸iu, T.: Introduction to Mechanics and Symmetry: A Basic Exposition of Classical Mechanical above reduce to well-known Runge-Kutta methods.

Implementation Issues For efficient implementation of LGI, it is important to employ fast algorithms for computing commutators and exponentials. A significant volume of research has been devoted to this. Important techniques involve replacing the exponential map with other coordinate maps on Lie groups [13, 20]. For special groups, there exist specialized algorithms for computing matrix exponentials [4, 21]. Time reversible LGI is discussed in [22], but these are all implicit methods and thus costly. Optimization of the number of commutators and exponentials has been considered in [3, 18].

References 1. Blanes, S., Casas, F., Oteo, J., Ros, J.: The Magnus expansion and some of its applications. Phys. Rep. 470(5–6), 151–238 (2009) 2. Calvo, M., Iserles, A., Zanna, A.: Numerical solution of isospectral flows. Math. Comput. 66(220), 1461–1486 (1997) 3. Casas, F., Owren, B.: Cost efficient Lie group integrators in the RKMK class. BIT Numer. Math. 43(4), 723–742 (2003) 4. Celledoni, E., Iserles, A.: Methods for the approximation of the matrix exponential in a Lie-algebraic setting. IMA J. Numer. Anal. 21(2), 463 (2001) 5. Celledoni, E., Marthinsen, A., Owren, B.: Commutatorfree Lie group methods. Future Gen. Comput. Syst. 19(3), 341–352 (2003) 6. Crouch, P., Grossman, R.: Numerical integration of ordinary differential equations on manifolds. J. Nonlinear Sci. 3(1), 1–33 (1993) 7. Engø, K., Faltinsen, S.: Numerical integration of LiePoisson systems while preserving coadjoint orbits and energy. SIAM J. Numer. Anal. 39, 128–145 (2002) 8. Hairer, E., Lubich, C., Wanner, G.: Geometric Numerical Integration: Structure-Preserving Algorithms for Ordinary Differential Equations, vol 31. Springer, Berlin/New York (2006) 9. Hochbruck, M., Ostermann, A.: Exponential integrators. Acta Numer. 19, 209–286 (2010) 10. Iserles, A., Nørsett, S.: On the solution of linear differential equations in Lie groups. Philos. Trans. A 357(1754), 983 (1999) 11. Iserles, A., Zanna, A.: Efficient computation of the matrix exponential by generalized polar decompositions. SIAM J. Numer. Anal. 42, 2218–2256 (2005) 12. Iserles, A., Munthe-Kaas, H., Nørsett, S., Zanna, A.: Liegroup methods. Acta Numer. 9(1), 215–365 (2000) 13. Krogstad, S., Munthe-Kaas, H., Zanna, A.: Generalized polar coordinates on Lie groups and numerical integrators. Numer. Math. 114(1), 161–187 (2009)

Systems. Springer, New York (1999) 15. Munthe-Kaas, H.: Lie-Butcher theory for Runge-Kutta methods. BIT Numer. Math. 35(4), 572–587 (1995) 16. Munthe-Kaas, H.: Runge-Kutta methods on Lie groups. BIT Numer. Math. 38(1), 92–111 (1998) 17. Munthe-Kaas, H.: High order Runge-Kutta methods on manifolds. Appl. Numer. Math. 29(1), 115–127 (1999) 18. Munthe-Kaas, H., Owren, B.: Computations in a free Lie algebra. Philos. Trans. R. Soc. Lond. Ser. A 357(1754), 957 (1999) 19. Owren, B., Marthinsen, A.: Runge-Kutta methods adapted to manifolds and based on rigid frames. BIT Numer. Math. 39(1), 116–142 (1999) 20. Owren, B., Marthinsen, A.: Integration methods based on canonical coordinates of the second kind. Numer. Math. 87(4), 763–790 (2001) 21. Zanna, A., Munthe-Kaas, H.: Generalized polar decompositions for the approximation of the matrix exponential. SIAM J. Matrix Anal. Appl. 23, 840 (2002) 22. Zanna, A., Engø, K., Munthe-Kaas, H.: Adjoint and selfadjoint Lie-group methods. BIT Numer. Math. 41(2), 395–421 (2001)

Linear Elastostatics Tarek I. Zohdi Department of Mechanical Engineering, University of California, Berkeley, CA, USA

Notation Throughout this work, boldface symbols denote vectors or tensors. For the inner product of two vectors (first-order tensors), u and v, we have u  v D ui vi D u1 v1 C u2 v2 C u3 v3 in three dimensions, where Cartesian basis and Einstein index summation notation are used. In this introduction, for clarity of presentation, we will ignore the difference between second-order tensors and matrices. Furthermore, we exclusively employ a Cartesian basis. Accordingly, if we consider the second-order tensor A D Aik ei ˝ ek , then a first-order contraction (inner product) of two second-order tensors A  B is defined by the matrix product ŒA ŒB , with components of Aij Bjk D Cik . It is clear that the range of the inner index j must be the same for ŒA and ŒB . For three dimensions, we have i; j D 1; 2; 3. The second-order inner product of two tensors or matrices is defined as A W B D Aij Bij D tr.ŒA T ŒB /.

Linear Elastostatics

791

Kinematics of Deformations The term deformation refers to a change in the shape of a continuum between a reference configuration and current configuration. In the reference configuration, a representative particle of a continuum occupies a point P in space and has the position vector (Fig. 1) X D X 1 e1 C X 2 e2 C X 3 e3 ;

(1)

where e1 ; e2 ; e3 is a Cartesian reference triad and X1 ; X2 ; X3 (with center O) can be thought of as labels for a material point. Sometimes the coordinates or labels .X1 ; X2 ; X3 / are called the referential or material coordinates. In the current configuration, the particle originally located at point P (at time t D 0) is located at point P 0 and can be also expressed in terms of another position vector x, with coordinates .x1 ; x2 ; x3 /. These are called the current coordinates. In this framework, the displacement is u D x  X for a point originally at X and with final coordinates x. When a continuum undergoes deformation (or flow), its points move along various paths in space. This motion may be expressed as a function of X and t as (Frequently, analysts consider the referential configuration to be fixed in time, thus, X ¤ X.t/.) x.X; t/ D u.X; t/ C X.t/ ;

(2)

interpreted as a mapping of the initial configuration onto the current configuration. In classical approaches, it is assumed that such a mapping is one to one and continuous, with continuous partial derivatives to whatever order is required. The description of motion or deformation expressed previously is known as the Lagrangian formulation. Alternatively, if the independent variables are the coordinates x and time t, then x.x1 ; x2 ; x3 ; t/ D u.x1 ; x2 ; x3 ; t/ C X.x1 ; x2 ; x3 ; t/, and the formulation is denoted as Eulerian (Fig. 1). Deformation of Line Elements Partial differentiation of the displacement vector u D x  X, with respect to X, produces the following displacement gradient: rX u D F  1;

(3)

where 2

@x1 6 6 @X1 6 6 @x2 def def @x D6 F D rX x D 6 @X1 @X 6 6 @x 4 3 @X1

@x1 @X2 @x2 @X2 @x3 @X2

3 @x1 7 @X3 7 7 @x2 7 7: @X3 7 7 @x3 7 5 @X3

L (4)

which gives the present location of a point at time F is known as the material deformation gradient. Now, consider the length of a differential element in t, written in terms of the referential coordinates X1 ; X2 ; X3 . The previous position vector may be the reference configuration d X and d x in the current configuration, d x D rX x  d X D F  d X. Taking the difference in the squared magnitudes of these elements X3, x3 yields u+du

d x  d x  d X  d X D .rX x  d X/  .rX x  d X/ d X  d X

dx dX

X+dX

u

P X O

D d X  .FT  F  1/  d X

P’

def

x

D 2 d X  E  d X:

X2, x2

(5)

Equation (5) defines the so-called strain tensor: 1 T .F  F  1/ 2 1 D ŒrX u C .rX u/T C .rX u/T  rX u : 2

def

X1, x1

Linear Elastostatics, Fig. 1 Different descriptions of a deforming body

ED

(6)

792

Linear Elastostatics

Remark 1 It should be clear that d x can be reinterpreted as the result of a mapping F  d X ! d x or a change in configuration (reference to current). One may develop so-called Eulerian formulations, employing the current configuration coordinates to generate Eulerian strain tensor measures. An important quantity is the Jacobian of the deformation gradient, def J D det F, which relates differential volumes in the reference configuration (d!0 ) to differential volumes in the current configuration (d!) via d! D J d!0 . The Jacobian of the deformation gradient must remain positive; otherwise, we obtain physically impossible “negative” volumes. For more details, we refer the reader to the texts of Malvern [3], Gurtin [2], and Chandrasekharaiah and Debnath [1].

t.n/A.n/ C t.1/ A.1/ C t.2/ A.2/ C t.3/ A.3/ CbV D V uR ;

(8)

where A.n/ is the surface area of the face of the tetrahedron with normal n and V is the tetrahedron volume. As the distance (h) between the tetrahedron base (located at (0,0,0)) and the surface center goes to V zero (h ! 0), we have A.n/ ! 0 ) A .n/ ! 0. def

.i /

A Geometrically, we have A .n/ D cos.xi ; xn / D ni , and .n/ .1/ cos.x1 ; xn / C t.2/ cos.x2 ; xn / C therefore t C t .3/ cos.x3 ; xn / D 0. It is clear that forces on the t surface areas could be decomposed into three linearly independent components. It is convenient to introduce the concept of stress at a point, representing the surface forces there, pictorially represented by a cube surrounding a point. The fundamental issue that must be Equilibrium/Kinetics of Solid Continua resolved is the characterization of these surface forces. We can represent the surface force density vector, The balance of linear momentum in the deformed the so-called traction, on a surface by the component (current) configuration is representation: 8 9 ˆ Z Z Z =

d .i / def (9) t D i 2 ; t da C b d! D uP d! ; (7) ˆ dt ; : > „@!ƒ‚ … „! ƒ‚ … „ !ƒ‚ … i 3 surface forces

body forces

inertial forces

where !  is an arbitrary portion of the continuum, with boundary @!,  is the material density, b is the body force per unit mass, and uP is the time derivative of the displacement. The force densities, t, are commonly referred to as “surface forces” or tractions.

where the second index represents the direction of the component and the first index represents components of the normal to corresponding coordinate plane. Henceforth, we will drop the superscript notation of def t.n/ , where it is implicit that t D t.n/ D  T  n, where 3 2 11 12 13 def 6 7  D 421 22 23 5 ; 31 32 33

Postulates on Volume and Surface Quantities Now, consider a tetrahedron in equilibrium, as shown in Fig. 2, where a balance of forces yields x2

σ22 σ21 σ21 σ23 σ11 σ

t(n) t(-1)

t(-3)

32

x2

x1

x3

t(-2)

σ33

σ31

x1 x3

Linear Elastostatics, Fig. 2 (Left) Cauchy tetrahedron: a “sectioned point” and (Right) stress at a point.

σ13

(10)

Linear Elastostatics

793

or explicitly (t.1/ D t.1/ , t.2/ D t.2/ , t.3/ D t.3/ ) t D t.1/ n1 C t.2/ n2 C t.3/ n3 D  T  n 3T 8 9 2 11 12 13 ˆ

= 7 6 n2 ; D 421 22 23 5 ˆ : > ; 31 32 33 n3

(11)

Remark 2 In the absence of couple stresses, a balance of angular momentum implies a symmetry of stress,  D  T, and thus the difference in notations becomes immaterial. Explicitly, starting with an angular momentum balance, under the assumptions that no infinitesimal “micro-moments” or so-called couple stresses exist, then it can Rbe shown that Rthe stress tensor x t da C ! x b d! D must R be symmetric, i.e., @! d T x uP d!; that is,  D  . It is somewhat easier dt ! to consider a differential element, such as in Fig. 2, and to simply sum moments about the center. Doing this, one immediately obtains 12 D 21 ; 23 D 32 , and 13 D 31 . Consequently, t D   n D  T  n. Balance Law Formulations Substitution of (11) into (7) yields .! / Z

Z d   n da C b d! D uP d! : dt „@! ƒ‚ … „! ƒ‚ … „ !ƒ‚ … surface forces

body forces

inertial forces

(13)

The First Law of Thermodynamics: An Energy Balance

where  is the so-called Cauchy stress tensor.

Z

R rx   C b D u:

(12)

The interconversions of mechanical, thermal, and chemical energy in a system are governed by the first law of thermodynamics, which states that the time rate of change of the total energy, K C I, is equal to the mechanical power, P, and the net heat d .K C I/ D P C H C Q. supplied, H C Q, i.e., dt Here the kinetic energy of a subvolume of material def R contained in ˝, denoted !, is K D ! 12 uP  uP d!; the power (rate of work) of the external forces acting on ! R def R is given by P D ! b  uP d! C @!   n  uP da; the heat flow into the volume by conduction is R R def Q D  @! q  n da D  ! rx  q d!, q being the heat flux; the heat generated due to sources, such as def R chemical reactions, is H D ! z d!, where z is the reaction source rate per unit mass; and the internal def R energy is I D ! w d!, w being the internal energy per unit mass. Differentiating the kinetic energy yields Z dK d 1 D uP  uP d! dt dt ! 2 Z d 1 P d!0 .J uP  u/ D !0 dt 2  Z

d 1 uP  uP d!0 0 D dt 2 !0 Z d 1 P d! .uP  u/ C  dt 2 ! Z D uP  uR d!;

A relationship can be determined between the densities in the configurations, R current and reference R R d! D Jd! D  d! 0 0 . Therefore, ! !0 !0 0 the Jacobian can also be interpreted as the ratio of (14) ! material densities at a point. Since the volume is arbitrary, we can assume that J D 0 holds at every point in the body. Therefore, we may write where we have assumed that the mass in the system is d d constant. We also have dt .0 / D dt .J / D 0, when the system is mass conservative over time. This leads to writing the last R R Z Z / d P d! D !0 d.J uP d!0 C d I term in (12) as Rdt d d ! u dt R D w d! D J w d!0 R d!0 D ! uR d!. From Gauss’s divergence dt dt ! dt !0 !0  uJ theorem and an implicit assumption that  is Z Z Z R d R d! D differentiable, we have ! .rx   C b  u/ .0 / w d!0 C w D P d! D w P d!: dtƒ‚ … !0 „ ! ! 0. If the volume is argued as being arbitrary, then D0 the integrand must be equal to zero at every point, (15) yielding

L

794

Linear Elastostatics

1 D .ru C .ru/T /: 2

By using the divergence theorem, we obtain Z

Z   n  uP da D @!

(19)

P d! rx  .  u/ !

Z

Z

Linear Elastic Constitutive Laws P D D .rx   /  uP d! C  W rx uP d!: If we neglect thermal effects, (18) implies w ! ! P  W rx u which, in the infinitesimal strain linearly P D  W P . From the chain rule of (16) elastic case, is w differentiation, we have

Combining the results, and enforcing a balance of linear momentum, leads to w P D

Z

@w @w d  W D  W P )  D  : @ dt @

(20)

.w P C uP  .uR  rx    b/ !

 W rx uP C rx  q  z/ d! (17) The starting point to develop a constitutive theory is Z to assume a stored elastic energy function exists, a D .w P   W rx uP C rx  q  z/ d! D 0: def ! function denoted W D w, which depends only on the mechanical deformation. The simplest function that Since the volume ! is arbitrary, the integrand must fulfills  D  @w is W D 1  W IE W , where IE is the @ 2 hold locally and we have fourth-rank elasticity tensor. Such a function satisfies the intuitive physical requirement that, for any small w P   W rx uP C rx  q  z D 0: (18) strain from an undeformed state, energy must be stored in the material. Alternatively, a small strain material @W When dealing with multifield problems, this equation law can be derived from  D @ and W  c0 C c1 W 1  C 2  W IE W  C : : : which implies   c1 C IE W is used extensively.  C : : :. We are free to set c0 D 0 (it is arbitrary) in order to have zero strain energy at zero strain, and, furthermore, we assume that no stresses exist in the Linearly Elastic Constitutive Equations reference state (c1 D 0). With these assumptions, we obtain the familiar relation We now discuss relationships between the stress and strain, so-called material laws or constitutive relations for linearly elastic cases (infinitesimal deformations). The Infinitesimal Strain Case In infinitesimal deformation theory, the displacement gradient components are considered small enough that higher-order terms like .rX u/T  rX u and .rx u/T  rx u can be neglected in the strain measure E D 1 T T 2 .rX u C .rX u/ C .rX u/  rX u/, leading to E  def

 D 12 ŒrX u C .rX u/T . If the displacement gradients are small compared with unity,  coincides closely to @ @  @x , we may use E or E. If we assume that @X  interchangeably. Usually  is the symbol used for infinitesimal strains. Furthermore, to avoid confusion, when using models employing the geometrically linear infinitesimal strain assumption, we use the symbol of r with no X or x subscript. Hence, the infinitesimal strains are defined by

 D IE W :

(21)

This is a linear relation between stresses and strains. The existence of a strictly positive stored energy function in the reference configuration implies that the linear elasticity tensor must have positive eigenvalues at every point in the body. Typically, different materials are classified according to the number of independent components in IE. In theory, IE has 81 components, since it is a fourth-order tensor relating 9 components of stress to strain. However, the number of components can be reduced to 36 since the stress and strain tensors are symmetric. This is observed from the matrix representation (The symbol Πis used to indicate the matrix notation equivalent to a tensor form, while fg is used to indicate the vector representation.) of IE:

Linear Elastostatics

795

8 9 2 E1111 11 > ˆ ˆ ˆ > > ˆ > 6E2211 ˆ > 22 ˆ < > = 6 6E3311 33 D6 6E1211 12 > ˆ 6 ˆ > ˆ > ˆ > 4E2311 ˆ 23 > ˆ > : ; 31 E1311 „ ƒ‚ … „ def Df g

E1122 E2222 E3322 E1222 E2322 E1322

E1133 E1112 E2233 E2212 E3333 E3312 E1233 E1212 E2333 E2312 E1333 E1312 ƒ‚ def D ŒIE

The existence of a scalar energy function forces IE to be symmetric since the strains are symmetric; in other words, W D 12  W IE W  D 12 . W IE W /T D 12 T W IET W T D 12  W IET W  which implies IET D IE. Consequently, IE has only 21 independent components. The nonnegativity of W imposes the restriction that IE remains positive definite. At this point, based on many factors that depend on the material microstructure, it can be shown that the components of IE may be written in terms of anywhere between 21 and 2 independent parameters. Accordingly, for isotropic materials, we have two planes of symmetry and an infinite number of planes of directional independence (two free components), yielding 2

4 2 2 C     0 6 3 3 3 6 6 2 4 2 6    C     0 6 3 3 3 6 6 2 2 4 def 6        C  0 IE D 6 6 3 3 3 6 6 0 0 0  6 6 6 6 0 0 0 0 4 0

0

0

3 0 0 0 0 0 

7 7 7 07 7 7 7 7 07 : 7 7 07 7 7 7 07 5

0 0 

E1123 E2223 E3323 E1223 E2323 E1323

9 38 11 > E1113 ˆ > ˆ ˆ > ˆ 22 > E2213 7 > 7ˆ > ˆ = < 7 E3313 7 33 : 7 E1213 7 ˆ 212 > > ˆ > ˆ > ˆ E2313 5 ˆ 2 > ˆ ; : 23 > E1313 231 … „ ƒ‚ … def Dfg

(22)

All of the material components of IE may be spatially variable, as in the case of composite media. Material Component Interpretation There are a variety of ways to write isotropic constitutive laws, each time with a physically meaningful pair of material values. Splitting the Strain It is sometimes important to split infinitesimal strains into two physically meaningful parts:

 tr  tr 1C  1 : D 3 3

L (25)

An expansion of the Jacobian of the deformation gradient yields J D det.1 C rX u/  1 C trrX u C O.rX u/ D 1 C tr C : : :. Therefore, with infinitesimal strains, .1 C tr/d!0 D d!, and we can write tr D d!d!0 . Hence, tr is associated with the volumetric d!0 (23) part of the deformation. Furthermore, since 0 def D  tr 1, the so-called strain deviator describes distortion 3 in the material. Infinitesimal Strain Material Laws The stress  can be split into two parts (dilatational and a deviatoric):

In this case, we have  D

 tr  def tr 1C   1 D p1 C  0 ; 3 3

(26)  tr 2 tr IE W  D 3 1 C 20 )  W IE W  D 9 3 3 where we call the symbol p the hydrostatic pressure 0 0 and  0 the stress deviator. With (24), we write C2 W  ; (24)  tr 

and  0 D 2  0 : (27) is the deviatoric where tr D ii and  D   3 strain. The eigenvalues of an isotropic elasticity tensor are .3; 2; 2; ; ; /. Therefore, we must have This is one form of Hooke’s law. The resistance to  > 0 and  > 0 to retain positive definiteness of IE. change in the volume is measured by . We note that 0

1 3 .tr/1

p D 3 

796

Linear Programming

. tr3 1/0 D 0, which indicates that this part of the stress References produces no distortion. 1. Chandrasekharaiah, D.S., Debnath, L.: Continuum MechanAnother fundamental form of Hooke’s law is  D

E 1C

 C



 .tr/1 ; 1  2

(28)

ics. Academic, Boston (1994) 2. Gurtin, M.: An Introduction to Continuum Mechanics. Academic, New York (1981) 3. Malvern, L.: Introduction to the Mechanics of a Continuous Medium. Prentice Hall, Englewood Cliffs (1968)

and the inverse form D

1C    .tr /1 : E E

(29)

To interpret the material values, consider an idealized uniaxial tension test (pulled in the x1 direction inducing a uniform stress state) where 12 D 13 D 23 D 0, which implies 12 D 13 D 23 D 0. Also, we have 22 D 33 D 0. Under these conditions, we have 11 D E11 and 22 D 33 D 11 . Therefore, E, Young’s modulus, is the ratio of the uniaxial stress to the corresponding strain component. The Poisson ratio, , is the ratio of the transverse strains to the uniaxial strain. Another commonly used set of stress-strain forms is the Lam´e relations:  D .tr/1 C 2

or

 D .tr 1/ C : 2.3 C 2/ 2

Linear Programming Robert J. Vanderbei Department of Operations Research and Financial Engineering, Princeton University, Princeton, NJ, USA

Mathematics Subject Classification Primary 90C05; Secondary 49N15

Synonyms (30) Linear optimization (LP)

To interpret the material values, consider a homogeneous pressure test (uniform stress) where 12 D 13 D Short Definition 23 D 0, and where 11 D 22 D 33 . Under these conditions, we have The Linear Programming Problem (LP) is the problem of maximizing or minimizing a linear function of one E E 2 and D ; or more, and typically thousands of, variables subject  D C  D 3 3.1  2/ 2.1 C / to a similarly large number of equality and/or inequal(31) ity constraints. and consequently,  2.1 C / D :  3.1  2/

(32)

We observe that  ! 1 implies  ! 12 and  ! 0 implies )  ! 1. Therefore, since both  and  must be positive and finite, this implies 1 <  < 1=2 and 0 < E < 1. For example, some polymeric foams exhibit  < 0, steels   0:3, and some forms of rubber have  ! 1=2. We note that can be positive or negative. For more details, see Malvern [3], Gurtin [2], and Chandrasekharaiah and Debnath [1].

Description Although Leonid Kantorovich [3] is generally credited with being the first to recognize the importance of linear programming as a tool for solving many practical operational problems, much credit goes to George Dantzig for independently coming to this realization a few years later (see [1, 2]). Originally, most applications arose out of military operations. However, it was quickly appreciated that important applications

Linear Programming

797

the dual problem and c T x D b T y, then x is optimal for the primal problem and y is optimal for the dual problem. There is also a strong duality theorem. It says that, if x is optimal for the primal problem, then there exists a y that is optimal for the dual problem and the two maximize c T x objective function values agree: c T x D b T y. subject to Ax  b (1) All algorithms for linear programming are based on x  0: simultaneously finding an optimal solution for both the primal and the dual problem (or showing that either Here, A is an m n matrix whose .i; j /-th element that the primal problem is infeasible or unbounded). is ai;j , b is an m-vector whose i -th element is bi , The value of the dual is that it proves that the primal and c is an n-vector whose j -th element is cj . The solution is optimal. linear function c T x is called the objective function. A particular choice of x is said to be feasible if it satisfies Slack Variables and Complementarity the constraints of the problem. It is useful to introduce slack variables into the primal It is easy to convert any linear programming prob- and dual problems so that all inequalities are simple lem into an equivalent one in standard form. For nonnegativities: example, any greater-than-or-equal-to constraint can be converted to a less-than-or-equal-to constraint by Primal Problem: multiplying by minus one, any equality constraint can maximize cT x be replaced with a pair of inequality constraints, a subject to Ax C w D b minimization problem can be converted to maximizax; w  0: tion by negating the objective function, and every appear in all areas of science, engineering, and business analytics. A problem is said to be in symmetric standard form if all the constraints are inequalities and all of the variables are nonnegative:

unconstrained variable can be replaced by a difference of two nonnegative variables. Duality Associated with every linear programming problem is a dual problem. The dual problem associated with (1) is minimize b T y subject to AT y  c (2) y  0: Written in standard form, the dual problem is maximize b T y subject to AT y  c y0 : From this form we see that the dual of the dual is the primal. We also see that the dual problem is in some sense the negative-transpose of the primal problem. The weak duality theorem states that, if x is feasible for the primal problem and y is feasible for the dual problem, then c T x  b T y. The proof is trivial: c T x  y T Ax  y T b. The weak duality theorem is useful in that it provides a certificate of optimality: if x is feasible for the primal problem and y is feasible for

Dual Problem: minimize bT y T subject to A y  z D c y; z  0: It is trivial to check that .c C z/T x D y T Ax D y T .b  w/. Hence, if x and w are feasible for the primal problem and y and z are feasible for the dual problem and c T x D b T y, then it follows that x is optimal for the primal, y is optimal for the dual and zT x C y T w D 0. Since all of the terms in these inner products are nonnegative, it follows that zj xj D 0

for all j

and

yi wi D 0 for all i:

This condition is called complementarity. Geometry The feasible set is an n-dimensional polytope defined by the intersection of n C m halfspaces where each halfspace is determined either by one of the m constraint inequalities, Ax  b, or one of the n nonnegativity constraints on the variables, x  0. Generally speaking, the vertices of this polytope

L

798

Linear Programming

correspond to the intersection of n hyperplanes defined as the boundaries of a specific choice of n out of the n C m halfspaces. Except in degenerate cases, the optimal solution to the LP occurs at one of the vertices. Ignoring, momentarily, which side of a hyperplane is feasible and which is not, the n C m hyperplanes generate up to .n C m/Š=nŠmŠ possible vertices corresponding to the many ways that one can choose n hyperplanes from the n C m. Assuming that these points of intersection are disjoint one from the other, these points in n-space are called basic solutions. The intersections that lie on the feasible set itself are called basic feasible solutions.

While doing jumps from one basic solution to another, this system of equations is rearranged so that the basic variables always remain on the left and the nonbasics appear on the right. Down the road, these equations become (5) xB D xB  B 1 N xN

where B denotes the m m invertible matrix consisting of the columns of the matrix ŒA I associated with the basic variables B, N denotes those columns of that matrix associated with the nonbasic variables N , and xB D B 1 b. Equation (5) is called a primal dictionary because it defines the primal basic variables in terms of the primal nonbasic variables. The process of updating equation (5) from one iteration to the next is called a simplex pivot. Simplex Methods Associated with each dictionary is a basic solution Inspired by the geometric view of the problem, George obtained by setting the nonbasic variables to zero and Dantzig introduced a class of algorithms, called simreading from the dictionary the values of the basic plex methods, that start at the origin and repeatedly jump from one basic solution to an adjacent basic variables solution in a systematic manner such that eventually and xB D xB : xN D 0 a basic feasible solution is found and then ultimately an optimal vertex is found. With the slack variables defined, the problem has In going from one iteration to the next, a single n C m variables. As the slack variables w and the element of N , say j  , and a single element of B, say original variables x are treated the same by the simplex i  , are chosen and these two variables are swapped in method, it is convenient to use a common notation: these two sets. The variable xj  is called the entering  variable and xi  is called the leaving variable. x In complete analogy with the primal problem, one : x  w can write down a dual dictionary and read off a dual A basic solution corresponds to choosing n of these basic solution. The initial primal/dual pair had a symmetry that we called the negative-transpose property. variables to be set to zero. The m equations given by It turns out that this symmetry is preserved by the pivot operation. As a consequence, it follows that priAx C w D b (3) mal/dual complementarity holds in every primal/dual basic solution. Hence, a basic solution is optimal if and can then be used to solve for the remaining m variables. only if it is primal feasible and dual feasible. Let N denote a particular choice of n of the n C m indices and let B denote the complement of this set (so that B [ N D f1; : : : ; n C mg). Let xN denote the n-vector consisting of the variables xj , j 2 N . These variables are called nonbasic variables. Let xB denote the m-vector consisting of the rest of the variables. They are called basic variables. Initially, xN D Œx1    xn T and xB D ŒxnC1    xnCm T so that (3) can be rewritten as xB D b  AxN :

(4)

Degeneracy and Cycling Every variant of the simplex method chooses the entering and leaving variables at each iteration with the intention of improving some specific measure of a distance either from feasibility or optimality. If such a move does indeed make a strict improvement at every iteration, then it easily follows that the algorithm will find an optimal solution in a finite number of pivots because there are only a finite number of ways to partition the set f1; 2; : : : ; n C mg into m basic and n nonbasic components. If the metric is always making

Linear Programming

a strict improvement, then it can never return to a place it has been before. However, it can happen that a simplex pivot can make zero improvement in one or more iterations. Such pivots are called degenerate pivots. It is possible, although exceedingly rare, for simple variants of the simplex method to produce a sequence of degenerate pivots eventually returning to a basic solution already visited. If the algorithm chooses the entering and leaving variables according to a deterministic rule, then returning once implies returning infinitely often and the algorithm fails. This failure is called cycling. There are many safe-guards to prevent cycling, perhaps the simplest being to add a certain random aspect to the entering/leaving variable selection rules. All modern implementations of the simplex method have anti-cycling safeguards.

Empirical Average-Case Performance Given the anti-cycling safeguards, it follows that the simplex method is a finite algorithm. But, how fast is it in practice? The answer is that, on average, most variants of the simplex method take roughly order min.n; m/ pivots to find an optimal solution. Such average case performance is about the best that one could hope for and accounts for much of the practical usefulness of linear programming in solving important everyday problems.

Worst-Case Performance One popular variant of the simplex method assumes that the initial primal dictionary is feasible and, at each iteration, selects for the entering variable the non-basic variable that provides the greatest rate of increase of the objective function and it then chooses the leaving variable so as to preserve primal feasibility. In 1972, Klee and Minty [6] constructed a simple family of LPs in which the n-th instance involved n variables and a feasible polytope that is topologically equivalent to an n-cube but for which the pivot rule described above takes short steps in directions of high rate of increase rather than huge steps in directions with a low rate of increase and in so doing visits all 2n vertices of this distorted n-cube in 2n1 pivots thus showing that this particular variant of the simplex method has exponential complexity. It is an open question whether or not there exists some variant of the simplex method whose worst-case performance is better than exponential.

799

Interior-Point Methods For years it was unknown whether or not there existed an algorithm for linear programming that is guaranteed to solve problems in polynomial time. In 1979, Leonid Khachiyan [5] discovered the first such algorithm. But, in practice, his algorithm was much slower than the simplex method. In 1984, Narendra Karmarkar [4] developed a completely different polynomial time algorithm. It turns out that his algorithm and the many variants of it that have appeared over time are also highly competitive with the simplex method. The class of algorithms inspired by Karmarkar’s algorithm are called interior-point algorithms. Most implementations of algorithms of this type belong to a generalization of this class called infeasible interiorpoint algorithms. These algorithms are iterative algorithms that approach optimality only in the limit – that is, they are not finite algorithms. But, for any  > 0, they get within  of optimality in polynomial time. The adjective “infeasible” points to the fact that these algorithms may, and often do, approach optimality from outside the feasible set. The adjective “interior” means that even though the iterates may be infeasible, it is required that all components of all primal and dual variables be strictly positive at every iteration. Complexity In the worst case, Karmarkar’s algorithm requires on p the order of n log.1=) iterations to get within  of an optimal solution. But, an iteration of an interiorpoint method is more computationally intensive (order n3 ) than an iteration of the simplex method (order n2 ). Comparing arithmetic operations, one gets that interior-point methods require on the order of n3:5 log.1=) arithmetic operations in the worst case, which is comparable to the average case performance of the simplex method.

References 1. Dantzig, G.: Programming in a linear structure. Econometrica 17, 73–74 (1949) 2. Dantzig, G.: Maximization of a linear function of variables subject to linear inequalities. In: Koopmans, T. (ed.) Activity Analysis of Production and Allocation, pp. 339–347. Wiley, New York (1951) 3. Kantorovich, L.: A new method of solving some classes of extremal problems. Dokl. Akad. Sci. USSR 28, 211–214 (1940)

L

800 4. Karmarkar, N.: A new polynomial time algorithm for linear programming. Combinatorica 4, 373–395 (1984) 5. Khachian, L.: A polynomial algorithm in linear programming. Dokl. Acad. Nauk SSSR 244, 191–194 (1979), in Russian. English Translation: Sov. Math. Dokl. 20, 191–194 6. Klee, V., Minty, G.: How good is the simplex algorithm? In: Shisha, O. (ed.) Inequalities–III, pp. 159–175. Academic, New York (1972)

Linear Sampling Michele Piana Dipartimento di Matematica, Universit`a di Genova, CNR – SPIN, Genova, Italy

Mathematics Subject Classification 34L25; 15A29

Linear Sampling

Far-field equation Linear integral equation relating the far-field operator with the far-field pattern of the field generated by a point source. Wavenumber Real positive number given by the ratio between 2 and the wavelength of the incident wave. Refractive index Complex-valued function where the real part is proportional to the electrical permittivity and the imaginary part is proportional to the electrical conductivity. Herglotz wave function Wave function which is a weighted linear superposition of plane waves. Tikhonov regularization Method for the solution of linear ill-posed problems based on the minimization of a convex functional with L2 penalty term. 2 L Hilbert space Linear space made of functions with bounded L2 norm. Maxwell equations The set of four equations describing classical electrodynamics. Lippmann-Schwinger equation Integral equation at the basis of both classical and quantum scattering. Poynting vector Vector field provided by the outer product between the electric and magnetic fields.

Synonyms Linear sampling method; LSM

Short Definition

Glossary/Definition Terms

The linear sampling method (LSM) is a linear visualization method for solving nonlinear inverse scattering problems.

Direct scattering problem Problem of determining the total acoustic or electromagnetic field from the knowledge of the geometrical and physical properties of the scatterer. Inverse scattering problem Problem of recovering the geometrical and physical properties of an inhomogeneity for the knowledge of the acoustic or electromagnetic scattered field. Ill-posed problem In the sense of Hadamard, it is a problem whose solution does not exist unique or does not depend continuously on the data. Far-field pattern In the asymptotic factorization of the far-field pattern, it is the term depending just on the observation angle. Far-field operator Linear intergral operator whose intergral kernel is the far-field pattern. Hankel function Complex function which is a linear combination of Bessel functions.

Description Inverse Scattering Methods Electromagnetic or acoustic scattering is a physical phenomenon whereby, in the presence of an inhomogeneity, an electromagnetic or acoustic incident wave is scattered and the total field at any point of the space is written as the sum of the original incident field and the scattered field. The direct scattering problem is the problem of determining this total field starting from the knowledge of the geometrical and physical properties of the scatterer. On the contrary, the inverse scattering problem is the problem of recovering information on the inhomogeneity from the knowledge of the scattered field. Solving inverse scattering problems is particularly challenging for two reasons. First, all inverse scattering problems significant in applications belong

Linear Sampling

to the class of the so-called ill-posed problems in the sense of Hadamard [1], and therefore, any reliable approach to their solution must face at some stage issues of uniqueness and numerical stability. Second, inverse scattering problems are often nonlinear, and there are physical conditions notably significant in the applied sciences where such nonlinearity is genuine and cannot be linearized by means of weak-scattering approximations. Most computational approaches for the solution of inverse scattering problems can be divided into three families: (1) nonlinear optimization schemes, where the restoration is performed iteratively from an initial guess of the position and shape of the scatterer; (2) weak-scattering approximation methods, where a linear inverse problem is obtained by means of lowor high-frequency approximations; and (3) qualitative methods, which provide visualization of the inhomogeneity but are not able to reconstruct the point values of the scattering parameters. The linear sampling method (LSM) [2–4] is, historically, the first qualitative method, the most theoretically investigated, and the most experimentally tested. In this approach, a linear integral equation of the first kind is written for each point of a computational grid containing the scatterer, the integral kernel of such equation being the farfield pattern of the scattered field, and the right-hand side being an exactly known analytical function. This integral equation is approximately solved for each sampling point by means of a regularization method [5], and the object profile is recovered by exploiting the fact that the norm of this regularized solution blows up when the sampling point approaches the boundary from inside. The main advantages of the linear sampling method are that it is fast, simple to implement, and not particularly demanding from a computational viewpoint. The method of course has also some disadvantages. The main one is that it only provides a visualization of the support of the scatterer and it is not possible to infer information about the point values of the refractive index.

801

8 for x 2 R2 n @D u.x/ C k 2 n.x/ u.x/ D 0 ˆ ˆ ˆ ˆ < O for x 2 R2 u.x/ D e ikxd C us .x/  

ˆ ˆ p @us ˆ ˆ : lim  ikus D 0; r r!1 @r (1) where D R2 is a C 2 -domain, @D is its boundary, dO D dO . / D .cos ; sin / is the incidence direction, and k is the wavenumber; n.x/ is the refractive index  .x/ 1 ".x/ C i n.x/ WD "B !

(2)

p where i D 1 and ! denote the angular frequency of the wave and ".x/ and .x/ are the electrical permittivity and conductivity, respectively. We assume that ".x/ is uniform in R2 n DN and equal to the background value "B > 0, while  D 0 in the same region. For each incidence direction dO , there exists a unique solution to problem (1) [6], and the corresponding scattered field us D us . I / has the following asymptotic behavior (holding uniformly in all directions xO WD x=jxj):   e ikr us .xI / D p u1 .'I /CO r 3=2 as r D jxj ! 1; r (3) where .r; '/ are the polar coordinates of the observation point x and the function u1 D u1 . I / 2 L2 Œ0; 2 is known as the far-field pattern of the scattered field us . Define the linear and compact far-field operator F W L2 Œ0; 2 ! L2 Œ0; 2 corresponding to the inhomogeneous scattering problem (1) as Z

2

.F g/.'/ WD

u1 .'; /g. /d

8g 2 L2 Œ0; 2 :

0

(4)

The operator F is injective with dense range if k 2 is not a transmission eigenvalue [7]. Next consider the outgoing scalar field ˆ.x; z/ D

Formulation of the Linear Sampling Method As a test case, consider the two-dimensional scattering problem [4, 6] of determining u D u. I / 2 C 2 .R2 n @D/ \ C 1 .R2 / such that

8x 2 R2 ;

i .1/ H .kjx  zj/ 4 0

8x ¤ z;

(5)

generated by a point source located at z 2 R2 , where .1/ H0 ./ denotes the Hankel function of the first kind and of order zero. The corresponding far-field pattern is given by

L

802

Linear Sampling

e i =4 ik x.'/z ˆ1 .'; z/ D p ; e O 8 k x.'/ O WD .cos '; sin '/ 8' 2 Œ0; 2 :

Q1

rections fdj D .cos j ; sin j /gj D0 . In the following, P D Q D N and 'i D i i D 0; N 1. These values are placed into the far-field matrix F, whose elements (6) are defined as

For each z 2 R2 , the far-field equation is defined as .F gz /.'/ D ˆ1 .'; z/:

Fij WD u1 .xO i ; dj /: (7)

(10)

In practical applications, the far-field matrix is affected by the measurement noise, and therefore, only a noisy version Fh of the far-field matrix is at disposal, such that Fh D F C H; (11)

The linear sampling method is inspired by a general theorem [7], concerning the existence of -approximate solutions to the far-field equation and their qualitative behavior. According to this theorem, if z 2 D, then for every  > 0, there exists a solution gz 2 L2 Œ0; 2 of where H is the noise matrix with kHk  h. Furtherthe inequality more, for each z D r.cos ; sin / 2 Z containing the scatterer,    F g  ˆ1 .; z/ 2  (8) z L Œ0;2

˚ 1 .z/ WD

such that for every z 2 @D,   lim gz L2 Œ0;2 D 1 and

z!z

T e i 4  ikr cos.'0  / e p ; : : : ; e ikr cos.'N 1  / : 8 k (12)

  lim vgz L2 .D/ D 1;

z!z

(9) where vgz is the Herglotz wave function with kernel gz . If z … D, the approximate solution remains unbounded. On the basis of this theorem, the algorithm of the linear sampling method may be described as follows [3]. Consider a sampling grid that covers a region containing the scatterer. For each point z of the grid, compute a regularized solution g˛ .z/ of the (discretized) far-field equation (7) by applying Tikhonov regularization coupled with the generalized discrepancy principle [5]. The boundary of the scatterer is visualized as the set of grid points in which the (discretized) L2 -norm of g˛ .z/ becomes mostly large. Computational Issues The main drawback of this first formulation of the LSM is that the regularization algorithm for the solution of the far-field equation is applied point-wise, i.e., a different regularization parameter must be chosen for each sampling point z. A much more effective implementation is possible by formulating the method in a functional framework which is the direct sum of many L2 spaces. The first step of this formulation is to observe that, in real experiments, the far-field 1 pattern is measured for P observation angles f'i giPD0 Q1 and Q incidence angles f j gj D0 , i.e., for observation 1 directions fxO i D .cos 'i ; sin 'i /gPi D0 and incidence di-

Therefore, the one-parameter family of linear integral equations (7) can be replaced by the one-parameter family of ill-conditioned square linear systems Fh g.z/ D

N ˚ 1 .z/: 2

(13)

Then consider the direct sum of Hilbert spaces: 

L2 .Z/

N

WD L2 .Z/ ˚    ˚ L2 .Z/; „ ƒ‚ …

(14)

N times

and define the linear operator Fh W  2 N L .Z/ such that

 2 N L .Z/ !

8 9N 1 1 m mn

1 X ind X qm .ra  rm /  2 a a m jra  rm j3

(4)

O solute .ra / represent the electrostatic where VO .rm / and E a potential and the electric field operators due to the solute electrons and nuclei calculated at the MM sites. On el describes the electrostatic the other hand, in (4) HO MM pol self-energy of the MM charges, while HO MM represents the polarization interaction between such charges and el term the induced dipoles. We recall that the HO MM enters in the effective Hamiltonian only as a constant pol energetic quantity, while the HO MM contribution explicitly depends on the QM wavefunction. Numerical Aspects of Polarizable MM Approaches The dipoles induced on each MM polarizable site can be obtained assuming a linear approximation,

814

Liquid-Phase Simulation: Theory and Numerics of Hybrid Quantum-Mechanical/Classical Approaches

neglecting any contribution of magnetic character related to the total electric field, and using an isotropic polarizability for each selected point in the MM part of the system. The electric field which determines such dipoles contains a sum of contributions from the solute, from the solvent point charges, and from the induced dipole moments themselves. This mutual polarization between the dipoles can be solved through a matrix inversion approach, by introducing a matrix equation: Kind c D Ec

(5)

where the matrix K is of dimension 3N 3N, N being the number of polarizable sites, and the vector Ec collects the c-th component of the electric field from the solute and the solvent permanent charge distribution. The form of matrix K will be determined uniquely by the position of the polarizable sites and the polarizability values, namely: Ki;i D Ki CN;i CN D Ki C2N;i C2N D 1=˛i Ki;i CN D Ki;i C2N D Ki CN;i D Ki CN;i C2N D Ki C2N;i D Ki C2N;i CN D 0 kl Ki CmN;j CnN D Ti;j

with n; m D 0; 1; 2 and k; l D x; y; z where the index i and j ¤ i run from 1 to N , and the dipole field tensor is given by: 2

Ti;j

important for extensions of QM/MM approaches to molecular dynamics. The recent development of linear-scaling for correlation methods has significantly extended the size of systems that can be treated with such methods, up to several tens of atoms, and has made them a very accurate alternative to be coupled with an MM description of the environment. As far as the choice of MM method is concerned, all the many force fields available in the literature can, in principle, be coupled with a QM description.

3

r2 r r r r 1 3 4 x x 2y x z 5 D 3 I  5 ry rx ry ry rz rij rij rz rx rz ry rz2

(6)

QM/Continuum The analysis of QM/classical methods is less straightforward if we adopt a continuum description. The basic formulation of continuum models requires the solution of a classical electrostatic problem (Poisson problem): h i E  ".Er /rV E .Er / D 4 M .Er / r

(7)

where M .Er / is the solute charge distribution and ".Er / is the general position-dependent permittivity. If we assume that the charge distribution is contained in a molecular cavity C of proper shape and dimension built within a homogeneous and isotropic solvent, ".Er / assumes the simple form: ".Er / D

1 rE 2 C " rE … C

(8)

where " is the dielectric constant of the solvent. Using the definition (8) with the appropriate boundary conditions, the electrostatic problem (7) can be solved in terms of a potential V which is the sum of the solute potential plus the contribution due to the reaction of the solvent (e.g., the polarization of the dielectric), namely V .Er / D VM .Er / C V .Er /. Under the assumption that the charge distribution is entirely supported inside the cavity C , an integral representation of the reaction potential can be derived which introduces a fictitious (or apparent) charge distribution  on the boundary between the solute and the solvent, that is, the surface of the cavity C ,  D @C , namely:

The QM/MM formalism can accommodate almost any combination of QM and MM methods. The choice of the QM method follows the same criteria as in pure QM studies. Essentially, the QM code must be able to perform the self-consistent field (SCF)  Hartree–Fock Type Methods treatment in the presence of the external point-charge (or dipole) field that represents the MM charge model in the case of Z electronic (or polarized) embedding. In practice, many .Es / ˇ d sE ˇ current QM/MM applications use density-functional V .Er / D (9) ˇrE  sEˇ theory (DFT)  Density Functional Theory as the  QM method owing to its favorable computationaleffort/accuracy ratio. Traditionally, semiempirical QM The surface charge  is solution of an integral equation methods have been most popular, and they remain on  , that is of an equation of the form [3–5]:

Liquid-Phase Simulation: Theory and Numerics of Hybrid Quantum-Mechanical/Classical Approaches

Z

kA .Es ; sE 0 / .Es 0 / d sE 0 D b .Es / 8Es 2 

.A/.Es / D 

815

Galerkin method could also be used); however, it is the most natural and easiest to implement for the specific case of apparent surface charge calculations [14]. The necessary preliminary step in the strategy is the generation of the surface elements (i.e., the surface mesh, see Fig. 1) as, once the mesh has been defined, the apparent charges q are obtained by solving a matrix equation, of the type

(10) where kA is the Green kernel of some integral operator A and b depends linearly on the charge distribution M . This formulation has been adopted in different continuum solvation models, the most famous ones being the polarizable continuum model (PCM) [15] (in its different versions) and the conductor-like screening model (COSMO) [9]. Each different formulation (13) Qq D RVM corresponds to different choices for A, but in all cases, it is obtained in terms of a specific combination of the where q and VM are the vectors containing the N following kernels: values of the charge and the solute potential at the surface points, respectively. Q and R are the matrix 8 1 ˆ analogs of the integral operators introduced in (10) to < jEs Es 0 j 1 (11) obtain the apparent charge distribution . In particular, kA .Es ; sE 0 / D @@nO s jEs E s 0j ˆ : @ 1 the different kernels reported in the (11) can be written @nO s0 jEs Es 0 j in terms of the following matrices: where nO s represents the unit vector normal to the surface at point sE and pointing toward the dielectric. Also b changes according to the different formulation of the model. For instance, the original version of COSMO is obtained with: Z b .Es / D f ./

I3 R

M .Er 0 / d rE 0 jEs  rE 0 j

(12)

where f ./ D .  1/=. C 0:5/. Numerical Aspects of Polarizable Continuum Approaches The reduction of the source of the solvent reaction potential to a charge distribution limited to a closed surface greatly simplifies the electrostatic problem with respect to other formulations in which the whole dielectric medium is considered as source of the reaction potential. In spite of this remarkable simplification, the integration of (10) over a surface of complex shape is computationally challenging. The solutions are generally based on a discretization of the integral into a finite number of elements. This discretization of  automatically leads to a discretization of .Es / in terms of point-like charges, namely if we assume that on each surface element .Es / does not significantly change, its effect can be simulated with that of a point charge of value q.Esi / D .Esi /ai where ai is the area of the surface element i and sEi its representative point. This numerical method, which can be defined as P0 collocation method, is not the only possible one (e.g., a

Sij D

1 jEsi  sEj j

Dij D

.Esi  sEj /  nO j jEsi  sEj j3

Dij D

.Esj  sEi /  nO i jEsi  sEj j3

(14)

As concerns the diagonal elements of S, D and D different numerical solutions have been proposed. In p particular, those commonly used are Si i D k 4 =ai P and Di i D .2 C j ¤i Dij aj /=ai where the former derives from the exact formula of a flat circular element with k taking into account that the element is spherical, and the latter becomes exact when the size of all the elements tends to zero. The approximation method described above belongs to the class of boundary element methods (BEM) [2]. BEM follows the same lines as finite element methods (FEM). In both cases, the approximation space is constructed from a mesh. In the context of continuum solvation models, FEM solves the (local) partial differential equation (7), complemented with convenient boundary conditions, a 3D mesh [6, 8], while BEM solves one of the (nonlocal) integral equations derived above, on a 2D mesh. In the former case, the resulting linear system is very large, but sparse. In the latter case, it is of much lower size, but full. If we now reintroduce a QM description of the charge distribution M in terms of the wavefunction

L

816

Liquid-Phase Simulation: Theory and Numerics of Hybrid Quantum-Mechanical/Classical Approaches

which is solution of the (1), we can rewrite the solvent we can in fact tune the boundary between the two cominduced term HO env as: ponents of the system as well as extend the dimensions of the classical system and change its description using X q.Esi /VO .Esi / (15) either an atomistic (MM) or a continuum approach. In HO env D HO QM=cont D m addition, both QM/MM and QM/continuum methods can be applied to environments of increasing complexwhere q are the solvent apparent charges and VO is ity [10, 11], from standard isotropic and homogeneous the electrostatic potential operator corresponding to the liquids, to gas–liquid or liquid–liquid interfaces and/or solute charge distribution. By comparing (15) with (4), anisotropic liquid crystalline phases, just to quote few. it might seem that there is a perfect equivalence be- The most important aspect of these methods, however, tween the nonpolarizable part of the QM/MM method is that the QM approach, even if limited to just a part and the QM/continuum one. As a matter of fact this of the system, allows for a more accurate description of equivalence is only apparent as the apparent charges all those processes and phenomena which are mostly entering in (15) are not external parameters as it is based on the electronic structure of the molecules for the MM charges but they are obtained solving a constituting the liquid. In more details, QM/classical matrix equation which depends on the solute charge methods should be preferred over other fully classical distribution. In (4), the induced dipoles ind a depend on approaches when the interest is not on the properties the solute charge distribution exactly as the apparent of the liquid itself but instead on the effects that the charges. liquid exerts on a property or a process which can The analogies and differences between QM/ Con- be localized on a specific part of the system. The tinuum and QM/MM approaches however are not only realm of (bio)chemical reactivity in solution as well on the methodological aspects of their formulation and as the world of spectroscopies in condensed phase are implementation. It is important to recall that the two examples where QM/classical methods really represent approaches also present fundamental specificities from the most effective approach. Of course there are also a physical point of view. By definition, continuum drawbacks; in particular, the computational cost can models introduce an averaged (bulk) description of increase enormously with respect to classical methods the environment effects. This is necessarily reflected especially when the QM/Classical approach is couin the results that can be obtained with these meth- pled to molecular dynamics simulations. Moreover, ods. While continuum models can be successfully the choice of the specific combination of the QM applied in all cases in which the environment acts description and the classical one is not straightforward, as a mean-field perturbation, solvent-specific effects but it has to be carefully chosen on the basis of the such as hydrogen bondings are not well reproduced. specific problem under investigation and the specific By contrast, QM/MM methods can properly describe chemical system of interest. It is however clear that many specific effects but, at the same time, they cannot QM/classical methods represent one of the most powbe applied to simulate longer-to-bulk effects if they erful approaches to combine accuracy with complexity are not coupled to a sampling of the configurational while still keeping a physically founded representation space of the solute–solvent system. For this, a molecu- of the main interactions determining the behavior of lar dynamics (MD) or Monte Carlo (MC) simulation the liquids. approach is needed with significant increase of the computational cost.

Conclusions Many alternative strategies are available to simulate the liquid phase, each with its advantages and weaknesses. Here, in particular, the attention has been focused on the class of methods which combine a QM description of the subsystem of interest with a classical one for the remainder. This hybrid approach is extremely versatile,

References 1. Allen, M.P., Tildesley, D.: Computer Simulations of Liquids. Oxford University Press, London (1987) 2. Beskos, D.E. (ed.): Boundary Element Methods in Mechanics, vol 3. North–Holland, Amsterdam (1989) 3. Cances, E.: Integral equation approaches for continuum models. In: Mennucci, B., Cammi, R. (eds.) Continuum Solvation Models in Chemical Physics, From Theory to Applications, pp. 29–48. Wiley, Hoboken (2007)

Lobatto Methods 4. Cances, E., Mennucci, B.: New applications of integral equations methods for solvation continuum models: ionic solutions and liquid crystals. J. Math. Chem. 23, 309–326 (1998) 5. Cances, E., Le Bris, C., Mennucci, B., Tomasi, J.: Integral equation methods for molecular scale calculations in the liquid phase. Math. Models Methods Appl. Sci. 9, 35–44 (1999) 6. Cortis, C., Friesner, R.: An automatic three-dimensional finite element mesh generation system for the poissonboltzmann equation. J. Comput. Chem. 18(13), 1570–1590 (1997) 7. Gao, J.L.: Hybrid quantum and molecular mechanical simulations: an alternative avenue to solvent effects in organic chemistry. Acc. Chem. Res. 29(6), 298–305 (1996) 8. Holst, M., Baker, N., Wang, F.: Adaptive multilevel finite element solution of the Poisson–Boltzmann equation i. algorithms and examples. J. Comput. Chem. 21, 1319–1342 (2000) 9. Klamt, A.: The COSMO and COSMORS solvation models. Wiley Interdiscip. Rev. Comput. Mol. Sci. 1(5), 699–709 (2011) 10. Lin, H., Truhlar, D.G.: QM/MM: What have we learned, where are we, and where do we go from here? Theor. Chem. Acc. 117, 185–199 (2007) 11. Mennucci, B.: Continuum solvation models: what else can we learn from them? J. Phys. Chem. Lett. 1(10), 1666–1674 (2010) 12. Senn, H.M., Thiel, W.: QM/MM methods for biomolecular systems. Angew. Chem. Int. Ed. 48(7), 1198–1229 (2009) 13. Tomasi, J., Persico, M.: Molecular interactions in solution – an overview of methods based on continuous distributions of the solvent. Chem. Rev. 94(7), 2027–2094 (1994) 14. Tomasi, J., Mennucci, B., Laug, P.: The modeling and simulation of the liquid phase. In: Le Bris, C. (ed.) Handbook of Numerical Analysis: Special Volume. Computational Chemistry, pp. 271–323. Elsevier, Amsterdam (2003) 15. Tomasi, J., Mennucci, B., Cammi, R.: Quantum mechanical continuum solvation models. Chem. Rev. 105(8), 2999–3093 (2005)

817

for the government in the fields of life insurance and of weights and measures. In 1842, he was appointed professor of mathematics at the Royal Academy in Delft (known nowadays as Delft University of Technology). Lobatto methods are characterized by the use of approximations to the solution at the two end points tn and tnC1 of each subinterval of integration Œtn ; tnC1 . Two well-known Lobatto methods based on the trapezoidal quadrature rule which are often used in practice are the (implicit) trapezoidal rule and the St¨ormer-Verlet-leapfrog method. The (Implicit) Trapezoidal Rule Consider a system of ordinary differential equations (ODEs): d y D f .t; y/ (1) dt where f W R Rd ! Rd . Starting from y0 at t0 one step .tn ; yn / 7! .tnC1 ; ynC1 / of the (implicit) trapezoidal rule applied to (1) is given by the implicit relation: ynC1

hn D yn C .f .tn ; yn / C f .tnC1 ; ynC1 // 2

where hn D tnC1  tn is the step size. The (implicit) trapezoidal rule is oftentimes called the CrankNicholson method when considered in the context of time-dependent partial differential equations (PDEs). This implicit method requires the solution of a system of d equations for ynC1 2 Rd that can be expressed as: F .ynC1 / WD ynC1  yn 

Cf .tnC1 ; ynC1 // D 0

Lobatto Methods Laurent O. Jay Department of Mathematics, The University of Iowa, Iowa City, IA, USA

Introduction

hn .f .tn ; yn / 2

and which is nonlinear when f .t; y/ is nonlinear in y. .0/ Starting from an initial guess ynC1  ynC1 , the solution ynC1 can be approximated iteratively by modified Newton iterations as follows: .kC1/

.k/

.k/

ynC1 D ynC1 C pnC1 ;

.k/

.k/

Jn pnC1 D F .ynC1 /

Lobatto methods for the numerical integration using, for example, an approximate Jacobian: of differential equations are named after Rehuel hn .k/ Lobatto. Rehuel Lobatto (1796–1866) was a Dutch Jn D Id  Dy f .tn ; yn /  Dy F .ynC1 /: 2 mathematician working most of his life as an advisor

L

818

Lobatto Methods

Taking Jn D Id leads to fixed-point iterations: .kC1/

ynC1 D yn C

 hn  .k/ f .tn ; yn / C f .tnC1 ; ynC1 / : 2

¨ The Generalized Newton-Stormer-VerletLeapfrog Method Consider now a partitioned system of ODEs:

Depending on the field of applications, this method is known under different names: the St¨ormer method in astronomy; the Verlet method in molecular dynamics; the leapfrog method in the context of timedependent PDEs, in particular for wave equations. This method can be traced back to Newton’s Principia (1687), see [10].

Lobatto Methods (2) In this entry, we consider families of Runge-Kutta (RK) methods based on Lobatto quadrature formulas whose simplest member is the trapezoidal quadrature where v W R Rdq Rdp ! Rdq and f W R rule. When applied to (1) Lobatto RK methods can be Rdq Rdp ! Rdp . Starting from .q0 ; p0 / at t0 one step expressed as follows: .tn ; qn ; pn / 7! .tnC1 ; qnC1 ; pnC1 / of the generalized Newton-St¨ormer-Verlet-leapfrog method applied to (2) s X reads: aij f .tn C cj h; Ynj / Yni D yn C hn d q D v.t; p; q/; dt

d p D f .t; q; p/ dt

hn f .tn ; qn ; pnC1=2 /; 2 hn  v.tn ; qn ; pnC1=2 / qnC1 D qn C 2

j D1

pnC1=2 D pn C

for i D 1; : : : ; s;



Cv.tnC1 ; qnC1 ; pnC1=2 / ; pnC1 D pnC1=2 C

(3)

hn f .tnC1 ; qnC1 ; pnC1=2 / 2

where hn D tnC1  tn is the step size. The first equation is implicit for pnC1=2 , the second equation is implicit for qnC1 , and the last equation is explicit for pnC1 . When v.t; q; p/ D v.t; p/ is independent of q, and f .t; q; p/ D f .t; q/ is independent of p the method is fully explicit. If in addition v.t; q; p/ D v.p/ is independent of t and q, the method can be simply expressed as: hn f .tn ; qn /; 2 qnC1 D qn C hn v.pnC1=2 /;

pnC1=2 D pn C

pnC1 D pnC1=2 C

hn f .tnC1 ; qnC1 /: 2

This explicit method is often applied as follows: 1 pnC1=2 D pn1=2 C .hn1 C hn /f .tn ; qn / ; 2 qnC1 D qn C hn v.pnC1=2 /:

ynC1 D yn C hn

s X

bj f .tn C cj h; Ynj /

(4)

(5)

j D1

where the stage value s satisfies s  2 and the coefficients aij ; bj ; cj characterize the Lobatto RK method. The s intermediate values Ynj for j D 1; : : : ; s are called the internal stages and can be considered as approximations to the solution at tn C cj hn , the main numerical RK approximation at tnC1 D tn C hn is given by ynC1 . Lobatto RK methods are characterized by c1 D 0 and cs D 1. They can also be considered in combination with other families of RK methods, for example, with Gauss methods in the context of certain systems of differential-algebraic equations (DAEs), see the section “Lobatto Methods for DAEs” below. The symbol III is usually found in the literature associated to Lobatto methods, the symbols I and II being reserved for the two types of Radau methods. The (implicit) trapezoidal rule is the simplest member (s D 2) in the Lobatto IIIA family. The generalized Newton-St¨ormerVerlet-leapfrog method seen above can be interpreted as a partitioned Runge-Kutta (PRK) resulting from the combination of the (implicit) trapezoidal rule and the Lobatto IIIB method for s D 2, see the section “Additive Lobatto Methods for Split and Partitioned ODEs” below.

Lobatto Methods

819

where

Families of Lobatto Methods

 1 dk  2 .x  1/k k k kŠ2 dx For a fixed value of s, the various families of Lobatto methods described below all share the same coeffi- is the kth Legendre polynomial. Lobatto quadrature cients bj ; cj of the corresponding Lobatto quadrature formulas are symmetric, that is their nodes and weights satisfy: formula. Pk .x/ D

Lobatto Quadrature Formulas The problem of approximating a Riemann integral: Z

bsC1j D bj ;

csC1j D 1  cj

for j D 1; : : : ; s:

tn Chn

(6) For s D 3, we obtain the famous Simpson’s rule:

f .t/dt tn

with f assumed to be continuous is equivalent to the problem of solving the initial value problem at t D tn C hn : d y D f .t/; y.tn / D 0 dt R t Ch since y.tn C hn / D tnn n f .t/dt. The integral (6) can be approximated by using a standard quadrature formula: ! Z tn Chn s X f .t/dt  hn bi f .tn C ci hn /

.b1 ; b2 ; b3 / D .1=6; 2=3; 1=6/; .c1; c2 ; c3 / D .0; 1=2; 1/:

Procedures to compute numerically accurately the nodes and weights of high order Lobatto quadrature formulas can be found in [7] and [23]. The subroutine GQRUL from the IMSL/MATH-LIBRARY can compute numerically these nodes and weights.

L

Lobatto Families The families of Lobatto RK methods differ only in the with s node coefficients c1 ; : : : ; cs , and s weight coef- values of their coefficients aij . Various equivalent defficients b1 ; : : : ; bs . Lobatto quadrature formulas, also initions can be found in the literature. The coefficients known as Gauss-Lobatto quadrature formulas in the aij of these families can be linearly implicitly defined literature, are given for s  2 by a set of nodes and with the help of so-called simplifying assumptions: weights satisfying conditions described hereafter. The s nodes cj are the roots of the polynomial of degree s: s X ck aij cjk1 D i C.q/ W k d s2 s1 j D1 .t .1  t/s1 /: s2 dt for i D 1; : : : ; s and k D 1; : : : ; q; tn

i D1

These nodes satisfy c1 D 0 < c2 < : : : < cs D 1. The weights bj and nodes cj satisfy the condition B.2s2/ where: B.p/ W

s X j D1

bj cjk1 D

1 k

s X i D1

bi cik1 aij D

 bj  1  cjk k

for j D 1; : : : ; s and k D 1; : : : ; r: for k D 1; : : : ; p;

implying that the quadrature formula is of order 2s  2. There exists an explicit formula for the weights bj D

D.r/ W

1 >0 s.s  1/Ps1 .2cj  1/2

for j D 1; : : : ; s b1 D bs D

1 s.s  1/



The importance of these simplifying assumptions comes from a fundamental result due to Butcher, see [5, 9], saying that a RK method satisfying the simplifying assumptions B.p/, C.q/, and D.r/ is of order at least min.p; 2qC2; qCr C1/. The coefficients aij ; bj ; cj characterizing the Lobatto RK method (4) and (5) will be displayed below in the form of a table called a Butcher-tableau:

820

Lobatto Methods

c1 D 0 a11 c2 a21 :: :: : : cs1 as1;1 cs D 1 as1 b1

a12    a22    :: : : : : as1;2    as2    b2   

a1;s1 a2;s1 :: :

1 Yn2 D .yn C ynC1 / 2 hn C .f .tn ; yn /  f .tnC1 ; ynC1 //; 8 hn  f .tn ; yn / C 4f .tnC1=2 ; Yn2 / ynC1 D yn C 6  Cf .tnC1 ; ynC1 /

a1s a2s :: :

as1;s1 as1;s as;s1 ass bs1 bs

In the four main families of Lobatto methods described below, namely Lobatto IIIA, Lobatto IIIB, Lobatto where tnC1=2 D tn C hn =2. It can be even further IIIC, and Lobatto IIIC , only one method does not reduced by rewriting satisfy the relation C.1/, that is, hn .f .tn ; yn / C f .tnC1 ; ynC1 // ynC1 D yn C s 6 X

aij D ci for i D 1; : : : ; s; 1 2hn f tnC1=2 ; .yn C ynC1 / C j D1 3 2  hn this is the Lobatto IIIB method for s D 2, see C .f .tn ; yn /  f .tnC1 ; ynC1 // : 8 below. The Lobatto IIIA, IIIB, IIIC, and IIIC methods can all be interpreted as perturbed colloLobatto IIIB cation methods [19] and discontinuous collocation The coefficients aijB of Lobatto IIIB methods can be methods [11]. defined by D.s/ (Table 2). They satisfy C.s  2/, B ai1 D b1 for i D 1; : : : ; s and aiBs D 0 for i D 1; : : : ; s. Lobatto IIIB methods are symmetric and of nonstiff Lobatto IIIA The coefficients aijA of Lobatto IIIA methods can be order 2s  2. Their stability function R.z/ is given by the .s  1; s  1/-Pad´e approximation to e z . They are A defined by C.s/ (Table 1). They satisfy D.s 2/, asj D A-stable, but not L-stable since R.1/ D .1/sC1 . A bj for j D 1; : : : ; s, and a1j D 0 for j D 1; : : : ; s. They are not B-stable and thus not algebraically Lobatto IIIA methods are symmetric and of nonstiff stable. The coefficients aijB can also be obtained order 2s  2. Their stability function R.z/ is given A by the .s  1; s  1/-Pad´e approximation to e z . They from the coefficients aij of Lobatto IIIA through the are A-stable, but not L-stable since R.1/ D .1/sC1 . relations: They are not B-stable and thus not algebraically stable. They can be interpreted as collocation methods. Since the first internal stage Yn1 of Lobatto IIIA methods is explicit (Yn1 D yn and f .tn C c1 hn ; Yn1 / D f .tn ; yn /) and the last internal stage satisfies Yns D ynC1 (and thus f .tnC1 ; ynC1 / D f .tn C cs hn ; Yns /), these methods are comparable in terms of computational work to Gauss methods with s  1 internal stages since they also have the same nonstiff order 2s  2. For s D 2, we obtain the (implicit) trapezoidal rule which is often expressed without its two internals stages Yn1 ; Yn2 since they are respectively equal to yn and ynC1 . The method for s D 3 is sometimes called the Hermite-Simpson (or Clippinger-Dimsdale) method and it has been used, for example, in trajectory optimization problems [4]. This method can be equivalently expressed in a compact form as:

bi aijB C bj ajAi  bi bj D 0

for i; j D 1; : : : ; s;

or A aijB D bj  asC1i;sC1j

for i; j D 1; : : : ; s:

Lobatto IIIC The coefficients aijC of Lobatto IIIC methods can be C defined by ai1 D b1 for i D 1; : : : ; s and C.s  1/ C D bj for j D (Table 3). They satisfy D.s  1/ and asj 1; : : : ; s. Lobatto IIIC methods are of nonstiff order 2s 2. They are not symmetric. Their stability function R.z/ is given by the .s  2; s/-Pad´e approximation to e z . They are L-stable. They are algebraically stable and thus B-stable. They are excellent methods for stiff problems.

Lobatto Methods

821

Lobatto Methods, Table 1 Coefficients of Lobatto IIIA for s D 2; 3; 4; 5 0

0 0

0 0

1

1 1 2 2

AsD2

1 1 2 2

0

0 0

1 2

5 24 1 6

1 AsD3

0

1 1  3 24 2 1 3 6

1 2 6 3

1 6

0

p 21 1  2 14

p 119 C 3 21 1960

1 2 p 1 21 C 2 14 1 AsD5

0

0

13 320 p 119  3 21 1960 1 20

p 343  9 21 2520 p 392 C 105 21 2880 p 343 C 69 21 2520 49 180

1 20

49 180

Lobatto IIIC Lobatto IIIC are also known as Lobatto III methods [5], Butcher’s Lobatto methods [9], and Lobatto IIIC methods [22] in the literature. (The name Lobatto IIIC was suggested by Robert P.K. Chan in an email correspondence with the author on June 13, 1995.)  The coefficients aijC of Lobatto IIIC methods can be

5 12

1 12

AsD4

0

0

0

p p p p p 5 11 C 5 25  5 25  13 5 1 C 5 1  2 10 120 120 120 120 p p p p p 1 5 11  5 25 C 13 5 25 C 5 1  5 C 2 10 120 120 120 120 5 5 1 1 1 12 12 12 12

0

5 12 0

p 392  96 21 2205

1 12 0 p 21 C 3 21 1960

p 392 C 96 21 2205 16 45

p 343  69 21 2520 p 392  105 21 2880 p 343 C 9 21 2520 49 180

16 45

49 180

1 20

8 45

3 320

p 21  3 21 1960 1 20

Other Families of Lobatto Methods Most Lobatto methods of interest found in the literature can be expressed as linear combinations of the four fundamental Lobatto IIIA, IIIB, IIIC, and IIIC methods. In fact, one can consider a very general family of methods with three real parameters .˛A ; ˛B ; ˛C / by considering Lobatto coefficients of the form:



defined by aiCs D 0 for i D 1; : : : ; s and C.s  1/ C D 0 for j D (Table 4). They satisfy D.s  1/ and a1j 1; : : : ; s. Lobatto IIIC methods are of nonstiff order 2s 2. They are not symmetric. Their stability function R.z/ is given by the .s; s2/-Pad´e approximation to e z . They are not A-stable. They are not B-stable and thus not algebraically stable. The Lobatto IIIC method for s D 2 is sometimes called the explicit trapezoidal rule.  The coefficients aijC can also be obtained from the coefficients aijC of Lobatto IIIC through the relations:  bi aijC

C

bj ajCi

 bi bj D 0

for i; j D 1; : : : ; s;



aij .˛A ; ˛B ; ˛C / D ˛A aijA C ˛B aijB C ˛C aijC C ˛C  aijC (7) where ˛C  D 1  ˛A  ˛B  ˛C . For any choice of .˛A ; ˛B ; ˛C / the corresponding Lobatto RK method is of nonstiff order 2s  2 [13]. The Lobatto IIIS methods presented in [6] depend on a real parameter . They can be expressed as: aijS ./

D .1  /



aijA

C aijB





 1  C  aij C aijC C  2

for i; j D 1; : : : ; s;

or 

C aijC D bj  asC1i;sC1j

for i; j D 1; : : : ; s:

corresponding to ˛A D ˛B D 1   and ˛C D ˛C  D   12 in (7). These methods satisfy C.s  2/ and

L

822

Lobatto Methods

Lobatto Methods, Table 2 Coefficients of Lobatto IIIB for s D 2; 3; 4; 5 0 0

1 0 2 1 0 2

0 1 BsD2

1 2 1

1 1 2 2

BsD3

1 1  0 6 6 1 1 0 6 3 1 5 0 6 6 1 6

2 3

1 6

1

1 20

p 7  21 120 p 343 C 9 21 2520 p 49 C 12 21 360 p 343 C 69 21 2520 p 119  3 21 360

BsD5

1 20

49 180

0 p 21 1  2 14 1 2 p 1 21 C 2 14

1 20 1 20 1 20 1 20

D.s  2/. They are symmetric and symplectic. Their stability function R.z/ is given by the .s  1; s  1/Pad´e approximation to e z . They are A-stable, but not Lstable. They are algebraically stable and thus B-stable. The Lobatto IIIS coefficients for  D 1=2 are given by: aijS .1=2/ D

 1 A aij C aijB 2

p 1 5  2 10 p 1 5 C 2 10

for i; j D 1; : : : ; s:

1 12 1 12 1 12

1

1 12

BsD4

1 12

p p 1  5 1 C 5 24 24 p p 25 C 5 25  13 5 120 120 p p 25 C 13 5 25  5 120 120 p p 11  5 11 C 5 24 24 5 12

13 45

p 7 C 21 120 p 343  69 21 2520 p 49  12 21 360 p 343  9 21 2520 p 119 C 3 21 360

16 45

49 180

1 15

p 56  15 21 315 8 45

p 56 C 15 21 315

5 12

0 0 0 0 1 12

0 0 0 0 0 1 20

˛A D 2, ˛B D 2, ˛C D 1, and ˛C  D 2 in (7). Their stability function R.z/ is given by the .s  2; s/Pad´e approximation to e z . These methods are L-stable. They are algebraically stable and thus B-stable. They are of nonstiff order 2s  2. They are not symmetric. They can be interpreted as perturbed collocation methods [19].

For  D 1 we obtain the Lobatto IIID methods [6, 13]:  1 C  aij C aijC aijD D aijS .1/ D 2

for i; j D 1; : : : ; s:

Additive Lobatto Methods for Split and Partitioned ODEs

Consider a split system of ODEs: These methods are called Lobatto IIIE in [19] and Lobatto IIIE in [22]. They satisfy C.s1/ and D.s1/, d and they can be interpreted as perturbed collocation (8) y D f1 .t; y/ C f2 .t; y/ dt methods [19]. Another family of Lobatto RK methods is given by the Lobatto IIID family of [19] called here Lobatto IIINW where the coefficients for s D 2; 3 where f1 ; f2 W R Rd ! Rd . Starting from y0 at t0 are given in Table 5. (Notice on p. 205 of [19] that one step .tn ; yn / 7! .tnC1 ; ynC1 / of an additive Lobatto 1 D 4.2m  1/.) These methods correspond to RK method applied to (8) reads:

Lobatto Methods

823

Lobatto Methods, Table 3 Coefficients of Lobatto IIIC for s D 2; 3; 4; 5

0

1 1  2 2 1 1 2 2

0 1 CsD2

1 2

1 2 1

1 2

CsD3

0

1 1 1  6 3 6 1 5 1  6 12 12 1 1 2 6 3 6 2 3

1 6

p 1 5  2 10 p 1 5 C 2 10 1

1 6

CsD4

0 p 1 21  2 14 1 2 p 1 21 C 2 14 1 CsD5

7 60

1 20



1 20

29 180

2 15

p 47  15 21 315

1 20 1 20

73 360 p 47 C 15 21 315 16 45

1 20

49 180

16 45

Yni D yn C hn

s X

1 12 1 12

p

5 12

5 12

1 5  12 12 p p 1 10  7 5 5 4 60 60 p p 10 C 7 5 1 5  60 4 60 5 5 1 12 12 12

1 12

7 60 p 203  30 21 1260 p 329  105 21 2880 29 180 49 180 49 180

1 12 1 20 

3 140

3 160 3 140 1 20



1 20

.a1;ij f1 .tn C cj h; Ynj /

j D1

Ca2;ij f2 .tn C cj h; Ynj // for i D 1; : : : ; s; ynC1 D yn C hn

1 12

p 5  12



p 329 C 105 21 2880 p 203 C 30 21 1260 49 180

1 20

1 12

s X

bj .f1 .tn C cj h; Ynj /

j D1

Cf2 .tn C cj h; Ynj //

f2 .t; q; p/ D

 0 : f .t; q; p/

Applying for s D 2 the Lobatto IIIA coefficients as a1;ij and the Lobatto IIIB coefficients as a2;ij , we obtain again the generalized Newton-St¨ormer-Verletleapfrog method (3). Additive Lobatto methods have been considered in multibody dynamics in [13, 21]. Additive methods are more general than partitioned methods since partitioned system of ODEs can always be reformulated as a split system of ODEs, but the reverse is false in general.

where s  2 and the coefficients a1;ij ; a2;ij ; bj ; cj characterize the additive Lobatto RK method. Consider, for example, any coefficients a1;ij and a2;ij from the family (7), the additive method is of nonstiff order Lobatto Methods for DAEs 2s  2 [13]. The partitioned system of ODEs (2) can be expressed in the form (8) by having d D dq C dp , An important use of Lobatto methods is for the solution y D .q; p/ 2 Rdq Rdp , and: of differential-algebraic equations (DAEs). DAEs consist generally of coupled systems of differential equa

v.t; q; p/ tions and nonlinear relations. They arise typically in f1 .t; q; p/ D ; 0 mechanics and electrical/electronic circuits simulation.

L

824

Lobatto Methods

Lobatto Methods, Table 4 Coefficients of Lobatto IIIC for s D 2; 3; 4; 5 0

0

0 0

1

1 0

 CsD2

1 1 2 2

0

0 0 0

1 2

1 1 0 4 4

1

0 1 0

 CsD3

1 2 1 6 3 6

1 2 1 2

 CsD4

0

0 p 21 1  2 14

1 14

1 2 p 21 1 C 2 14

1 32

1

0

 CsD5

1 20

1 14

1 2

1 2 1 2 1 2

0

5 12

1 12

0

0

0

1 9

p 13  3 21 63

p 14  3 21 126 p 91  21 21 576

p 91 C 21 21 576 p 14 C 3 21 126 7 18

11 72 p 13 C 3 21 63 2 9

49 180

16 45

Lobatto Methods, Table 5 Coefficients of Lobatto IIINW for s D 2; 3 [19]

1 0 2 1 1  2

0

1 6 1 1 2 12 1 1 2

5 12 1 3

1 6

2 3

0

0 

1 6

0 1 6 1 6

0

0

p p p 1 15  7 5 5 5C 5  0 10 60 6 60 p p p 1 5 5  5 15 C 7 5 C 0 10 60 60 6 p p 5 5 5C 5 1 1 0 6 12 12

1 9 7 18 49 180

5 12

1 12 0 0 0 0 0 1 20

hn .f .tn ; Yn1 ; n1 /  f .tnC1 ; Yn2 ; n2 //; 4 hn Yn2 D yn C .3f .tn ; Yn1 ; n1 / C f .tnC1 ; Yn2 ; n2 //; 4 hn ynC1 D yn C .f .tn ; Yn1 ; n1 / C f .tnC1 ; Yn2 ; n2 //; 2 1 0 D .k.tn ; Yn1 / C k.tnC1 ; Yn2 //; 2 0 D k.tnC1 ; ynC1 /: Yn1 D yn C

For such DAEs, a combination of Gauss and Lobatto coefficients is also considered in [18]. Consider now overdetermined system of DAEs (ODAEs) of the form:

Consider, for example, a system of DAEs of the form: d d d q D v.t; q; p/; p D f .t; q; p; /; 0 D g.t; q/; y D f .t; y; /; 0 D k.t; y/ dt dt dt 0 D Dt g.t; q/ C Dq g.t; q/v.t; q; p/ (9) where Dy k.t; y/D f .t; y; / is nonsingular. Lobatto methods can be applied to this class of problems while preserving their classical order of convergence [14]. where Dq g.t; q/Dp v.t; q; p/D f .t; q; p; / is nonFor example, the application of the two-stage Lobatto singular. Very general Lobatto methods can be applied to this type of ODAEs [13]. Hamiltonian and IIID method can be expressed as:

Lobatto Methods

825

Lagrangian systems with holonomic constraints can be expressed in the form (9). For such ODAEs, the application of Lobatto IIIA and IIIB methods can be shown to preserve their classical order of convergence, to be variational integrators, and to preserve a symplectic two-form [8, 11, 12, 17]. For example, the application of the two-stage Lobatto IIIA and IIIB method reads:

qnC1 D qn C

 hn   v tn ; qn ; pnC1=2 2   Cv tnC1 ; qnC1 ; pnC1=2 ;

 hn  f tn ; qn ; pnC1=2 ; n1 ; 2 0 D g.tnC1 ; qnC1 /;

pnC1=2 D pn C

 hn  f tnC1 ; qnC1 ; pnC1=2 ; n2 2 0 D Dt g.tnC1 ; qnC1 /

pnC1 D pnC1=2 C

CDq g.tnC1 ; qnC1 /v.tnC1 ; qnC1 ; pnC1 /:

Gauss methods with s stages can also be applied in combination with Lobatto methods with sC1 stages for this type of ODAEs when f .t; q; p; / is decomposed in f .t; q; p/ C r.t; q; / and they also possess these aforementioned properties while generally requiring less computational effort [15]. For example, the application of the midpoint-trapezoidal method (the .1; 1/Gauss-Lobatto SPARK method of Jay [15]) reads:

hn 1 v.tnC1=2 ; Qn1 ; Pn1 / D .qn C qnC1 /; 2 2 hn Pn1 D pn C f .tnC1=2 ; Qn1 ; Pn1 / 2 hn C r.tn ; qn ; n1 /; 2 qnC1 D qn C hn v.tnC1=2 ; Qn1 ; Pn1 /; Qn1 D qn C

pnC1 D pn C hn f .tnC1=2 ; Qn1 ; Pn1 /

 1 1 Chn r.tn ; qn ; n1 /C r.tnC1 ; qnC1 ; n2 / ; 2 2 0 D g.tnC1 ; qnC1 /; 0 D Dt g.tnC1 ; qnC1 / CDq g.tnC1 ; qnC1 /v.tnC1 ; qnC1 ; pnC1 /:

Lobatto Methods for Some Other Classes of Problems Lobatto IIIA methods have been considered for boundary value problems (BVP) due to their good stability properties [1, 2]. The MATLAB code bvp4c for BVP is based on three-stage collocation at Lobatto points, hence it is equivalent to the three-stage Lobatto IIIA method [16]. Lobatto methods have also been applied to delay differential equations (DDEs) [3]. The combination of Lobatto IIIA and IIIB methods has also been considered for the discrete multisymplectic integration of certain Hamiltonian partial differential equations (PDEs) such as the nonlinear Schr¨odinger equation and certain nonlinear wave equations [20].

References 1. Ascher, U.M., Mattheij, R.M.M., Russell, R.D.: Numerical Solution of Boundary Value Problems for Ordinary Differential Equations. Classics in Applied Mathematics, vol. 13. SIAM, Philadelphia (1995) 2. Bashir-Ali, Z., Cash, J.R., Silva, H.H.M.: Lobatto deferred correction for stiff two-point boundary value problems. Comput. Math. Appl. 36, 59–69 (1998) 3. Bellen, A., Guglielmi, N., Ruehli, A.E.: Methods for linear systems of circuit delay differential equations of neutral type. IEEE Trans. Circuits Syst. 46, 212–216 (1999) 4. Betts, J.T.: Practical Methods for Optimal Control and Estimation Using Nonlinear Programming. Advances in Design and Control, 2nd edn. SIAM, Philadelphia (2008) 5. Butcher, J.C.: Numerical Methods for Ordinary Differential Equations, 2nd edn. Wiley, Chichester (2008) 6. Chan, R.P.K.: On symmetric Runge-Kutta methods of high order. Computing 45, 301–309 (1990) 7. Gautschi, W.: High-order Gauss-Lobatto formulae. Numer. Algorithms 25, 213–222 (2000) 8. Hairer, E., Wanner, G.: Solving Ordinary Differential Equations II. Stiff and Differential-Algebraic Problems. Computational Mathematics, vol. 14, 2nd edn. Springer, Berlin (1996) 9. Hairer, E., Nørsett, S.P., Wanner, G.: Solving Ordinary Differential Equations I. Nonstiff Problems. Computational Mathematics, vol. 18, 2nd edn. Springer, Berlin (1993) 10. Hairer, E., Lubich, C., Wanner, G.: Geometric numerical integration illustrated by the St¨ormer/Verlet method. Acta Numer. vol. 12, 1–51 (2003) 11. Hairer, E., Lubich, C., Wanner, G.: Geometric Numerical Integration. Computational Mathematics, vol. 31, 2nd edn. Springer, Berlin (2006) 12. Jay, L.O.: Symplectic partitioned Runge-Kutta methods for constrained Hamiltonian systems. SIAM J. Numer. Anal. 33, 368–387 (1996) 13. Jay, L.O.: Structure preservation for constrained dynamics with super partitioned additive Runge-Kutta methods. SIAM J. Sci. Comput. 20, 416–446 (1998)

L

826 14. Jay, L.O.: Solution of index 2 implicit differential-algebraic equations by Lobatto Runge-Kutta methods. BIT 43, 91–104 (2003) 15. Jay, L.O.: Specialized partitioned additive Runge-Kutta methods for systems of overdetermined DAEs with holonomic constraints. SIAM J. Numer. Anal. 45, 1814–1842 (2007) 16. Kierzenka, J., Shampine, L.F.: A BVP solver based on residual control and the MATLAB PSE. ACM Trans. Math. Softw. 27, 299–316 (2001) 17. Leimkuhler, B., Reich, S.: Simulating Hamiltonian Dynamics, Cambridge Monographs on Applied and Computational Mathematics, vol. 14. Cambridge University Press, Cambridge (2005) 18. Murua, A.: Partitioned Runge-Kutta methods for semiexplicit differential-algebraic systems of index 2. Technical Report EHU-KZAA-IKT-196, University of the Basque country (1996) 19. Nørsett, S.P., Wanner, G.: Perturbed collocation and RungeKutta methods. Numer. Math. 38, 193–208 (1981) 20. Ryland, B.N., McLachlan, R.I.: On multisymplecticity of partitioned Runge-Kutta methods. SIAM J. Sci. Comput. 30, 1318–1340 (2008) 21. Schaub, M., Simeon, B.: Blended Lobatto methods in multibody dynamics. Z Angew. Math. Mech. 83, 720–728 (2003) 22. Sun, G.: A simple way of constructing symplectic RungeKutta methods. J. Comput. Math. 18, 61–68 (2000) 23. von Matt, U.: Gauss quadrature. In: Gander, W., Hˇreb´ıcˇ ek, J. (eds.) Solving Problems in Scientific Computing Using Maple and Matlab, vol. 14, 4th edn. Springer, Berlin (2004)

Logarithmic Norms

Logarithmic Norms

as well as boundary value problems and their discretizations. Some special fields in mathematics, such as semigroup theory, rely on notions that are strongly related to the logarithmic norm. Let j  j denote an arbitrary vector norm on Cd , as well as its subordinate operator norm on Cd d . The classical definition of the logarithmic norm of A 2 Cd d is M ŒA D lim

h!0C

Introduction The logarithmic norm is a real-valued functional on operators, quantifying the notions of definiteness for matrices; monotonicity for nonlinear maps; and ellipticity for differential operators. It is defined either in terms of an inner product in Hilbert space, or in terms of the operator norm on a Banach space. The logarithmic norm has a wide range of applications in matrix theory, stability theory, and numerical analysis. It offers various quantitative bounds on (functions of) operators, operator spectra, resolvents, Rayliegh quotients, and the numerical range. It also offers error bounds and stability estimates in initial

(1)

It is easily computed for the most common norms, see Table 1. In Hilbert space, where the norm is generated by an inner product jxj2 D hx; xi, one may alternatively define the least upper bound logarithmic norm M ŒA and the greatest lower bound logarithmic norm mŒA such that for all x mŒA  jxj2  Re hx; Axi  M ŒA  jxj2 :

(2)

Unlike (1), this also admits unbounded operators, while still agreeing with (1) if A is bounded, in which case it also holds that mŒA D lim

h!0

Gustaf S¨oderlind Centre for Mathematical Sciences, Numerical Analysis, Lund University, Lund, Sweden

jI C hAj  1 : h

jI C hAj  1 : h

(3)

The functionals M Œ and mŒ can further be extended to nonlinear maps, both in a Banach and a Hilbert space setting, so that the above definitions become special cases for linear operators. The logarithmic norm has a large number of useful properties and satisfy several important inequalities. For A; B 2 Cd d , ˛ 2 R and z 2 C, some of the most important are: 1. glbŒA  M ŒA  jAj 2. M Œ˛A D ˛M ŒA ; ˛  0 3. M ŒA C zI D M ŒA C Re z 4. mŒA D M ŒA

5. M ŒA C mŒB  M ŒA C B  M ŒA C M ŒB

6. jM ŒA  M ŒB j  jA  Bj 7. jmŒA  mŒB j  jA  Bj 8. et mŒA  jetA j  etM ŒA ; t  0 9. M ŒA < 0 ) jA1 j  1=M ŒA

10. mŒA > 0 ) jA1 j  1=mŒA :

Logarithmic Norms

827

Logarithmic Norms, Table 1 Computation of l p vector, matrix, and logarithmic norms. Here Œ and ˛Œ denote the spectral radius and spectral abscissa of a matrix, respectively (From [3]) Vector norm P jxj1 D i jxi j jxj2 D

pP i

jxi

Matrix norm jAj1 D maxj j2

jAj2 D

jxj1 D maxi jxi j

p

P i

Logarithmic norm X   M1 ŒA D max Re ajj C jaij j

jaij j

j

jAj1 D maxi

P j

i¤j

M2 ŒA D ˛Œ.A C A /=2

H

ŒAH A

jaij j

X   M1 ŒA D max Re ai i C jaij j i

j ¤i

(8) gives jA1 rj  jrj=M ŒA for all r. Therefore, even the inverse of A can be bounded in terms of the The logarithmic norm was originally introduced for logarithmic norm, as matrices, [3, 12], in order to establish bounds for solutions to a linear system 1 : (9) M ŒA < 0 ) jA1 j   M ŒA

xP D Ax C r: (4) This inequality is of particular importance also in The norm of x satisfies the differential inequality boundary value problems, where it provides a bound for the inverse of an elliptic operator. DC (5) t jxj  M ŒA  jxj C jr.t/j ;

Differential Inequalities

where M ŒA is the logarithmic norm of A and DC t jxj Spectral Bounds is the upper right Dini derivative of jxj with respect to time. Consider first the homogeneous case r  0; this For the spectrum of a general matrix A it holds that is akin to the Gr¨onwall lemma. Then x.t/ D etA x.0/, and (5) provides the matrix exponential bound ŒA  jAj I ˛ŒA  M ŒA ; (10) jetA j  etM ŒA I

t  0:

(6)

Thus the condition M ŒA < 0 implies that the matrix exponential is a contraction (semi-)group. Consider next the case x.0/ D 0, with r ¤ 0. By integration of (5), the solution is then bounded on compact intervals by jx.t/j 

etM ŒA  1 krk1 ; M ŒA

(7)

where krk1 D sup jr./j. If M ŒA < 0, the bound also holds as t ! 1, in which case kxk1  

krk1 ; M ŒA

(8)

showing that x depends continuously on the data r. Finally, consider xP D Ax C r with r  const. If M ŒA < 0, homogeneous solutions decay to a unique equilibrium x D A1 r. Taking x.0/ D A1 r,

where ŒA D maxi j i j is the spectral radius of A and ˛ŒA D maxi Re i is the spectral abscissa. The operator norm is an upper bound for the magnitude of the eigenvalues, while the logarithmic norm is an upper bound for the real part of the eigenvalues. Equality is usually not attained, except in important special cases. For example, the Euclidean norms j  j2 and M2 Œ are sharp for the entire class of normal matrices. All eigenvalues of A are thus contained in the strip mŒA  Re  M ŒA (for any choice of norm). They are also contained in the annulus glbŒA  j j  jAj. Further, from (2) it follows that M ŒA and mŒA are the maximum and minimum of the Rayleigh quotient. This implies that mŒA > 0 generalizes and quantifies the notion of a positive definite matrix, while M ŒA < 0 generalizes negative definiteness. Moreover, M ŒA and mŒA are also the maximal and minimal real parts, respectively, of the numerical range of an operator [16]. Resolvents can also be bounded in half-planes. Thus, as a generalization of (9), one has

L

828

Logarithmic Norms

for u; v 2 D, the domain of f . The lub Lipschitz constant is an operator semi-norm that generalizes the matrix norm: if f D A is a linear map, then A similar bound can be obtained in the half-plane LŒA D jAj. One can then define two more functionals Re z < mŒA . on D, the lub logarithmic Lipschitz constant and the While the bounds above hold for all norms, some glb logarithmic Lipschitz constant, by less obvious results can be obtained in Hilbert space. LŒI C hf  1 According to the well-known spectral theory of von I M Œf D lim h!0C h Neumann, [17], if a polynomial has the property jzj  1 ) jP .z/j  1, then this property can be extended to LŒI C hf  1 : (14) mŒf D lim matrices and norms. Thus, if a matrix is a contraction h!0 h with respect to an inner product norm, then so is P .A/, i.e., jAjH  1 ) jP .A/jH  1, where the subscript Naturally, these definitions only apply to “bounded H refers to the Hilbert space topology. This result also operators,” which here correspond to Lipschitz maps. holds for rational functions, as well as over half-planes In Hilbert space, however, one can also include unin C. Thus, if R is a rational function such that Re z  bounded operators; in analogy with (2), one then de0 ) jR.z/j  1, then MH ŒA  0 ) jR.A/jH  1. fines mH Œ and MH Œ as the best constants such that This is of particular importance in the stability the- the inequalities ory of Runge–Kutta methods for ordinary differential equations. When such a method is applied to the linear mH Œf  ju  vj2H  Re hu  v; f .u/  f .v/iH test equation xP D x with step size h, the solution is  MH Œf  ju  vj2H (15) advanced by a recursion of the form xnC1 D R.h /xn , where the stability function R.z/ approximates ez . The hold for all u; v 2 D. For Lipschitz maps, these method is called A-stable if Re z  0 ) jR.z/j  1. It definitions are compatible with (14), and the linear then follows that every A-stable Runge–Kutta method theory is fully extended to nonlinear problems. All has the property that, when applied to a linear system previously listed general properties of the logarithmic xP D Ax, norm are preserved, although attention must be paid to the domains of the operators involved. The terminology MH ŒA  0 ) jR.hA/jH  1: (11) is also different. Thus, a map with M Œf < 0 (or mŒf > 0) is usually called strongly monotone. Such This implies that the method has stability properties a map is one-to-one from D to f .D/ with a Lipschitz similar to those of the differential equation, as both are inverse: contractive when MH ŒA < 0; by (6), we have 1 : (16) M Œf < 0 ) LŒf 1   hA MH ŒA  0 ) je jH  1: (12) M Œf

M ŒA < Re z ) j.A  zI /1 j
0 and i D 1; y  0, provided that f .x; N y/  b.x; N y/ for all x; N y. The addition “on notation” refers to the fact that this definition is most naturally understood if one thinks of natural numbers in binary notation.

Logical Characterizations of Complexity Classes

Cobham’s Theorem [4]. The class of P-functions is the closure of the basic functions x 7! 0 (“constant 0”), .x1 ; : : : ; xk / 7! xi for all i  k (“projections”), x 7! 2x and x 7! 2x C 1 (“successor functions”), and .x; y/ 7! 2jxjjyj (“smash function”), where jxj denotes the length of the binary representation of x, under composition and bounded primitive recursion on notation.

Similar characterizations are known for other complexity classes. What is slightly unsatisfactory about Cobham’s characterization of the P-functions is the explicit time bound b in the bounded primitive recursion scheme. Bellantoni and Cook [1] devised a refined primitive recursion scheme that distinguishes between different types of variables and how they may be used and characterize the P-functions without an explicit time bound. This is the starting point of the area of “implicit computational complexity” ([10] is a survey). While Bellantoni and Cook’s recursion scheme is still fairly restrictive, in the sense that the type system excludes natural definitions of P-functions by primitive recursion, subsequently researchers have developed a variety of full (mostly functional) programming languages with very elaborate type systems guaranteeing that precisely the K-functions (for many of the standard complexity classes K) have programs in this language. The best known of these is Hofmann’s functional language for the P-functions with a type system incorporating ideas from linear logic [11]. Proof Theory and Bounded Arithmetic There is yet another line of logical characterizations of complexity classes. It is based on provability in formal system rather than just definability. Again, these characterizations have precursors in computability theory, in particular the characterization of the primitive recursive functions as precisely those functions that are ˙1 -definable in the fragment i˙1 of Peano arithmetic. The setup is fairly complicated, and we will only be able to scratch the surface; for a thorough treatment, we refer the reader to the survey [3] and the textbook [5]. Our basic logic is first-order logic in the language of arithmetic, consisting of the standard symbols  (order), C (addition),  (multiplication), 0,

833

1 (constants 0 and 1), and possibly additional function symbols. In the standard model of arithmetic N , all these symbols get their standard interpretations over the natural numbers. A theory is a set of first-order sentences that is closed under logical consequence. For example, Th.N / is the set of all sentences that are true in the standard model N . It follows from G¨odel’s First Incompleteness Theorem that Th.N / has no decidable axiom system. A decidable, yet still very powerful, theory that contained Th.N / is Peano arithmetic PA. It is axiomatized by a short list of basic axioms making sure that the basic symbols are interpreted right together with inductionaxioms of the  form .0/ ^ 8x..x/ ! .x C 1// ! 8x.x/ for all first-order formulas . Here, we are interested in fragments i˚ of PA obtained by restricting the induction axioms to formulas  2 ˚ for sets ˚ of first-order formulas. 0 denotes the set of all bounded first-order formulas, that is, formulas where all quantifications are of the form 9x  t or 8x  t for some term t that does not contain the variable x. Almost everything relevant for complexity theory takes place within 0 , but let us mention that ˙1 is the set of all first-order formulas of the form 9x, where  is a 0 formula. We say that a function f on the natural numbers is definable in a theory T if there is a formula .x; y/ such that the theory T proves that for all x there is exactly one y such that .x; y/ and for all natural numbers m; n the standard model N satisfies .m; n/ if and only if f .m/ D n. For example, it can be shown that the functions in the linear time hierarchy LTH are precisely the functions that are 0 -definable in the theory i0 . To characterize the classes P and NP and the other classes of the polynomial hierarchy, Buss introduced a hierarchy of very weak arithmetic theories Si2 . They are obtained by even restricting the use of bounded quantifiers in 0 -formulas, defining a hierarchy of ˙ib formulas within 0 but at the same time using an extended language that also contains functions symbols like # (for the “smash” function .x; y/ 7! 2jxjjyj ) and j j (for the binary length).

Buss’s Theorem [2]. For all i  1, the functions ˙ib definable in Si2 are precisely the ˙iP1 -functions, where ˙0P D P, ˙1P D NP, and ˙iP are the i th level of the polynomial hierarchy.

L

834

Lyapunov Exponents: Computation

concepts to investigate stability in differential equations. These are collectively known as Lyapunov Bellantoni, S., Cook, S.: A new recursion-theoretic charac- Stability Theory. Lyapunov was concerned with the terization of the polytime functions. Comput. Complex. 2, asymptotic stability of solutions with respect to 97–110 (1992) perturbations of initial data. Among other techniques Buss, S.: Bounded Arithmetic. Bibliopolis, Napoli (1986) Buss, S.: First-order proof theory of arithmetic. In: Buss, S. (e.g., what are now known as first and second (ed.) Handbook of Proof Theory, pp. 79–147. Elsevier, New Lyapunov methods), he introduced a new tool to York (1998) analyze the stability of solutions of linear timeCobham, A.: The intrinsic computational difficulty of functions. In: Proceedings of the International Conference on varying systems of differential equations, the soLogic, Methodology, and Philosophy of Science, pp. 24–30. called characteristic numbers, now commonly and North-Holland, Amsterdam (1962) appropriately called Lyapunov exponents. Cook, S., Nguyen, P.: Logical Foundations of Proof ComSimply put, these characteristic numbers play the plexity. Perspectives in Logic. Cambridge University Press, role that the (real parts of the) eigenvalues play for Cambridge/New York (2010) time-invariant linear systems. Lyapunov considered the Ebbinghaus, H.-D., Flum, J.: Finite Model Theory. Springer, Berlin/New York (1995) n-dimensional linear system

References 1.

2. 3.

4.

5.

6.

7. Fagin, R.: Generalized first–order spectra and polynomial– time recognizable sets. In: Karp, R. (ed.) Complexity of Computation, SIAM-AMS Proceedings, New York, vol. 7, pp. 43–73 (1974) 8. Gr¨adel, E., Kolaitis, P., Libkin, L., Marx, M., Spencer, J., Vardi, M., Venema, Y., Weinstein, S.: Finite Model Theory and Its Applications. Springer, Berlin/New York (2007) 9. Grohe, M.: From polynomial time queries to graph structure theory. Commun. ACM 54(6), 104–112 (2011) 10. Hofmann, M.: Programming languages capturing complexity classes. ACM SIGACT News 31(1), 31–42 (2000) 11. Hofmann, M.: Linear types and non-size-increasing polynomial time computation. Inf. Comput. 183, 57–85 (2003) 12. Immerman, N.: Relational queries computable in polynomial time (extended abstract). In: Proceedings of the 14th ACM Symposium on Theory of Computing, San Francisco, pp. 147–152 (1982) 13. Immerman, N.: Descriptive Complexity. Springer, New York (1999) 14. Vardi, M.: The complexity of relational query languages. In: Proceedings of the 14th ACM Symposium on Theory of Computing, San Francisco, pp. 137–146 (1982)

Lyapunov Exponents: Computation Luca Dieci1 and Erik S. Van Vleck2 1 School of Mathematics, Georgia Institute of Technology, Atlanta, GA, USA 2 Department of Mathematics, University of Kansas, Lawrence, KS, USA

xP D A.t/x; t  0 ;

(1)

where A is continuous and bounded: supt kA.t/k< 1. He showed that “if all characteristic numbers (see below for their definition) of (1) are negative, then the zero solution of (1) is asymptotically (in fact, exponentially) stable.” He further proved an important characterization of stability relative to the perturbed linear system xP D A.t/x C f .t; x/ ;

(2)

where f .t; 0/ D 0, so that x D 0 is a solution of (2), and further f .t; x/ is assumed to be “small” near x D 0 (this situation is what one expects from a linearized analysis about a bounded solution trajectory). Relative to (2), Lyapunov proved that “if the linear system (1) is regular, and all its characteristic numbers are negative, then the zero solution of (2) is asymptotically stable.” About 30 years later, it was shown by Perron in [38] that the assumption of regularity cannot generally be removed.

Definition We refer to the monograph [1] for a comprehensive definition of Lyapunov exponents, regularity, and so forth. Here, we simply recall some of the key concepts. Consider (1) and let us stress that the matrix funcHistory and Scope tion A.t/ may be either given or obtained as the linearization about the solution of a nonlinear differential In 1892, in his doctoral thesis The general problem of equation; e.g., yP D f .y/ and A.t/ D Df .y.t// (note the stability of motion (reprinted in its original form in that in this case, in general, A will depend on the initial [33]), Lyapunov introduced several groundbreaking condition used for the nonlinear problem). Now, let X

Lyapunov Exponents: Computation

835

be a fundamental matrix solution of (1), and consider important results on the prevalence of this property the quantities within the class of linear systems. i D lim sup t !1

1 ln jjX.t/ei jj ; i D 1; : : : ; n; t

(3)

where ei denotes the i th standard unit vector, i D n P i is minimized with respect to 1; : : : ; n. When i D1

all possible fundamental matrix solutions, then the i are called the characteristic numbers, or Lyapunov exponents, of the system. It is customary to consider them ordered as 1  2      n . Similar definitions can be given for t ! 1 and/or with lim inf replacing the lim sup, but the description above is the prevailing one. An important consequence of regularity of a given system is that in (3) one has limits instead of lim sups. More Recent Theory Given that the condition of regularity is not easy to verify for a given system, it was unclear what practical use one was going to make of the Lyapunov exponents in order to study stability of a trajectory. Moreover, even assuming that the system is regular, it is effectively impossible to get a handle on the Lyapunov exponents except through their numerical approximation. It then becomes imperative to have some comfort that what one is trying to approximate is robust; in other words, it is the Lyapunov exponents themselves that will need to be stable with respect to perturbations of the function A in (1). Unfortunately, regularity is not sufficient for this purpose. Major theoretical advances to resolve the two concerns above took place in the late 1960s, thanks to the work of Oseledec and Millionshchikov (e.g., see [36] and [34]). Oseledec was concerned with stability of trajectories on a (bounded) attractor, on which one has an invariant measure. In this case, Oseledec’s Multiplicative Ergodic Theorem validates regularity for a broad class of linearized systems; the precise statement of this theorem is rather technical, but its practical impact is that (with respect to the invariant measure) almost all trajectories of the nonlinear system will give rise to a regular linearized problem. Millionshchikov introduced the concept of integral separation, which is the condition needed for stability of the Lyapunov exponents with respect to perturbations in the coefficient matrix, and further gave

Further Uses of Lyapunov Exponents Lyapunov exponents found an incredible range of applicability in several contexts, and both theory and computational methods have been further extended to discrete dynamical systems, maps, time series, etc. In particular: (i) The largest Lyapunov exponent of (2), 1 , characterizes the rate of separation of trajectories (with infinitesimally close initial conditions). For this reason, a positive value of 1 (coupled with compactness of the phase space) is routinely taken as an indication that the system is chaotic (see [37]). (ii) Lyapunov exponents are used to estimate dimension of attractors through the Kaplan-Yorke formula (Lyapunov dimension): DimL D k C . 1 C 2 C    C k /=j kC1 j

(iii)

(iv)

(v)

(vi)

where k is the largest index i such that 1 C 2 C    C i > 0. See [31] for the original derivation of the formula and [9] for its application to the 2-d Navier-Stokes equation. The sum of all the positive Lyapunov exponents is used to estimate the entropy of a dynamical system (see [3]). Lyapunov exponents have also been used to characterize persistence and degree of smoothness of invariant manifolds (see [26] and see [12] for a numerical study). Lyapunov exponents have even been used in studies of piecewise-smooth differential equations, where a formal linearized problem as in (1) does not even exist (see [27, 35]). Finally, there has been growing interest also in approximating bases for the growth directions associated to the Lyapunov exponents. In particular, there is interest in obtaining representations for the stable (and unstable) subspaces of (1) and in their use to ascertain stability of traveling waves. For example, see [23, 39].

Factorization Techniques Many of the applications listed above are related to nonlinear problems, which in itself is witness

L

836

Lyapunov Exponents: Computation

to the power of linearized analysis based on the Lyapunov exponents. Still, the computational task of approximating some or all of the Lyapunov exponents for dynamical systems defined by the flow of a differential equation is ultimately related to the linear problem (1), and we will thus focus on this linear problem. Techniques for numerical approximation of Lyapunov exponents are based upon smooth matrix factorizations of fundamental matrix solutions X , to bring it into a form from which it is easier to extract the Lyapunov exponents. In practice, two techniques have been studied: based on the QR factorization of X and based on the SVD (singular value decomposition) of X . Although these techniques have been adapted to the case of incomplete decompositions (useful when only a few Lyapunov exponents are needed) or to problems with Hamiltonian structure, we only describe them in the general case when the entire set of Lyapunov exponents is sought, the problem at hand has no particular structure, and the system is regular. For extensions, see the references.

have two differential equations, for Q and for R. Given X.0/ D Q0 R0 , we have QP D QS.Q; A/ ; RP D B.t/R ;

Q.0/ D Q0 ;

R.0/ D R0 ;

B W D QT AQ  S.Q; A/

P AQR D QRP C QR

or QP D AQ  QB ;

(4)

where RP D BR; hence, B must be upper triangular. Now, let us formally set S D QT QP and note that since Q is orthogonal then S must be skew symmetric. Now, from B D QT AQ  QT QP it is easy to determine at once the strictly lower triangular part of S (and from this, all of it) and the entries of B. To sum up, we

(6)

The diagonal entries of R are used to retrieve the exponents: 1 t !1 t

Z

t

i D lim

.QT .s/A.s/Q.s//i i ds ; i D 1; : : : ; n: 0

(7) A unit upper triangular representation for the growth directions may be further determined by limt !1 diag.R1 .t//R.t/ (see [13, 22, 23]). Discrete QR Here one seeks the QR factorization of the fundamental matrix X at discrete points 0 D t0 < t1 <    < tk <    , where tk D tk1 C hk , hk  hO > 0. Let X0 D Q0 R0 , and suppose we seek the QR factorization of X.tkC1 /. For j D 0; : : : ; k, progressively define Zj C1 .t/ D X.t; tj /Qj , where X.t; tj / solves (1) for t  tj , X.tj ; tj / D I , and Zj C1 is the solution of

QR Methods The idea of QR methods is to seek the factorization of a fundamental matrix solution as X.t/ D Q.t/R.t/, for all t, where Q is an orthogonal matrix valued function and R is an upper triangular matrix valued ZP j C1 D A.t/Zj C1 ; tj  t  tj C1 function with positive diagonal entries. The validity of Zj C1 .tj / D Qj : this factorization has been known since Perron [38] and Diliberto [25], and numerical techniques based upon Update the QR factorization as the QR factorization date back at least to [4]. QR techniques come in two flavors, continuous Zj C1 .tj C1 / D Qj C1 Rj C1 ; and discrete, and methods for quantifying the error in approximation of Lyapunov exponents have been and finally observe that developed in both cases (see [15–17, 21, 40]). Continuous QR Upon differentiating the relation X D QR and using (1), we have

(5)

X.tkC1 / D QkC1 ŒRkC1 Rk    R1 R0

(8)

(9)

(10)

is the QR factorization of X.tkC1 /. The Lyapunov exponents are obtained from the relation k 1 X log.Rj /i i ; i D 1; : : : ; n : k!1 tk j D0

lim

(11)

SVD Methods Here one seeks to compute the SVD of X : X.t/ D U.t/†.t/V T .t/, for all t, where U and V are orthogonal and † D diag.i ; i D 1 : : : ; n/, with

Lyapunov Exponents: Computation

837

1 .t/  2 .t/      n .t/. If the singular values are distinct, the following differential equations U; V; and † hold. Letting G D U T AU , they are UP D UH;

VP T D K V T ;

P D D†; †

(12)

where D D diag.G/, H T D H , and K T D K, and for i ¤ j , Hij D

Gij j2 C Gj i i2 j2  i2

;

Kij D

.Gij C Gj i /i j : j2  i2 (13)

From the SVD of X , the Lyapunov exponents may be obtained as lim

t !1

1 ln i .t/ : t

(14)

Finally, an orthogonal representation for the growth directions may be determined by limt !1 V .t/ (see [10, 13, 22, 23]). Numerical Implementation Although algorithms based upon the above techniques appear deceivingly simple to implement, much care must be exercised in making sure that they perform as one would expect them to. (For example, in the continuous QR and SVD techniques, it is mandatory to maintain the factors Q; U , and V orthogonal.) Fortran software codes for approximating Lyapunov exponents of linear and nonlinear problems have been developed and tested extensively and provide a combined state of the knowledge insofar as numerical methods suited for this specific task. See [14, 20, 24]. Acknowledgements Erik Van Vleck acknowledges support from NSF grant DMS-1115408.

References 1. Adrianova, L.Ya.: Introduction to Linear Systems of Differential Equations (Trans. from the Russian by Peter Zhevandrov). Translations of Mathematical Monographs, vol. 146, pp. x+204. American Mathematical Society, Providence (1995) 2. Aston, P.J., Dellnitz, M.: The computation of Lyapunov exponents via spatial integration with application to blowout bifurcations. Comput. Methods Appl. Mech. Eng. 170, 223–237 (1999)

3. Barreira, L., Pesin, Y.: Lyapunov Exponents and Smooth Ergodic Theory. University Lecture Series, vol. 23. American Mathematical Society, Providence (2001) 4. Benettin, G., Galgani, L., Giorgilli, A., Strelcyn, J.-M.: Lyapunov exponents for smooth dynamical systems and for Hamiltonian systems: a method for computing all of them. Part 1: Theory, and . . . Part 2: Numerical applications. Meccanica 15, 9–20, 21–30 (1980) 5. Bridges, T., Reich, S.: Computing Lyapunov exponents on a Stiefel manifold. Physica D 156, 219–238 (2001) 6. Bylov, B.F., Vinograd, R.E., Grobman, D.M., Nemyckii, V.V.: The Theory of Lyapunov Exponents and Its Applications to Problems of Stability. Nauka, Moscow (1966) 7. Calvo, M.P., Iserles, A., Zanna, A.: Numerical solution of isospectral flows. Math. Comput. 66(220), 1461–1486 (1997) 8. Christiansen, F., Rugh, H.H.: Computing Lyapunov spectra with continuous Gram-Schmidt orthonormalization. Nonlinearity 10, 1063–1072 (1997) 9. Constantin, P., Foias, C.: Global Lyapunov exponents, Kaplan-Yorke formulas and the dimension of the attractors for 2D Navier-Stokes equations. Commun. Pure Appl. Math. 38, 1–27 (1985) 10. Dieci, L., Elia, C.: The singular value decomposition to approximate spectra of dynamical systems. Theoretical aspects. J. Differ. Equ. 230(2), 502–531 (2006) 11. Dieci, L., Lopez, L.: Smooth SVD on symplectic group and Lyapunov exponents approximation. CALCOLO 43(1), 1–15 (2006) 12. Dieci, L., Lorenz, J.: Lyapunov type numbers and torus breakdown: numerical aspects and a case study. Numer. Algorithms 14, 79–102 (1997) 13. Dieci, L., Van Vleck, E.S.: Lyapunov spectral intervals: theory and computation. SIAM J. Numer. Anal. 40(2), 516– 542 (2002) 14. Dieci, L., Van Vleck, E.S.: LESLIS and LESLIL: codes for approximating Lyapunov exponents of linear systems. Technical report, Georgia Institute of Technology. http:// www.math.gatech.edu/dieci (2004) 15. Dieci, L., Van Vleck, E.S.: On the error in computing Lyapunov exponents by QR methods. Numer. Math. 101(4), 619–642 (2005) 16. Dieci, L., Van Vleck, E.S.: Perturbation theory for approximation of Lyapunov exponents by QR methods. J. Dyn. Differ. Equ. 18(3), 815–840 (2006) 17. Dieci, L., Van Vleck, E.S.: On the error in QR integration. SIAM J. Numer. Anal. 46(3), 1166–1189 (2008) 18. Dieci, L., Russell, R.D., Van Vleck, E.S.: Unitary integrators and applications to continuous orthonormalization techniques. SIAM J. Numer. Anal. 31(1), 261–281 (1994) 19. Dieci, L., Russell, R.D., Van Vleck, E.S.: On the computation of Lyapunov exponents for continuous dynamical systems. SIAM J. Numer. Anal. 34, 402–423 (1997) 20. Dieci, L., Jolly, M., Van Vleck, E.S.: LESNLS and LESNLL: codes for approximating Lyapunov exponents of nonlinear systems. Technical report, Georgia Institute of Technology. http://www.math.gatech.edu/dieci (2005) 21. Dieci, L., Jolly, M., Rosa, R., Van Vleck, E.: Error on approximation of Lyapunov exponents on inertial manifolds: the Kuramoto-Sivashinsky equation. J. Discret. Contin. Dyn. Syst. Ser. B 9(3–4), 555–580 (2008)

L

838 22. Dieci, L., Elia, C., Van Vleck, E.S.: Exponential dichotomy on the real line: SVD and QR methods. J. Differ. Equ. 248(2), 287–308 (2010) 23. Dieci, L., Elia, C., Van Vleck, E.S.: Detecting exponential dichotomy on the real line: SVD and QR algorithms. BIT 51(3), 555–579 (2011) 24. Dieci, L., Jolly, M.S., Van Vleck, E.S.: Numerical techniques for approximating Lyapunov exponents and their implementation. ASME J. Comput. Nonlinear Dyn. 6, 011003–1–7 (2011) 25. Diliberto, S.P.: On systems of ordinary differential equations. In: Lefschetz, S. (ed.) Contributions to the Theory of Nonlinear Oscillations. Annals of Mathematics Studies, vol. 20, pp. 1–38. Princeton University Press, Princeton (1950) 26. Fenichel, N.: Persistence and smoothness of invariant manifolds for flows. Indiana Univ. Math. J. 21, 193–226 (1971) 27. Galvanetto, U.: Numerical computation of Lyapunov exponents in discontinuous maps implicitly defined. Comput. Phys. Commun. 131, 1–9 (2000) 28. Geist, K., Parlitz, U., Lauterborn, W.: Comparison of different methods for computing Lyapunov exponents. Prog. Theor. Phys. 83, 875–893 (1990) 29. Goldhirsch, I., Sulem, P.L., Orszag, S.A.: Stability and Lyapunov stability of dynamical systems: a differential approach and a numerical method. Physica D 27, 311–337 (1987) 30. Greene, J.M., Kim, J.-S.: The calculation of Lyapunov spectra. Physica D 24, 213–225 (1987) 31. Kaplan, J.L., Yorke, J.A.: Chaotic behavior of multidimensional difference equations. In: Peitgen, H.-O., Walter,

Lyapunov Exponents: Computation

32.

33. 34.

35.

36.

37. 38. 39.

40.

41.

H.-O. (eds.) Functional Differential Equations and Approximations of Fixed Points. Lecture Notes in Mathematics, vol. 730. Springer, Berlin (1979) Leimkuhler, B.J., Van Vleck, E.S.: Orthosymplectic integration of linear Hamiltonian systems. Numer. Math. 77(2), 269–282 (1997) Lyapunov, A.: Probl´em G´eneral de la Stabilit´e du Mouvement. Int. J. Control 53, 531–773 (1992) Millionshchikov, V.M.: Systems with integral division are everywhere dense in the set of all linear systems of differential equations. Differ. Uravn. 5, 1167–1170 (1969) M¨uller, P.: Calculation of Lyapunov exponents for dynamic systems with discontinuities. Chaos Solitons Fractals 5, 1671–1681 (1995) Oseledec, V.I.: A multiplicative ergodic theorem. Lyapunov characteristic numbers for dynamical systems. Trans. Mosc. Math. Soc. 19, 197–231 (1968) Ott, E.: Chaos in Dynamical Systems, 2nd edn. Cambridge University Press, Cambridge (2002) Perron, O.: Die Ordnungszahlen Linearer Differentialgleichungssysteme. Math. Z. 31, 748–766 (1930) Sandstede, B.: Stability of travelling waves. In: Hasselblatt, B., Katok, A.B. (eds.) Handbook of Dynamical Systems, vol. 2, pp. 983–1055. North-Holland, Amsterdam (2002) Van Vleck, E.S.: On the error in the product QR decomposition. SIAM J. Matrix Anal. Appl. 31(4), 1775–1791 (2009/2010) Wiesel, W.E.: Continuous-time algorithm for Lyapunov exponents: Part 1, and Part 2. Phys. Rev. E 47, 3686–3697 (1993)

M

Machine Learning Algorithms Ding-Xuan Zhou Department of Mathematics, City University of Hong Kong, Hong Kong, China

Machine learning algorithms are motivated by the mission of extracting and processing information from massive data which challenges scientists and engineers in various fields such as biological computation, computer vision, data mining, image processing, speech recognition, and statistical analysis. Tasks of machine learning include regression, classification, dimension reduction, clustering, ranking, and feature selection. Learning algorithms aim at learning from sample data structures or function relations (responses or labels) to events by means of efficient computing tools. The main difficulty of learning problems lies in the huge sample size or the large number of variables. Solving these problems relies on suitable statistical modelling and powerful computational methods to tackle the involved large-size optimization problems. There are essentially three categories of learning problems: supervised learning, unsupervised learning, and semi-supervised learning. The input space X for a learning problem contains its possible events x. A typical case is when X is a subset of a Euclidean space IRn with an element x D .x 1 ; : : : ; x n / 2 X corresponding to n numerical measures for a practical event. For a supervised learning problem, the output space Y contains all possible responses or labels y, and it might be a set of real numbers Y  IR for regression or a finite set of labels for classification.

A supervised learning algorithm produces a function fz W X ! Y based on a given set of examples z D f.xi ; yi / 2 X Y gm i D1 . It predicts a response fz .x/ 2 Y to each future event x 2 X . The prediction accuracy or learning ability of an output function f W X ! Y may be measured quantitatively by means of a loss function V W Y  Y ! IRC as V .f .x/; y/ for an input-output pair .x; y/ 2 X  Y . For regression, the least squares loss V .f .x/; y/ D .f .x/  y/2 is often used and it gives the least squares error when the output function value f .x/ (predicted value) approximates the true output value y. The first family of supervised learning algorithms can be stated as empirical risk minimization (ERM). Such an algorithm [1, 16] is implemented by minimizing the empirical risk or empirical error Ez .f / WD 1 Pm i D1 V .f .xi /; yi / over a set H of functions from X m to Y (called a hypothesis space) fz D arg min Ez .f /: f 2H

(1)

Its convergence can be analyzed by the theory of uniform convergence or uniform law of large numbers [1, 4, 6, 16]. The second family of supervised learning algorithms can be stated as Tikhonov regularization schemes in .HK ; k  kK /, a reproducing kernel Hilbert space (RKHS) associated with a reproducing kernel K W X  X ! IR, a symmetric and positive semidefinite function. Such a regularization scheme is a kernel method [3, 7, 9, 12, 16, 17] defined as

© Springer-Verlag Berlin Heidelberg 2015 B. Engquist (ed.), Encyclopedia of Applied and Computational Mathematics, DOI 10.1007/978-3-540-70529-1

˚  fz D arg min Ez .f / C kf k2K ; f 2HK

(2)

840

Machine Learning Algorithms

where  > 0 is a regularization parameter (which might be determined by cross-validation). The Tikhonov regularization scheme (with a general function space) has a long history in various areas [14]. It is powerful for learning due to the important property of the RKHS: f .x/ D hf; K.; x/iK . Taking the orthogonal projection onto the subspace P m f m i D1 ci K.; xi / W c 2 IR g does not change the empirical error of f 2 HK . Hence, the minimizer fz of (2) must lie in this subspace (representer theorem [17]) and its coefficients can be computed by minimizing the induced function over IRm . This reduces the computational complexity while the strong approximation ability provided by the possibly infinitely many dimensional function space HK may be maintained [5, 11]. Moreover, when V is convex with respect to the first variable, the algorithm is implemented by a convex optimization problem in IRm . A well-known setting for (2) is support vector machine (SVM) algorithms [15, 16, 18] for which the optimization problem is a convex quadratic programming one: for binary classification with Y D f1; 1g, V is the hinge loss given by V .f .x/; y/ D maxf1yf .x/; 0g; for SVM regression, V .f .x/; y/ D  .y  f .x// is induced by the insensitive loss  .t/ D maxfjtj  ; 0g with  > 0. When V takes the least squares loss, the coefficient vector for fz satisfies a linear system of equations. The third family of supervised learning algorithms are coefficient-based regularization schemes. With a general (not necessarily positive semi-definite or symmetric) kernel K, the scheme takes the form

fz D

m X

ciz K.; xi /; where .ciz /m i D1

i D1

(

D arg minm Ez c2IR

m X

!

)

ci K.; xi / C .c/ ; (3)

i D1

where  W IRm ! IRC is a regularizer or penalty. Scheme (3) has the advantage of possibly producing sparse representations [16]. One well-known algorithm is Lasso [13] which takes the least squares loss with linear kernel K.x; u/ D x  u and the `1 -regularizer: .c/ D kck`1 . In addition to sparsity, the non-smooth optimization problem (3) can be tackled by an efficient least angle regression algorithm. The `1 -regularizer also plays a crucial role for compressive sensing. For sparsity and approximation ability of scheme (3) with

a general or data dependent kernel and a general regularizer, see the discussion in [10]. There are many other supervised learning algorithms such as k-nearest neighbor methods, Bayesian methods, maximum likelihood methods, expectationminimization algorithm, boosting methods, tree-based methods, and other non-kernel-based methods [6, 8, 9]. Modelling of supervised learning algorithms is usually stated under the assumption that the sample z is randomly drawn according to a (unknown) probability measure  on X  Y with both X and Y being metric spaces. Mathematical analysis of a supervised learning algorithm studies the convergence of the genReralization error or expected risk defined by E.f / D X Y V .f .x/; y/d, in the sense that E.fz / converges with confidence or in probability to the infimum of E.f / when f runs over certain function set. The error analysis and learning rates of supervised learning algorithms involves uniform law of large numbers, various probability inequalities or concentration analysis, and capacity of function sets [1, 3, 5, 12, 16, 19]. Unsupervised learning aims at understanding properties of the distribution of events in X from a sample n m x D fxi gm i D1 2 X . In the case X  IR , an essential difference from supervised learning is the number n of variables is usually much larger. When n is small, unsupervised learning tasks may be completed by finding from the sample x good approximations of the density function of the underlying probability measure X on X . Curse of dimensionality makes this approach difficult when n is large. Principal component analysis (PCA) can be regarded as an unsupervised learning algorithm. It attempts to find some information about covariances of variables and to reduce the dimension for representing the data in X efficiently. Kernel PCA is an unsupervised learning algorithm generalizing this idea [9]. It is based on a kernel K which assigns a value K.x; u/ measuring dissimilarity or association between the events x and u. The sample x yields a matrix ŒK D .K.xi ; xj //m i;j D1 and it can be used to analyze the feature map mapping data points x 2 X to K.; x/ 2 HK . Thus kernel PCA overcomes some limitations of linear PCA. Graph Laplacian is another unsupervised learning algorithm. With a sample dependent matrix ŒK, the Laplacian matrix L is defined as L D D  ŒK, where D is a diagonal matrix with diagonal entries P Di i D m j D1 K.xi ; xj /. Let 0 D 1  2  : : :  m be the eigenvalues of the generalized eigenproblem

Markov Random Fields in Computer Vision: MAP Inference and Learning

Lf D f and f1 ; : : : ; fm be the associated normalized eigenvectors in IRn . Then by setting a low dimension s < m, the graph Laplacian eigenmap [2] embeds the data point xi into IRs as the vector ..f2 /i ; : : : ; .fsC1 /i /, which reduces the dimension and possibly keeps some data structures. Graph Laplacian can also be used for clustering. One way is to cluster the data x into two sets fi W .f2 /i  0g and fi W .f2 /i < 0g. Other unsupervised learning algorithms include local linear embedding, isomap, and diffusion maps. In many practical applications, getting labels would be expensive and time consuming while large unlabelled data might be available easily. Making use of unlabelled data to improve the learning ability of supervised learning algorithms is the motivation of semi-supervised learning. It is based on the expectation that the unlabelled data reflect the geometry of the underlying input space X such as manifold structures. Let us state a typical semi-supervised learning algorithm associated with a Mercer kernel K, labelled data mCu z D f.xi ; yi /gm i D1 and unlabelled data u D fxi gi DmC1 . mCu With a similarity matrix .!ij /i;j D1 such as truncated Gaussian weights, the semi-supervised learning algorithm takes the form 8 < fz;u;; D arg min Ez .f / C kf k2K f 2HK : 9 mCu = X  2 C ! .f .x /  f .x // ; ij i j ; .m C u/2 i;j D1 (4)

841

4. Cucker, F., Smale, S.: On the mathematical foundations of learning. Bull. Am. Math. Soc. 39, 1–49 (2001) 5. Cucker, F., Zhou, D.X.: Learning Theory: An Approximation Theory Viewpoint. Cambridge University Press, Cambridge/New York (2007) 6. Devroye, L., Gy¨orfi, L., Lugosi, G.: A Probabilistic Theory of Pattern Recognition. Springer, New York (1997) 7. Evgeniou, T., Pontil, M., Poggio, T.: Regularization networks and support vector machines. Adv. Comput. Math. 13, 1–50 (2000) 8. Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning, 2nd edn. Springer, New York (2009) 9. Sch¨olkopf, B., Smola, A.J.: Learning with Kernels. MIT, Cambridge (2002) 10. Shi, L., Feng, Y.L., Zhou, D.X.: Concentration estimates for learning with `1 -regularizer and data dependent hypothesis spaces. Appl. Comput. Harmon. Anal. 31, 286–302 (2011) 11. Smale, S., Zhou, D.X.: Estimating the approximation error in learning theory. Anal. Appl. 1, 17–41 (2003) 12. Steinwart, I., Christmann, A.: Support Vector Machines. Springer, New York (2008) 13. Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. 58, 267–288 (1996) 14. Tikhonov, A., Arsenin, V.: Solutions of Ill-Posed Problems. W. H. Winston, Washington, DC (1977) 15. Tsybakov, A.B.: Optimal aggregation of classifiers in statistical learning. Ann. Stat. 32, 135–166 (2004) 16. Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998) 17. Wahba, G.: Spline Models for Observational Data. SIAM, Philadelphia (1990) 18. Zhang, T.: Statistical behavior and consistency of classification methods based on convex risk minimization. Ann. Stat. 32, 56–85 (2004) 19. Zhou, D.X.: The covering number in learning theory. J. Complex 18, 739–767 (2002)

Markov Random Fields in Computer

where ;  > 0 are regularization parameters. It is Vision: MAP Inference and Learning unknown whether rigorous error analysis can be done to show that algorithm (4) has better performance than Nikos Komodakis1;4 , M. Pawan Kumar2;3 , and algorithm (2) when u >> m. In general, mathemat- Nikos Paragios1;2;3 ical analysis for semi-supervised learning is not well 1 Ecole des Ponts ParisTech, Universite Paris-Est, understood compared to that of supervised learning or Champs-sur-Marne, France 2´ unsupervised learning algorithms. Ecole Centrale Paris, Chˆatenay-Malabry, France 3´ Equipe GALEN, INRIA Saclay, ˆIle-de-France, France References 4 UMR Laboratoire d’informatique Gaspard-Monge, 1. Anthony, M., Bartlett, P.L.: Neural Network Learning: The- CNRS, Champs-sur-Marne, France oretical Foundations. Cambridge University Press, Cambridge/New York (1999) 2. Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. 15, 1373–1396 (2003) 3. Cristianini, N., Shawe-Taylor, J.: An Introduction to Support Vector Machines. Cambridge University Press, Cambridge/ New York (2000)

Introduction A Markov random field (MRF) can be visualized as a graph G D .V; E/. Associated with each of its n vertices Va (where a 2 f1;    ; ng) is a discrete

M

842

Markov Random Fields in Computer Vision: MAP Inference and Learning

random variable Xa , which can take a value from a finite, discrete label set Xa . We will refer to a particular assignment of values to the random variables from the corresponding label sets as a labeling. In other words, a labeling x 2 X1  X2     Xn implies that the random variable Xa is assigned the value xa . The probability of a labeling is specified by potential functions. For simplicity, we will assume a pairwise MRF parameterized by w, whose potential functions are either unary, denoted by a .xa I w/, or pairwise, denoted by ab .xa ; xb I w/. For a discussion on high-order MRFs, we refer the interested reader to the following relevant books [2, 9, 20]. The joint probability of all the random variables can be expressed in terms of the potential functions as follows: Pr.xI w/ / exp.E.xI w//; E.xI w/ X X a .xa I w/ C ab .xa ; xb I w/: (1) D

MAP Inference Maximum a posteriori (MAP) inference refers to the estimation of the most probable labeling. Formally, it is defined as

MAPG ./  min x

X Va 2V

a .xa / C

X

ab .xa ; xb /;

.Va ;Vb /2E

(2) where we have dropped the parameters w from the notation of the potential functions to avoid clutter. The above problem is known to be NP-hard in general. However, given its importance, several approximate algorithms have been proposed in the literature, which we review below.

Belief Propagation Belief propagation (BP) [21] is an iterative message The function E.xI w/ is called the Gibbs energy (or passing algorithm, where the messages at iteration t are given by simply the energy) of x. Va 2V

.Va ;Vb /2E

mtab .j / D min

i 2Xa

8 < :

9 =

X

a .i / C ab .i; j / C

c¤b;.Va ;Vc /2E

t 1 mca .i / ; 8.a; b/ 2 E; j 2 Xb : ;

(3)

At convergence (when the change in messages is below a minimum st-cut on a graph [10, 23] if the energy tolerance), the approximate MAP labeling is estimated function is submodular, that is, it satisfies as 8
0 and location Rk 2 Y for every 1  k  K, we denote: mD

K X

zk ı.  Rk /

kD1

the pattern, where ı is the Dirac mass and Z D PK kD1 zk the total nuclear charge by unit cell. This distribution of charge creates an attractive Coulomb-type P P zk potential. The sum  2 K kD1 jRk C j is infinite due to the long range of the Coulomb potential in R3 , but it can be renormalized. Actually, the potential due to P the nuclei is the periodic potential Gm D K kD1 zk G .  Rk /, where G is the  -periodic Coulomb kernel; that is, the unique square-integrable function on Y solving the Poisson equation: 8 X 1  ˆ 0. In this latter case, the thermodynamic limit for the ground-state density is based on a careful analysis of the Euler–Lagrange equation resulting in the limit. This equation belongs to the class of non local and non linear elliptic PDEs without boundary conditions, and the task is to show the existence of a unique and periodic nonnegative solution [8]. Hartree–Fock Type Models We now define the periodic Hartree–Fock functional (and its reduced version) as introduced in Catto et al. [9]; see also [18]. This is the analog of the standard Hartree–Fock model for molecules when expressed in terms of the one-particle density matrix, in the periodic setting; see I. Catto’s entry on  Hartree–Fock Type Methods in this encyclopedia. The main object of interest is the one-particle density matrix of the electrons , that is, a self-adjoint operator on L2 .R3 /, satisfying    2 , or equivalently, 0    1, where 1 is the identity operator on L2 .R3 /, and that commutes with the translations that preserve the un-

 dx D Z

o

Y

derlying lattice  . As in the independent electron case, the Bloch theorem allows to decompose the density matrix according R ˚to the Bloch waves decomposition L2 .R3 / D jY1 j Y  d L2 .Y /. This leads to:

Y

C

Z

Z

˚ Y

d  ;

where, for almost every 2 Y  ,  is a self-adjoint operator on L2 .Y / that is trace class and such that  2   , or equivalently, 0    1, where 1 is the identity operator on L2 .Y /. Using the same notation for the Hilbert–Schmidt kernel of  , we have:  .x; y/ D

X

n . / e i .xy/ un . ; x/ uN n . ; y/;

n1

where fe i x un . ; x/gn1 is a complete set of eigenfunctions of  on L2 .Y / corresponding to eigenvalues n . / 2 Œ0; 1 and where zN denotes the complex conjugate of the complex number z. For almost every

2 Y  , the function x 7!  .x; x/ is non negative,  -periodic and locally integrable on Y , and Z TrL2 .Y /  D

 .x; x/ dx D Y

X

n . /:

n1

The full electronic density is the  -periodic function  that is defined by

Mathematical Theory for Quantum Crystals

 .x/ D

1 jY  j

861

Z Y

d  .x; x/:

(2)

HF Iper

D inf

n

Z HF Eper ./

j  2 Dper ;

 dx D Z

o

Y

R

In particular, Y  .y/ dy is the number of electrons with per unit cell. An extra condition on the  s is necessary

Z 1 2 1 HF in order to give sense to the kinetic energy term in the  r  d

Tr 2 Eper ./ D  jY j Y  L .Y / 2 periodic setting. This condition reads: Z Z Z  Gm .y/  .y/ dy 2 d TrL2 .Y / Œr   D d n . / Y

“ Y Y 1 Z  .x/ G .x  y/  .y/ dxdy C 2 Y Y jrx un . ; x/j2 dx < C1; Y “ j.x; y/j2 1 dxdy (3) R  ˝ 2 Y R3 jx  yj where r 2 D jY1 j Y  d Œr 2 ; that is, r 2 is the Laplace operator on Y with quasi periodic boundary and where Dper is the set of admissible periodic density conditions with quasi momentum . The Hartree–Fock model for crystals (HF, in short) matrix reads: n Dper D  W L2 .R3 / ! L2 .R3 /;  D   ; 0    1; D

1 jY  j

Z

Z

˚

Y

 d ;

Y

o   TrL2 .Y / .1  r 2 /1=2  .1  r 2 /1=2 d < C1 :

M

The last term in the energy functional, namely (3), is the exchange term; it can also be rephrased as:

W .; x/ D 4 e i x

X e i kx I j  kj 

k2

“

j.x; y/j2 dxdy Y R3 jx  yj “ 1 D 2 d d 0 jY j  Y Y  “ dxdy  .x; y/ W .  0 ; x  y/  0 .x; y/; Y Y

thereby shedding light on its non local nature, where X e i   W .; z/ D ; jz C  j  2

see [9]. The reduced Hartree–Fock model for crystals (rHF, in short) is obtained from the Hartree–Fock model by getting rid of the exchange term in the energy functional; that is, rHF Eper ./

Z 1 1 D  Tr 2 Πr 2   d

jY j Y  L .Y / 2

Z  Gm .y/  .y/ dy Y

1 C 2

“  .x/ G .x  y/  .y/ dxdy: Y Y

; z 2 R3 :

From a mathematical point of view, this latter model has nicer properties, the energy functional being strictly convex with respect to the density. Existence rHF HF The function e i x W .; x/ is  -periodic with respect of a minimizer for Eper and Eper R on the set of density to x when  is fixed. The Fourier series expansion of matrices  2 Dper such that Y  .x/ dx D Z was proved by Catto, Le Bris, and Lions in [9]. Uniqueness W writes as follows:

862

Mathematical Theory for Quantum Crystals

in the rHF case has been proven later by Canc`es  D .1;F  .HrHF /; et al. [6]. The periodic mean-field Hartree–Fock Hamiltonian HHF corresponding to a minimizer  is decomposed where, here again, the real F is a Lagrange multiplier, according toR the Bloch  waves decomposition as identified with a Fermi level. In particular, the mini ˚ HHF D jY1 j Y  d HHF with mizer is also a projector in that case. These properties are crucial for the proper construction of a reduced  HF  Hartree–Fock model for a crystal with defect; see 1 2 H D  r  Gm C  ?Y G [7]. 2 Z 1   W . 0  ; x  y/  0 .x; y/ d 0 : jY j Y  The self-consistent equation satisfied by a minimizer  is then:  D .1;F / .HHF / C s fF g .HHF /;

(4)

where I is the spectral projection onto the interval I , the real F is a Lagrange multiplier, identified to a Fermi level like in the linear case, and the real parameter s is 0 or 1, as proved by Ghimenti and Lewin in [11]. In particular, they also have proved that every minimizer of the periodic Hartree–Fock functional is necessarily a projector, a fact that was only known so far for the Hartree–Fock model for molecules; see [2, 15] and entry  Hartree–Fock Type Methods in this encyclopedia. Therefore, for almost every 2 Y  , the eigenvalues fn . /gn1 that appear in the decomposition of  are either 0 or 1 for a minimizer. The Euler– Lagrange equation (4) may be rewritten in terms of the eigenfunctions fun . ; /gn1 of the operators  s. We obtain a system of infinitely many non linear, non local coupled partial differential equations of Schr¨odinger type: for every n  1 and for almost every 2 Y  : (  HHF un . ; x/ D n . / un . ; x/; on Y; n . /  F :

Extensions Crystals with Defects Real crystals feature defects or irregularities in the ideal arrangements described above, and it is these defects that explain many of the electrical and mechanical properties of real materials [14]. The first mathematical result in this direction is due to Canc`es, Deleurence, and Lewin who introduced and studied in [6, 7] a rHF-type model for crystals with a defect, that is, a vacancy, an interstitial atom, or an impurity with possible rearrangement of the neighboring atoms.

Crystal Problem It is an unsolved problem in the study of matter to understand why matter is in crystalline state at low temperature. A few mathematical results have contributed to partially answer this fundamental issue, known as the crystal problem. The pioneering work is due to Radin for electrons considered as classical particles and in one space dimension [19,20]. In two dimensions, the crystallization phenomenon in classical mechanics has been solved by Theil [23]. For quantum electrons, the first mathematical result goes back to Blanc and Le Bris, within the framework of a onedimensional TFW model [3].

Analogous results for the reduced Hartree–Fock model were proved by Canc`es, Deleurence, and Lewin in [6]. In that case, the minimizer  is unique, and denoting by

References 1 HrHF D  r 2  Gm C  ?Y G 2 the corresponding periodic mean-field Hamiltonian, it solves the non linear equation

1. Ashcroft, N., Mermin, N.: Solid State Physics. Saunders, Philadelphia (1976) 2. Bach, V.: Error bound for the Hartree-Fock energy of atoms and molecules. Commun. Math. Phys. 147(3), 527–548 (1992)

Matrix Functions: Computation 3. Blanc, X., Bris, C.L.: Periodicity of the infinite-volume ground state of a one-dimensional quantum model. Nonlinear Anal. 48(6, Ser. A: Theory Methods), 791–803 (2002) ´ Ehrlacher, V.: Local defects are always neutral 4. Canc`es, E., in the Thomas–Fermi–von Weisz¨acker model for crystals. Arch. Ration. Mech. Anal. 202(3), 933–973 (2011) 5. Canc`es, E., Defranceschi, M., Kutzelnigg, W., Le Bris, C., Maday, Y.: Computational quantum chemistry: a primer. In: Ciarlet, P.G. (ed.) Handbook of Numerical Analysis, vol. X, pp. 3–270. North-Holland, Amsterdam (2003) ´ Deleurence, A., Lewin, M.: A new approach 6. Canc`es, E., to the modelling of local defects in crystals: the reduced Hartree-Fock case. Commun. Math. Phys. 281(1), 129–177 (2008) ´ Deleurence, A., Lewin, M.: Non-perturbative 7. Canc`es, E., embedding of local defects in crystalline materials. J. Phys. 20, 294, 213 (2008) 8. Catto, I., Le Bris, C., Lions, P.L.: The Mathematical Theory of Thermodynamic Limits: Thomas-Fermi Type Models. Oxford Mathematical Monographs. Clarendon/Oxford University Press, New York (1998) 9. Catto, I., Le Bris, C., Lions, P.L.: On the thermodynamic limit for Hartree-Fock type models. Ann. Inst. H. Poincar´e Anal. Non Lin´eaire 18(6), 687–760 (2001) 10. Fefferman, C.: The thermodynamic limit for a crystal. Commun. Math. Phys. 98(3), 289–311 (1985) 11. Ghimenti, M., Lewin, M.: Properties of periodic HartreeFock minimizers. Calc. Var. Partial Differ. Equ. 35(1), 39–56 (2009) 12. Hainzl, C., Lewin, M., Solovej, J.P.: The thermodynamic limit of quantum Coulomb systems. Part I. General theory. Adv. Math. 221, 454–487 (2009) 13. Hainzl, C., Lewin, M., Solovej, J.P.: The thermodynamic limit of quantum Coulomb systems. Part II. Applications. Adv. Math. 221, 488–546 (2009) 14. Kittel, C.: Introduction to Solid State Physics, 8th edn. Wiley, New York (2004) 15. Lieb, E.H.: Variational principle for many-fermion systems. Phys. Rev. Lett. 46, 457–459 (1981) 16. Lieb, E.H., Lebowitz, J.L.: The constitution of matter: existence of thermodynamics for systems composed of electrons and nuclei. Adv. Math. 9, 316–398 (1972) 17. Lieb, E.H., Simon, B.: The Thomas-Fermi theory of atoms, molecules and solids. Adv. Math. 23(1), 22–116 (1977) 18. Pisani, C.: Quantum-mechanical ab-initio calculation of the properties of crystalline materials. In: Pisani, C. (ed.) Quantum-Mechanical Ab-initio Calculation of the Properties of Crystalline Materials. Lecture Notes in Chemistry, vol. 67, Springer, Berlin (1996) 19. Radin, C.: Classical ground states in one dimension. J. Stat. Phys. 35(1–2), 109–117 (1984) 20. Radin, C., Schulman, L.S.: Periodicity of classical ground states. Phys. Rev. Lett. 51(8), 621–622 (1983) 21. Reed, M., Simon, B.: Methods of Modern Mathematical Physics. IV. Analysis of Operators. Academic, New York (1978) 22. Ruelle, D.: Statistical Mechanics. Rigorous Results. World Scientific, Singapore/Imperial College Press, London (1999) 23. Theil, F.: A proof of crystallization in two dimensions. Commun. Math. Phys. 262(1), 209–236 (2006)

863

Matrix Functions: Computation Nicholas J. Higham School of Mathematics, The University of Manchester, Manchester, UK

Synonyms Function of a matrix

Definition A matrix function is a map from the set of complex n  n matrices to itself defined in terms of a given scalar function in one of various, equivalent ways. For example, if the scalar function has a power series exP1 P i i pansion f .x/ D 1 i D1 ai x , then f .A/ D i D1 ai A for any n  n matrix A whose eigenvalues lie within the radius of convergence of the power series. Other definitions apply more generally without restrictions on the spectrum [6].

Description Transformation Methods Let A be an n  n matrix. A basic property of matrix functions is that f .X 1 AX / D X 1 f .A/X for any nonsingular matrix X . Hence, if A is diagonalizable, so that A D XDX 1 for some diagonal matrix D D diag.di / and nonsingular X , then f .A/ D Xf .D/X 1 D X diag.f .di //X 1 . The task of computing f .A/ is therefore trivial when A has a complete set of eigenvectors and the eigendecomposition is known. However, in general the diagonalizing matrix X can be arbitrarily ill conditioned and the evaluation in floating point arithmetic can therefore be inaccurate, so this approach is recommended only for matrices for which X can be assured to be well conditioned. For Hermitian, symmetric, or more generally normal matrices (those satisfying AA D A A), X can be taken unitary and evaluation by diagonalization is an excellent approach. For general matrices, it is natural to restrict to unitary similarity transformations, in which case the Schur decomposition A D QTQ can be exploited, where

M

864

Matrix Functions: Computation

Q is unitary and T is upper triangular. Now f .A/ D Qf .T /Q and the problem reduces to computing a function of a triangular matrix. In the 2  2 case there is an explicit formula:

 f .1 / t12 f Œ2 ; 1  1 t12 ; D f 0 2 0 f .2 /

(1)

where f Œ2 ; 1  is a first-order divided difference and the notation reflects that i D ti i is an eigenvalue of A. More generally, when the eigenvalues are distinct, f .T / can be computed by an elegant recurrence due to Parlett [10]. This recurrence breaks down for repeated eigenvalues and can be inaccurate when two eigenvalues are close. These problems can be avoided by employing a block form of the recurrence, in which T D .Tij / is partitioned into a block m  m matrix with square diagonal blocks Ti i . The Schur–Parlett algorithm of Davies and Higham [4] uses a unitary similarity to reorder the blocks of T so that no two distinct diagonal blocks have close eigenvalues while within every diagonal block the eigenvalues are close, then applies a block form of Parlett’s recurrence. Some other method must be used to compute the diagonal blocks f .Ti i /, such as a Taylor series taken about the mean of the eigenvalues of the block. The Schur– Parlett algorithm is the best general-purpose algorithm for evaluating matrix functions and is implemented in the MATLAB function funm. For the square root function, f .T / can be computed by a different approach: the equation U 2 D T can be solved for the upper triangular matrix U by a recurrence of Bj¨orck and Hammarling [3] that runs to completion even if A has repeated eigenvalues. A generalization of this recurrence can be used to compute pth roots [11]. Approximation Methods Another class of methods is based on approximations to the underlying scalar function. Suppose that for some rational function r, r.A/ approximates f .A/ well for A within some ball. Then we can consider transforming a general A to a matrix B lying in the ball, approximating f .B/ r.B/, then recovering an approximation to f .A/ from r.B/. The most important example of this approach is the scaling and squaring method for the matrix exponential, which approxis mates e A rm .A=2s /2 , where m and s are nonnega-

tive integers and rm is the Œm=m Pad´e approximant to e x . Backward error analysis can be used to determine a choice of the parameters s and m that achieves a given backward error (in exact arithmetic) at minimal computational cost [1, 7]. The analogue for the matrix logarithm is the inverse scaling and squaring method, which uses the approxs imation log.A/ 2s rm .A1=2  I /, where rm .x/ is the Œm=m Pad´e approximant to log.1 C x/. Here, amongst the many logarithms of a matrix, log denotes the principal logarithm: the one whose eigenvalues have imaginary parts lying in . ; /; there is a unique such logarithm for any A having no eigenvalues on the closed negative real axis. Again, backward error analysis can be used to determine an optimal choice of the parameters s and m [2]. The derivation of (inverse) scaling and squaring algorithms requires attention to many details, such as how to evaluate a Pad´e approximant at a matrix argument, how to obtain the sharpest possible error bounds while using only norms, and how to avoid unnecessary loss of accuracy due to rounding errors. Approximation methods can be effectively used in conjunction with a Schur decomposition, in which case the triangularity can be exploited [1, 2, 8]. Matrix Iterations For functions that satisfy an algebraic equation, matrix iterations can be set up that, under appropriate conditions, converge to the matrix function. Many different derivations are possible, one of which is to apply Newton’s method to the relevant equation. For example, for the equation X 2 D A, Newton’s method can be put in the form XkC1 D

1 .Xk C Xk1 A/; 2

(2)

under the assumption that X0 commutes with A. This iteration does not always converge. But if A has no eigenvalues on the closed negative real axis and we take X0 D A, then Xk converges quadratically to A1=2 , the unique square root of A whose spectrum lies in the open right half-plane. Matrix iterations potentially suffer from two problems: they may be slow to converge initially, before the asymptotic fast convergence (in practice of quadratic or higher rate) sets in, and they may be unstable in finite precision arithmetic. Iteration (2) suffers from both these

Mechanical Systems

865

problems. However, (2) is mathematically equivalent References to the coupled iteration  1 Xk C Yk1 ; 2  1 YkC1 D Yk C Xk1 ; 2

XkC1 D

X0 D A; (3) Y0 D I;

of Denman and Beavers [5]: the Xk from (3) are identical to those from (2) with X0 D I and Yk  A1 Xk . This iteration is numerically stable. Various other equivalent and practically useful forms of (2) are available [6, Chap. 6]. The convergence of matrix iterations in the early stages can be accelerated by including scaling parameters. Consider the Newton iteration XkC1 D

1 .Xk C Xk1 /; 2

X0 D A:

(4)

Assuming that A has no pure imaginary eigenvalues, Xk converges quadratically to sign.A/, which is the matrix function corresponding to the scalar sign function that maps points in the open right half-plane to 1 and points in the open left half-plane to 1. Although the iteration converges at a quadratic rate, convergence can be extremely slow initially. To accelerate the iteration we can introduce a positive scaling parameter k : XkC1 D

 1 1 k Xk C 1 ; k Xk 2

1. Al-Mohy, A.H., Higham, N.J.: A new scaling and squaring algorithm for the matrix exponential. SIAM J. Matrix Anal. Appl. 31(3), 970–989 (2009). doi:http://dx.doi.org/10.1137/ 09074721X 2. Al-Mohy A.H., Higham, N.J.: Improved inverse scaling and squaring algorithms for the matrix logarithm. SIAM J. Sci. Comput. 34(4), C152–C169, (2012) ˚ Hammarling, S.: A Schur method for the square 3. Bj¨orck, A., root of a matrix. Linear Algebra Appl. 52/53, 127–140 (1983) 4. Davies, P.I., Higham, N.J.: A Schur–Parlett algorithm for computing matrix functions. SIAM J. Matrix Anal. Appl. 25(2), 464–485 (2003). doi:http://dx.doi.org/10.1137/ S0895479802410815 5. Denman, E.D., Beavers, A.N., Jr.: The matrix sign function and computations in systems. Appl. Math. Comput. 2, 63–94 (1976) 6. Higham, N.J.: Functions of Matrices: Theory and Computation. Society for Industrial and Applied Mathematics, Philadelphia (2008) 7. Higham, N.J.: The scaling and squaring method for the matrix exponential revisited. SIAM Rev. 51(4), 747–764 (2009). doi:http://dx.doi.org/10.1137/090768539 8. Higham, N.J., Lin, L.: A Schur–Pad´e algorithm for fractional powers of a matrix. SIAM J. Matrix Anal. Appl. 32(3), 1056–1078 (2011). doi:http://dx.doi.org/10.1137/ 10081232X 9. Nakatsukasa, Y., Bai, Z., Gygi, F.: Optimizing Halley’s iteration for computing the matrix polar decomposition. SIAM J. Matrix Anal. Appl. 31(5), 2700–2720 (2010) 10. Parlett, B.N.: A recurrence among the elements of functions of triangular matrices. Linear Algebra Appl. 14, 117–121 (1976) 11. Smith, M.I.: A Schur algorithm for computing matrix pth roots. SIAM J. Matrix Anal. Appl. 24(4), 971–989 (2003)

X0 D A:

Various choices of k are available, with differing motivations. One is the determinantal scaling k D j det.Xk /j1=n , which tries to bring the eigenvalues of Xk close to the unit circle. The number of iterations required for convergence to double precision accuracy (unit roundoff about 1016 ) varies with the iteration (and function) and the scaling but in some cases can be strictly bounded. For certain scaled iterations for computing the unitary polar factor of a matrix, it can be proved that less than ten iterations are needed for matrices with condition number less than 1016 (e.g., [9]). Moreover, for these iterations only one or two iterations might be needed if the starting matrix is nearly unitary.

Mechanical Systems Bernd Simeon Department of Mathematics, Felix-Klein-Zentrum, TU Kaiserslautern, Kaiserslautern, Germany

Synonyms Constrained mechanical system; Differential-algebraic equation (DAE); Euler–Lagrange equations; Multibody system (MBS); Time integration methods

M

866

Overview A mechanical multibody system (MBS) is defined as a set of rigid bodies and massless interconnection elements such as joints that constrain the motion and springs and dampers that act as compliant elements. Variational principles dating back to Euler and Lagrange characterize the dynamics of a multibody system and are the basis of advanced simulation software, so-called multibody formalisms. The corresponding specialized time integrators adopt techniques from differential-algebraic equations (DAE) and are extensively used in various application fields ranging from vehicle dynamics to robotics and biomechanics. This contribution briefly introduces the underlying mathematical models, discusses alternative formulations of the arising DAEs, and gives then, without claiming to be comprehensive, a survey on the most successful integration schemes.

Mathematical Modeling In Fig. 1, a multibody system with typical components is depicted. The motion of the bodies is described by the vector q.t/ 2 Rnq , which comprises the coordinates for position and orientation of each body depending on time t. We leave the specifics of the chosen coordinates open at this point but will come back to this issue below. Differentiation with respect to time is expressed by a dot, and thus, we write q.t/ P and Mechanical Systems, Fig. 1 Sketch of a multibody system with rigid bodies and typical interconnections

Mechanical Systems

q.t/ R for the corresponding velocity and acceleration vectors. Revolute, translational, universal, and spherical joints are examples for bonds in a multibody system. They may constrain the motion q and hence determine its kinematics. If constraints are present, we express the resulting conditions on q in terms of n constraint equations 0 D g.q/ : (1) Obviously, a meaningful model requires n < nq . The Eq. (1) that restrict the motion q are called holonomic constraints, and the matrix G .q/ WD

@g.q/ 2 Rn nq @q

is called the constraint Jacobian. We remark that there exist constraints, e.g., driving constraints, that may explicitly depend on time t and that are written as 0 D g.q; t/. For notational simplicity, however, we omit this dependence in (1). A standard assumption on the constraint Jacobian is the full rank condition rank G .q/ D n ;

(2)

which means that the constraint equations are linearly independent. In this case, the difference ny WD nq  n is the number of degrees of freedom (DOF) in the system.

Mechanical Systems

867

Two routes branch off at this point. The first modeling approach expresses the governing equations in terms of the redundant variables q and uses additional Lagrange multipliers .t/ 2 Rn to take the constraint equations into account. Alternatively, the second approach introduces minimal coordinates y.t/ 2 Rny such that the redundant variables q can be written as a function q.y/ and that the constraints are satisfied for all choices of y:

d dt



@ T @Pq



@ T D f a .q; q; P t/  G .q/T  ; @q 0 D g.q/

(7)

Lagrange Equations of Type One and Type Two Using both the redundant position variables q and additional Lagrange multipliers  to describe the dynamics leads to the equations of constrained mechanical motion, also called the Lagrange equations of type one:

with applied forces f a . Carrying out the differentiations and defining the force vector f as sum of f a and Coriolis and centrifugal forces result in the equations of motion (5). Note that f a D rU in the conservative case. It should be remarked that for ease of presentation, we omit the treatment of generalized velocities resulting from 3-dimensional rotation matrices. For that case, an additional kinematic equation qP D S .q/v with transformation matrix S and velocity vector v needs to be taken into account [9]. The Lagrange equations of type one are a system of second-order differential equations with additional constraints (5), which is a special form of a DAE. Applying minimal coordinates y, on the other hand, eliminates the constraints and allows generating a system of ordinary differential equations. If we insert the coordinate transformation q D q.y.t// into the principle (6) or apply it directly to (5), the constraints and Lagrange multipliers cancel due to the property (3). The resulting Lagrange equations of type two then take the form

M .q/ qR D f .q; q; P t/  G .q/T  ;

C .y/ yR D h.y; y; P t/ :

g.q.y//  0 :

(3)

As a consequence, by differentiation of the identity (3) with respect to y, we get the orthogonality relation G .q.y// N .y/ D 0

(4)

with the null space matrix N .y/ WD @q.y/=@y 2 Rnq ny .

0 D g.q/

(5a)

(8)

(5b) This system of second-order ordinary differential equations bears also the name state space form. For a closer where M .q/ 2 Rnq nq stands for the mass matrix and look at the structure of (8), we recall the null space matrix N from (4) and derive the relations f .q; q; P t/ 2 Rnq for the vector of applied and internal forces. d In case of a conservative multibody system where q.y/ D N .y/y; P the applied forces can be written as the gradient of a dt potential U , the equations of motion (5) follow from d2 @N .y/ .y; P y/ P q.y/ D N .y/yR C Hamilton’s principle of least action: dt 2 @y Z t1   T  U  g.q/T  dt ! stationary ! (6) for the velocity and acceleration vectors. Inserting t0 these relations into the dynamic equations (5a) and Here, the kinetic energy possesses a representation as premultiplying by N T lead directly to the state space P and the Lagrange form (8). quadratic form T D 12 qP T M .q/q, multiplier technique is applied for coupling the dyThe analytical complexity of the constraint namics with the constraints (1). In the nonconservative equations (1) makes it often impossible to obtain case, the Lagrange equations of type one read [23, 25] a set of minimal coordinates y that is valid for all

M

868

configurations of the system. Moreover, although we know from the implicit function theorem that such a set exists in a neighborhood of the current configuration, it might lose its validity when the configuration changes. This holds in particular for multibody systems with so-called closed kinematic loops.

Remarks 1. The differential-algebraic model (5) does not cover all aspects of multibody dynamics. In particular, features such as control laws, non-holonomic constraints, and substructures require a more general formulation. Corresponding extensions are discussed in [9,26]. A detailed treatment of systems with non-holonomic constraints is given by Rabier and Rheinboldt [22]. 2. In the conservative case, which is of minor relevance in engineering applications, the Lagrange equations of type one and type two can be reformulated by means of Hamilton’s canonical equations. 3. Different methodologies for the derivation of the governing equations are commonly applied in multibody dynamics. Examples are the principle of virtual work, the principle of Jourdain, and the Newton–Euler equations in combination with the principle of D’Alembert. These approaches are, in general, equivalent and lead to the same mathematical model. In practice, the crucial point lies in the choice of coordinates and in the corresponding computer implementation. 4. With respect to the choice of coordinates, one distinguishes between absolute and relative coordinates. Absolute or Cartesian coordinates describe the motion of each body with respect to an inertial reference frame, while relative or joint coordinates are based on relative motions between interacting bodies. Using absolute coordinates results in a large number of equations which have a clear and sparse structure and are inexpensive to compute. Furthermore, constraints always imply a differential-algebraic model [16]. Relative coordinates, on the other hand, lead to a reduced number of equations and, in case of systems with a tree structure, allow to eliminate all kinematic bonds, thus leading to a global state space form. In general, the system matrices are full and require more complicated computations than for absolute coordinates.

Mechanical Systems

Index Reduction and Stabilization The state space form (8) represents a system of secondorder ordinary differential equations. The equations of constrained mechanical motion (5), on the other hand, constitute a differential-algebraic system of index 3, as we will see in the following. For this purpose, it is convenient to rewrite the equations as a system of first order: qP D v ;

(9a)

M .q/ vP D f .q; v; t/  G .q/T  ; 0 D g.q/

(9b) (9c)

with additional velocity variables v.t/ 2 Rnq . By differentiating the constraints (9c) with respect to time, we obtain the constraints at velocity level: 0D

d g.q/ D G .q/ qP D G .q/ v : dt

(10)

A second differentiation step yields the constraints at acceleration level: d2 g.q/ D G .q/ vP C .q; v/ ; dt 2 @G .q/ .v; v/ ; (11) .q; v/ W D @q 0D

where the two-form  comprises additional derivative terms. Combining the dynamic equation (9b) and the acceleration constraints (11), we finally arrive at the linear system !   f .q; v; t/ M .q/ G .q/T vP : D  G .q/ 0 .q; v/

(12)

The matrix on the left-hand side has a saddle point structure. We presuppose that  M .q/ G .q/T G .q/ 0

is invertible

(13)

in a neighborhood of the solution. A necessary but not sufficient condition for (13) is the full rank of the constraint Jacobian G as stated in (2). If in addition the mass matrix M is symmetric positive definite, (13) obviously holds (We remark that there are applications

Mechanical Systems

869

where the mass matrix is singular, but the prerequisite (13) nevertheless is satisfied). Assuming (13) and a symmetric positive definite mass matrix, we can solve the linear system (12) for the acceleration vP and the Lagrange multiplier  by block Gaussian elimination. This leads to an ordinary differential equation for the velocity variables v and an explicit expression for the Lagrange multiplier  D .q; v; t/. Since two differentiation steps result in the linear system (12) and a final third differentiation step yields an ordinary differential equation for , the differentiation index of the equations of constrained mechanical motion is 3. Note that the above differentiation process involves a loss of integration constants. However, if the initial values .q 0 ; v0 / are consistent, i.e., if they satisfy the original constraints and the velocity constraints, 0 D g.q 0 / ;

This system is obviously of index 1, and at first sight, one could expect much less difficulties here. But a closer view shows that (15) lacks the information of the original position and velocity constraints, which have become invariants of the system. In general, these invariants are not preserved under discretization, and the numerical solution may thus turn unstable, which is called the drift-off phenomenon. Instead of the acceleration constraints, one can also use the velocity constraints (10) to replace (9c). This leads to qP D v ; M .q/ vP D f .q; v; t/  G .q/T  ;

(16)

0 D G .q/ v :

0 D G .q 0 / v0 ;

(14) Now the index is 2, but similar to the index 1 case, the information of the position constraint is lost. The the solution of (9a) and (12) also fulfills the original resulting drift off is noticeable but stays linear, which means a significant improvement compared to (15) system (9). where the drift off grows quadratically in time (see Higher index DAEs such as (9) suffer from several (19) below). Nevertheless, additional measures such drawbacks. For one, in a numerical time integration scheme, a differentiation step is replaced by a dif- as stabilization by projection are often applied when ference quotient, i.e., by a division by the stepsize. discretizing (16). Therefore, the approximation properties of the numerical scheme deteriorate and we observe phenomena GGL and Overdetermined Formulation like order reduction, ill-conditioning, or even loss of On the one hand, we have seen that it is desirable convergence. Most severely affected are typically the for the governing equations to have an index as small Lagrange multipliers. Also, an amplification of pertur- as possible. On the other hand, though simple differbations may occur (cf. the concept of the perturbation entiation lowers the index, it may lead to drift off. index [15]). For these reasons, it is mostly not advis- The formulation of Gear, Gupta, and Leimkuhler [12] able to tackle DAEs of index 3 directly. Instead, it has combines the kinematic and dynamic equations (9a-b) become standard in multibody dynamics o lower the with the constraints at velocity level (10). The position index first by introducing alternative formulations. constraints (5b) are interpreted as invariants and appended by means of extra Lagrange multipliers, which Formulations of Index 1 and Index 2 results in The differentiation process for determining the index revealed the hidden constraints at velocity and qP D v  G .q/T  ; at acceleration level. It is a straightforward idea to M .q/ vP D f .q; v; t/  G .q/T  ; (17) replace now the original position constraint (9c) by one of the hidden constraints. Selecting the acceleration 0 D G .q/ v ; equation (11) for this purpose, one obtains 0 D g.q/ qP D v ; with .t/ 2 Rn . A straightforward calculation shows (15) M .q/ vP D f .q; v; t/  G .q/T  ;  D 0 if G .q/ of full rank. With the additional 0 D G .q/ vP C .q; v/ : multipliers  vanishing, (17) and the original equations

M

870

Mechanical Systems

of motion (5) coincide along any solution. Yet, the index of (17) is 2 instead of 3. Some authors refer to (17) also as stabilized index 2 system. From an analytical point of view, one could drop the extra multiplier  in (17) and consider instead the overdetermined system qP D v ; M .q/ vP D f .q; v; t/  G .q/T  ;

and velocity variables. This means that any standard ODE integrator can be easily applied in this way to solve the equations of constrained mechanical motion. However, as the position and velocity constraints are in general not preserved under discretization, the arising drift off requires additional measures. More specifically, if the integration method has order k, the numerical solution q n and vn after n time steps satisfies (18) the bound [13]

0 D G .q/ v ; 0 D g.q/ :

kg.q n /k  hkmax .Atn CBtn2 /;

kG .q n /vn k  hkmax C tn (19)

Though there are more equations than unknowns in (18), the solution is unique and, given consistent initial values, coincides with the solution of the original system (9). Even more, one could add the acceleration constraint (11)–(18) so that all hidden constraints are explicitly stated. After discretization, however, a generalized inverse is required to define a meaningful method.

with constants A; B; C . The drift off from the position constraints grows thus quadratically with the length of the integration interval but depends also on the order of the method. If the constraints are linear, however, there is no drift off since the corresponding invariants are preserved by linear integration methods. A very common cure for the drift off is a two-stage projection method where after each integration step, the numerical solution is projected onto the manifold of position and velocity constraints. Let q nC1 and Local State Space Form If one views the equations of constrained mechani- vnC1 denote the numerical solution of (15). Then, the cal motion as differential equations on a manifold, projection consists of the following steps: it becomes clear that it is always possible to find at least a local set of minimal coordinates to set up the 0 D M .qQ nC1 /.qQ nC1  q nC1 / C G .qQ nC1 /T ; state space form (8) and compute a corresponding null solve .20a/ for qQ nC1 ;  space matrix. We mention in this context the coordi0 D g.qQ nC1 /; nate partitioning method [28] and the tangent space parametrization [21], which both allow the application 0 D M .qQ nC1 /.vQ nC1  vnC1 / C G .qQ nC1 /T I of ODE integration schemes in the local coordinates. solve .20b/ for vQ nC1 ;  The class of null space methods [5] is related to these 0 D G .qQ nC1 / vQ nC1 : approaches.

Time Integration Methods We discuss in the following a selection of time integration methods that are typically used for solving the constrained mechanical system (9). For this purpose, we assume consistent initial values (14) and denote by t0 < t1 < : : : < tn the time grid, with hi D ti C1  ti being the stepsize. Projection Methods By solving the linear system (12), the formulation (15) of index 1 can be reduced to an ODE for the position

A simplified Newton method can be used to solve the nonlinear system (20a), and since the corresponding iteration matrix is just (13) evaluated at q nC1 and already available in decomposed form due to the previous integration step, this projection is inexpensive to compute. Furthermore, (20b) represents a linear system for vQ nC1 and  with similar structure where the corresponding matrix decomposition can be reused for solving (12) in the next integration step [24]. As the projection (20) reflects a metric that is induced by the mass matrix M [18], the projected value qQ nC1 is the point on the constraint manifold that has minimum distance to q nC1 in this metric. An analysis of the required number of Newton iterations and of the

Mechanical Systems

871

relation to alternative stabilization techniques including the classical Baumgarte method [4] is provided by Ascher et al. [3]. Projection methods are particularly attractive in combination with explicit ODE integrators. The combination with implicit methods, on the other hand, is also possible but not as efficient as the direct discretization by DAE integrators discussed below. Half-Explicit Methods Half-explicit methods for DAEs discretize the differential equations explicitly, while the constraint equations are enforced in an implicit fashion. Due to the linearity of the velocity constraint (10), the formulation (16) of index 2 is a good candidate for this method class. Several one-step and extrapolation methods have been tailored to the needs and peculiarities of mechanical systems. The half-explicit Euler method as generic algorithm for the method class reads q nC1 D q n C hvn ;

A xP D .x; t/

(22)

with singular matrix A, right-hand side , and with the vector x.t/ collecting the position and velocity coordinates as well as the Lagrange multipliers. A state-dependent mass matrix M .q/ can be (formally) inverted and moved to the right-hand side or, alternatively, treated by introducing additional acceleration variables a.t/ WD v.t/ P and writing the dynamic equations as 0 D M .q/ a  f .q; v; t/ C G .q/T . BDF Methods The backward differentiation methods (BDFs) are successfully used as a multistep discretization of stiff and differential-algebraic equations [11]. For the linear implicit system (22), the BDF discretization with fixed stepsize h simply replaces x.t P nCk / by the difference scheme 1X ˛i x nCi D .x nCk ; tnCk / ; h i D0 k

A

(23)

M .q n / vnC1 D M .q n / vn

with coefficients ˛i , i D 0; : : : ; k. Since the difference operator on the left evaluates the derivative of a polynomial passing through the points x n ; x nC1 ; : : : ; x nCk , 0 D G .q nC1 / vnC1 : (21) this discretization can be interpreted as a collocation method and extends easily to variable stepsizes. The Similar to the index 1 case above, only a linear system new solution x nCk is given by solving the nonlinear of the form system Chf .q n ; vn ; tn /  hG .q n /T n ;

 vnC1 M .q n / G .q n /T G .q nC1 / 0 hn  M .q n /vn C hf .q n ; vn ; tn / D 0



arises in each step. The scheme (21) forms the basis for a class of half-explicit Runge–Kutta methods [2, 15] and extrapolation methods [18]. These methods have in common that only information of the velocity constraints is required. As remedy for the drift off, which grows only linearly here but might still be noticeable, the projection (20) can be applied. Implicit DAE Integrators For the application of general DAE methods, it is convenient to write the different formulations from above as linear implicit system

1 X ˛k A x nCk  .x nCk ; tnCk / C A ˛i x nCi D 0 h h i D0 (24) k1

for x nCk , where ˛k is the leading coefficient of the method. The convergence properties of the BDFs when applied to the equations of constrained mechanical motion depend on the index of the underlying formulation [6]. In case of the original system (9) of index 3, convergence is only guaranteed for fixed stepsize, and additional numerical difficulties arise that are typical for higher index DAEs. The formulations (15) and (16) behave better under discretization but suffer from drift off, and for this reason, the GGL formulation (17) is, in general, preferred when using the BDFs for constrained mechanical systems. However, since (17) is still of index 2, the local error in the different

M

872

Mechanical Systems

components is, assuming n D 0, exact history data and hence required in the simplified Newton iteration and constant stepsize: in the error estimation. In practice, the 5th-order Radau IIa method with y.tk /  y k D O.hkC1 /; z.tk /  zk D O.hk / (25) s D 3 stages has stood the test as versatile and robust integration scheme for constrained mechanical systems where y D .q; v/ collects the differential components (see also the hints on resources for numerical software and z D .; / the algebraic components. To cope below). An extension to a variable order method with with this order reduction phenomenon in the algebraic s D 3; 5; 7 stages and corresponding order 5; 9; 13 is components and related effects, a scaled norm kyk C presented in [14]. Both the BDFs and the implicit Runge–Kutta methhkzk is required both for local error estimation and for convergence control of Newton’s method. Global ods require a formulation of the equations of motion convergence of order k for the k-step BDF method as first-order system, which obviously increases the when applied to (17) can nevertheless be shown both size of the linear systems within Newton’s method. For an efficient implementation, it is crucial to apply for the differential and the algebraic components. As analyzed in [10], the BDF discretization of (17) block Gaussian elimination to reduce the dimension is equivalent to solving the corresponding discretized in such a way that only a system similar to (12) with overdetermined system (18) in a certain least squares two extra Jacobian blocks of the right-hand side vector sense where the least squares objective function inher- f has to be solved. When comparing these implicit DAE integrators with explicit integration schemes for its certain properties of the state space form (8). the formulation of index 1, stabilized by the projection scheme (20), and with half-explicit methods, Implicit Runge–Kutta Methods the performance depends on several parameters such Like the BDFs, implicit Runge–Kutta schemes constitute an effective method class for stiff and as problem size, smoothness, and, most of all, stiffdifferential-algebraic equations. Assuming a stiffly ness. The adjective “stiff” typically characterizes an ODE accurate method where the weights are simply given by the last row of the coefficient matrix, such a method whose eigenvalues have strongly negative real parts. However, numerical stiffness may also arise in case with s stages for the linear implicit system (22) reads of second-order systems with large eigenvalues on or s close to the imaginary axis. If such high frequenX aij .X n;j ; tn C cj h/ ; AX n;i D Ax n C h cies are viewed as a parasitic effect which perturbs a j D1 slowly varying smooth solution, implicit time integrai D 1; : : : ; s: (26) tion methods with adequate numerical dissipation are an option and usually superior to explicit methods. For Here, X n;i denotes the intermediate stage vectors and a mechanical system, this form of numerical stiffness ci for i D 1; : : : ; s the nodes of the underlying is directly associated with large stiffness forces, and quadrature formula, while .aij /i;j D1;:::;s is the coeffi- thus the notion of a stiff mechanical system has a cient matrix. The numerical solution after one step is twofold meaning. If the high-frequency information carries physical significance and needs to be resolved, given by the last stage vector, x nC1 WD X n;s . Efficient methods for solving the nonlinear system even implicit methods are compelled to taking tiny (26) by means of simplified Newton iterations and a stepsizes. Most often, however, it suffices to track a transformation of the coefficient matrix are discussed smooth motion where the high-frequency part reprein [13]. In the DAE context, collocation methods of sents a singular perturbation [27]. In case of a stiff mechanical system with high fretype Radau IIa [7] have proved to possess particularly favorable properties. For ODEs and DAEs of index quencies, the order of a BDF code should be restricted 1, the convergence order of these methods is 2s  1, to k D 2 due to the loss of A-stability for higher while for higher index DAEs, an order reduction occurs order. L-stable methods such as the Radau IIa schemes, [15]. In case of the equations of motion in the GGL however, are successfully applied in such situations formulation (17), a scaled norm like in the BDF case is (see [19] for an elaborate theory).

Mechanical Systems

Generalized ˛-Method The generalized ˛-method [8] represents the method of choice for time-dependent solid mechanics applications with deformable bodies discretized by the finite element method (FEM). Since this field of structural dynamics and the field of multibody systems are more and more growing together, extensions of the ˛-method for constrained mechanical systems have recently been introduced. While the algorithms of [17, 20] are based on the GGL formulation (17), the method by Arnold and Br¨uls [1] discretizes the equations of motion (5) directly in the second-order formulation. In brief, the latter algorithm uses discrete values q nC1 ; qP nC1 ; qR nC1 ; nC1 that satisfy the dynamic equations (5) and auxiliary variables for the accelerations

873 Mechanical Systems, Table 1 Internet downloading numerical software

resources

for

1: pitagora.dm.uniba.it/testset/ 2: www.netlib.org/ode/ 3: www.cs.ucsb.edu/cse/software.html 4: www.unige.ch/hairer/software.html 5: www.zib.eu/Numerik/numsoft/CodeLib/codes/mexax/

Resources for Numerical Software A good starting point for exploring the available numerical software is the initial value problem (IVP) test set [1] (see Table 1), which contains several examples of constrained and unconstrained mechanical systems along with various results and comparisons for a wide selection of integration codes. The BDF code DASSL by L. Petzold can be obtained from the IVP site but also from netlib [2]. The more recent version DASPK with extensions for large-scale .1˛m /anC1 C ˛m an D .1˛f /qR nC1 C˛f qR n : (27) systems is available at [3]. The implicit Runge– Kutta codes RADAU5 and RADAU by E. Hairer These are then integrated via and G. Wanner can be downloaded from [4] where also the half-explicit Runge–Kutta code PHEM56 by  A. Murua is provided. The extrapolation code MEXAX 1 q nC1 D q n C hqP n C h2  ˇ an C h2 ˇanC1 ; by Ch. Lubich and coworkers is in the library [5]. 2 Finally, the half-explicit Runge–Kutta code HEDOP5 qP nC1 D qP n C h.1   /an C h anC1 : (28) by M. Arnold method and a projection method by the author of this contribution are contained in the library The free coefficients ˛f ; ˛m ; ˇ;  determine the MBSPACK [26]. method. Of particular interest is the behavior of this scheme for a stiff mechanical system where the high frequen- References cies need not be resolved. An attractive feature in this context is controllable numerical dissipation, which is 1. Arnold, M., Br¨uls, O.: Convergence of the generalizeda scheme for constrained mechanical systems. Multibody mostly expressed in terms of the spectral radius 1 Syst. Dyn. 18, 185–202 (2007) at infinity. More specifically, it holds 1 2 Œ0; 1 2. Arnold, M., Murua, A.: Non-stiff integrators for differentialwhere 1 D 0 represents asymptotic annihilation of algebraic systems of index 2. Numer. Algorithms 19, 25–41 the high-frequency response, i.e., the equivalent of L(1998) stability. On the other hand, 1 D 1 stands for the case 3. Ascher, U., Chin, H., Petzold, L., Reich, S.: Stabilization of constrained mechanical systems with DAEs and invariant of no algorithmic dissipation. A-stability, also called manifolds. J. Mech. Struct. Mach 23, 135–158 unconditional stability, is achieved for the parameters 4. Baumgarte, J.: Stabilization of constraints and integrals of motion in dynamical systems. Comput. Methods Appl. Mech. 1, 1–16 (1972) 1 21  1 ; ˛m D ; ˇ D 14 .1  ˛m C ˛f /2 : 5. Betsch, P., Leyendecker, S.: The discrete null space method ˛f D 1 C 1 1 C 1 for the energy consistent integration of constrained mechan(29) ical systems. Int. J. Numer. Methods Eng. 67, 499–552

The choice  D 1=2  ˛m C ˛f guarantees secondorder convergence.

(2006) 6. Brenan, K.E., Campbell, S.L., Petzold, L.R.: The Numerical Solution of Initial Value Problems in Ordinary DifferentialAlgebraic Equations. SIAM, Philadelphia (1996)

M

874

Medical Applications in Bone Remodeling, Wound Healing, Tumor Growth, and Cardiovascular Systems

7. Butcher, J.C.: Integration processes based on Radau quadrature formulas. Math. Comput. 18, 233–244 (1964) 8. Chang, J., Hulbert, G.: A time integration algorithm for structural dynamics with improved numerical dissipation. ASME J Appl Mech 60, 371–375 (1993) 9. Eich-Soellner, E., F¨uhrer, C.: Numerical Methods in Multibody Dynamics. Teubner, Stuttgart (1998) 10. F¨uhrer, C., Leimkuhler, B.: Numerical solution of differential-algebraic equations for constrained mechanical motion. Numer. Math. 59, 55–69 (1991) 11. Gear, C.: Numerical Initial Value Problems in Ordinary Differential Equations. Prentice-Hall, Englewood Cliffs (1971) 12. Gear, C., Gupta, G., Leimkuhler, B.: Automatic integration of the Euler–Lagrange equations with constraints. J. Comput. Appl. Math. 12 & 13, 77–90 (1985) 13. Hairer, E., Wanner, G.: Solving Ordinary Differential Equations II: Stiff and Differential-Algebraic Problems, 2nd edn. Springer, Berlin (1996) 14. Hairer, E., Wanner, G.: Stiff differential equations solved by Radau methods. J. Comput. Appl. Math. 111, 93–111 (1999) 15. Hairer, E., Lubich, C., Roche, M.: The Numerical Solution of Differential-Algebraic Equations by Runge– Kutta Methods. Lecture Notes in Mathematics, vol. 1409. Springer, Heidelberg (1989) 16. Haug, E.: Computer-Aided Kinematics and Dynamics of Mechanical Systems, vol. I. Allyn and Bacon, Boston (1989) 17. Jay, L., Negrut, D.: Extensions of the hht-method to differential-algebraic equations in mechanics. ETNA 26, 190–208 (2007) 18. Lubich, C.: h2 extrapolation methods for differentialalgebraic equations of index-2. Impact Comput. Sci. Eng. 1, 260–268 (1989) 19. Lubich, C.: Integration of stiff mechanical systems by Runge–Kutta methods. ZAMP 44, 1022–1053 (1993) 20. Lunk, C., Simeon, B.: Solving constrained mechanical systems by the family of Newmark and ˛-methods. ZAMM 86, 772–784 (2007) 21. Potra, F., Rheinboldt, W.: On the numerical integration for Euler–Lagrange equations via tangent space parametrization. J. Mech. Struct. Mach. 19(1), 1–18 (1991) 22. Rabier, P., Rheinboldt, W.: Nonholonomic Motion of Rigid Mechanical Systems from a DAE Point of View. SIAM, Philadelphia (2000) 23. Roberson, R., Schwertassek, R.: Dynamics of Multibody Systems. Springer, Heidelberg (1988) 24. Schwerin, R.: Multibody System Simulation. Springer, Berlin (1999) 25. Shabana, A.: Dynamics of Multibody Systems. Cambridge University Press, Cambridge/New York (1998) 26. Simeon, B,: MBSPACK – numerical integration software for constrained mechanical motion. Surv. Math. Ind. 5, 169–202 (1995) 27. Simeon, B.: Order reduction of stiff solvers at elastic multibody systems. Appl. Numer. Math. 28, 459–475 (1998) 28. Wehage, R.A., Haug, E.J.: Generalized coordinate partitioning for dimension reduction in analysis of constrained dynamic systems. J. Mech. Des. 134, 247–255 (1982)

Medical Applications in Bone Remodeling, Wound Healing, Tumor Growth, and Cardiovascular Systems Yusheng Feng and Rakesh Ranjan NSF/CREST Center for Simulation, Visulization and Real-Time Prediction, The University of Texas at San Antonio, San Antonio, TX, USA

Synonyms Bone remodeling; Cardiovascular flow; Computational bioengineering; Continuum model; Mixture theory; Predictive medicine; Tumor growth; Wound healing

Description Predictive medicine is emerging both as a research field and a potential medical tool for designing optimal treatment options. It can also advance understanding of biological and biomedical processes and provide patient-specific prognosis. In order to characterize macro-level tissue behavior, mixture theory can be introduced for modeling both hard and soft tissues. In continuum mixture theory, an arbitrary point in a continuous medium can be occupied simultaneously by many different constituents differentiated only through their volume fractions. The advantage of this mathematical representation of tissues permits direct reconstruction of patientspecific geometry from medical imaging, inclusion of species from different scales as long as they can be characterized by either density or volume fraction functions, and explicit consideration of interactions among species included in the mixture. Furthermore, the mathematical models based on the notion of mixture can be derived from the first principles (conservation laws and the second law of thermodynamics). The applications considered here include bone remodeling, wound healing, and tumor growth. The cardiovascular system can also be included if soft tissues such as the heart and vessels are treated as separate species different than fluid (blood).

Medical Applications in Bone Remodeling, Wound Healing, Tumor Growth, and Cardiovascular Systems

Bone remodeling is a natural biological process during the course of maturity or after injuries, which can be characterized by a reconfiguration of the density of the bone tissue due to mechanical forces or other biological stimuli. Wound healing (or cicatrization), on the other hand, mainly involves skins or other soft organ tissues that repair themselves after the protective layer and/or tissues are broken and damaged. In particular, wound healing in fasciated muscle occurs due to the presence of traction forces that accelerate the healing process. Both bone remodeling and wound healing can be investigated under the general framework of continuum mixture theory at the tissue level. Another important application is tumor growth modeling, which is crucial in cancer biology, treatment planning, and outcome prediction. The mixture theory framework can provide a convenient vehicle to simulate growth (or shrinking) phenomenon under various biological conditions.

Continuum Mixture Theory There are several versions of mixture theories [1]. In general, mixture theory is a comprehensive framework that allows multiple species to be included under an abstract notion of continuum. In this framework, the biological tissue can be considered as a multiphasic system with different species including solid tissue, body fluids, cells, extra cellular matrix (ECM), nutrients, etc. Each of the species (or constituents) is denoted by ˛ .˛ D 1; 2; : : :; / where  is the number of species in the mixture. The nominal densities of each constituent are denoted by ˛ , and the true densities are denoted by ˛R . To introduce the volume fraction concept, a domain occupying the control space BS is defined with the boundary @BS , in which all the constituents ˛ occupy the volume fractions ˛ , which satisfy the constraint  X

˛ .x; t/ D

 X

˛ D 1; ˛R ˛D1

875

As noted in the introductory entry, two frames of reference are used to describe the governing principles of continuum mechanics. The Lagrangian frame of reference is often used in solid mechanics, while the Eulerian frame of reference is used in fluid mechanics. The Lagrangian description is usually suitable to establish mathematical models for stress-induced growth such as bone remodeling and wound healing (e.g., [2]), while the Eulerian description is used for developing mass transfer-driven tumor growth models [3–6] with a few exceptions when tumors undergo large deformations [7]. To develop mathematical models for each application, the governing equations are provided by the conservation laws, and the constitutive relations are usually developed through empirical relationships subject to constraints such as invariance condition, consistency with thermodynamics, etc. Specifically, the governing equations can be obtained from conservations of mass, momentum, and energy for each species as well as the mixture. Moreover, the conservation of energy is often omitted from the governing equations under the isothermal assumption, unless bioheat transfer is of interest (e.g., in thermotherapies). When the free energy of the system is given as a function of dependent field variables such as strain, temperature, etc., the second law of thermodynamics (the Clausius-Duhem inequality) provides a means for determining forms of some constitutive equations via the well-known method of Coleman and Noll [8].

Bone Remodeling and Wound Healing Considering the conservation of mass for each species  ˛ in a control volume, the mass production and fluxes across the boundary of the control volume are required to be equal: @˛ C r  .˛ v/ D O˛ : @t

(2)

(1) In Eq. (2), the velocity of the constituent is denoted by v, and the mass supplies between the phases are denoted by O˛ . From a mechanical point of view, the where x is the position vector of the actual placement processes of bone remodeling and wound healing are and t denotes the time. mainly induced by traction forces. To develop the mass ˛D1

M

876

Medical Applications in Bone Remodeling, Wound Healing, Tumor Growth, and Cardiovascular Systems

conservation equation, we may include all the necessary species of interest. For simplicity, however, we choose a triphasic system comprised of solid, liquid, and nutrients to illustrate the modeling process [2]. The mass exchange terms are subject to the constraint  X

medium (solid phase). Various types of material behavior can be described in terms of principal invariants of structural tensor M and right Cauchy-Green tensor CS , where M D A ˝ A and CS D FTS FS ;

(9)

O˛ D 0 or OS C ON C OL D 0:

(3) and A is the preferred direction inside the material and FS is the deformation gradient for a solid undergoing large deformations. The expressions for the stress in Moreover, if the liquid phase is not involved in the mass the solid are dependent on the deformation graditransition, then ent and consequently the displacements of the solid. Summation of the momentum conservation equations OS D ON and OL D 0: (4) provides the equation for the solid displacements. Mass conservation equations with incorporation of the satNext, the momentum of the constituent ˛ is defined uration condition provide the equation for interstitial by Z pressure. In addition, the mass conservation equations ˛ v˛ dv (5) for each species provide the equations for the evolution m˛ D B˛ of volume fractions. By including total change of linear momentum in B˛ Assuming the fluid phase (F ) is comprised of the by m˛ and the interaction of the constituents ˛ by pO ˛ , liquid (L) and the nutrient phases (N ), we obtain the standard momentum equation (Cauchy equation of (F D L C N ) motion) for each constituent becomes ˛D1

r  T˛ C ˛ .b  a˛ / C pO ˛  O˛ v˛ D 0

(6) r 

S;L;N X ˛

T˛ Cb

S;L;N X ˛

˛ C

S;L;N X

pO ˛  OS vS  OF vF D 0

˛

(10) where the expression O v˛ represents the exchange of F S S N F O O O D   O and p C p C p D 0, we obtain Since  O linear momentum through the density supply O˛ . The term T˛ denotes the partial Cauchy stress tensor, and S;L;N S;L;N X X ˛ b specifies the volume force. In addition, the terms ˛ T C b ˛ C OS .vF  vS / D 0 (11) r  ˛ pO , where ˛ D S; L; N are required to satisfy the ˛ ˛ constraint condition The definition of the seepage velocity wF S provides the S L N following equation pO C pO C pO D 0: (7) ˛

S;F S;F X X In the case of either bone remodeling or wound healing, r T˛ C b ˛ C OS .wF S / D 0 (12) the velocity field is nearly steady state. Thus, the ˛ ˛ acceleration can be neglected by setting a˛ D 0. The resulting system of equations can then be written as The strong form for the pressure equation can be written as follows: r  T˛ C ˛ b C pO ˛ D O˛ v˛ : (8)    1 1 D 0 (13)  r  F wF S C I W DS  OS SR NR The second law of thermodynamics (entropy inequality) provides expressions for the stresses in the solid and fluid phases that are dependent on the displace- The strong form of mass conservation equation for the ments and the seepage velocity, respectively. The seep- solid phase is

age velocity is a relative velocity between the liquid and solid phases, which are often obtained from explicit Darcy-type expressions for flow through a porous

D S .S / OS C S I W DS D SR Dt 

(14)

Medical Applications in Bone Remodeling, Wound Healing, Tumor Growth, and Cardiovascular Systems

877

Finally, the balance of mass for the nutrient phase can (ODE), e.g., [10–13], extensions of ODE’s to partial be described as differential equations [4, 14], or continuum mechanicsbased descriptions that study both vascular and   D S .N / ON avascular tumor growth. Continuum mechanics-based  NR C N I W DS C r  N wF S D 0 (15) formulations consider either a Lagrangian [2] or Dt  a Eulerian description of the medium [4]. Various In the above, wFS is the seepage velocity, DS denotes considerations such as modifications of the ordinary the symmetric part of the spatial velocity gradient, differential equations (ODE’s) to include effects S and DDt./ denotes the total derivative of quantities with of therapies [12], studying cell concentrations in respect to the solid phase. The seepage velocity is capillaries during vascularization with and without inhibitors, multiscale modeling [15–19], and cell obtained from transport equations in the extracellular matrix i 1 h (ECM) [5] have been included. rF  pOF (16) wFS D SF Modeling tumor growth can also be formulated under the framework of mixture theory with a multiHere, SF is the permeability tensor,  denotes the constituent description of the medium. It is convenient pressure, and F is the volume fraction of the fluid. to use an Eulerian frame of reference. Other descripEquations (8)–(15) are required to be solved for the tions have considered the tumor phase with diffused bone remodeling problem with the mixture theory. The interface [6]. Consider the volume fraction of cells deprimary variables to be solved are fuS ; ; nS ; nN g the noted by ( ), extracellular liquid (l), and extracellular solid displacements, interstitial pressure, and the solid matrix (m) [4]. The governing equations are derived and nutrient volume fractions. One example of bone from conservation laws for each constituent of the remodeling is the femur under the traction loadings, individual phases. The cells can be further classified which drive the process so that the bone density is as tumor cells, epithelial cells, fibroblasts, etc. denoted redistributed. Based on the stress distribution, the bone by subscript ˛. Similarly we can distinguish different usually becomes stiffer in the areas of higher stresses. components of the extracellular matrix (ECM), namely, Importantly, the same set of equations can also be collagen, elastin, fibronectin, vitronectin, etc. [20] deused to study the process of wound healing. It is obvi- noted by subscript ˇ. The ECM component velocities ous, however, that the initial and boundary conditions are assumed to be the same, based on the constrained are specified differently. It is worth noting that traction sub-mixture assumption [5]. The concentrations of forces inside the wound can facilitate the closure of chemicals within the liquid are of interest in the extrathe wound. From the computational point of view, the cellular liquid. The above assumptions provide us the specification of solid and liquid volume fractions as mass conservation equations for the constituents as ( , well as pressure is required on all interior and exterior m, and l) boundaries of the computational domain. The interior boundary (inner face) of the wound can @ ˛ C r  . ˛ v ˛ / D  ˛ be assumed to possess sufficiently large quantity of the @t solid and liquid volume fractions, which is implicated @mˇ biologically with sufficient nutrient supplies. On the C r  .mˇ vm / D mˇ (17) @t other hand, the opening of the wound can be prescribed by natural boundary conditions with seepage velocity. The equations above v ˛ and vm denote the velocities of the respective phases. Note that there is no subscript on vm (constrained sub-mixture assumption). Mass Modeling Tumor Growth balance equation expressed as concentrations in the Attempts at developing computational mechanics liquid phase is expressed as models of tumor growth date back over half a century (see, e.g., [9]). Various models have been proposed based on ordinary differential equations

@c D r  .Drc/ C G @t

(18)

M

878

Medical Applications in Bone Remodeling, Wound Healing, Tumor Growth, and Cardiovascular Systems

Here, D denotes the effective diffusivity tensor in the mixture and G contains the production/source terms and degradation/uptake terms relative to the entire mixture. The system of equations requires the velocities of each component to obtain the closure. The motion of the volume fraction of the cells is governed by the momentum equations  

@v

C v  rv

@t



D r  TQ C  b C mQ

(19)

Here,  denotes the volume fraction, J denotes the fluxes that account for the mechanical interactions among the different species, and the source term S accounts for the inter-component mass exchange as well as gains due to proliferation and loss due to cell death. The above Eq. (21) is interpreted as the evolution equation for  which characterizes the phase of the system. This approach modifies the equation for the interface to provide both for convection of the interface and with an appropriate diminishing of the total energy of the system. The free energy of a system of two immiscible fluids consists of mixing, bulk distortion, and anchoring energy. For simple two-phase flows, only mixing energy is retained, which results in a rather simple expression for the free energy .

Similar expressions hold for the extracellular matrix and the liquid phases. The presence of the saturation constraint requires one to introduce a Lagrange multiplier into the Clausius-Duhem inequality and provides expressions for the excess stress TQ and excess interaction force m . The Lagrange multiplier is classically Z  1 2 identified with the interstitial pressure P . Body forces,  jrj2 C f .; T / dV F .; r; T / D 2 b, are ignored for the equations for the ECM, and Z the excess stress tensor in the extracellular liquid is D ftot dV (22) assumed to be negligible in accordance with the low viscous forces in porous media flow studies. With these assumptions, we obtain the following equations: Thus the total energy is minimized with the definition of the chemical potential which implements an energy  ˛ rP C r  . ˛ T ˛ / C m ˛ C  ˛ b˛ D 0 cost proportional to the interface width . The following equation describes evolution of the phase field mrP C r  .mTm/ C mm D 0 parameter: lrP C ml D 0

(20)

The set of equations above provides the governing differential equations required to solve tumor growth problems. The primary variables to be solved are f ˛ ; mˇ ; P g. The governing equations can be solved with suitable boundary conditions of specified volume fractions of the cells, extracellular liquid, and pressures. Fluxes of these variables across the boundaries also need to be specified for a complete description of the problem. Other approaches in modeling tumor growth involve tracking the moving interface of the growing tumor. Among them is the phase field approach. The derivation of the basic governing equations is given in Wise [21]. From the continuum advection-reactiondiffusion equations, the volume fractions of the tissue components obey @ C r  .u/ D r  J C S @t

@ C ur D r   r @t



@ftot @ftot r @ @r

(23)

where ftot is the total free energy of the system. The above Eq. (23) seeks to minimize the total free energy of the system with a relaxation time controlled by the mobility  . With some further approximations, the partial differential equation governing the phase field variable is obtained as the Cahn-Hilliard equation: @ C ur D r   rG @t

(24)

where G is the chemical potential and  is the mobility. The mobility determines the time scale of the CahnHilliard diffusion and must be large enough to retain a constant interfacial thickness but small enough so that the convective terms are not overly damped. The mobility is defined as a function of the interface thickness (21) as  D  2 . The chemical potential is provided by

Medical Applications in Bone Remodeling, Wound Healing, Tumor Growth, and Cardiovascular Systems

. 2  1/ 2 G D  r  C 2

(25)

The Cahn-Hilliard equation forces  to take values of 1 or C1 except in a very thin region on the fluid-fluid interface. The introduction of the phase field interface allows the above equation to be written as a set of two second-order PDEs:  @ C u  r D r  2 r @t  D r   2 r C . 2  1/

(26) (27)

The above equation is the simplest phase field model and is known as model A in the terminology of phase field transitions [3, 6, 22]. Phase field approaches have been applied for solving the tumor growth, and multiphase descriptions of an evolving tumor have been obtained with each phase having its own interface and a characteristic front of the moving interface obtained with suitable approximations. When specific applications of the phase field approach to tumor growth are considered, the proliferative and nonproliferative cells are described by the phase field parameter . The relevant equations in the context of tumor growth are provided by the following [6, 23]:   @ D M r 2  C  3  r 2  C ˛p .T /./ @t (28) Here, M denotes the mobility coefficient, T stands for the concentration of hypoxic cell-produced angiogenic factor, and ./ denotes the Heaviside function which takes a value of 1 when its argument is positive. The proliferation rate is denoted by ˛p .T / and as usual  denotes the width of the capillary wall. The equation above is solved with the governing equation for the angiogenic factor T . The angiogenic factor diffuses randomly from the hypoxic tumor area where it is produced and obeys the following equation: @Ti D r  .DrT /  ˛T T ./ @t

Modeling Cardiovascular Fluid Flow Cardiovascular system modeling is another important field in predictive medicine. Computational modeling of blood flow requires solving, in the general sense, three-dimensional transient flow equations in deforming blood vessels [24]. The appropriate framework for problems of this type is the arbitrary LagrangianEulerian (ALE) description of the continuous media in which the fluid and solid domains are allowed to move to follow the distensible vessels and deforming fluid domain. Under the assumption of zero wall motion, the problem reduces to the Eulerian description of the fixed spatial domain. The strong form of the problem governing incompressible Newtonian fluid flow in a fixed domain consists of the Navier-Stokes equations and suitable initial and boundary conditions. Direct analytical solutions of these equations are not available for complex domains, and numerical methods must be used. The finite element method has been the most widely used numerical method for solving the equations governing blood flow [24]. In the Eulerian frame of reference, the conservation of mass is expressed as the continuity equation, and the conservation of momentum closes the system of equations with expressions of the stress tensor for the Newtonian fluid derived from the second law of thermodynamics. The flow of blood inside the arteries and the heart comprises some of the examples in biological systems. The governing equations for laminar fluid flow in cardiovascular structures are provided by the incompressible Navier-Stokes equations when the fluid flow is in the laminar regime with assumptions of constant viscosity [25]. We provide here the basic conservation laws for fluid flow expressed as the Navier-Stokes equations. Consider the flow of a nonsteady Newtonian fluid with density  and viscosity . Let  2 Rn and t 2 Œ0; T  be the spatial and temporal domains, respectively, where n is the number of space dimensions. Let  denote the boundary of . We consider the following velocity-pressure formulation of Navier-Stokes equations governing unsteady incompressible flows. 

(29)

In the equation above, D denotes the diffusion coefficient of the factor in the tissue and ˛T denotes the rate of consumption by the endothelial cells.

879



@u C u  ru  f  r   D 0 on  8 t 2 Œ0; T  @t (30) r  u D 0; on  8 t 2 Œ0; T 

(31)

M

880

Medical Applications in Bone Remodeling, Wound Healing, Tumor Growth, and Cardiovascular Systems

where , u, f, and œ are the density, velocity, body of 0:04 poise (or 0:004 kg/m-s). Three-dimensional force, and stress tensor, respectively. The stress tensor transient simulations require meshing the domain with is written as a sum of the isotropic and deviatoric parts: significant degree of freedom and require considerable computing time [25].  D pI C T D pI C 2 .u/ (32)  1 ru C ruT (33) 2 References Here, I is the identity tensor,  D , p is the pressure, and u is the fluid velocity. The part of the 1. Atkin, R.J., Craine, R.E.: Continuum theory of mixtures: basic theory and historical development. Q. J. Mech. Appl. boundary at which the velocity is assumed to be speciMath. 29, 209–244 (1976) fied is denoted by g 2. Ricken, T., Schwarz, A., Bluhm, J.A.: Triphasic model of .u/ D

u D g on g 8 t 2 Œ0; T 

(34)

The natural boundary conditions associated with Eq. (30) are the conditions on the stress components, and these are the conditions assumed to be imposed on the remaining part of the boundary, n   D h on Th 8 t 2 Œ0; T 

(35)

where g and h are the complementary subsets of the boundary , or  D g [ h . As the initial condition, a divergence-free velocity field, u0 .x/, is imposed. To simulate realistic flow conditions, one needs to consider a pulsatile flow as the boundary conditions at the inlet. The governing equations along with the boundary conditions characterize the flow through a cardiovascular system, which can be solved to obtain the descriptions of the velocity profiles and pressure inside the domain. In general, stabilized finite element methods have been used for solving incompressible flow inside both arteries and the heart [24]. Realistic simulations of the blood flow have required three-dimensional patient-specific solid models of pulmonary tree by integrating combined magnetic resonance imaging (MRI) and computational fluid dynamics. An extension of MRI is magnetic resonance angiography (MRA), which has also been used for reconstructing the three-dimensional coarse mesh from MRA data. Three-dimensional subject-specific solid models of the pulmonary tree have been created from these MRA images as well [25]. The finite element mesh discretization of the problem is effective in capturing multiple levels of blood vessel branches. Different conditions of the patient resting and exercise conditions have been simulated. Blood is usually assumed to behave as a Newtonian fluid with a viscosity

3.

4.

5.

6.

7. 8.

9.

10. 11.

12.

13.

14.

15.

16.

transversely isotropic biological tissue with applications to stress and biologically induced growth. Comput. Mater. Sci. 39, 124–136 (2007) Wise, S.M., Lowengrub, J.S., Friebos, H.B., Cristini, V.: Three-dimensional diffuse-interface simulation of multispecies tumor growth-I model and numerical method. J. Theor. Biol. 253, 523–543 (2008) Ambrosi, D., Preziosi, L.: On the closure of mass balance models for tumor growth. Math. Models Methods Appl. Sci. 12, 737–754 (2002) Preziosi, L.: Cancer Modeling and Simulation. Chapman and Hall/CRC Mathematical Biology and Medicine Series. London (2003) Oden, J.T., Hawkins, A., Prudhomme, S.: General diffuseinterface theories and an approach to predictive tumor growth modeling. Math. Models Methods Appl. Sci. 20(3), 477–517 (2010) Ambrosi, D., Mollica, F.: On the mechanics of a growing tumor. Int. J. Eng. Sci. 40, 1297–1316 (2002) Coleman, B.D., Noll, W.: The thermodynamics of elastic materials with head conduction and viscosity. Arch. Ration. Mech. Anal. 13, 167–178 (1963) Armitage, P., Doll, R.: The age distribution of cancer and a multi-stage theory of carcinogenesis. Br. J. Cancer 8, 1 (1954) Ward, J.P., King, J.R.: Mathematical modelling of avascular tumor growth. J. Math. Appl. Med. Biol. 14, 39–69 (1997) Grecu, D., Carstea, A.S., Grecu, A.T., Visinescu, A.: Mathematical modelling of tumor growth. Rom. Rep. Phys. 59, 447–455 (2007) Tian, J.T., Stone, K., Wallin, T.J.: A simplified mathematical model of solid tumor regrowth with therapies. Discret. Contin. Dyn. Syst. 771–779 (2009) Lloyd, B.A., Szczerba, D., Szekely, G.: A coupled finite element model of tumor growth and vascularization. Med. Image Comput. Comput. Assist. Interv. 2, 874–881 (2007) Roose, T., Chapman, S.J., Maini, P.K.: Mathematical models of avascular tumor growth. SIAM Rev. 49(2), 179–208 (2007) Macklin, P., McDougall, S., Anderson, A.R.A., Chaplain, M.A.J., Cristini, V., Lowengrub, J.: Multiscale modelling and nonlinear simulation of vascular tumor growth. J. Math. Biol. 58(4–5), 765–798 (2009) Cristini, V., Lowengrub, J.: Multiscale Modeling of Cancer. An Integrated Experimental and Mathematical Modeling Approach. Cambridge University Press, Cambridge (2010)

Medical Imaging 17. Tan, Y., Hanin, L.: Handbook of Cancer Models with Applications. Series in Mathematical Biology and Medicine. World Scientific, Singapore (2008) 18. Deisboeck, T.S., Stmatakos, S.: Multiscale Cancer Modeling. Mathematical and Computational Biology Series. CRC/Taylor and Francis Group, Boca Raton (2011) 19. Wodarz, D., Komarova, N.L.: Computational Biology of Cancer. Lecture Notes and Mathematical Modeling. World Scientific, Hackensack (2005) 20. Astanin, S., Preziosi, L.: Multiphase models of tumor growth. In: Selected Topics in Cancer Modeling. Modeling and Simulation in Science, Engineering, and Technology, pp. 1–31. Birkh¨auser, Boston (2008) 21. Wise, S.M., Lowngrub, J.S., Frieboes, H.B., Cristini, V.: Three-dimensional multispecies nonlinear tumor growth-I model and numerical method. J. Theor. Biol. 253, 524–543 (2008) 22. Travasso, R.D.M., Castro, M., Oliveira, J.C.R.E.: The phase-field model in tumor growth. Philos. Mag. 91(1), 183–206 (2011) 23. Travasso, R.D.M., Poire, E.C., Castro, M., Rodrguez, J.C.M.: Tumor angiogenesis and vascular patterning: a mathematical model. PLos One 6(5), 1–9 (2011) 24. Taylor, C.A., Hughes, J.T.R., Zarins, C.K.: Finite element modeling of blood flow in arteries. Comput. Methods Appl. Mech. Eng. 158, 155–196 (1998) 25. Tang, B.T., Fonte, T.A., Chan, F.P., Tsao, P.S., Feinstein, J.A., Taylor, C.A.: Three dimensional hemodynamics in the human pulmonary arteries under resting and exercise conditions. Ann. Biomed. Eng. 39(1), 347–358 (2011)

Medical Imaging Charles L. Epstein Departments of Mathematics and Radiology, University of Pennsylvania, Philadelphia, PA, USA

Synonyms Magnetic resonance imaging; Radiological imaging; Ultrasound; X-ray computed tomography

Description Medical imaging is a collection of technologies for noninvasively investigating the internal anatomy and physiology of living creatures. The prehistory of modern imaging includes various techniques for physical examination, which employ palpation and other external observations. Though the observations are indirect

881

and require considerable interpretation to relate to the internal state of being, each of these methods is based on the principle that some observable feature differs between healthy and sick subjects. While new technologies have vastly expanded the collection of available measurements, this basic principle remains the central tenet of medical imaging. Modern medical imaging is divided into different modalities according to the physical principles underlying the measurement process. These differences in underlying physics lead to contrasts in the images that reflect different aspects of anatomy or physiology. The utility of a modality is largely governed by three interconnected considerations: contrast, resolution, and noise. Contrast refers to the physical or chemical distinctions that produce the image itself, and the magnitude of these differences in the reconstructed image. Resolution is usually thought of as the size of the smallest objects discernible in the image. Finally noise is an inevitable consequence of real physical measurements. The ratio between the size of the signal and the size of the noise which contaminates it, called SNR, limits both the contrast and resolution attainable in any reconstructed image. Technological advances in the nineteenth and twentieth centuries led to a proliferation of methods for medical imaging. The first such advances were the development of photographic imaging, and the discovery of x-rays. These were the precursors of projection x-rays, which led, after the development of far more sensitive solid-state detectors, to x-ray tomography. Sonar, which was used by the military to detect submarines, was adapted, along with ideas from radar, to ultrasound imaging. In this modality highfrequency acoustic energy is used as a probe of internal anatomy. Taking advantage of the D¨oppler effect, ultrasound can also be used to visualize blood flow, see [7]. Nuclear magnetic resonance, which depends on the subtle quantum mechanical phenomenon of spin, was originally developed as a spectroscopic technique in physical chemistry. With the advent of powerful, large, high-quality superconducting magnets, it became feasible to use this phenomenon to study both internal anatomy and physiology. In its simplest form the contrast in MRI comes from the distribution of water molecules within the body. The richness of the spinresonance phenomenon allows the use of other experimental protocols to modulate the contrast, probing

M

882

many aspects of the chemical and physical environment. The four imaging modalities in common clinical use are (1) x-ray computed tomography (x-ray CT), (2) ultrasound (US), (3) magnetic resonance imaging (MRI), and (4) emission tomography (PET and SPECT). In this article we only consider the details of x-ray CT and MRI. Good general references for the physical principles underlying these modalities are [4, 7]. There are also several experimental techniques, such as diffuse optical tomography (DOT) and electrical impedance tomography (EIT), which, largely due to intrinsic mathematical difficulties, have yet to produce useful diagnostic tools. A very promising recent development involves hybrid modalities, which combine a high-contrast (low-resolution) modality with a high-resolution (low-contrast) modality. For example, photo-acoustic imaging uses infrared light for excitation of acoustic vibrations and ultrasound for detection, see [1]. Each measurement process is described by a mathematical model, which in turn is used to “invert” the measurements and build an image of some aspect of the internal state of the organism. The success of an imaging modality relies upon having a stable and accurate inverse algorithm, usually based on an exact inversion formula, as well as the availability of sufficiently many measurements with an adequate signal-to-noise ratio. The quality of the reconstructed image is determined by complicated interactions among the size and quality of the data set, the available contrast, and the inversion method.

Medical Imaging

Medical Imaging, Fig. 1 A projection x-ray image (Image courtesy: Dr. Ari D. Goldberg)

Here x.s/ is the point along the line, `; and s is arclength parametrization. If the intersection ` \ B lies between smin and smax ; then Beer’s law predicts that: 

X-Ray Computed Tomography The first “modern” imaging method was the projection x-ray, introduced in the late 1800s by Roentgen. X-rays are a high-energy form of electromagnetic radiation, which pass relatively easily through the materials commonly found in living organisms. The interaction of xrays with an object B is modeled by a function B .x/; called the attenuation coefficient. Here x is a location within B: If we imagine that an x-ray beam travels along a straight line, `; then Beer’s law predicts that I.s/; the intensity of the beam satisfies the differential equation: dI D B .x.s//I.s/: (1) ds

Iout log Iin



Zsmax B .x.s//ds: .`/ D 

(2)

smin

Early x-ray images recorded the differential attenuation of the x-ray beams by different parts of the body, as differing densities on a photographic plate. In the photograph highly attenuating regions appear light, and less dense regions appear dark. An example is shown in Fig. 1. X-ray images display a good contrast between bone and soft tissues, though there is little contrast between different types of soft tissues. While the mathematical model embodied in Beer’s law is not needed to interpret projection x-ray images, it is an essential step to go from this simple modality to x-ray computed tomography.

Medical Imaging

883

X-ray CT was first developed by Alan Cormack in the early 1960s, though the lack of powerful computers made the idea impractical. It was rediscovered by Godfrey Hounsfield in the early 1970s. Both received the Nobel prize for this work in 1979, see [6]. Hounsfield was inspired by the recent development of solid-state x-ray detectors, which were more sensitive and had a much larger dynamic range than photographic film. This is essential for medical applications of x-ray CT, as the attenuation coefficients of different soft tissues in the human body differ by less than 3 %. By 1971, solidstate detectors and improved computers made x-ray tomography a practical possibility. Medical Imaging, Fig. 2 Radon transform data, shown as a The mathematical model embodied in Beer’s law sinogram, for the Shepp–Logan phantom. The horizontal axis is leads to a simple description of the measurements  and the vertical axis t available in an x-ray CT-machine. Assuming that we have a monochromatic source of x-rays the measurement described in (2) is the Radon (in two dimensions), or x-ray transform (in three dimensions) of the attenuation coefficient, B .x/: For simplicity we consider the two-dimensional case. The collection, L; of oriented lines in R2 is conveniently parameterized by S 1  R1 ; with .t; / corresponding to the oriented line: `t; D ft.cos ; sin / C s. sin ; cos / W s 2 R1 g: (3) The Radon transform can then be defined by: Z RB .t; / D

B .t.cos ; sin / `t;

C s. sin ; cos //ds:

(4) Medical Imaging, Fig. 3 Filtered back-projection reconstruction of the Shepp–Logan phantom from the data in Fig. 2

The measurements made by an x-ray CT-machine are modeled as samples of RB .t; /: The actual physical design of the machine determines exactly which Z1 samples are collected. The raw data collected by an x- f R.t; /e i t  dt D F ..cos ; sin //: R.; / D ray CT-machine can be represented as a sinogram, as 1 shown in Fig. 2. The reconstructed image is shown in (5) Fig. 3. This theorem and the inversion formula for the The inversion formula for the Radon transform is called the filtered back-projection formula. It is derived two-dimensional Fourier transform show that we can reconstruct B by first filtering the Radon transform: by using the Central Slice theorem: Theorem 1 (Central Slice Theorem) The Radon transform of ; R; is related to its two-dimensional Fourier transform, F ; by the one-dimensional Fourier transform of R in t W

1 GRB .; / D 2

Z1

e

RB .r; /e i r jrjdr: 1

(6)

M

884

Medical Imaging

The total magnetic field is therefore: B.x; t/ D and then back-projecting, which is R ; the adjoint of the Radon transform itself: B0 .x/ C G.t/  x C B1 .t/: The response of the bulk nuclear magnetization, M; to such a field is governed Z 1 by Bloch’s phenomenological equation: GRB .h.cos ; sin /; .x; y/i; /d: B .x; y/D 2  0 1 dM (7) .x; t/ D  M.x; t/  B.x; t/  dt T1 .x/  The filtration step RB ! GRB is implemented 1 using a fast Fourier transform. The multiplication .Mk .x; t/  M0 .x//  M? .x; t/: (9) T .x/ 2 by jrj in the frequency domain makes it mildly ill-conditioned; nonetheless the high quality of the data available in a modern CT-scanner allows for Here Mk is the component of M parallel to B0 and stable reconstructions with a resolution of less than M? is the orthogonal component. The terms with a millimeter. As a map from a function g.t; / on L coefficients T1 and T2 describe relaxation processes to functions on R2 ; back-projection can be understood which tend to relax M toward the equilibrium state M0 : as half the average of g on the set of lines that pass The components Mk and M? relax at different rates through .x; y/: A detailed discussion of x-ray CT can T1 > T2 : In most medical applications their values lie in the range of 50 ms–2 s. The spatial dependence of be found in [2]. T1 and T2 provides several possibilities for contrast in MR-images, sometimes called T1 - or T2 -weighted images. Note that (9) is a system of ordinary differential Magnetic Resonance Imaging equations in time, t; and that the spatial position, x; Magnetic resonance imaging takes advantage of the appears as a pure parameter. Ignoring the relaxation terms for the moment and fact that the protons in water molecules have both an intrinsic magnetic moment  and an intrinsic angular assuming that B is independent of time, we see that (9) momentum, J; known as spin. As both of these quan- predicts that the magnetization M.x/ will precess tum mechanical observables transform by the standard around the B0 .x/ with angular velocity ! D  kB0 .x/k: representation of SO.3/ on R3 ; the Wigner–Eckert This is the resonance phenomenon alluded to in the Theorem implies that there is a constant ; called the name “nuclear magnetic resonance.” Faraday’s Law gyromagnetic ratio, so that  D  J: For a water proton predicts that such a precessing magnetization will  42:5 MHz/T. If an ensemble of water protons is produce an E.M.F. in a coil C with placed in a static magnetic field B0 ; then, after a short Z time, the protons become polarized producing a bulk d M.x; t/  n.x/dS; (10) EMF / magnetization M0 : If .x/ now represents the density dt ˙ of water, as a function of position, then thermodynamic considerations show that there is a constant C for for ˙ a surface spanning C: A simple calculation which: C.x/ B0 .x/: M0 .x/

(8) shows that the strength of the signal is proportional T to ! 2 ; which explains the utility of using a very ı At room temperature (T 300 K) this field is quite strong background field. The noise magnitude in MRsmall and is, for all intents and purposes, not directly measurements is proportional to !; hence the SNR is detectable. proportional to ! as well. A clinical MRI scanner consists of a large For the remainder of this discussion we assume solenoidal magnet, which produces a strong, that B0 is a homogeneous field of the form B0 D homogeneous background field, B0 ; along with coaxial .0; 0; b0 /: The frequency !0 D b0 is called the Larelectromagnets, which produce gradient fields G.t/  x; mor frequency. The main magnet of a clinical scanner used for spatial encoding, and finally a radio-frequency typically has a field strength between 1.5 and 7 T, (RF) coil, which produces an excitation field, B1 .t/; which translates to Larmor frequencies between 64 and and is also used for signal detection. 300 MHz.

Medical Imaging

885

The RF-component of the field B1 .t/ is assumed to take the form: .a.t/ cos !0 t; a.t/ sin !0 t; 0/; with a.t/ nonzero for a short period of time. As implied by the notation, the gradient fields are designed to have a linear spatial dependence, and therefore take the form: G.t/  x D .g1 .t/x3  g3 .t/x1 ; g2 .t/x3 ; g1 .t/x1 Cg2 .t/x2 C g3 .t/x3 /:

(11)

Here g.t/ D .g1 .t/; g2 .t/; g3 .t// is a spatially independent vector describing the time course of the gradient field. Typically kgk NS is used, which functions [40]. An application of meshfree methods leads to an overdetermined system in Eq. (22) which is for problems with higher-order differentiation, such can be solved by the least-squares method, the QR as the Kirchhoff-Love plate and shell problems [35], decomposition, or the singular value decomposition where meshfree approximation functions with higher(SVD). The few commonly used radial basis functions order continuities can be employed. Meshfree methods are are shown to be effective for large deformation and  3  2 fragment-impact problems (Fig. 3) [16, 25], where 2 n 2 Multiquadrics (MQ): gI .x/ D rI C c ; mesh entanglement in the finite element method can  2 r be greatly alleviated. Another popular application of (24) Gaussian: gI .x/ D exp  I2 meshfree method is for the modeling of evolving disc continuities, such as crack propagation simulations. where rI D kx  x I k and c is called the shape The extended finite element method [49, 61] that comparameter that controls the localization of the function. bines the FEM approximation and the crack-tip enIn MQ RBF function in Eq. (24), the function is called richment functions [9, 11] under the partition of unity reciprocal MQ RBF if n D 1, linear MQ RBF if n D 2, framework has been the recent focal point in fracture and cubic MQ RBF if n D 3, and so forth. There modeling.

Meshless and Meshfree Methods

893

Meshless and Meshfree Methods, Fig. 3 Experimental and numerical damage patterns on the exit face of a concrete plate penetrated by a bullet

References 1. Aluru, N.R.: A point collocation method based on reproducing kernel approximations. Int. J. Numer. Methods Eng. 47, 1083–1121 (2000) 2. Arroyo, M., Ortiz, M.: Local maximum-entropy approximation schemes: a seamless bridge between finite elements and meshfree methods. Int. J. Numer. Methods Eng. 65, 2167– 2202 (2006) 3. Babuˇska, I.: The finite element method with Lagrangian multipliers. Numer. Math. 20, 179–192 (1973) 4. Babuˇska, I., Melenk, J.M.: The partition of unity method. Int. J. Numer. Methods Eng. 40, 727–758 (1997) 5. Beissel, S., Belytschko, T.: Nodal integration of the elementfree Galerkin method. Comput. Methods Appl. Mech. Eng. 139, 49–74 (1996) 6. Belikov, V.V., Ivanov, V.D., Kontorovich, V.K., Korytnik, S.A., Semenov, A.Yu.: The non-Sibsonian interpolation: a new method of interpolation of the value of a function on an arbitrary set of points. Comput. Math. Math. Phys. 37, 9–15 (1997) 7. Belytschko, T., Krongauz, Y., Organ, D., Fleming, M., Krysl, P.: Meshless methods: an overview and recent developments. Comput. Methods Appl. Mech. Eng. 139, 3–47 (1996) 8. Belytschko, T., Lu, Y.Y., Gu, L.: Element-free Galerkin methods. Int. J. Numer. Methods Eng. 37, 229–256 (1994) 9. Belytschko, T., Lu, Y.Y., Gu, L.: Crack propagation by element-free Galerkin methods. Eng. Fract. Mech. 51, 295– 315 (1995) 10. Belytschko, T., Organ, D., Krongauz, Y.: A coupled finite element-element-free Galerkin method. Comput. Mech. 17, 186–195 (1995) 11. Belytschko, T., Tabbara, M.: Dynamic fracture using element-free Galerkin methods. Int. J. Numer. Methods Eng. 39, 923–938 (1996) 12. Bonet, J., Kulasegaram, S.: Correction and stabilization of smooth particle hydrodynamics methods with applications in metal forming simulations. Int. J. Numer. Methods Eng. 47, 1189–1214 (2000) 13. Braun, J., Sambridge, M.: A numerical method for solving partial differential equations on highly irregular evolving grids. Nature 376, 655–660 (1995)

14. Brezzi, F.: On the existence, uniqueness and approximation of saddle-point problems arising from Lagrangian multipliers. Rev. Franc¸aise Autom. Inf. Recherche Op´erationnelle S´er. Rouge 8, 129–151 (1974) 15. Chen, J.S., Han, W., You, Y., Meng, X.: A reproducing kernel method with nodal interpolation property. Int. J. Numer. Methods Eng. 56, 935–960 (2003) 16. Chen, J.S., Pan, C., Wu, C.T., Liu, W.K.: Reproducing kernel particle methods for large deformation analysis of non-linear structures. Comput. Methods Appl. Mech. Eng. 139, 195–227 (1996) 17. Chen, J.S., Wang, D.: A constrained reproducing kernel particle formulation for shear deformable shell in Cartesian coordinates. Int. J. Numer. Methods Eng. 68, 151–172 (2006) 18. Chen, J.S., Wang, H.P.: New boundary condition treatments in meshfree computation of contact problems. Comput. Methods Appl. Mech. Eng. 187, 441–468 (2000) 19. Chen, J.S., Wu, C.T., Yoon, S., You, Y.: A stabilized conforming nodal integration for Galerkin meshfree methods. Int. J. Numer. Methods Eng. 50, 435–466 (2001) 20. Chen, J.S., Yoon, S., Wu, C.T.: Non-linear version of stabilized conforming nodal integration for Galerkin meshfree methods. Int. J. Numer. Methods Eng. 53, 2587–2615 (2002) 21. Duarte, C.A., Babuˇska, I., Oden, J.T.: Generalized finite element methods for three-dimensional structural mechanics problems. Comput. Struct. 77, 215–232 (2000) 22. Duarte, C.A., Oden, J.T.: An h-p adaptive method using clouds. Comput. Methods Appl. Mech. Eng. 139, 237–262 (1996) 23. Fern´andez-M´endez, S., Huerta, A.: Imposing essential boundary conditions in mesh-free methods. Comput. Methods Appl. Mech. Eng. 193, 1257–1275 (2004) 24. Gingold, R.A., Monaghan, J.J.: Smoothed particle hydrodynamics: theory and application to non-spherical stars. Mon. Not. R. Astron. Soc. 181, 375–389 (1977) 25. Guan, P.C., Chi, S.W., Chen, J.S., Slawson, T.R., Roth, M.J.: Semi-Lagrangian reproducing kernel particle method for fragment-impact problems. Int. J. Impact Eng. 38, 1033– 1047 (2011)

M

894 26. Han, W., Meng, X.: Error analysis of reproducing kernel particle method. Comput. Methods Appl. Mech. Eng. 190, 6157–6181 (2001) 27. Hardy, R.L.: Multiquadric equations of topography and other irregular surfaces. J. Geophys. Res. 76, 1905–1915 (1971) 28. Hu, H.Y., Chen, J.S., Hu, W.: Weighted radial basis collocation method for boundary value problems. Int. J. Numer. Methods Eng. 69, 2736–2757 (2007) 29. Hu, H.Y., Chen, J.S., Hu, W.: Error analysis of collocation method based on reproducing kernel approximation. Numer. Methods Partial Differ. Equ. 27, 554–580 (2011) 30. Huerta, A., Fern´andez-M´endez, S.: Enrichment and coupling of the finite element and meshless methods. Int. J. Numer. Methods Eng. 48, 1615–1636 (2000) 31. Jaynes, E.T.: Information theory and statistical mechanics. Phys. Rev. 106, 620–630 (1957) 32. Kaljevi´c, I., Saigal, S.: An improved element free Galerkin formulation. Int. J. Numer. Methods Eng. 40, 2953–2974 (1997) 33. Kansa, E.J.: Multiquadrics—A scattered data approximation scheme with applications to computational fluiddynamics—I surface approximations and partial derivative estimates. Comput. Math. Appl. 19, 127–145 (1990) 34. Kansa, E.J.: Multiquadrics—A scattered data approximation scheme with applications to computational fluiddynamics—II solutions to parabolic, hyperbolic and elliptic partial differential equations. Comput. Math. Appl. 19, 147– 161 (1990) 35. Krysl, P., Belytschko, T.: Analysis of thin plates by the element-free Galerkin method. Comput. Mech. 17, 26–35 (1995) 36. Lancaster, P., Salkauskas, K.: Surfaces generated by moving least squares methods. Math. Comput. 37, 141–158 (1981) 37. Li, S., Liu, W.K.: Meshfree Particle Methods, 2nd edn. Springer, New York (2007) 38. Liszka, T., Orkisz, J.: The finite difference method at arbitrary irregular grids and its application in applied mechanics. Comput. Struct. 11, 83–95 (1980) 39. Liu, G.R.: Meshfree Methods: Moving Beyond the Finite Element Method, 2nd edn. CRC, Boca Raton (2010) 40. Liu, W.K., Chen, Y.: Wavelet and multiple scale reproducing kernel methods. Int. J. Numer. Methods Fluids 21, 901–931 (1995) 41. Liu, W.K., Chen, Y., Jun, S., Chen, J.S., Belytschko, T., Pan, C., Uras, R.A., Chang, C.T.: Overview and applications of the reproducing kernel particle methods. Arch. Comput. Methods Eng. 3, 3–80 (1996) 42. Liu, W.K., Han, W., Lu, H., Li, S., Cao, J.: Reproducing kernel element method. Part I: theoretical formulation. Comput. Methods Appl. Mech. Eng. 193, 933–951 (2004) 43. Liu, W.K., Jun, S., Li, S., Adee, J., Belytschko, T.: Reproducing kernel particle methods for structural dynamics. Int. J. Numer. Methods Eng. 38, 1655–1679 (1995) 44. Liu, W.K., Jun, S., Zhang, Y.F.: Reproducing kernel particle methods. Int. J. Numer. Methods Fluids 20, 1081–1106 (1995) 45. Liu, W.K., Li, S., Belytschko, T.: Moving least-square reproducing kernel methods (I) methodology and convergence. Comput. Methods Appl. Mech. Eng. 143, 113–154 (1997)

Meshless and Meshfree Methods 46. Lu, Y.Y., Belytschko, T., Gu, L.: A new implementation of the element free Galerkin method. Comput. Methods Appl. Mech. Eng. 113, 397–414 (1994) 47. Madych, W.R., Nelson, S.A.: Bounds on multivariate polynomials and exponential error estimates for multiquadric interpolation. J. Approx. Theory 70, 94–114 (1992) 48. Melenk, J.M., Babuˇska, I.: The partition of unity finite element method: basic theory and applications. Comput. Methods Appl. Mech. Eng. 139, 289–314 (1996) 49. Mo¨es, N., Dolbow, J., Belytschko, T.: A finite element method for crack growth without remeshing. Int. J. Numer. Methods Eng. 46, 131–150 (1999) 50. Nayroles, B., Touzot, G., Villon, P.: Generalizing the finite element method: diffuse approximation and diffuse elements. Comput. Mech. 10, 307–318 (1992) ¨ 51. Nitsche, J.: Uber ein Variationsprinzip zur L¨osung von Dirichlet-Problemen bei Verwendung von Teilr¨aumen, die keinen Randbedingungen unterworfen sind. Abh. Math. Semin. Univ. Hambg. 36, 9–15 (1971) 52. Oden, J.T., Duarte, C.A., Zienkiewicz, O.C.: A new cloudbased hp finite element method. Comput. Methods Appl. Mech. Eng. 153, 117–126 (1998) 53. O˜nate, E., Idelsohn, S., Zienkiewicz, O.C., Taylor, R.L.: A finite point method in computational mechanics. Application to convective transport and fluid flow. Int. J. Numer. Methods Eng. 39, 3839–3866 (1996) 54. Perrone, N., Kao, R.: A general finite difference method for arbitrary meshes. Comput. Struct. 5, 45–57 (1975) 55. Puso, M.A., Chen, J.S., Zywicz, E., Elmer, W.: Meshfree and finite element nodal integration methods. Int. J. Numer. Methods Eng. 74, 416–446 (2008) 56. Randles, P.W., Libersky, L.D.: Smoothed particle hydrodynamics: some recent improvements and applications. Comput. Methods Appl. Mech. Eng. 139, 375–408 (1996) 57. Schaback, R., Wendland, H.: Using compactly supported radial basis functions to solve partial differential equations. Bound. Elem. Technol. XIII, 311–324 (1999) 58. Sibson, R.: A vector identity for the Dirichlet tessellation. Math. Proc. Camb. Phil. Soc. 87, 151–155 (1980) 59. Strouboulis, T., Copps, K., Babuˇska, I.: The generalized finite element method. Comput. Methods Appl. Mech. Eng. 190, 4081–4193 (2001) 60. Sukumar, N.: Construction of polygonal interpolants: a maximum entropy approach. Int. J. Numer. Methods Eng. 61, 2159–2181 (2004) 61. Sukumar, N., Mo¨es, N., Moran, B., Belytschko, T.: Extended finite element method for three-dimensional crack modelling. Int. J. Numer. Methods Eng. 48, 1549–1570 (2000) 62. Sukumar, N., Moran, B., Belytschko, T.: The natural element method in solid mechanics. Int. J. Numer. Methods Eng. 43, 839–887 (1998) 63. Wang, D., Chen, J.S.: Locking-free stabilized conforming nodal integration for meshfree Mindlin-Reissner plate formulation. Comput. Methods Appl. Mech. Eng. 193, 1065– 1083 (2004) 64. You, Y., Chen, J.S., Lu, H.: Filters, reproducing kernel, and adaptive meshfree method. Comput. Mech. 31, 316–326 (2003)

Metabolic Networks, Modeling

Metabolic Networks, Modeling

895

the reaction. The simplest kind of kinetics is massaction kinetics in which a unimolecular reaction (one k

Michael C. Reed1 , Thomas Kurtz2 , and H. Frederik Nijhout3 1 Department of Mathematics, Duke University, Durham, NC, USA 2 University of Wisconsin, Madison, WI, USA 3 Duke University, Durham, NC, USA

Mathematics Subject Classification 34; 37; 60; 92

Synonyms Biochemical network

Definition Suppose that a system has m different chemicals, A1 ; : : : ; Am ; and define a complex to be an m-vector of nonnegative integers. A metabolic network is a directed graph, not necessarily connected, whose vertices are complexes. There is an edge from complex C to complex D if there exists a chemical reaction in which the chemicals in C with nonzero components are changed into the chemicals in D with nonzero components. The nonzero integer components represent how many molecules of each chemical are used or produced in the reaction. Metabolic networks are also called biochemical networks.

substrate), A ! B, proceeds at a rate proportional to the concentration of the substrate, that is, kŒA, and a k

bimolecular reaction, A C B ! C , proceeds at a rate proportional to the product of the concentrations of the substrates, kŒAŒB, and so forth. Given a chemical reaction diagram, such as Fig. 1, the differential equations for the concentrations of the substrates simply state that the rate of change of each substrate concentration is equal to the sum of the rates of the reactions that make it minus the rates of the reactions that use it. A simple reaction diagram and corresponding differential equations are shown in Fig. 1. Figure 2 shows the simplest reaction diagram for an enzymatic reaction in which an enzyme, E, binds to a substrate, S , to form a complex, C . The complex then dissociates into the product, P , and the enzyme that can be used again. One can write down the four differential equations for the variables S; E; C; P but they cannot be solved in closed form. It is very useful to have a closed form formula for the overall rate of the reaction S ! P because that formula can be compared to experiments and the constants can be determined. Such an approximate formula was derived by Leonor Michaelis and Maud Menten (see Fig. 2).

d[A] A

k1 k2

dt B+C

d[B] dt

C

k3 k4

d[C] D

dt

Description

d[D] dt

Chemicals inside of cells are normally called substrates and the quantity of interest is the concentration of the substrate that could be measured as mass per unit volume or, more typically, number of molecules per unit volume. In Fig. 3, the substrates are indicated by rectangular boxes that contain their acronyms. A chemical reaction changes one or more substrates into other substrates and the function that describes how the rate of this process depends on substrate concentrations and other variables is said to give the kinetics of

= k2[B][C] − k1[A]

= −k2[B][C] + k1[A]

= −k2[B][C] + k1[A]−k3[C] + k4[D]

= k3[C] − k4[D]

Metabolic Networks, Modeling, Fig. 1 On the right are the differential equations corresponding to the reaction diagram if one assumes mass-action kinetics

S+E

k1 k–1

C

k2

E+P

V =

k2Etot [S] K m + [S ]

Metabolic Networks, Modeling, Fig. 2 A simple enzymatic reaction and the Michaelis–Menten formula

M

896

Metabolic Networks, Modeling

Metabolic Networks, Modeling, Fig. 3 Folate and methionine metabolism. The rectangular boxes represent substrates whose acronyms are in the boxes. All the pink boxes are different forms of folate. Each arrow represents a biochemical reaction. The acronyms for the enzymes that catalyze the reactions are

in the blue ellipses. The TS and AICART reactions are important steps in pyrimidine and purine synthesis, respectively. The DNMT reaction methylates cytosines in DNA and is important for gene regulation

Here Etot is the total enzyme concentration, k2 is indicated in Fig. 2, and Km is the so-called Michaelis– Menten constant. The quantity k2 Etot is called the Vmax of the reaction because that is the maximum rate obtained as ŒS  ! 1: There is a substantial mathematical literature about when this approximation is a good one [33]. For further discussion of kinetics and references, see [24]. The biological goal is to understand how large biochemical systems that accomplish particular tasks work, that is, how the behavior of the whole system depends on the components and on small and large changes in inputs. So, for example, the folate cycle in Fig. 3 is central to cell division since it is involved in the production of purines and pyrimidines necessary for copying DNA. Methotrexate, a chemotherapeutic agent, binds to the enzyme DHFR and slows down cell division. Why? And how much methotrexate do you need to cut the rate of cell division in half? The enzyme DNMT catalyzes the methylation of DNA. How does the rate of the DNMT reaction depend on the folate status of the individual, that is, the total concentration of the six folate substrates?

Difficulties It would seem from the description so far that the task of an applied mathematician studying metabolism should be quite straightforward. A biologist sets the questions to be answered. The mathematician writes down the differential equations for the appropriate chemical reaction network. Using databases or original literature, the constants for each reaction, like Km and Vmax , are determined. Then the equations are solved by machine computation and one has the answer. For many different reasons the actual situation is much more difficult and much more interesting. What is the network? The metabolism of cells is an exceptionally large biochemical network and it is not so easy to decide on the “correct” relatively small network that corresponds to some particular cellular task. Typically, the substrates in any small network will also be produced and used up by other small networks and thus the behavior in those other networks affects the one under study. How should one draw the boundaries of a relatively small network so that everything that is important for the effect one is studying is included?

Metabolic Networks, Modeling

Enzyme properties. The rates of reactions depend on the properties of the enzymes that catalyze them. Biochemists often purify these enzymes and study their properties when they are combined with substrates in a test tube. These experiments are typically highly reproducible. However, enzymes may behave very differently in the context of real cells. They are affected by pH and by the presence or absence of many other molecules that activate them or inhibit them. Thus their Km and Vmax may depend on the context in which they are put. Many metabolic pathways are very ancient, for example, the folate cycle occurs in bacteria, and many different species have the “same” enzymes. But, in reality, the enzymes may have different properties because of differences in the genes that code for them.

897

reticulum. In this case, there will clearly be gradients, the well-mixed assumption is not valid, and partial differential equations will be required.

Are these systems at steady state? It is difficult to choose the right network and determine enzyme constants. However, once that is done surely the traditional approach in applied mathematics to large nonlinear systems of ODEs should work. First one determines the steady-states and then one linearizes around the steady-states to determine which ones are asymptotically stable. Unfortunately, many cellular systems are not at or even near steady state. For example, amino acid concentrations in the blood for the hours shortly after meals increase by a factor of 2–6. This means that cells are subject to enormous fluctuations in the inputs Gene expression levels. Enzymes are proteins that of amino acids. The traditional approach has value, of are coded for by genes. The Vmax is roughly propor- course, but new tools, both technical and conceptual, tional to the total enzyme concentration, which is itself are needed for studying these systems of ODEs. dependent on gene expression level and the rate of degradation of the enzyme. The expression level of the Long-range interactions. Many biochemical gene that codes for the enzyme will depend on the cell reaction diagrams do not include the fact that some type (liver cell or epithelial cell) and on the context in substrates influence distant enzymes in the network. which the cell finds itself. This expression level will These are called long range interactions and several are vary between different cells in the same individual, indicated by red arrows in Fig. 3. The substrate SAM between individuals of the same species, and between activates the enzyme CBS and inhibits the enzymes different species that have the same gene. Furthermore, MTHFR and BHMT. The substrate 5mTHF inhibits the expression level may depend on what other genes the enzyme GNMT. We note that “long range” does are turned on or the time of day. Even more daunting is not indicate distance in the cell; we are assuming the the fact that identical cells (same DNA) in exactly the cell is well mixed. “Long-range” refers to distance same environment often show a 30 % variation in gene in the network. It used to be thought that it was easy expression levels [36]. Thus, it is not surprising that the to understand the behavior of chemical networks by Km and Vmax values (that we thought the biochemists walking through the diagrams step by step. But if there would determine for us) vary sometimes by two or are long-range interactions this is no longer possible; three orders of magnitude in public enzyme databases. one must do serious mathematics and/or extensive machine experimentation to determine the system Is the mean field approximation valid? When we properties of the network. write down the differential equations for the concenBut what do these long-range interactions do in the trations of substrates using mass-action, Michaelis– cases indicated in Fig. 3? After meals the methionine Menten, or other kinetics, we are assuming that the input goes way up and the SAM concentration rises cell can be treated as a well-mixed bag of chemicals. dramatically. This activates CBS and inhibits BHMT, There are two natural circumstances where this is not which means that more mass is sent away from the true. First, the number of molecules of a given substrate methionine cycle via the CBS reaction and less mass may be very small; this is particularly true in bio- is recycled within the cycle via the BHMT reaction. chemical networks related to gene expression. In this So these two long-range interaction, roughly conserve case stochastic fluctuations play an important role. mass in the methionine cycle. The other two long range Stochastic methods are discussed below. Second, some interactions keep the DNMT reaction running at almost biochemical reactions occur only in special locations, a constant rate despite large fluctuations in methionine for example, the cell membrane or the endoplasmic input. Here is a verbal description of how this works.

M

898

If SAM starts to go up, the enzyme MTHFR is more inhibited so there will be less of the substrate 5mTHF. Since there is less 5mTHF, the inhibition of GNMT is partly relieved and the extra SAMs that are being produced are taken down the GNMT pathway, leaving the rate of the DNMT reaction about constant [28]. We see that in both cases the long-range interactions have specific regulatory roles and probably evolved for just those reasons. The existence of such longrange interactions makes the study of chemical reaction networks much more difficult.

Theoretical Approaches to Complex Metabolic Systems Cell metabolism is an extremely complex system and the large number of modeling studies on particular parts of the system cannot be summarized in this short entry. However, we can discuss several different theoretical approaches. Metabolic Control Analysis (MCA). This theory, which goes back to the original papers of Kacser and Burns [21, 22], enables one to calculate “control coefficients” that give some information about the system properties of metabolic networks. Let x D < x1 ; x2 ; : : : > denote the substrate concentrations in a large metabolic network and suppose that the network is at a steady state x s .c/, where c denotes a vector of constants that the steady state depends on. These constants may be kinetic constants like Km or Vmax values, initial conditions, input rates, enzyme concentrations, etc. If we assume that the constants are not at critical values where behavior changes, then the mapping c ! x s .c/ will be smooth and we can compute its partial derivatives. Since the kinetic formulas tell us how the fluxes along each pathway depend on the substrate concentrations, we can also compute the rates of change of the fluxes as the parameters c are varied. These are called the “flux control coefficients.” In practice, this can be done by hand only for very simple networks, and so is normally done by machine computation. MCA gives information about system behavior very close to a steady state. One of the major contributions of MCA was to emphasize that local behavior, for example, a flux, was a system property in that it depended on all or many of the constants in c. So, for example, there is no single rate-limiting step for

Metabolic Networks, Modeling

the rate of production of a particular metabolite, but, instead, control is distributed throughout the system. Biochemical Systems Theory (BST). This theory, which goes back to Savageau [32], replaces the diverse nonlinear kinetic formulas for different enzymes with a common power-law formulation. So, the differential equation for each substrate concentration looks like Q ˇ Q ı P P x 0 .t/ D i ˛ij j xijij  i ij j xijij . In the first term, the sum over i represents all the different reactions that produce x and the product over j gives the variables that influence each of those reactions. Similarly, the second sum contains the reactions that use x. The powers ˇij and ıij , which can be fractional or negative, are to be obtained by fitting the model to experimental data. The idea is that one needs to know the network and the influences, but not the detailed kinetics. A representation of the detailed kinetics will emerge from determining the powers by fitting data. Note that the influences would naturally include the long-range interactions mentioned above. From a mathematical point of view there certainly will be such a representation near a (noncritical) steady state if the variables represent deviations from that steady state. One of the drawbacks of this method is that biological data is highly variable (for the reasons discussed above) and therefore the right choice of data set for fitting may not be clear. BST has also been used to simulate gene networks and intracellular signaling networks [31, 34]. Metabolomics. With the advent of high-throughput studies in molecular biology, there has been much interest in applying concepts and techniques from bioinformatics to understanding metabolic systems. The idea is that one measures the concentrations of many metabolites at different times, in different tissues, or cells. Statistical analysis reveals which variables seem to be correlated, and one uses this information to draw a network of influences. Clusters of substrates that vary together could be expected to be part of the same “function.” The resulting networks can be compared, between cells or species, in an effort to understand how function arises from network properties; see, for example [29]. Graph theory. A related approach has been to study the directed graphs that correspond to known metabolic

Metabolic Networks, Modeling

(or gene) networks with the substrates (genes) as nodes and the directed edges representing biochemical reactions (or influences). One is interested in large-scale properties of the networks, such as mean degree of nodes and the existence of large, almost separated, clusters. One is also interested in local properties, such as a particular small connection pattern, that is repeated often in the whole graph. It has been proposed by Alon [1] that such repeated “motifs” have specific biological functions. From the biological point of view, the graph theoretic approaches have a number of pitfalls. It is very natural to assume that graph properties must have biological function or significance, for example, to assume that a node with many edges must be “important,” or clusters of highly connected nodes are all contributing to a single “function.” Nevertheless, it is interesting to study the structure of the graphs independent of the dynamics and to ask what influence or significance the graph structure has. Deficiency zero systems. The study of graphs suggests a natural question about the differential equations that represent metabolic systems. When are the qualitative properties of the system independent of the local details? As discussed in Difficulties, the details will vary considerably from species to species, from tissue to tissue, from cell to cell, and even from time to time in the same cell. Yet large parts of cell metabolism keep functioning in the same way. Thus, the biology tells us that many important system properties are independent of the details of the dynamics. This must be reflected in the mathematics. But how? A major step to understanding the answer to this question was made by Marty Feinberg and colleagues [14]. Let m be the number of substrates. For each reaction in the network, we denote by  the m-component vector of integers that indicates how many molecules of different substrates are used in the reaction;  0 indicates how many are produced by the reaction. Each  is called a complex and we denote the number of complexes by c. The span of the set of vectors of the form    0 is called the stoichiometric subspace and it is invariant under the dynamics. We denote its dimension by s and let ` denote the number of connected components of the graph. The deficiency of the network is defined as ı D c  s  `: The network is weakly reversible if whenever a sequence of reactions allows us to go from complex 1 to complex 2 then there exists a sequence of reactions from 2 to

899

complex 1 . Feinberg formulated the deficiency zero theorem which says that a weakly reversible deficiency zero network with mass-action kinetics has a unique globally stable equilibrium in the interior of each stoichiometric compatibility class. This is true independent of the choice of rate constants. Feinberg gave a proof in the case that there are no boundary equilibria on the faces of the positive orthant. Since then, the proof has been extended to many cases that allow boundary equilibria [2, 9, 35]. Stochastic Models There are many sources of stochasticity in cellular networks. For example, the initial conditions for a cell will be random due to the random assignment of resources at cellular division, and the environment of the cell is random due to fluctuations in such things as temperature and the chemical environment of the cell. If these were the only sources of randomness, then one would only need to modify the coefficients and initial conditions of the differential equation models to obtain reasonable models taking these stochastic effects into account. But many cellular processes involve substrates and enzymes present in the system in very small numbers, and small (random) fluctuations in these numbers may have significant effects on the behavior of the system. Consequently, it is the discreteness of the system as much as its inherent stochasticity that demands a modeling approach different from the classical differential equations. Markov chain models. The idea of modeling a chemical reaction network as a discrete stochastic process at the molecular level dates back at least to [12], with a rapid development beginning in the 1950s and 1960s; see, for example, [7, 8, 27]. The simplest and most frequently used class of models are continuoustime Markov chains. The state X.t/ of the model at time t is a vector of nonnegative integers giving the numbers of molecules of each species in the system at that time. These models are specified by giving transition intensities (or propensities in much of the reaction network literature) l .x/ that determine the infinitesimal probabilities of seeing a particular change or transition X.t/ ! X.t C t/ D X.t/ C l in the next small interval of time .t; t C t, that is, P fX.t C t/ D X.t/ C l jX.t/g l .X.t// t:

M

900

Metabolic Networks, Modeling

In the chemical network setting, each type of tran2A ! C (4) sition corresponds to a reaction in the network, and l D l0  l , where l is a vector giving the number of would give an intensity molecules of each chemical species consumed in the ! lth reaction and l0 is a vector giving the number of  XA .t/ molecules of each species produced in the reaction. D XA .t/.XA .t/  1/ .X.t// D  2 2 The intuitive notion of a transition intensity can be translated into a rigorous specification of a model D kXA .t/.XA .t/  1/; in a number of different ways. The most popular approach in the chemical networks literature is through where we replace =2 by k. the master (or Kolmogorov forward) equation For unary reactions, for example, A ! C , the assumption is that the molecules behave independently ! X X and the intensity becomes .X.t// D kXA .t/. l .y  l /pyl .t/  l .y/ py .t/; pPy .t/ D l

l

Relationship to deterministic models. The larger the volume of the system the less likely a particular pair of molecules is to come close enough together to react, so it is natural to assume that intensities for binary reactions should vary inversely with respect to some measure of the volume. If we take that measure, N , to be Avogadro’s number times the volume in liters, then (2) the intensity for (3) becomes

(1) where py .t/ D P fX.t/ D yg, and the sum is over the different reactions in the network. Another useful approach is through a system of stochastic equations X.t/ D X.0/ C

X

Z

t

l Yl

l .X.s//ds ;

0

where the Yl are independent unit Poisson processes. Rt Note that Rl .t/ D Yl . 0 l .X.s//ds/ simply counts the number of times that the transition taking the state x to the state xCl occurs by time t, that is, the number of times the lt h reaction occurs. The master equation and the stochastic equation determine the same models in the sense that if X is a solution of the stochastic equation, py .t/ D P fX.t/ D yg is a solution of the master equation, and any solution of the master equation can be obtained in this way. See [4] for a survey of these models and additional references. The stochastic law of mass action. The basic assumption of the simplest Markov chain model is the same as that of the classical law of mass action: the system is thoroughly mixed at all times. That assumption suggests that the intensity for a binary reaction

.X.t// D

k XA .t/XB .t/ D N kŒAt ŒBt ; N

where ŒAt D XA .t/=N is the concentration of A measured in moles per liter. The intensity for (4) becomes .X.t// D kŒAt .ŒAt  N 1 / kŒA2t , assuming, as is likely, that N is large and that XA .t/ is of the same order of magnitude as N (which may not be the case for cellular reactions). If we assume that our system consists of the single reaction (3), the stochastic equation for species A, written in terms of the concentrations, becomes Z t 1 kŒAs ŒBs ds/ ŒAt D ŒA0  Y .N N 0 Z t

ŒA0  kŒAs ŒBs ds; 0

where, again assuming that N is large, the validity (3) of the approximation follows by the fact that the law of large numbers for the Poisson process implies should be proportional to the number of pairs con- N 1 Y .N u/ u. Analysis along these lines gives a sisting of one molecule of A and one molecule of B, derivation of the classical law of mass action starting that is, .X.t// D kXA .t/XB .t/. The same intuition from the stochastic model; see, for example, Kurtz applied to the binary reaction [25, 26], or Ethier and Kurtz [13], Chap. 10. ACB ! C

Metabolic Networks, Modeling

901

Simulation. Among the basic properties of a continuous-time Markov chain (with intensities that do not depend on time) is that the holding time in a state x is exponentially distributed and is independent of the value of the next state occupied by the chain. To be specific, the parameter of the holding time is N .x/ D

X

l .x/;

l

and the probability that the next state is x C l is N This observation immediately suggests a l .x/=.x/. simulation algorithm known in the chemical literature as Gillespie’s direct method or the stochastic simulation algorithm (SSA)[16, 17]. Specifically, given two independent uniform Œ0; 1 random variables U1 and U2 and noting that  log U1 is exponentially distributed with mean 1, the length of time the process remains in 1 . log U1 /. Assuming state x is simulated by D .x/ N that there are m reactions indexed by 1 Pl  m and N 1 l k .x/, defining 0 .x/ D 0 and l .x/ D .x/ kD1 the new state is given by xC

X

XO .n / D X.0/

l

l Yl

X

Z

l .X.s//ds ;

0

l2Rd

Xk .t/ D Xk .0/ C

t

l Yl

X l2Rc

Z

Z l

k 2 Sd ; t

l .X.s//ds 0

t

D Xk .0/ C

Fk .X.s//ds; 0

k 2 Sc ;

M

that is, the new state is x C l if l1 .x/ < U2  l .x/. If one simulates the process by simulating the Poisson processes Yl and solving the stochastic equation (2), one obtains the next reaction (next jump) method as defined by Gibson and Bruck [15]. If we define an Euler-type approximation for (2), that is, for 0 D 0 < 1 <    , recursively defining

C

Xk .t/ D Xk .0/ C

l 1.l1 .x/;l .x/ .U2 /;

l

X

numbers that need to be modeled as discrete variables and others species present in much larger numbers that would be natural to model as continuous variables. This observation leads to hybrid or piecewise deterministic models (in the sense of Davis [11]) as considered in the chemical literature by Crudu et al. [10], Haseltine and Rawlings [19], Hensel et al. [20], and Zeiser et al. [39]. We can obtain these models as solutions of systems of equations of the form

n1 X

! l .XO .k //.kC1  k / ;

kD0

we obtain Gillespie’s -leap method, which provides a useful approximation to the stochastic model in N situations where .x/ is large for values of the state x of interest [18]. See [3, 5] for additional analysis and discussion. Hybrid and multiscale models. A discrete model is essential if the chemical network consists of species present in small numbers, but a typical biochemical network may include some species present in small

where Rd and Sd are the indices of the reactions and the species that are modeled discretely, Rc and Sc are the indices for the reactions and species modeled P continuously, and Fk .x/ D l2Rc l l .x/. Models of this form are in a sense “multiscale” since the numbers of molecules in the system for the species modeled continuously are typically many orders of magnitude larger than the numbers of molecules for the species modeled discretely. Many of the stochastic models that have been considered in the biochemical literature are multiscale for another reason in that the rate constants vary over several orders of magnitude as well (see, e.g., [37, 38].) The multiscale nature of the species numbers and rate constants can be exploited to identify subnetworks that function naturally on different timescales and to obtain reduced models for each of the timescales. Motivated in part by Rao and Arkin [30] and Haseltine and Rawlings [19], a systematic approach to identifying the separated timescales and reduced models is developed in Refs. [6] and [23]. Acknowledgements The authors gratefully acknowledge the support of the National Science Foundation (USA) through grants DMS 08-05793 (TK), DMS-061670 (HFN, MR), and EF1038593 (HFN, MR).

902

References 1. Alon, U.: An Introduction to Systems Biology: Design Principles of Biological Circuits. CRC Press, Boca Raton (2006) 2. Anderson, D.: Global asymptotic stability for a class of nonlinear chemical equations. SIAM J. Appl. Math. 68(5), 1464–1476 (2008) 3. Anderson, D.F.: Incorporating postleap checks in tauleaping. J. Chem. Phys. 128(5), 054103 (2008). doi:10.1063/1.2819665. http://link.aip.org/link/?JCP/128/ 054103/1 4. Anderson, D.F., Kurtz, T.G.: Continuous time markov chain models for chemical reaction networks. In: Koeppl, H., Setti, G., di Bernardo, M., Densmore D. (eds.) Design and Analysis of Biomolecular Circuits. Springer, New York (2010) 5. Anderson, D.F., Ganguly, A., Kurtz, T.G.: Error analysis of tau-leap simulation methods. Ann. Appl. Prob. To appear (2010) 6. Ball, K., Kurtz, T.G., Popovic, L., Rempala, G.: Asymptotic analysis of multiscale approximations to reaction networks. Ann. Appl. Probab. 16(4), 1925–1961 (2006) 7. Bartholomay, A.F.: Stochastic models for chemical reactions. I. Theory of the unimolecular reaction process. Bull. Math. Biophys. 20, 175–190 (1958) 8. Bartholomay, A.F.: Stochastic models for chemical reactions. II. The unimolecular rate constant. Bull. Math. Biophys. 21, 363–373 (1959) 9. Chavez, M.: Observer design for a class of nonlinear systems, with applications to biochemical networks. Ph.D. thesis, Rutgers (2003) 10. Crudu, A., Debussche, A., Radulescu, O.: Hybrid stochastic simplifications for multiscale gene networks. BMC Syst. Biol. 3, 89 (2009). doi:10.1186/1752-0509-3-89 11. Davis, M.H.A.: Markov Models and Optimization. Monographs on Statistics and Applied Probability, vol. 49. Chapman & Hall, London (1993) 12. Delbr¨uck, M.: Statistical fluctuations in autocatalytic reactions. J. Chem. Phys. 8(1), 120–124 (1940). doi: 10.1063/1.1750549. http://link.aip.org/link/?JCP/8/120/1 13. Ethier, S.N., Kurtz, T.G.: Markov Processes: Characterization and Convergence. Wiley Series in Probability and Mathematical Statistics. Wiley, New York (1986) 14. Feinberg, M.: Chemical reaction network structure and the stability of complex isothermal reactors–i. the deficiency zero and deficiency one theorems. Chem. Eng. Sci. 42, 2229–2268 (1987) 15. Gibson, M.A., Bruck, J.: Efficient exact simulation of chemical systems with many species and many channels. J. Phys. Chem. A 104(9), 1876–1889 (2000) 16. Gillespie, D.T.: A general method for numerically simulating the stochastic time evolution of coupled chemical reactions. J. Comput. Phys. 22(4), 403–434 (1976) 17. Gillespie, D.T.: Exact stochastic simulation of coupled chemical reactions. J. Phys. Chem. 81, 2340–2361 (1977) 18. Gillespie, D.T.: Approximate accelerated stochastic simulation of chemically reacting systems. J. Chem. Phys. 115(4), 1716–1733 (2001). doi:10.1063/1.1378322. http://link.aip. org/link/?JCP/115/1716/1

Metabolic Networks, Modeling 19. Haseltine, E.L., Rawlings, J.B.: Approximate simulation of coupled fast and slow reactions for stochastic chemical kinetics. J. Chem. Phys. 117(15), 6959–6969 (2002) 20. Hensel, S.C., Rawlings, J.B., Yin, J.: Stochastic kinetic modeling of vesicular stomatitis virus intracellular growth. Bull. Math. Biol. 71(7), 1671–1692 (2009). doi:10.1007/s11538-009-9419-5 http://dx.doi.org.ezproxy. library.wisc.edu/10.1007/s11538-009-9419-5 21. Kacser, H., Burns, J.A.: The control of flux. Symp. Soc. Exp. Biol. 27, 65–104 (1973) 22. Kacser, H., Burns, J.A.: The control of flux. Biochem. Soc. Trans. 23, 341–366 (1995) 23. Kang, H.W., Kurtz, T.G.: Separation of time-scales and model reduction for stochastic reaction networks. Ann. Appl. Prob. To appear (2010) 24. Keener, J., Sneyd, J.: Mathematical Physiology. Springer, New York (2009) 25. Kurtz, T.G.: The relationship between stochastic and deterministic models for chemical reactions. J. Chem. Phys. 57(7), 2976–2978 (1972) 26. Kurtz, T.G.: Approximation of Population Processes. CBMS-NSF Regional Conference Series in Applied Mathematics, vol. 36. Society for Industrial and Applied Mathematics (SIAM), Philadelphia (1981) 27. McQuarrie, D.A.: Stochastic approach to chemical kinetics. J. Appl. Probab. 4, 413–478 (1967) 28. Nijhout, H.F., Reed, M., Anderson, D., Mattingly, J., James, S., Ulrich, C.: Long-range allosteric interactions between the folate and methionine cycles stabilize dna methylation. Epigenetics 1, 81–87 (2006) 29. Pepin, J.A., Price, N.D., Wiback, S.J., Fell, D.A., Palsson, B.O.: Metabolic pathways in the post-genome era. Trends Biochem. Sci. 28, 250–258 (2003) 30. Rao, C.V., Arkin, A.P.: Stochastic chemical kinetics and the quasi-steady-state assumption: application to the gillespie algorithm. J. Chem. Phys. 118(11), 4999–5010 (2003) 31. Reinitz, J., Sharp, D.H.: Mechanism of even stripe formation. Mech. Dev. 49, 133–158 (1995) 32. Savageau, M.A.: Biochemical systems analysis: I. some mathematical properties of the rate law for the component enzymatic reactions. J. Theor. Biol. 25(3), 365–369 (1969) 33. Segal, L.E.: On the validity of the steady state assumption of enzyme kinetics. Bull. Math. Biol. 50, 579–593 (1988) 34. Sharp, D.H., Reinitz, J.: Prediction of mutant expression patterns using gene circuits. Biosystems 47, 79–90 (1998) 35. Shiu, A., Sturmfels, B.: Siphons in chemical reaction networks. Bull. Math. Biol. 72(6), 1448–1463 (2010) 36. Sigal, A., Milo, R., Cohen, A., Geva-Zatorsky, N., Klein, Y., Liron, Y., Rosenfeld, N., Damon, T., Perzov, N., Alon, U.: Variability and memory of protein levels in human cells. Nature 444, 643–646 (2006) 37. Srivastava, R., Peterson, M.S., Bentley, W.E.: Stochastic kinetic analysis of Escherichia coli stress circuit using sigma(32)-targeted antisense. Biotechnol. Bioeng. 75, 120– 129 (2001) 38. Srivastava, R., You, L., Summers, J., Yin, J.: Stochastic vs. deterministic modeling of intracellular viral kinetics. J. Theor. Biol. 218(3), 309–321 (2002)

Methods for High-Dimensional Parametric and Stochastic Elliptic PDEs 39. Zeiser, S., Franz, U., Liebscher, V.: Autocatalytic genetic networks modeled by piecewise-deterministic Markov processes. J. Math. Biol. 60(2), 207–246 (2010). doi:10.1007/s00285-009-0264-9. http://dx.doi.org/10.1007/ s00285-009-0264-9

Methods for High-Dimensional Parametric and Stochastic Elliptic PDEs Christoph Schwab Seminar for Applied Mathematics (SAM), ETH Z¨urich, ETH Zentrum, Z¨urich, Switzerland

Synonyms Generalized Polynomial Chaos; Partial Differential Equations with Random Input Data; Sparse Finite Element Methods; Sparsity; Uncertainty Quantification

Motivation and Outline A large number of stationary phenomena in the sciences and in engineering are modeled by elliptic partial differential equations (elliptic PDEs), and during the past decades, numerical methods for their solution such as the finite element method (FEM), the finite difference method (FDM), and the finite volume method (FVM) have matured. Well-posed mathematical problem formulations as well as the mathematical analysis of the numerical methods for their approximate solution were based on the paradigm (going back to Hadamard’s notion of well-posedness) that all input data of interest are known exactly and that numerical methods should be convergent, i.e., able to approximate the unique solution at any required tolerance (neglecting effects of finite precision arithmetic). In recent years, due to apparent limited predictive capability of high-precision numerical simulations, and in part due to limited measurement precision of PDE input data in applications,

Research supported by the European Research Council (ERC) under Grant No. AdG 247277

903

the numerical solution of PDEs with random inputs has emerged as key area of applied and computational mathematics with the aim to quantify uncertainty in predictive engineering computer simulations. At the same time, in many application areas, the data deluge has become reality, due to the rapid advances in digital data acquisition such as digital imaging. The question how to best computationally propagate uncertainty, e.g., from digital data, from multiple observations and statistical information on measurement errors through engineering simulations mandate, once more, the (re)formulation of elliptic PDEs of engineering interest as stochastic elliptic PDEs, for which all input data can be random functions in suitable function spaces. Often (but not always), the function spaces will be the spaces in which the deterministic counterparts of the PDEs of interest admit unique solutions. In the formulation of stochastic elliptic (and other) PDEs, one distinguishes two broad classes of random inputs (or “noises”): first, so-called colored noise, where the random inputs have realizations in classical function spaces and the statistical moments are bounded and exhibit, as functions of the spatial variable, sufficient smoothness to allow for the classical differential calculus (in the sense of distributions), and second, so-called white, or more generally, rough noises. One characteristic of white noise being the absence of spatial or temporal correlation, solutions of PDEs with white noise inputs can be seen, in a sense, as stochastic analogues to fundamental solutions (in the sense of distributions) of deterministic PDEs: as in the deterministic setting, they are characterized by rather low regularity and by low integrability. As in the deterministic setting, classical differential calculus does not apply any more. Extensions such as Ito Calculus (e.g., [25, 73]), white noise calculus (e.g., [57, 64]), or rough path calculus (e.g., [35]) are required in the mathematical formulations of PDE for such inputs. In the present notes, we concentrate on efficient numerical treatment of colored noise models. For stochastic PDEs (by which we mean partial differential equations with colored random inputs, which we term “SPDEs” in what follows), well-posedness can be established when inputs and outputs of SPDEs are viewed as random fields.

M

904

Methods for High-Dimensional Parametric and Stochastic Elliptic PDEs

(or random function, RF for short) is an E-valued random variable (RV for short), i.e., a measurable mapping from .; A; P/ to .E; E/. We denote by L.X / the image measure of the probability measure P under the mapping X on the measurable space .E; E/, defined for any A 2 E by L.X /.A/ D P .! 2  W X.! 2 A/. The measure  D L.X / is the distribution or law of the RV X . If E is a separable Banach space, and X is a RV on .; A/ taking values in E, then the real valued function kX./kE is measurable (i.e., a random number) (e.g., [25, Lemma 1.5]). For 1  p  1, and for a separable Banach space E, denote by Lp .; A; PI E/ the Bochner space of all RF X W  7! E which are p-integrable, i.e., for which the norms

Random Fields in Stochastic PDEs The formulation of elliptic SPDEs with stochastic input data (such as stochastic coefficient functions, stochastic source or volume terms, or stochastic domains) necessitates random variables which take values in function spaces. Some basic notions are as follows (see, e.g., [25, Chap 1] or [2] for more details). Consider “stochastic” PDE within the probabilistic framework due to Kolmogoroff (other approaches toward randomness are fuzzy sets, belief functions, etc.): let .; A; P/ denote probability space, and let E be a metric space, endowed with a sigma algebra E. A strongly measurable mapping X W  7! E is a mapping such that for every A 2 A, the set f! 2  W X.!/ 2 Ag 2 A. A random field

kX kLp .;A;PIE/ WD

8 Z < :

1=p p

!2

kX.!/kE d P.!/

esssup!2 kX.!/kE < 1 ;

We also write Lp .I E/ if the probability space is clear from the context. If X 2 L1 .I E/ is a RF, the mathematical expectation EŒX  (also referred to as “mean field” or “ensemble average” of X ) is well defined as an element of E by Z X.!/d P.!/ 2 E ; (2) EŒX  WD

0 there exists a constant Cs > 0 and a convergence rate t.s/ > 0 such that for all ` 2 N holds the error estimate 8u 2 Vt W

inf ku  v` kV  CN`t kukVs :

v2S`

(5)

In the context of the Dirichlet problem of the Poisson equation with random source term (cf., e.g., [80, 80]) in a bounded, Lipschitz domain D  Rd , for example, we think of V D H01 .D/ and of Vs D .H 1Cs \ H01 /.D/. Then, for S` denoting the space of continuous, piecewise polynomial functions of degree k  1

EM Œu`  WD

M 1 X u` .!i / : M i D1

(6)

There are two contributions to the error EŒu  EM Œu` : a sampling error and of a discretization error (see [83]): assuming that u 2 L2 .I Vs /, for M D 1; 2; : : : ; ` D 1; 2; : : : holds the error bound kEŒu  EM Œu` kL2 .IV /  kEŒu  EM ŒukL2 .IV / C kEM Œu  EM Œu` kL2 .IV / 1  p kukL2 .IV / C Cs N`t kukL2 .IVs / : M (7) Relation (7) gives an indication on the selection of the number of degrees of freedom N` in the discretization scheme versus the sample size M in order to balance both errors: to reach error O."/ in L2 .I V /, work of order O.MN` / D O."21=t / D O."2d=s / is required (assuming work for the realization of one sample u` 2 S` being proportional to dimS` , as is typically the case when multilevel solvers are used for the approximate solution of the discretized equations). We note that, even for smooth solutions (where s is large), the convergence rate of error versus work never exceeds 1=2. Methods which allow to achieve higher rates of convergence are so-called quasi-Monte Carlo methods (QMC methods for short) (see, e.g., [61, 62] for recent results). The naive use of MC methods for the numerical solution of SPDEs is, therefore, limited either to small model problems or requires the use of massive computational resources. The so-called multilevel Monte Carlo (MLMC for short), proposed by M. Giles in [39] (after earlier work of Heinrich on quadrature) to accelerate path simulation of Ito SDEs, can dramatically improve the situation. These methods are also effective in particular for RF solutions u with only low regularity, i.e., u 2 Vs for small s > 0, whenever hierarchic discretizations of

M

906

Methods for High-Dimensional Parametric and Stochastic Elliptic PDEs

stochastic PDEs are available. To derive it, we assume this is naturally the case for multigrid methods). By that for each draw u.!i / of the RF u, a sequence the linearity of the mathematical expectation, with the u` .!i / of approximate realizations is available (e.g., convention that u1 WD 0, we may write

" EŒu  uL  D E u 

L X

# .u`  u`1 / D EŒu 

`D0

L X

EŒu`  u`1  :

`D0

(8) L Rather than applying the MC estimator now to uL , X L Œu WD EM` Œu`  u`1  : (9) E we estimate separately the expectation EŒu`  u`1  `D0 of each discretization increment, amounting to the numerical solution of the same realization of the SPDE on two successive mesh levels as is naturally available Efficiency gains in MLMC stem from the possibility to in multilevel solvers for elliptic PDEs. This results in use, at each level of discretization, a level-dependent number M` of MC samples: combining (4) and (5), the MLMC estimator

kEM` Œu`  u`1 kL2 .IV /  kEŒuEM` Œu` kL2 .IV / CkEŒuEM` Œu`1 kL2 .IV / 1=2

. M`

This error bound may now be used to optimize the sample numbers M` with respect to the discretization errors at each mesh level. In this way, computable estimates for the ensemble average EŒu of the RF u can be obtained with work of the order of one multilevel solve of one single instance of the deterministic problem at the finest mesh level (see, e.g., [12] for a complete analysis for a scalar, second-order elliptic problem with random diffusion coefficients and [10, 11, 65] for other types of SPDEs, and [41] for subsurface flow models with lognormal permeability). For quasi-MC methods, similar constructions are possible; this is an ongoing research (see, e.g., [47, 61, 62]).

t .s/

N`

kukL2 .IVs / :

Au D f .!/ ;

f 2 L2 .I V  / :

(10)

This equation admits a unique solution, a RF u 2 L2 .I V /. Since the operator equation (10) is linear, application of the operator A and tensorization commute, and there holds the deterministic equation for the covariance function M.2/ Œu WD EŒu ˝ u 2 V ˝ V : .A ˝ A/M.2/ Œu D M.2/ Œf  :

(11)

Equation (11) is exact, i.e., no statistical moment closures, e.g., in randomly forced turbulence, are necessary. For the Poisson equation, this approach was first proposed in [5]. If A is V -coercive, the operator A˝A is not elliptic in the classical sense but boundedly invertible in scales of tensorized spaces and naturally admits regularity shifts in the tensorized smoothness Moment Approximation by Sparse Tensor Galerkin Finite Element Methods scale Vs ˝ Vs . This allows for deterministic Galerkin approximation of 2- and of k-point correlation funcFor a boundedly invertible, deterministic operator A 2 tions in log-linear complexity w.r. to the number of L.V; V  / on some separable Hilbert space V over R, degrees of freedom used for the solution of one realizaconsider operator equation with random loading tion of the mean-field problem (e.g., [51,76,78,80,83]).

Methods for High-Dimensional Parametric and Stochastic Elliptic PDEs

The non-elliptic nature of A ˝ A implies, however, possible enlargement of the singular support of the RF’s k-point correlation functions; see, e.g., [71, 72] for a simple example and an hp-error analysis with exponential convergence estimates and [31] and the references there for more general regularity results for elliptic pseudodifferential equations with random input data, covering in particular also strongly elliptic boundary integral equations. For nonlinear problems with random inputs, deterministic tensor equations such as (11) for k-point correlation functions of random solutions do not hold, unless some closure hypothesis is imposed. In the case of an a priori assumption on P-a.s. smallness of solution fluctuations about mean, deterministic tensor equations like (11) for the second moments of the random solution can be derived from a first-order moment closure; we refer to [28] for an early development of this approach in the context subsurface flow; to [52] for an application to elliptic problems in random domains, where the perturbation analysis is based on the shape derivative of the solution at the nominal domain; and to [18] for a general formulation and for error estimates of both discretization and closure error. For an analysis of the linearization approach of random elliptic PDEs in uncertainty quantification, we refer to [7]. We emphasize that the Galerkin solution of the tensorized perturbation equations entails cost O.NLk / where NL denotes the number of degrees of freedom necessary for the discretization of the nominal problem and k  1 denotes the order of the statistical moment of interest. Using sparse tensor Galerkin discretizations as in [76, 80, 80], this can be reduced to O.NL .log NL /k / work and memory, rendering this approach widely applicable.

Generalized Polynomial Chaos Representations Generalized polynomial chaos (“gpc” for short) representations aim at a parametric, deterministic representation of the law L.u/ of a random solution u of a SPDE. For PDEs with RF inputs, they are usually infinite-dimensional, deterministic parametrizations, in the sense that a countable number of variables are required. Representations of this type go back to the spectral representation of random fields introduced by N. Wiener [84]. A general representation theorem for

907

RFs in L2 .I H / was obtained in [16], for Gaussian RFs over separable Hilbert spaces H . This result shows that the classical Wiener-Hermite polynomial chaos is, in a sense, universal. Representations in terms of chaos expansions built from polynomials which are orthogonal with respect to non-Gaussian probability measures were proposed in [86]; these so-called generalized polynomial chaos expansions often allow finitely truncated approximations with smaller errors for strongly non-Gaussian RF finite second moments. Special cases of polynomial chaos expansions are the so-called Karhunen-Lo`eve expansions. These can be consider as a particular instance of so-called principal component approximations. Karhunen-Lo`eve expansions allow to parametrize a RF a 2 L2 .I H / taking values in a separable Hilbert space H in terms of countably many eigenfunctions f'i gi 1 of its covariance operator Ca W H 7! H : the unique compact, self-adjoint nuclear, and trace-class operator whose kernel is the two-point correlation of the RF a, i.e., a D EŒa ˝ a; see, e.g., [82] or, for the Gaussian case, [2, Thm. 3.3.3]. Importantly, the enumeration of eigenpairs .k ; 'k /k1 of Ca is such that 1  2  : : : ! 0 so that the Karhunen-Lo`eve eigenfunctions constitute principal components of the RF a, ordered according to decreasing importance (measured in terms of their contribution to the variance of the RF a): a.; !/ D a./ N C

Xp k Yk .!/'k ./ :

(12)

k1

In (12), the normalization of the RVs Yk and of the 'k still needs to be specified: assuming that the 'k are H -orthonormal, i.e., .'i ; 'j / D ıij , the RVs Yk 2 L2 .I R/ defined in (12) are given by (.; / denoting the H inner product): 1=2

Yk .!/ D k

.a.; !/  a./; N 'k / ;

k D 1; 2; : : : (13)

The sequence fYk gk1 constitutes a sequence of pairwise uncorrelated RVs which, in case a is a Gaussian RF over H , are independent. Recently, for scalar, elliptic problems in divergence form with lognormal Gaussian RF permeability typically appearing in subsurface flow models (see, e.g., [66, 67] and the references there), several rigorous mathematical formulations were given in [17,36,43]. It was shown that the stochastic diffusion problem admits

M

908

Methods for High-Dimensional Parametric and Stochastic Elliptic PDEs

a unique RF solution which belongs to Lp .;  I V / where  is a Gaussian measure on V (see, e.g., [15]). In particular, in [17,34,61], dimension truncation error analyses were performed. Here, two broad classes of discretization approaches are distinguished: stochastic collocation (SC) and stochastic Galerkin (SG). SC is algorithmically similar to MC sampling in that only instances of the deterministic PDEs need to be solved. We refer to [6, 8, 69, 70]. Recently, adaptive stochastic collocation algorithms for the full, infinite-dimensional problem were developed in [19]. For solutions with low regularity with respect to the stochastic variable, also quasi-Monte Carlo (“QMC” for short) is effective; we refer to [62] for a general introduction to the mathematical foundation of QMC quadrature as applied to infinite-dimensional parametric operator equations and to [29, 30, 47, 48, 63] for recent applications of QMC to elliptic SPDEs. Numerical optimal control of stochastic elliptic (and parabolic) PDEs with countably parametric operators has been investigated in [42, 60]. Regularity and efficient numerical methods for stochastic elliptic multiscale problems were addressed in the papers [1, 54]; there, multilevel Monte Carlo and generalized polynomial chaos approximations were proposed, and convergence rates independent of the scale parameters were established under the (natural, for this class of problems) assumption of a multiscale discretization in physical space. Bayesian inverse problems for stochastic, elliptic PDEs have also been addressed from the point of view of sparsity of forward maps. We refer to [79] and the references there for the formulation and the basic sparsity result for parametric diffusion problems. The result and approach was generalized to large classes of countably parametric operator equations which cover random elliptic and parabolic PDEs in [4, 37, 74, 75, 77]. Parametric Algebraic Equations The Karhunen-Lo`eve expansion (12) can be viewed as a parametrization of the RF a.; !/ 2 L2 .I H / in terms of the sequence Y D .Yk .!//k1 of R-valued RVs Yk .!/. Assuming that the RVs Yk are independent, (12) could be interpreted as parametric, deterministic function of infinitely many parameters y D .yk /k1 , a.; y/ D a./ N C

Xp k yk './ k1

(14)

which is evaluated at Y D .Yk .!//k1 . We illustrate this in the most simple setting: given real numbers f; 2 R and a parametric function a.y/ D 1 C y where y 2 U WD Œ1; 1, we wish to find the function U 3 y 7! u.y/ such that a.y/u.y/ D f ;

for y 2 U :

(15)

Evidently, u.y/ D f =a.y/ provided that a.y/ ¤ 0 for all y 2 U which is easily seen to be ensured by a smallness condition on : if j j   < 1, then a.y/  1   > 0 for every y 2 U and (15) admits the unique solution u.y/ D f =.1 C y / which depends analytically on the parameter y 2 U . Consider next the case where the coefficient a.y/ depends on a sequence y D .y1 ; y2 ; : : :/ D .yj /j 1 of parameters yj , for which we assume once more that jyj j  1 for all j 2 N or, symbolically, that P y 2 U D Œ1; 1N . Then a.y/ D 1 C j 1 yj j and a minimal condition to render a.y/ well defined is that the infinite series converges, which is ensured by the summability condition D . j /j 1 2 `1 .N/. Note that this condition is always satisfied if there are only finitely many parameters y1 ; : : : ; yJ for some J < 1 which corresponds to the case that j D 0 for all j > J . Once again, in order to solve (15) for the function u.y/ (which now depends on infinitely many variables y1 ; y2 ; : : :), a smallness condition is required:  WD k k`1 .N/ D

X

j

jj

0, and for every y 2 U , (18) admits also called nonintrusive. a unique solution u.; y/ 2 L2 .D/, which is given by An alternative approach to SC is stochastic Galerkin u.; y/ D .a.; y//1 f . (SG for short) discretizations, which are based on The element bj D k j kL1 .D/ quantifies the sensi- mean square projection (w.r. to P) of the parametric tivity of the “input” a.; y/ with respect to coordinate solution u.x; y/ of (21) onto finite spans of tensorized yj : there holds generalized polynomial chaos (gpc) expansions on U . Recent references for mathematical formulations  b and convergence analysis of SG methods are [6, 27, kf kL2 .D/ ; where sup k@y u.; y/kL2 .D/  34, 40, 44, 45, 68]. Efficient implementations, includ1 y2U ing a posteriori error estimates and multi-adaptive Y j bj D b11 b22 : : : ;  2 F : (20) AFEM, are addressed in [32]. SG-based methods feab  WD j 1 ture the significant advantage of Galerkin orthogonality in L2 .I V / of the gpc approximation, which N Here  D .1 ; 2 ; : : :/ 2 F  N0 , the set of implies the perspective of adaptive discretization of sequences of nonnegative integers which are “finitely random fields. Due to the infinite dimension of the supported,” i.e., which have only finitely many nonzero parameter space U , these methods differ in essential terms j ; due to bj0 D 1, the infinite product in (20) is respects from the more widely known adaptive FEM: meaningful for  2 F . an essential point is sparsity in the gpc expansion of the parametric solution u.x; y/ of (21). In [22, 23], a fundamental sparsity observation has been made Parametric Elliptic PDEs Efficient methods for parametric, elliptic PDEs with for equations like (21): sparsity in the random in(infinite-dimensional) parametric coefficients of the puts’ parametric expansion implies the same sparsity form (14), (17), such as the scalar model elliptic in the gpc representation of the parametric solution u.x; y/. equation

M

910

Methods for High-Dimensional Parametric and Stochastic Elliptic PDEs

The results in [22, 23] are not limited to (21) but hold for rather large classes of elliptic (as well as parabolic) SPDEs (see [20, 24, 49, 55, 56]), implying with the results in [19, 45, 46] quasi best N -term, dimension-independent convergence rates of SC and SG algorithms. Dimension-independent approximation rates for large, nonlinear systems of random initial value ODEs were proved in [49] and computationally investigated in [50]. For implementational and mathematical aspects of adaptive stochastic Galerkin FEM with computable, guaranteed upper error bounds and applications to engineering problems, we refer to [32, 33].

5. 6.

7.

8.

9.

Further Results and New Directions For further indications, in particular on the efficient algorithmic realization of collocation approaches for the parametric, deterministic equation, we refer to [38,85]. Numerical solution of SPDEs based on sparse, infinitedimensional, parametric representation of the random solutions also allows the efficient numerical treatment of Bayesian inverse problems in the non-Gaussian setting. We refer to [74, 79] and the references there. For the use of various classes of random elliptic PDEs in computational uncertainty quantification, we refer to [53]. The fully discretized, parametric SPDEs (21) can be viewed as high-dimensional, multi-linear algebra problems; here, efficient discretizations which directly compress matrices arising in the solution process of SGFEM are currently emerging (we refer to [59, 76] and the references there for further details). For an SC approach to eigenvalue problems for (21) (and more general problems), we refer to [3].

10.

11.

12.

13.

14.

15.

References 1. Abdulle, A., Barth, A., Schwab, C.: Multilevel Monte Carlo methods for stochastic elliptic multiscale PDEs. SIAM J. Multiscale Methods Simul. 11(4), 1033–1070 (2013). doi:http://dx.doi.org/10.1137/120894725 2. Adler, R.J.: The Geometry of Random Fields. Wiley Series in Probability and Mathematical Statistics. Wiley, Chichester (1981) 3. Andreev, R., Schwab, C.: Sparse Tensor Approximation of Parametric Eigenvalue Problems. Lecture Notes in Computational Science and Engineering, vol. 83, pp. 203–241. Springer, Berlin (2012) 4. Arnst, M., Ghanem, R., Soize, C.: Identification of Bayesian posteriors for coefficients for chaos expansions. J. Comput. Phys. 229(9), 3134–3154 (2010).

16.

17.

18.

19.

doi:10.1016/j.jcp.2009.12.033, http://dx.doi.org/10.1016/j. jcp.2009.12.033 Babuˇska, I.: On randomised solutions of Laplace’s equation. ˇ Casopis Pˇest Mat. 86, 269–276 (1961) Babuˇska, I., Tempone, R., Zouraris, G.E.: Galerkin finite element approximations of stochastic elliptic partial differential equations. SIAM J. Numer. Anal. 42(2), 800–825 (2004) (electronic). doi:10.1137/S0036142902418680 Babuˇska, I., Nobile, F., Tempone, R.: Worst case scenario analysis for elliptic problems with uncertainty. Numer. Math. 101(2), 185–219 (2005). doi:10.1007/s00211-005-0601-x, http://dx.doi.org/10.1007/ s00211-005-0601-x Babuˇska, I., Nobile, F., Tempone, R.: A stochastic collocation method for elliptic partial differential equations with random input data. SIAM J. Numer. Anal. 45(3), 1005–1034 (2007) (electronic) Bacuta, C., Nistor, V., Zikatanov, L.: Improving the rate of convergence of high-order finite elements on polyhedra. I. A priori estimates. Numer. Funct. Anal. Optim. 26(6), 613–639 (2005) Barth, A., Lang, A.: Simulation of stochastic partial differential equations using finite element methods. Stochastics 84(2–3), 217–231 (2012). doi:10.1080/17442508.2010.523466, http://dx.doi.org/ 10.1080/17442508.2010.523466 Barth, A., Lang, A., Schwab, C.: Multi-level Monte Carlo finite element method for parabolic stochastic partial differential equations. Technical report 2011/30, Seminar for Applied Mathematics, ETH Z¨urich (2011) Barth, A., Schwab, C., Zollinger, N.: Multi-level Monte Carlo finite element method for elliptic PDEs with stochastic coefficients. Numer. Math. 119(1), 123–161 (2011). doi:10.1007/s00211-011-0377-0, http://dx.doi.org/10.1007/ s00211-011-0377-0 Bieri, M.: A sparse composite collocation finite element method for elliptic SPDEs. SIAM J. Numer. Anal. 49(6), 2277–2301 (2011). doi:10.1137/090750743, http://dx.doi. org/10.1137/090750743 Bieri, M., Schwab, C.: Sparse high order FEM for elliptic sPDEs. Comput. Methods Appl. Mech. Eng. 198(37–40), 1149–1170 (2009) Bogachev, V.I.: Gaussian Measures. Mathematical Surveys and Monographs, vol. 62. American Mathematical Society, Providence (1998) Cameron, R.H., Martin, W.T.: The orthogonal development of non-linear functionals in series of Fourier-Hermite functionals. Ann. Math. (2) 48, 385–392 (1947) Charrier, J.: Strong and weak error estimates for elliptic partial differential equations with random coefficients. SIAM J. Numer. Anal. 50(1), 216–246. doi:10.1137/100800531, http://dx.doi.org/10.1137/100800531 Chernov, A., Schwab, C.: First order k-th moment finite element analysis of nonlinear operator equations with stochastic data. Math. Comput. 82, 1859–1888 (2013). doi:http:// dx.doi.org/10.1090/S0025-5718-2013-02692-0 Chkifa, A., Cohen, A., DeVore, R., Schwab, C.: Adaptive algorithms for sparse polynomial approximation of parametric and stochastic elliptic PDEs. M2AN Math. Model. Numer. Anal. 47(1), 253–280 (2013). doi:http://dx.doi.org/ 10.1051/m2an/2012027

Methods for High-Dimensional Parametric and Stochastic Elliptic PDEs 20. Chkifa, A., Cohen, A., Schwab, C.: High-dimensional adaptive sparse polynomial interpolation and applications to parametric PDEs. J. Found. Comput. Math. 14(4), 601–633 (2013) 21. Ciarlet, P.: The Finite Element Method for Elliptic Problems. Studies in Mathematics and Its Applications, vol. 4. North-Holland Publishing, Amsterdam/New York (1978) 22. Cohen, A., DeVore, R., Schwab, C.: Convergence rates of best n-term Galerkin approximations for a class of elliptic sPDEs. J. Found. Comput. Math. 10(6), 615–646 (2010) 23. Cohen, A., Devore, R., Schwab, C.: Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDE’s. Anal. Appl. (Singap) 9(1), 11– 47 (2011). doi:10.1142/S0219530511001728, http://dx.doi. org/10.1142/S0219530511001728 24. Cohen, A., Chkifa, A., Schwab, C.: Breaking the curse of dimensionality in sparse polynomial approximation of parametric PDEs. J. Math. Pures et Appl. (2014). doi:http:// dx.doi.org/10.1016/j.matpur.2014.04.009 25. Da Prato, G., Zabczyk, J.: Stochastic Equations in Infinite Dimensions. Encyclopedia of Mathematics and Its Applications, vol. 44. Cambridge University Press, Cambridge (1992) 26. Davis, P.J.: Interpolation and Approximation. Dover Publications, New York (1975) (republication, with minor corrections, of the 1963 original, with a new preface and bibliography) 27. Deb, M.K., Babuˇska, I.M., Oden, J.T.: Solution of stochastic partial differential equations using Galerkin finite element techniques. Comput. Methods Appl. Mech. Eng. 190(48), 6359–6372 (2001). doi:10.1016/S0045-7825(01)00237-7, http://dx.doi.org/10.1016/S0045-7825(01)00237-7 28. Dettinger, M., Wilson, J.L.: First order analysis of uncertainty in numerical models of groundwater flow part 1. Mathematical development. Water Resour. Res. 17(1), 149– 161 (1981) 29. Dick, J., Kuo, F.Y., LeGia, Q.T., Nuyens, D., Schwab, C.: Higher order QMC Petrov-Galerkin discretization for affine parametric operator equations with random field inputs. SIAM J. Numer. Anal. 52(6), 2676–2702 (2014). doi:http:// dx.doi.org/10.1137/130943984 30. Dick, J., Kuo, F.Y., LeGia, Q.T., Schwab, C.: Multi-level higher order QMC Galerkin discretization for affine parametric operator equations. Technical report 2014-14, Seminar for Applied Mathematics, ETH Z¨urich (2014) 31. D¨olz, J., Harbrecht, H., Schwab, C.: Covariance regularity and H -matrix approximation for rough random fields. Technical report 2014-19, Seminar for Applied Mathematics, ETH Z¨urich (2014) 32. Eigel, M., Gittelson, C., Schwab, C., Zander, E.: Adaptive stochastic Galerkin FEM. Comput. Methods Appl. Mech. Eng. 270, 247–269 (2014). doi:http://dx.doi.org/10.1016/j. cma.2013.11.015 33. Eigel, M., Gittelson, C., Schwab, C., Zander, E.: A convergent adaptive stochastic Galerkin finite element method with quasi-optimal spatial meshes. Technical report 201401, Seminar for Applied Mathematics, ETH Z¨urich (2014) 34. Frauenfelder, P., Schwab, C., Todor, R.A.: Finite elements for elliptic problems with stochastic coefficients. Comput. Methods Appl. Mech. Eng. 194(2–5), 205–228 (2005)

911

35. Friz, P.K., Victoir, N.B.: Multidimensional Stochastic Processes as Rough Paths: Theory and Applications. Cambridge Studies in Advanced Mathematics, vol. 120. Cambridge University Press, Cambridge (2010) 36. Galvis, J., Sarkis, M.: Approximating infinity-dimensional stochastic Darcy’s equations without uniform ellipticity. SIAM J. Numer. Anal. 47(5), 3624–3651 (2009). doi:10.1137/080717924, http://dx.doi.org/10.1137/ 080717924 37. Gantner, R.N., Schillings, C., Schwab, C.: Binned multilevel Monte Carlo for Bayesian inverse problems with large data. To appear in Proc. 22nd Int. Conf. on Domain Decomposition (2015) 38. Ghanem, R.G., Spanos, P.D.: Stochastic Finite Elements: A Spectral Approach. Springer, New York (1991). doi:10.1007/978-1-4612-3094-6, http://dx.doi.org/10.1007/ 978-1-4612-3094-6 39. Giles, M.B.: Multilevel Monte Carlo path simulation. Oper. Res. 56(3), 607–617 (2008) 40. Gittelson, C.: An adaptive stochastic Galerkin method for random elliptic operators. Math. Comput. 82(283), 1515– 1541 (2013) 41. Gittelson, C., K¨onn¨o, J., Schwab, C., Stenberg, R.: The multi-level Monte Carlo finite element method for a stochastic Brinkman problem. Numer. Math. 125(2), 347–386 (2013). doi:http://dx.doi.org/10.1007/s00211-013-0537-5 42. Gittelson, C., Andreev, R., Schwab, C.: Optimality of adaptive Galerkin methods for random parabolic partial differential equations. J. Comput. Appl. Math. 263, 189–201 (2014). doi:http://dx.doi.org/10.1016/j.cam.2013.12.031 43. Gittelson, C.J.: Stochastic Galerkin discretization of the log-normal isotropic diffusion problem. Math. Models Methods Appl. Sci. 20(2), 237–263 (2010). doi:10.1142/S0218202510004210 44. Gittelson, C.J.: Adaptive Galerkin methods for parametric and stochastic operator equations. PhD thesis, ETH Z¨urich (2011). doi:10.3929/ethz-a-006380316, http://dx. doi.org/10.3929/ethz-a-006380316 45. Gittelson, C.J.: Adaptive stochastic Galerkin methods: beyond the elliptic case. Technical report 2011/12, Seminar for Applied Mathematics, ETH Z¨urich (2011) 46. Gittelson, C.J.: Stochastic Galerkin approximation of operator equations with infinite dimensional noise. Technical report 2011/10, Seminar for Applied Mathematics, ETH Z¨urich (2011) 47. Graham, I.G., Kuo, F.Y., Nuyens, D., Scheichl, R., Sloan, I.H.: Quasi-Monte Carlo methods for elliptic PDEs with random coefficients and applications. J. Comput. Phys. 230(10), 3668–3694 (2011) 48. Graham, I.G., Kuo, F.Y., Nichols, J.A., Scheichl, R., Schwab, C., Sloan, I.H.: Quasi-Monte Carlo finite element methods for elliptic PDEs with lognormal random coefficients. Numer. Math. 128(4) (2014). doi:http://dx.doi.org/ 10.1007/s00211-014-0689-y 49. Hansen, M., Schwab, C.: Sparse adaptive approximation of high dimensional parametric initial value problems. Vietnam J. Math. 41(2), 181–215 (2013). doi:http://dx.doi.org/ 10.1007/s10013-013-0011-9 50. Hansen, M., Schillings, C., Schwab, C.: Sparse approximation algorithms for high dimensional parametric initial

M

912

51.

52.

53.

54.

55.

56.

57.

58.

59.

60.

61.

62.

63.

64.

Methods for High-Dimensional Parametric and Stochastic Elliptic PDEs value problems. In: Proceedings of the Fifth International Conference on High Performance Scientific Computing 2012, Hanoi (2014). doi:http://dx.doi.org/10.1007/978-3319-09063-4 1 Harbrecht, H.: A finite element method for elliptic problems with stochastic input data. Appl. Numer. Math. 60(3), 227–244 (2010). doi:10.1016/j.apnum.2009.12.002, http:// dx.doi.org/10.1016/j.apnum.2009.12.002 Harbrecht, H., Schneider, R., Schwab, C.: Sparse second moment analysis for elliptic problems in stochastic domains. Numer. Math. 109(3), 385–414 (2008). doi:10.1007/s00211-008-0147-9, http://dx.doi.org/10.1007/ s00211-008-0147-9 Hlav´acˇ ek, I., Chleboun, J., Babuˇska, I.: Uncertain Input Data Problems and the Worst Scenario Method. NorthHolland Series in Applied Mathematics and Mechanics, vol. 46. Elsevier Science B.V., Amsterdam (2004) Hoang, V.H., Schwab, C.: High-dimensional finite elements for elliptic problems with multiple scales. Multiscale Model. Simul. 3(1), 168–194 (2004/2005). doi:10.1137/030601077, http://dx.doi.org/10.1137/030601077 Hoang, V.H., Schwab, C.: Analytic regularity and polynomial approximation of stochastic, parametric elliptic multiscale PDEs. Anal. Appl. (Singap.) 11(1), 1350001 (2013) Hoang, V.H., Schwab, C.: N-term Wiener chaos approximation rates for elliptic PDEs with lognormal Gaussian random inputs. Math. Model. Meth. Appl. Sci. 24(4), 797–826 (2014). doi:http://dx.doi.org/10.1142/S0218202513500681 Holden, H., Øksendal, B., Ubøe, J., Zhang, T.: Stochastic Partial Differential Equations. Probability and Its Applications: A Modeling, White Noise Functional Approach. Birkh¨auser, Boston (1996) Karniadakis, G.E., Sherwin, S.J.: Spectral/hp element methods for CFD. Numerical Mathematics and Scientific Computation. Oxford University Press, New York (1999) Khoromskij, B.N., Schwab, C.: Tensor-structured Galerkin approximation of parametric and stochastic elliptic PDEs. SIAM J. Sci. Comput. 33(1) (2011). doi:10.1137/100785715, http://dx.doi.org/10.1137/ 100785715 Kunoth, A., Schwab, C.: Analytic regularity and GPC approximation for control problems constrained by linear parametric elliptic and parabolic PDEs. SIAM J. Control Optim. 51(3), 2442–2471 (2013). doi:http://dx.doi.org/10. 1137/110847597 Kuo, F., Schwab, C., Sloan, I.H.: Quasi-Monte Carlo finite element methods for a class of elliptic partial differential equations with random coefficients. SIAM J. Numer. Anal. 62, 3351–3374 (2012) Kuo, F.Y., Schwab, C., Sloan, I.H.: Quasi-Monte Carlo methods for high dimensional integration – the standard (weighted hilbert space) setting and beyond. ANZIAM J. 53(1), 1–37 (2011). doi:http://dx.doi.org/10.1017/ S1446181112000077 Kuo, F.Y., Schwab, C., Sloan, I.H.: Quasi-Monte Carlo finite element methods for a class of elliptic partial differential equations with random coefficients. SIAM J. Numer. Anal. 50(6), 3351–3374 (2012). doi:http://dx.doi.org/10. 1137/110845537 Lototsky, S., Rozovskii, B.: Stochastic differential equations: a Wiener chaos approach. In: From Stochastic Calcu-

65.

66.

67.

68.

69.

70.

71.

72.

73.

74.

75.

76.

77.

78.

lus to Mathematical Finance, pp. 433–506. Springer, Berlin (2006). doi:10.1007/978-3-540-30788-4-23 Mishra, S., Schwab, C: Sparse tensor multi-level Monte Carlo finite volume methods for hyperbolic conservation laws with random initial data. Math. Comp. 81(280), 1979– 2018 (2012) Naff, R.L., Haley, D.F., Sudicky, E.: High-resolution Monte Carlo simulation of flow and conservative transport in heterogeneous porous media 1. Methodology and flow results. Water Resour. Res. 34(4), 663–677 (1998) Naff, R.L., Haley, D.F., Sudicky, E.: High-resolution Monte Carlo simulation of flow and conservative transport in heterogeneous porous media 2. Transport results. Water Resour. Res. 34(4), 679–697 (1998). doi:10.1029/97WR 02711 Nistor, V., Schwab, C.: High-order Galerkin approximations for parametric second-order elliptic partial differential equations. Math. Models Methods Appl. Sci. 23(9), 1729–1760 (2013) Nobile, F., Tempone, R., Webster, C.G.: An anisotropic sparse grid stochastic collocation method for partial differential equations with random input data. SIAM J. Numer. Anal. 46(5), 2411–2442 (2008). doi:10.1137/070680540, http://dx.doi.org/10.1137/070680540 Nobile, F., Tempone, R., Webster, C.G.: A sparse grid stochastic collocation method for partial differential equations with random input data. SIAM J. Numer. Anal. 46(5), 2309–2345 (2008). doi:10.1137/060663660, http://dx.doi. org/10.1137/060663660 Pentenrieder, B., Schwab, C.: hp-FEM for second moments of elliptic PDEs with stochastic data part 1: analytic regularity. Numer. Methods Partial Differ. Equ. (2012). doi:10.1002/num.20696, http://dx.doi.org/10.1002/ num.20696 Pentenrieder, B., Schwab, C.: hp-FEM for second moments of elliptic PDEs with stochastic data part 2: exponential convergence. Numer. Methods Partial Differ. Equ. (2012). doi:10.1002/num.20690, http://dx.doi.org/10.1002/ num.20690 Protter, P.E.: Stochastic Integration and Differential Equations. Stochastic Modelling and Applied Probability, vol. 21, 2nd edn. Springer, Berlin (2005) (Version 2.1, Corrected third printing) Schillings, C., Schwab, C.: Sparse, adaptive Smolyak quadratures for Bayesian inverse problems. Inverse Probl. 29(6), 1–28 (2013). doi:http://dx.doi.org/10.1088/02665611/29/6/065011 Schillings, C., Schwab, C.: Sparsity in Bayesian inversion of parametric operator equations. Inverse Probl. 30(6) (2014). doi:http://dx.doi.org/10.1088/0266-5611/30/6/065007 Schwab, C., Gittelson, C.J.: Sparse tensor discretizations of high-dimensional parametric and stochastic PDEs. Acta Numer. 20, 291–467 (2011). doi:10.1017/S0962492911000055 Schwab, C., Schillings, C.: Sparse quadrature approach to Bayesian inverse problems. Oberwolfach Rep. 10(3), 2237– 2237 (2013). doi:http://dx.doi.org/10.4171/OWR/2013/39 Schwab, C., Stevenson, R.: Adaptive wavelet algorithms for elliptic PDE’s on product domains. Math. Comput. 77(261), 71–92 (2008) (electronic). doi:10.1090/S0025-5718-07-02019-4, http://dx.doi.org/10.1090/S0025-5718-07-02019-4

Metropolis Algorithms 79. Schwab, C., Stuart, A.M.: Sparse deterministic approximation of Bayesian inverse problems. Inverse Probl. 28(4), (2012). doi:10.1088/0266-5611/28/4/045003, http://dx.doi. org/10.1088/0266-5611/28/4/045003 80. Schwab, C., Todor, R.A.: Sparse finite elements for elliptic problems with stochastic loading. Numer. Math. 95(4), 707– 734 (2003). doi:10.1007/s00211-003-0455-z, http://dx.doi. org/10.1007/s00211-003-0455-z 81. Schwab, C., Todor, R.A.: Sparse finite elements for stochastic elliptic problems—higher order moments. Computing 71(1), 43–63 (2003). doi:10.1007/s00607-003-0024-4. http://dx.doi.org/10.1007/s00607-003-0024-4 82. Schwab, C., Todor, R.A.: Karhunen-Lo`eve approximation of random fields by generalized fast multipole methods. J. Comput. Phys. 217(1), 100–122 (2006) 83. von Petersdorff, T., Schwab, C.: Sparse finite element methods for operator equations with stochastic data. Appl. Math. 51(2), 145–180 (2006). doi:10.1007/s10492-006-0010-1, http://dx.doi.org/10.1007/s10492-006-0010-1 84. Wiener, N.: The homogeneous chaos. Am. J. Math. 60(4), 897–936 (1938). doi:10.2307/2371268, http://dx.doi.org/10. 2307/2371268 85. Xiu, D.: Fast numerical methods for stochastic computations: a review. Commun. Comput. Phys. 5(2–4), 242–272 (2009) 86. Xiu, D., Karniadakis, G.E.: The Wiener-Askey polynomial chaos for stochastic differential equations. SIAM J. Sci. Comput. 24(2), 619–644 (2002) (electronic). doi:10.1137/S1064827501387826, http://dx.doi.org/10. 1137/S1064827501387826

913

Description

The origin of the Metropolis algorithm can be traced to the early 1950s when physicists were faced with the need to numerically study the properties of many particle systems. The state of the system is represented by a vector x D .x1 ; x2 ; : : : xn /, where xi is the coordinate of the ith particle in the system and the goal is to study properties such as pressure and kinetic energy, which can be obtained from computation of the averaged values of suitably defined functions of the state vector. The averaging is weighted with respect to the canonical weight exp.E.x/=kT /, where the constants k and T denote the Boltzmann constant and the temperature, respectively. The physics of the system is encoded in form of the energy function. For example, in a simple liquid model, one has the PP energy E.x/ D .1=2/ i ¤j V .jxi  xj j/, where V .:/ is a potential function giving the dependence of pair-wise interaction energy on the distance between two particles. Metropolis et al. [4] introduce the first Markov chain Monte Carlo method in this context by making sequential moves of the state vector by changing one particle at a time. In each move, a random change of a particle is proposed, say, by changing to a position chosen within a fixed distance from its current position, and the proposed change is either accepted or rejected according to a randomized deMetropolis Algorithms cision that depends on how much the energy of the system is changed by such a move. Metropolis et al. Martin A. Tanner justified the method via the concepts of ergodicity Department of Statistics, Northwestern University, and detailed balance as in kinetic theory. Although Evanston, IL, USA they did not explicitly mention “Markov chain,” it is easy to translate their formulation to the terminology of modern Markov chain theory. In subsequent deMathematics Subject Classification velopment, this method was applied to a variety of physical systems such as magnetic spins, polymers, 62F15; 65C40 molecular fluids, and various condense matter systems (reviewed in [1]). All these applications share the characteristics that n is large and the n comSynonyms ponents are homogeneous in the sense each takes value in the same space (say, 6-dimensional phase Markov chain Monte Carlo (MCMC) space, or up/down spin space, etc.) and interacts in identical manner with other components according to the same physical law as specified by the energy Short Definition function. An important generalization of the Metropolis A Metropolis algorithm is a MCMC computational algorithm, due to [3], is given as follows. Starting method for simulating from a probability distribution. with  .0/ (of dimension d ), iterate for t D 1; 2; : : :

M

914

Microlocal Analysis Methods

1. Draw  from a proposal distribution q.j .t 1/ /. 2. Compute

Wave Front Sets

The phase space in Rn is the cotangent bundle T  Rn that can be identified with Rn Rn . Given a distribution ./  q. .t 1/ j/ .t 1/ : / D min 1; ˛.j f 2 D0 .Rn /, a fundamental object to study is the wave . .t 1/ /  q.j .t 1/ / (1) front set WF.f /  T  Rn n0 viewed as the singularities .t 1/ .t / 3. With probability ˛.j /, set  / D , other- of f that we define below. Here, 0 stands for the zero section .x; 0/, in other words, we do not allow D 0. wise set  .t / D  .t 1/ . It can be shown that ./ is the stationary distribution of the Markov chain . .0/ ;  .1/ ;    /. Moreover, if the proposal distribution q.j/ is symmetric, so that Definition q.j/ D q.j/, then the algorithm reduces to the The basic idea goes back to the properties of the classic Metropolis algorithm. Note that neither algo- Fourier transform. If f is an integrable compactly R f is smooth rithm requires knowledge of the normalizing constant supported function, one can tell whether O. / D e ix f .x/ dx by looking at the behavior of f for . Tierney [6] discusses convergence theory for the algorithm, as well as choices for q.j/. See also [5]. (that is smooth, even analytic) when j j ! 1. It is known that f is smooth if and only if for any N , jfO. /j  CN j jN for some CN . If we localize this requirement to a conic neighborhood V of some 0 6D 0 References (V is conic if 2 V ) t 2 V; 8t > 0), then we can 1. Binder, K.: Monte Carlo Methods in Statistical Physics. think of this as a smoothness in the cone V . To localize in the base x variable, however, we first have to cut Springer, New York (1978) 2. Hammersley, J.M., Handscomb, D.C.: Monte Carlo Methods, smoothly near a fixed x0 . 2nd edn. Chapman and Hall, London (1964) We say that .x0 ; 0 / 2 Rn  .Rn n 0/ is not in the 3. Hastings, W.K.: Monte Carlo sampling methods using 0 n Markov chains and their applications. Biometrika 57, 97–109 wave front set WF.f / of f 2 D .R / if there exists 1 n  2 C0 .R / with .x0 / 6D 0 so that for any N , there (1970) 4. Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, exists CN so that



A.H., Teller, E.: Equation of state calculations by fast computing machines. J. Chem. Phys. 21, 1087–1091 (1953) 5. Tanner, M.A., and Wong, W.H.: From EM to data augmentation: The emergence of MCMC Bayesian computation in the 1980s. Stat. Sci. 25, 506–516 (2010) 6. Tierney, L.: Markov chains for exploring posterior distributions. Ann. Stat. 22, 1701–1762 (1994)

c . /j  CN j jN jf

for in some conic neighborhood of 0 . This definition is independent of the choice of . If f 2 D0 .˝/ with some open ˝  Rn , to define WF.f /  ˝  .Rn n 0/, we need to choose  2 C01 .˝/. Clearly, the wave front set is a closed conic subset of Rn  .Rn n 0/. Next, Microlocal Analysis Methods multiplication by a smooth function cannot enlarge the wave front set. The transformation law under coordiPlamen Stefanov nate changes is that of covectors making it natural to Department of Mathematics, Purdue University, think of WF.f / as a subset of T  Rn n 0, or T  ˝ n 0, West Lafayette, IN, USA respectively. The wave front set WF.f / generalizes the notion singsupp.f / – the complement of the largest open set One of the fundamental ideas of classical analysis is a where f is smooth. The points .x; / in WF.f / are thorough study of functions near a point, i.e., locally. referred to as singularities of f . Its projection onto the Microlocal analysis, loosely speaking, is analysis near base is singsupp.f /, i.e., points and directions, i.e., in the “phase space.” We review here briefly the theory of pseudodifferential operators and geometrical optics. singsupp.f / D fxI 9 ; .x; / 2 WF.f /g:

Microlocal Analysis Methods

Example 1 (a) WF.ı/ D f.0; /I 6D 0g. In other words, the Dirac delta function is singular at x D 0 and in all directions there. (b) Let x D .x 0 ; x 00 /, where x 0 D .x1 ; : : : ; xk /, x 00 D .xkC1 ; : : : ; xn / with some k. Then WF.ı.x 0 // D f.0; x 00 ; 0 ; 0/; 0 6D 0g, where ı.x 0 / is the Dirac delta function x 0 D 0, R on the00plane 0 00 defined by hı.x /; i D .0; x / dx . In other words, WF.ı.x 0 // consists of all (co)vectors 6D 0 with a base point on that plane, perpendicular to it. (c) Let f be a piecewise smooth function that has a nonzero jump across some smooth surface S . Then WF.f / consists of all nonzero (co)vectors at points of S , normal to it. This follows from a change of variables that flattens S locally and reduces the problem to that for the Heaviside function multiplied by a smooth function. (d) Let f D pv x1  iı.x/ in R, where pv x1 is the regularized 1=x in the principal value sense. Then WF.f / D f.0; /I > 0g.

915

˝ and simply write S m . There are other classes in the literature, for example, ˝ D Rn , and (1) is required to hold for all x 2 Rn . The estimates (1) do not provide any control of p when x approaches boundary points of ˝ or 1. Given p 2 S m .˝/, we define the pseudodifferential operator ( DO) with symbol p, denoted by p.x; D/, by p.x; D/f D .2 /n

Z

e ıx p.x; /fO. / d ;

f 2 C01 .˝/: (2)

The P definition is˛ inspired by the following. If P D j˛jm a˛ .x/D is a differential operator, where D D i@, then using the Fourier inversion formula we can P write P as in (2) with a symbol p D j˛jm a˛ .x/ ˛ that is a polynomial in with x-dependent coefficients. The symbol class S m allows for more general functions. The class of the pseudodifferential operators with symbols in S m is denoted usually by  m . The operator P is called a  DO if it belongs to  m for some m. By definition, S 1 D \m S m , and  1 D \m  m . An important subclass is the set of the classical symbols that have an asymptotic expansion of the form

In example (d) we see a distribution with a wave front set that is not even in the variable, i.e., not symmetric under the change 7!  . In fact, wave front sets do not have a special structure except for the requirement to be closed conic sets; given any such set, there is a distribution with a wave front set exactly that set. On the other hand, if f is real valued, then fO is an 1 even function; therefore WF.f / is even in , as well. X pmj .x; /; (3) p.x;

/

Two distributions cannot be multiplied in general. 0 j D0 However, if WF.f / and WF .g/ do not intersect, there is a “natural way” to define a product. Here, WF0 .g/ D where m 2 R, and pmj are smooth and positively f.x;  /I .x; / 2 WF.g/g. homogeneous in of order m  j for j j > 1, i.e., pmj .x;  / D mj pmj .x; / for j j > 1,  > 1; and the sign means that

Pseudodifferential Operators

N X Definition pmj .x; / 2 S mN 1 ; 8N  0: (4) p.x; / m We first define the symbol class S .˝/, m 2 R, as j D0 the set of all smooth functions p.x; /, .x; / 2 ˝  Rn , called symbols, satisfying the following symbol Any  DO p.x; D/ is continuous from C01 .˝/ to estimates: for any compact set K  ˝, and any multi- C 1 .˝/ and can be extended by duality as a continuous indices ˛, ˇ, there is a constant CK;˛;ˇ > 0 so that map from E 0 .˝/ to D0 .˝/.

j@˛ @ˇx p.x; /jCK;˛;ˇ .1Cj j/mj˛j ; 8.x; /2KRn : (1) m .˝/ with More generally, one can define the class S;ı 0  , ı  1 by replacing m  j˛j there by m  m .˝/. Often, we omit j˛j C ıjˇj. Then S m .˝/ D S1;0

Principal Symbol The principal symbol of a  DO in  m .˝/ given by (2) is the equivalence class S m .˝/=S m1.˝/, and any representative of it is called a principal symbol as well. In case of classical  DOs, the convention is to choose

M

916

Microlocal Analysis Methods

the principal symbol to be the first term pm that in Af particular is positively homogeneous in . “ n e i.xy/ a.x; y; / f .y/dy d ; f 2 C01 .˝/; D .2 / Smoothing Operators (7) Those are operators than map continuously E 0 .˝/ into C 1 .˝/. They coincide with operators with where the amplitude a satisfies smooth Schwartz kernels in ˝  ˝. They can always be written as  DOs with symbols in S 1 and j@˛ @ˇx @y a.x; y; /j vice versa – all operators in  1 are smoothing. Smoothing operators are viewed in this calculus as  CK;˛;ˇ; .1 C j j/mj˛j ; 8.x; y; / 2 K  Rn negligible and  DOs are typically defined modulo (8) smoothing operators, i.e., A D B if and only if A  B is smoothing. Smoothing operators are not for any compact set K  ˝  ˝ and for any ˛, “small.” ˇ,  . In fact, any such A is a  DO with symbol p.x; / (independent of y) with the formal asymptotic The Pseudolocal Property expansion For any  DO P and any f 2 E 0 .˝/, X p.x; / D ˛ @˛y a.x; x; /: singsupp.Pf /  singsuppf: (5) ˛0 In other words, a  DO cannot increase the singular support. This property is preserved if we replace In particular, the principal symbol of that operator can be taken to be a.x; x; /. singsupp by WF; see (13). Symbols Defined by an Asymptotic Expansion In many applications, a symbol is defined by consecutively constructing symbols pj 2 S mj , j D 0; 1; : : : , where mj & 1, and setting p.x; /

X j

pj .x; /:

(6)

Transpose and Adjoint Operators to a DO The mapping properties of any  DO A indicate that it has a well-defined transpose A0 and a complex adjoint A with the same mapping properties. They satisfy hAu; vi D hu; A0 vi; hAu; vi N D hu; A vi; 8u; v 2 C01 where h; i is the pairing in distribution sense; and in this particular case just an integral of uv. In particular, A u D A0 uN , and if A maps L2 to L2 in a bounded way, then A is the adjoint of A in L2 sense. The transpose and the adjoint are  DOs in the same class with amplitudes a.y; x;  / and a.y; N x; /, respectively; and symbols

The series on the right may not converge but we can make it convergent by using our freedom to modify each pj for in expanding compact sets without changing the large behavior of each term. This extends the Borel idea of constructing a smooth function with prescribed derivatives at a fixed point. The asymptotic (6) then is understood in a sense simX 1 X 1 ilar to (4). This shows that there exists a symbol @˛ Dx˛ p.x; .1/j˛j .@˛ Dx˛ p/.x;  /; N /; ˛Š ˛Š p 2 S m0 satisfying (6). That symbol is not unique ˛0 ˛0 but the difference of two such symbols is always in S 1 . if a.x; y; / and p.x; / are the amplitude and/or the symbol of that  DO. In particular, the principal symbols are p0 .x;  / and pN0 .x; /, respectively, where p0 Amplitudes is (any representative of) the principal symbol. A seemingly larger class of  DOs is defined by

Microlocal Analysis Methods

Composition of DOs and DOs with Properly Supported Kernels Given two  DOs A and B, their composition may not be defined even if they are smoothing ones because each one maps C01 to C 1 but may not preserve the compactness of the support. For example, if A.x; y/ and B.x; y/ are their Schwartz R kernels, the candidate for the kernel of AB given by A.x; z/B.z; y/ dz may be a divergent integral. On the other hand, for any  DO A, one can find a smoothing correction R, so that ACR has properly supported kernel, i.e., the kernel of A C R has a compact intersection with K  ˝ and ˝  K for any compact K  ˝. The proof of this uses the fact that the Schwartz kernel of a  DO is smooth away from the diagonal fx D yg, and one can always cut there in a smooth way to make the kernel properly supported at the price of a smoothing error.  DOs with properly supported kernels preserve C01 .˝/, and also E 0 .˝/, and therefore can be composed in either of those spaces. Moreover, they map C 1 .˝/ to itself and can be extended from D0 .˝/ to itself. The property of the kernel to be properly supported is often assumed, and it is justified by considering each  DO as an equivalence class. If A 2  m .˝/ and B 2  k .˝/ are properly supported  DOs with symbols a and b, respectively, then AB is again a  DO in  mCk .˝/ and its symbol is given by X ˛0

.1/j˛j

1 ˛ @ a.x; /Dx˛ b.x; /: ˛Š

917

Relation (9) shows that the transformation law under coordinate changes is that of a covector. Therefore, the principal symbol is a correctly defined function on the cotangent bundle T  ˝. The full symbol is not invariantly defined there in general. Let M be a smooth manifold and A W C01 .M / ! C 1 .M / be a linear operator. We say that A 2  m .M /, if its kernel is smooth away from the diagonal in M  M and if in any coordinate chart .A; /, where  W U ! ˝  Rn , we have .A.uı//ı1 2  m .˝/. As before, the principal symbol of A, defined in any local chart, is an invariantly defined function on T  M . Mapping Properties in Sobolev Spaces In Rn , Sobolev spaces H s .Rn /, s 2 R, are defined as the completion of S 0 .Rn / in the norm Z kf k2H s .Rn / D

.1 C j j2 /s jfO. /j2 d :

When s is a nonnegative integer, norm is R ˛ an equivalent P 2 the square root of j˛js j@ f .x/j dx. For such s, and a bounded domain ˝, one defines H s .˝/ as the N using the latter norm with the completion of C 1 .˝/ integral taken in ˝. Sobolev spaces in ˝ for other real values of s are defined by different means, including duality or complex interpolation. Sobolev spaces are also Hilbert spaces. Any P 2  m .˝/ is a continuous map from sm s .˝/ to Hloc .˝/. If the symbols estimates (1) Hcomp are satisfied in the whole Rn Rn , then P W H s .Rn / ! H sm .Rn /.

In particular, the principal symbol can be taken to be ab. Elliptic DOs and Their Parametrices The operator P 2  m .˝/ with symbol p is called elliptic of order m, if for any compact K  ˝, there Change of Variables and DOs on Manifolds Let ˝ 0 be another domain, and let  W ˝ ! ˝Q be a exists constants C > 0 and R > 0 so that diffeomorphism. For any P 2  m .˝/, PQ f WD .P .f ı Q into C 1 .˝/. Q It is a  DO in // ı  1 maps C01 .˝/ C j jm  jp.x; /j for x 2 K, and j j > R: (10) m Q  .˝/ with principal symbol Then the symbol p is called also elliptic of order m. (9) It is enough to require the principal symbol only to be elliptic (of order m). For classical  DOs, see (3); where p is the symbol of P , d is the Jacobi matrix the requirement can be written as pm .x; / 6D 0 for f@i =@xj g evaluated at x D  1 .y/, and .d/0 stands 6D 0. A fundamental property of elliptic operators is for the transpose of that matrix. We can also write that they have parametrices. In other words, given an .d/0 D ..d 1 /1 /0 . An asymptotic expansion for the elliptic  DO P of order m, there exists Q 2  m .˝/ whole symbol can be written down as well. so that p. 1 .y/; .d/0 /

M

918

Microlocal Analysis Methods

QP  Id 2  1 ;

PQ  Id 2  1 :

(11) K  U with a compact projection on the base “xspace,” (1) is fulfilled for any m. The essential support The proof of this is to construct a left parametrix first ES.P /, sometimes also called the microsupport of by choosing a symbol q0 D 1=p, cut off near the P , is defined as the smallest closed conic set on the possible zeros of p, that form a compact set any time complement of which the symbol p is of order 1. when x is restricted to a compact set as well. The Then WF.Pf /  WF.f / \ ES.P /: corresponding  DO Q0 will then satisfy Q0 P D Id C R, R 2  1 . Then we take a  DO E with asymptotic Let P have a homogeneous principal symbol pm . The expansion E Id  R C R2  R3 C : : : that would be characteristic set CharP is defined by the formal Neumann series expansion of .Id C R/1 , if the latter existed. Then EQ0 is a left parametrix that is CharP D f.x; / 2 T  ˝ n 0I pm .x; / D 0g: also a right parametrix. An important consequence is the following elliptic CharP can be defined also for general  DOs that regularity statement. If P is elliptic (and properly may not have homogeneous principal symbols. For any supported), then  DO P , we have singsupp.PF / D singsupp.f /;

8f 2 D0 .˝/; (12)

compared to (5). In particular, Pf 2 C 1 implies f 2 C 1. It is important to emphasize that elliptic  DOs are not necessarily invertible or even injective. For example, the Laplace-Beltrami operator  S n1 on the sphere is elliptic, and then so is  S n1  z for every number z. The latter however so not injective for z an eigenvalue. On the other hand, on a compact manifold M , an elliptic P 2  m .M / is “invertible” up to a compact error, because then QP  Id D K1 , PQ  Id D K2 , see (11) with K1;2 compact operators. As a consequence, such an operator is Fredholm and in particular has a finitely dimensional kernel and cokernel.

DOs and Wave Front Sets

WF.f /  WF.Pf / [ CharP;

8f 2 E 0 .˝/: (14)

P is called microlocally elliptic in the open conic set U , if (10) is satisfied in all compact subsets, similarly to the definition of ES.P / above. If it has a homogeneous principal symbol pm , ellipticity is equivalent to pm 6D 0 in U . If P is elliptic in U , then Pf and f have the same wave front set restricted to U , as follows from (14) and (13). The Hamilton Flow and Propagation of Singularities Let P 2  m .M / be properly supported, where M is a smooth manifold, and suppose that P has a real homogeneous principal symbol pm . The Hamiltonian vector field of pm on T  M n 0 is defined by Hpm

n  X @pm @ @pm @ : D  @xj @ j @ j @xj j D1

The microlocal version of the pseudolocal property is The integral curves of Hpm are called bicharacteristics of P . Clearly, Hpm pm D 0; thus pm is constant given by the following: along each bicharacteristic. The bicharacteristics along WF.Pf /  WF.f / (13) which pm D 0 are called zero bicharacteristics. The H¨ormander’s theorem about propagation of for any (properly supported)  DO P and f 2 D0 .˝/. singularities is one of the fundamental results in the In other words, a  DO cannot increase the wave front theory. It states that if P is an operator as above and set. If P is elliptic for some m, it follows from the P u D f with u 2 D0 .M /, then existence of a parametrix that there is equality above, WF.u/ n WF.f /  CharP i.e., WF.Pf / D WF.f /, which is a refinement of (12). We say that the  DO P is of order 1 in the open conic set U  T  ˝ n 0, if for any closed conic set and is invariant under the flow of Hpm .

Microlocal Analysis Methods

919

An important special case is the wave operator P D @2t  g , where g is the Laplace Beltrami operator associated with a Riemannian metric g. We may add lower-order terms without changing the bicharacteristics. Let .; / be the dual variables to .t; x/. The principal symbol is p2 D  2 C j j2g , where P ij g .x/ i j , and .g ij / D .gij /1 . The j j2g WD bicharacteristics equations then are P D 0, tP D 2, P P ij g .x/ i j , and xP j D 2 g ij i , and Pj D 2@x j they are null ones if  2 D j j2g . Here, xP D dx=ds, etc. The latter equations are the Hamiltonian curves of P two HQ WD g ij .x/ i j and they are known to coincide with the geodesics .; P / on TM when identifying vectors and covectors by the metric. They lie on the energy surface HQ D const. The first two equations imply that  is a constant, positive or negative; and up to rescaling, one can choose the parameter along the geodesics to be t. That rescaling forces the speed along the geodesic to be 1. The null condition  2 D j j2g defines two smooth surfaces away from .; / D .0; 0/:  D ˙j jg . This corresponds to geodesics starting from x in direction either or  . To summarize, for the homogeneous equation P u D 0, we get that each singularity .x; / of the initial conditions at t D 0 starts to propagate from x in direction either or 

or both (depending on the initial conditions) along the unit speed geodesic. In fact, we get this first for the singularities in T  .Rt  Rnx / first, but since they lie in CharP , one can see that they project to T  Rnx as singularities again.

with initial conditions u.0; x/ D f1 .x/ and ut .0; x/ D f2 . It is enough to assume first that f1 and f2 are in C01 and extend the resulting solution operator to larger spaces later. We are looking for a solution of the form u.t; x/ D .2 /

n

XZ

 e i .t;x; / a1; .x; ; t/fO1 . /

 D˙

 C j j1 a2; .x; ; t/fO2 . / d ; (16) modulo terms involving smoothing operators of f1 and f2 . The reason to expect two terms is already clear by the propagation of singularities theorem, and is also justified by the eikonal equation below. Here the phase functions ˙ are positively homogeneous of order 1 in

. Next, we seek the amplitudes in the form aj;

1 X

.k/

aj; ;

 D ˙; j D 1; 2;

.k/

where aj; is homogeneous in of degree k for large j j. To construct such a solution, we plug (16) into (15) and try to kill all terms in the expansion in homogeneous (in ) terms. Equating the terms of order 2 yields the eikonal equation .@t /2  c 2 .x/jrx j2 D 0:

Geometrical Optics Geometrical optics describes asymptotically the solutions of hyperbolic equations at large frequencies. It also provides a parametrix (a solution up to smooth terms) of the initial value problem for hyperbolic equations. The resulting operators are not  DOs anymore; they are actually examples of Fourier Integral Operators. Geometrical Optics also studies the large frequency behavior of solutions that reflect from a smooth surface (obstacle scattering) including diffraction, reflect from an edge or a corner, and reflect and refract from a surface where the speed jumps (transmission problems). As an example, consider the acoustic equation .@2t  c 2 .x/ /u D 0;

.t; x/ 2 Rn ;

(17)

kD0

(18)

R Write fj D .2 /n e ix fOj . / d , j D 1; 2, to get the following initial conditions for ˙ ˙ jt D0 D x  :

(19)

The eikonal equation can be solved by the method of characteristics. First, we determine @t  and rx  for t D 0. We get @t jt D0 D c.x/j j, rx jt D0 D . This implies existence of two solutions ˙ . If c D 1, we easily get ˙ D j jt C x  . Let for any .z; /, z; .s/ be unit speed geodesic through .z; /. Then C is constant along the curve .t; z; .t// that implies that C D z.x; /  in any domain in which .t; z/ can be chosen to be coordinates. Similarly,  is constant along the curve .t; z; .t//. In general, we cannot solve the eikonal equation globally, for (15) all .t; x/. Two geodesics z; and w; may intersect,

M

920

Minimal Surface Equation

for example, giving a nonunique value for ˙ . We always have a solution however in a neighborhood of t D 0. Equate now the order 1 terms in the expansion of .@2t  c 2 /u to get that the principal terms of the amplitudes must solve the transport equation 

 .0/ .@t ˙ /@t  c 2 rx ˙  rx C C˙ aj;˙ D 0;

(20)

with 2C˙ D .@2t  c 2 /˙ : This is an ODE along the vector field .@t ˙ ; c 2 rx /, and the integral curves of it coincide with the curves .t; z;˙ /. Given an initial condition at t D 0, it has a unique solution along the integral curves as long as  is well defined. Equating terms homogeneous in of lower order we .k/ get transport equations for aj; , j D 1; 2; : : : with the same left-hand side as in (20) with a right-hand side .k1/ determined by ak; . Taking into account the initial conditions, we get a1;C C a1; D 1;

a2;C C a2; D 0 for t D 0: .0/

This is true in particular for the leading terms a1;˙ and .0/

a2;˙ . Since @t ˙ D c.x/j j for t D 0, and ut D f2 for t D 0, from the leading order term in the expansion of ut , we get .0/

.0/

a1;C D a1; ;

.0/

.0/

ic.x/.a2;  a2;C / D 1

large, depending on the speed. On the other hand, the solution operator .f1 ; f2 / 7! u makes sense as a global Fourier Integral Operator for which this construction is just one if its local representations. Acknowledgements Author partly supported by a NSF FRG Grant DMS-1301646. This article first appeared as an appendix (pp. 310–320) to “Multi-wave methods via ultrasound” by Plamen Stefanov and Gunther Uhlmann, Inverse Problems and Applications: Inside Out II, edited by Gunther Uhlmann, Mathematical Sciences Research Institute Publications v.60, Cambridge University Press, New York, 2012, pp. 271–324.

References 1. H¨ormander, L.: The Analysis of Linear Partial Differential Operators. III, Pseudodifferential Operators. Volume 274 of Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]. Springer, Berlin (1985) 2. Melrose, R.: Introduction to Microlocal Analysis. (2003) http://www-math.mit.edu/rbm/iml90.pdf 3. Taylor, M.E.: Pseudodifferential Operators. Volume 34 of Princeton Mathematical Series. Princeton University Press, Princeton (1981) 4. Tr`eves, F.: Introduction to Pseudodifferential and Fourier Integral Operators. Pseudodifferential Operators. The University Series in Mathematics, vol. 1. Plenum Press, New York (1980)

for t D 0:

Minimal Surface Equation

Therefore, Einar M. Rønquist and Øystein Tr˚asdahl Department of Mathematical Sciences, Norwegian for tD0: D D University of Science and Technology, Trondheim, 2 (21) Norway Note that if c D 1, then ˙ D x   tj j, and a1;C D a1; D 1=2, a2;C D a2; D i=2. Using those initial .0/ conditions, we solve the transport equations for a1;˙ Minimal Surfaces 1 .0/ a1; D ;

.0/ a1;C

.0/ a2;C

.0/ a2; D

i 2c.x/

.0/

and a2;˙ . Similarly, we derive initial conditions for the lower-order terms in (17) and solve the corresponding transport equations. Then we define aj; by (17) as a symbol. The so constructed u in (16) is a solution only up to smoothing operators applied to .f1 ; f2 /. Using standard hyperbolic estimates, we show that adding such terms to u, we get an exact solution to (15). As mentioned above, this construction may fail for t too

Minimal surfaces arise many places in natural and man-made objects, e.g., in physics, chemistry, and architecture. Minimal surfaces have fascinated many of our greatest mathematicians and scientists for centuries. In its simplest form, the problem can be stated as follows: find the surface S of least area spanning a given closed curve C in R3 . In the particular case when C lies in a two-dimensional plane, the minimal surface

Minimal Surface Equation

is simply the planar region bounded by C . However, the general problem is very difficult to solve [2, 6]. The easiest way to physically study minimal surfaces is to look at soap films. In the late nineteenth century, the Belgian physicist Joseph Plateau [7] conducted a number of soap film experiments. He observed that, regardless of the shape of a closed and curved wire, the wire could always bound at least one soap film. Since capillary forces attach a potential energy proportional to the surface area, a soap film in stable equilibrium position corresponds to a surface of minimal area [4]. The mathematical boundary value problem for minimal surfaces is therefore also called the Plateau problem.

921

Z

q

AŒf  D ˝

1 C fx2 C fy2 dx dy:

(1)

The surface of minimum area is then given directly by the Euler-Lagrange equation for the area functional AŒf : .1 C fy2 /fxx C .1 C fx2 /fyy  2fx fy fxy D 0: (2)

Equation (2) is called the minimal surface equation. Hence, to determine S mathematically, we need to solve a nonlinear, second-order partial differential equation with specified boundary conditions (determined by the given curve C ). Despite the difficulty of finding closed form solutions, the minimal surface problem has created an intense mathematical activity over the past couple of centuries and spurred advances Mathematical Formulation in many fields like calculus of variation, differential Assume for simplicity that the surface S can be repre- geometry, integration and measure theory, and complex sented as a function z D f .x; y/; see Fig. 1. Note that analysis. With the advent of the computer, a range of computational algorithms has also been proposed to this may not always be possible. Using subscripts x and y to denote differentiation construct approximate solutions. with respect to x and y, respectively, the area AŒf  of a surface f .x; y/ can be expressed as the integral

Characterizations and Generalizations

A point on the minimal surface S is given by the coordinates .x; y; z/ D .x; y; f .x; y//. This represents an example of a particular parametrization P of S : for any point .x; y/ 2 ˝, there is a corresponding point P .x; y/ D .x; y; z/ D .x; y; f .x; y// on S 2 R3 . Two tangent vectors t1 and t2 spanning the tangent plane at P .x; y/ is then given as t1 D Px .x; y/ D .1; 0; fx /;

(3)

t2 D Py .x; y/ D .0; 1; fy /:

(4)

The normal vector at this point is then simply the cross product between t1 and t2 : nD

Minimal Surface Equation, Fig. 1 The minimal surface S is the surface of least area bounded by the given blue curve, C . The projection of S onto the xy-plane is the planar region ˝ bounded by the red curve, @˝. The minimal surface z D f .x; y/ for the particular choice of C shown here is called the Enneper surface

.fx ; fy ; 1/ t1  t2 ; Dq jt1  t2 j 1 C fx2 C fy2

(5)

where n is normalized to be of unit length. It can be shown that the divergence of n is equal to twice the mean curvature, H , at the point P .x; y/. Using (2), it follows that 2H D r  n D 0: (6)

M

922

Hence, a minimal surface is characterized by the fact that the mean curvature is zero everywhere. Note that this is in sharp contrast to a soap bubble where the mean curvature is nonzero. This is because the pressure inside the soap bubble is higher than on the outside, and the pressure difference is given as the product of the mean curvature and the surface tension, which are both nonzero. For a soap film, however, the pressure is the same on either side, consistent with the fact that the mean curvature is zero. In many cases, we cannot describe a surface as a simple function z D f .x; y/. However, instead of using the simple parametrization P .x; y/ D .x; y; f .x; y//, we can generalize the parametrization in the following way. For a point .u; v/ 2 ˝ 2 2 point P .u; v/ D R  , there is a corresponding  x.u; v/; y.u; v/; z.u; v/ on S 2 R3 , i.e., each individual coordinate x, y, and z is a function of the new coordinates u and v. Being able to choose

Minimal Surface Equation

different parametrizations for a surface is of great importance, both for the pure mathematical analysis and for numerical computations. Examples of Minimal Surfaces Enneper’s surface (see also Fig. 1) can be parametrized as [2]  1 2 2 x.u; v/ D u 1  u C v ; 3  1 (7) y.u; v/ D v 1  v 2 C u2 ; 3 z.u; v/ D u2  v 2 ; where u and v are coordinates on a circular domain of radius R. For R  1 the surface is stable and has a global area minimizer; see Fig. 2b which depicts the computed minimal surface for the case R D 0:8 using the surface in Fig. 2a as an initial condition [8].

Minimal Surface Equation, Fig. 2 Enneper’s surface is the minimal surface corresponding to the given, blue boundary curve (a) Initial surface. (b) Minimal surface

Minimal Surface Equation, Fig. 3 The three minimal surfaces in the Enneper case for R D 1:2. The unstable solution (a) is known analytically (see (7)) and is found by interpolating this known parametrization. The two other solutions are stable

and global area minimizers. The surfaces (b) and (c) are here obtained by starting from slightly perturbed versions of (a) (by adding random perturbations on the order of 1010 )

Model Reduction

923

p For 1 < R < 3 the given parametrization gives be done once, in an offline stage, to obtain a model of an unstable minimal surface. However, there also exist reduced complexity to be evaluated at little cost while two (symmetrically similar) stable minimal surfaces maintaining accuracy. Let us consider a generic dynamical system as which are global area minimizers; see Fig. 3. This illustrates a case where the minimal p surface problem @u.x; t; / C F .u; x; t; / D f .x; t; /; has more than one solution. For R  3 the boundary @t curve intersects itself. y.x; t; / D G T u; subject to appropriate initial and boundary values. Here u is an N -dimensional vector field, possibly depending on space x and time t, and  is a q-dimensional Chopp, D.L.: Computing minimal surfaces via level set parameter space. y represents an output of interest. curvature flow. J. Comput. Phys. 106, 77–91 (1993) Dierkes, U., Hildebrandt, S., K¨uster, A., Wohlrab, O.: Mini- If N is very large, e.g., when originating from the mal Surfaces I. Springer, Berlin/Heidelberg/New York (1991) discretization of a partial differential equation, the Gerhard, D., Hutchinson, J.E.: The discrete Plateau probevaluation of this output is potentially expensive. lem: algorithm and numerics. Math. Comput. 68(225), 1–23 This situation has led to the development of a (1999) Isenberg, C.: The Science of Soap Films and Soap Bubbles. myriad of methods to develop reduced models with the Dover Publications Inc, New York (1992) majority focusing on representing the solution, u, as a Jacobsen, J.: As flat as possible. SIAM Rev. 49(3), 491–507 linear combination of N -vectors as (2007)

References 1. 2. 3.

4. 5.

6. Osserman, R.: A Survey of Minimal Surfaces. Dover Publications Inc, New York (2002) 7. Plateau, J.A.F.: Statique exp´erimentale et th´eorique des liquides soumis aux seules forces mol´eculaires. GauthierVillars, Paris (1873) 8. Tr˚asdahl, Ø., Rønquist, E.M.: High order numerical approximation of minimal surfaces. J. Comput. Phys. 230(12), 4795–4810 (2011)

u ' uO D Va; where V is an N  m orthonormal matrix, representing a linear space, and a an m-vector. Inserting this into the model yields the general reduced model @aO C V T F .Va/ D V T f; y D .G T V /a; @t

Model Reduction

where we have left out the explicit dependence of the parameters for simplicity. In the special case where F .u/ D Lu is linear, the problem further reduces to

Jan S. Hesthaven Division of Applied Mathematics, Brown University, Providence, RI, USA

@aO C V T LVa D V T f; y D .G T V /a; @t

While advances in high-performance computing and mathematical algorithms have enabled the accurate modeling of problems in science and engineering of very considerable complexity, problems with strict time restrictions remain a challenge. Such situations are often found in control and design problems and in situ deployed systems and others, characterized by a need for a rapid online evaluation of a system response under the variation of one or several parameters, including time. However, to ensure this ability to perform many evaluations under parameter variation of a particular system, it is often acceptable that substantial work

which can be evaluated in complexity independent of N . Hence, if N  m, the potential for savings is substantial, reflecting the widespread interest in and use of reduced models. For certain classes of problems, lack of linearity can be overcome using nonlinear interpolation techniques, known as empirical interpolation methods [2], to recover models with evaluation costs independent of N . Considering the overall accuracy of the reduced model leads to the identification of different methods, with the key differences being in how V is formed and how the overall accuracy of the reduced model is estimated.

M

924

Proper Orthogonal Decompositions In the proper orthogonal decomposition (POD) [4], closely related to principal component analysis (PCA), Karhunen-Loeve (KL) transforms, and the Hotelling transform, the construction of the linear space to approximate u is obtained through processing of a selection of solution snapshots. Assume that a sequence of solutions, un , is obtained at regular intervals for parameters or, as in often done, at regular intervals in time. Collecting these in an N  n matrix X D Œu1 ; : : : ; un , the singular value decomposition (SVD) yields X D U˙W  where ˙ is an n  n diagonal matrix with the singular values, U is N  n and W is a n  n matrix, both of which are orthonormal. The POD basis is then formed by truncating ˙ at a tolerance, ", such that m  "  mC1 . The linear space used to represent u over parameter or time variation is obtained from the first m columns of U . An estimate of the accuracy of the linear space for approximating of the solution is recovered from the magnitude of the largest eliminated singular value. The success of the POD has led to numerous extensions, [4, 9, 12], and this approach has been utilized for the modeling of large and complex systems [3, 13]. To develop effective and accurate POD models for nonlinear problems, the discrete empirical interpolation method (DEIM) [5] has been introduced. A central disadvantage of the POD approach is the need to compute n snapshots, often in an offline approach, perform the SVD on this, possibly large, solution matrix, and then eliminate a substantial fraction through the truncation process. This results in a potentially large computational overhead. Furthermore, the relationship between the eliminated vectors associated with truncated singular values and the accuracy of the reduced model is generally not clear [8], and the stability of the reduced model, in particular for nonlinear problems, is known to be problematic. Some of these issues, in particular related to preservation of the asymptotic state, are discussed in [16].

Model Reduction

obtains a transfer function, H.s/ D G T .s C L/1 B, between input g and output y with s being the Laplace parameter. Introducing the matrix A D .L C s0 /1 and the vector r D .L C s0 /1 B, the transfer function becomes H.s/ D G T .I  .s  s0 /A/1 r. Hence for perturbations of s around s0 , we recover H.s/ D

1 X

mi .s  s0 /m ; mi D G T Am r;

i D0

where one recognizes that mi is obtained as a product of the vectors in the Krylov subspaces spanned by Aj r and AT G. These are recognized as the left and the right Krylov subspace vectors and can be computed in a stable manner using a Lanczos process. The union of the first m=2 left and right Krylov vectors spans the solution of the dynamical problem, and, as expected, larger intervals including s0 require longer sequences of Krylov vectors. While the computational efficiency of the Krylov techniques is appealing, a thorough error analysis is lacking [1]. However, there are several extensions to more complex problems and nonlinear systems [15] as well as to closely related techniques aimed to address time-dependent problems [15].

Certified Reduced Basis

Certified reduced basis methods (RBM) [11, 14] are fundamentally different from the previous two techniques in how the linear space is constructed and were originally proposed as an accurate and efficient way to construct reduced models for parametrized steady or harmonic partial differential equations. In this approach, based in the theory of variational problems and Galerkin approximations, one expresses the problem as a bilinear form, a.u; v; / D f .; v/, and seeks an approximation to the solution, u./, over variations of . In contrast to the POD, in the RBM, the basis is constructed through a greedy approach based on maximizing the residual a.Ou; v/  f in some approKrylov-Based Methods priate norm and an offline testing across the parameter space using a carefully designed residual-based error The majority of Krylov-based methods [1] consider the estimator. simplified linear problem with F D Lu and f D This yields a reduced method with a couple of Bg representing the input. In Laplace domain, one distinct advantages over POD in particular. On one

Modeling of Blood Clotting

hand, the greedy approximation enables a minimal computational effort since only snapshots required to build the basis in an max-norm optimal manner are computed. Furthermore, the error estimator enables one to rigorously certify the quality of the reduced model and the output of interest. This is a unique quality of the certified reduced basis methods and has been demonstrated for a large class of linear problems, including applications originating in solid mechanics, heat conduction, acoustics, and electromagnetics [6, 11, 14], and for geometric variations [6, 11] and applications formulated as integral equations [7]. This more rigorous approach is difficult to extend to nonlinear problems and general time-dependent problems although there are recent results in this direction [10, 17], combining POD and RBM to enable reduced models for time-dependent parametric problems.

925 11. Quarteroni, A., Rozza, G., Manzoni, A.: Certified reduced basis approximation for parametrized partial differential equations and applications. J. Math. Ind. 1, 3 (2011) 12. Rathinam, M., Petzold, L.: A new look at proper orthogonal decomposition. SIAM J. Numer. Anal. 41, 1893–1925 (2004) 13. Rowley, C.: Model reduction for fluids using balanced proper orthogonal decomposition. Int. J. Bifurc. Chaos 15, 997–1013 (2000) 14. Rozza, G., Huynh, D.B.P., Patera, A.T.: Reduced basis approximation and a posteriori error estimation for affinely parametrized elliptic coercive partial differential equations: application to transport and continuum mechanics. Arch. Comput. Methods Eng. 15(3), 229–275 (2008) 15. Schmid, P.J.: Dynamic mode decomposition of numerical and experimental data. J. Fluid Mech. 656, 5–28 (2010) 16. Sirisup, S., Karniadakis, G.E.: A spectral viscosity method for correcting the long-term behavior of POD-models. J. Comput. Phys. 194, 92–116 (2004) 17. Veroy, K., Prud’homme, C., Patera, A.T.: Reduced-basis approximation of the viscous Burgers equation: rigorous a posteriori error bounds. C. R. Math. 337(9), 619–624 (2003)

References

Modeling of Blood Clotting 1. Bai, Z.: Krylov subspace techniques for reduced-order modeling of large-scale dynamical systems. Appl. Numer. Math. 43, 9–44 (2002) 2. Barrault, M., Maday, Y., Nguyen, N.C., Patera, A.T.: An ‘empirical interpolation’ method: application to efficient reduced-basis discretization of partial differential equations. C. R. Math. 339(9), 667–672 (2004) 3. Berkooz, G., Holmes, P., Lumley, J.L.: The proper orthogonal decomposition in the analysis of turbulent flows. Ann. Rev. Fluid Mech. 25, 539–575 (1993) 4. Chatterjee, A.: An introduction to the proper orthogonal decomposition. Curr. Sci. 78, 808–817 (2000) 5. Chaturantabus, S., Sorensen, D.C.: Nonlinear model reduction via discrete empirical interpolation. SIAM J. Sci. Comput. 32, 2737–2764 (2010) 6. Chen, Y., Hesthaven, J.S., Maday, Y., Rodriguez, J., Zhu, X.: Certified reduced methods for electromagnetic scattering and radar cross section prediction. Comput. Methods Appl. Mech. Eng. 233, 92–108 (2012) 7. Hesthaven, J.S., Stamm, B., Zhang, S.: Certified reduced basis methods for the electric field integral equation. SIAM J. Sci. Comput. 34(3), A1777–A1799 (2011) 8. Homescu, C., Petzold, L.R., Serban, R.: Error estimation for reduced-order models of dynamical systems. SIAM Rev. 49, 277–299 (2007) 9. Ilak, M., Rowley, C.W.: Modeling of transitional channel flow using balanced proper orthogonal decomposition. Phys. Fluids 20, 034103 (2008) 10. Nguyen, N.C., Rozza, G., Patera, A.T.: Reduced basis approximation and a posteriori error estimation for the timedependent viscous Burgers’ equation. Calcolo 46(3), 157– 185 (2009)

Aaron L. Fogelson Departments of Mathematics and Bioengineering, University of Utah, Salt Lake City, UT, USA

Mathematics Subject Classification 92C05; 92C10; 92C35 (76Z05); 92C42; 92C45; 92C55

Synonyms Modeling of thrombosis; Modeling of platelet aggregation and coagulation; Modeling of platelet deposition and coagulation

Description Blood circulates under pressure through the human vasculature. The pressure difference across the vascular wall means that a hole in the vessel wall can lead to rapid and extensive loss of blood. The hemostatic (blood clotting) system has developed to seal a vascular

M

926

Modeling of Blood Clotting

injury quickly and minimize hemorrhage. The components of this system, so important to its necessarily rapid and robust response to overt injury, are implicated in the pathological processes of arterial and venous thrombosis that cause extensive death and morbidity. Intensive laboratory research has revealed much about the players involved in the clotting process and about their interactions. Yet much remains unknown about how the system as a whole functions. This is because the nature of the clotting response – complex, multifaceted, dynamic, spatially distributed, and multiscale – makes it very difficult to study using traditional experimentation. For this reason, mathematical models and computational simulations are essential to develop our understanding of clotting and an ability to make predictions about how it will progress under different conditions. Blood vessels are lined with a monolayer of endothelial cells. If this layer is disrupted, then exposure of the subendothelial matrix initiates the intertwined processes of platelet deposition and coagulation. Platelets are tiny cells which circulate with the blood in an unactive state. When a platelet contacts the exposed subendothelium, it may adhere by means of bonds formed between receptors on the platelet’s surface and molecules in the subendothelial matrix (see Fig. 1). These bonds also trigger a suite

vWF Fbg

Plt Fbg Fbg Plt

EC

vWF Plt

vWF

EC

of responses known as platelet activation which include change of shape, the mobilization of an additional family of binding receptors on the platelet’s surface, and the release of chemical agonists into the blood plasma (the most important of these being ADP from cytoplasmic storage granules and the coagulation enzyme thrombin synthesized on the surface of activated platelets). These agonists can induce activation of other platelets that do not directly contact the injured vascular tissue. By means of molecular bonds that bridge the gap between the newly mobilized binding receptors on two platelets’ surfaces, platelets can cohere to one another. As a result of these processes, platelets deposit on the injured tissue and form a platelet plug. Exposure of the subendothelium also triggers coagulation which itself can be viewed as consisting of two subprocesses. The first involves a network of tightly-regulated enzymatic reactions that begins with reactions on the damaged vessel wall and continues with important reactions on the surfaces of activated platelets. The end product of this reaction network is the enzyme thrombin which activates additional platelets and creates monomeric fibrin which polymerizes into a fibrous protein gel that mechanically stabilizes the clot. This polymerization process is the second subprocess of coagulation. Both platelet aggregation and the two parts of coagulation occur in the presence of moving blood, and are strongly affected by the fluid dynamics in ways that are as yet poorly understood. One indication of the effect of different flow regimes is that clots that form in the veins, where blood flow is relatively slow, are comprised mainly of fibrin gel (and trapped red blood cells), while clots that form under the rapid flow conditions in arteries are made up largely of platelets. Understanding why there is this fundamental difference between venous and arterial clotting should give important insights into the dynamics of the clotting process.

Collagen

Modeling of Blood Clotting, Fig. 1 Schematic of platelet adhesion and cohesion. Von Willebrand Factor (vWF ) adsorbed on the subendothelial collagen binds to platelet GPIb (Red Y ) or platelet ˛II b ˇIII (Blue Y ) receptors. Soluble vWF and fibrinogen (Fbg) bind to platelet ˛II b ˇIII receptors to bridge platelet surfaces. Platelet GPIb receptors are constitutively active, while ˛II b ˇIII receptors must be mobilized when the platelet is activated. Platelet GPVI and ˛2 ˇ1 receptors for collagen itself are not shown

Models Flow carries platelets and clotting chemicals to and from the vicinity of the vessel injury. It also exerts stress on the developing thrombi which must be withstood by the platelet adhesive and cohesive bonds in

Modeling of Blood Clotting

order for a thrombus to grow and remain intact. To look at clot development beyond initial adhesion to the vascular wall, the disturbance to the flow engendered by the growth of the thrombus must be considered. Hence, models of thrombus growth involve a coupled problem of fluid dynamics, transport of cells and chemicals, and perturbation of the flow by the growing platelet mass. Most of these models have looked at events at a scale for which it is feasible to track the motion and behavior of a collection of individual platelets. Because there are approximately 250,000 platelets/L of blood, this is possible only for small vessels of, say, 50 m in diameter, such as arterioles or venules, or for the parallel plate flow chambers often used for in vitro investigations of platelet deposition under flow. To look at platelet deposition in larger vessels of the size, say, of the coronary arteries (diameter 1–2 mm), a different approach is needed. In the next two sections, both the microscale and macroscale approaches to modeling platelet deposition are described. Microscale Platelet Deposition Models Microscale platelet deposition modeling was begun by Fogelson who combined a continuum description of the fluid dynamics (using Stokes equations) with a representation of unactivated and activated platelets using Peskin’s Immersed Boundary (IB) method [11]. This line of research continues as described shortly. Others have modeled this process using the Stokes or Navier Stokes equations for the fluid dynamics and the Cellular-Potts model [14], Force-Coupling Method [12], or Boundary-Integral Method [10] to represent the platelets. Another approach is to use particle methods to represent both the fluid and the platelets [1, 7]. Fogelson and Guy [3] describe the current state of IB-based models of platelet deposition on a vascular wall. These models track the motion and behavior of a collection of individual platelets as they interact mechanically with the suspending fluid, one another, and the vessel walls. More specifically, the models track the fluid motion, the forces the fluid exerts on the growing thrombus, and the adhesive and cohesive bond forces which resist these. An Eulerian description of the fluid dynamics by means of the Navier–Stokes equations is combined with Lagrangian descriptions of each of the platelets and vessel walls. In computations, the fluid variables are determined on a regular Cartesian grid, and each platelet is represented by a discrete set of

927

elastically-linked Lagrangian IB points arrayed along a closed curve (in 2D) or surface (in 3D). Forces generated because of deformation of a platelet or by stretching of its bonds with other platelets or the vessel wall are transmitted to the fluid grid in the vicinity of each IB point. The resulting highly-localized fluid force density is how the fluid “sees” the platelets. Each IB point moves at a velocity that is a local average of the newly computed fluid velocity. In the models, nonactivated platelets are activated by contact with reactive sites on the injured wall, or through exposure to a sufficiently high concentration of a soluble chemical activator. Activation enables a platelet to cohere with other activated platelets, and to secrete additional activator. The concentration of each fluid-phase chemical activator satisfies an advection– diffusion equation with a source term corresponding to the chemical’s release from the activated platelets. To model adhesion of a platelet to the injured wall or the cohesion of activated platelets to one another, new elastic links are created dynamically between IB points on the platelet and the other surface. The multiple links, which in the models can form between a pair of activated platelets or between a platelet and the injured wall, collectively represent the ensemble of molecular bridges binding real platelets to one another or to the damaged vessel. Figure 2 shows snapshots of a portion of the computational domain during a simulation using the two-dimensional IB model. In the simulation, part of the bottom vessel wall is designated as injured and platelets that contact it, adhere to it and become activated. Two small thrombi form early in the simulation. The more upstream one grows more quickly and partially shields the downstream portion of the injured wall, slowing growth of the other thrombus. Together these thrombi disturb the flow sufficiently that few platelets contact and adhere to the downstream portion of the injured wall. Linear chains of platelets bend in response to the fluid forces and bring platelets of the two aggregates into close proximity and lead to consolidation of the adherent platelets into one larger thrombus. When a thrombus projects substantially into the vessel lumen there is a substantial strain on its most upstream attachments to the vessel wall. These bonds can break allowing the aggregate to roll downstream. (See [3] for examples of results from 3D simulations.) For the simulation in Fig. 2, simple rules were used for platelet activation and the formation and breaking

M

928

Modeling of Blood Clotting

a

b

c

d

Modeling of Blood Clotting, Fig. 2 Snapshots from a simulation using the Immersed Boundary–based model of platelet deposition. Rings depict platelets, arrows show velocity field. Unactivated platelets are elliptical. Upon activation, a platelet

becomes both less rigid and more circular. Initially there are two small thrombi which due to growth and remodeling by fluid forces merge into one larger thrombus. Plots show only a portion of computational domain

of adhesive and cohesive bonds. In recent years, much new information has become available about how a platelet detects and responds to stimuli that can induce its activation and other behaviors. The detection of stimuli, be it from a soluble chemical activator or a molecule embedded in the subendothelium or on another platelet, is mediated by surface receptors of many types. These include tens of thousands of receptors on each platelet involved in adhesion and cohesion (see Fig. 1), as well as many other receptors for soluble platelet agonists including ADP and thrombin, and hundreds to thousands of binding sites for the different enzymes and protein cofactors involved in the coagulation reactions on the platelet’s surface that are described below. Including such surface reactions as well as more sophisticated treatment of the dynamics of platelet adhesive and cohesive bonds will be essential components of extended versions of the models described here. For work in this direction see [9, 10].

the rest of the model’s dynamics, and these stresses were computed by doing a weighted average over the microscale spatial variables for each macroscale location and time. By performing this average on each term of the PDE for the bond distribution function and devising an appropriate closure approximation, an evolution equation for the bond stress tensor, that involved only the macroscale spatial variables, was derived. Comparison with the multiscale model showed that this equation still captured essential features of the multiscale behavior, in particular, the sensitivity of the bond breaking rate to strain on the bond. The PDE for the stress tensor is closely related to the Oldroyd-B equation for viscoelastic flows, but has “elastic modulus” and “relaxation time” coefficients that evolve in space and time. The divergence of this stress tensor, as well as that of a similar one from platelet-wall bonds, appears as a force density in the Navier–Stokes equations. The model also includes transport equations for the nonactivated and activated platelet concentrations, the activating chemical concentration, and the platelet– platelet and platelet–wall bond concentrations. The model has been used to explore platelet thrombosis in response to rupture of an atherosclerotic plaque. The plaque itself constricts the vessel producing a complex flow with areas of high and low shear stress. The rupture triggers platelet deposition, the outcome of which depends on the location of the rupture in the plaque and features of the flow in addition to biological parameters. The thrombus can grow to occlude the vessel and thus stop flow, or it can be torn apart by shear stresses leading to one or more thrombus fragments that are carried downstream. Model results make clear the fact that flow matters as

Large Vessel Platelet Thrombosis Models Because of the vast number of platelets involved, to study platelet thrombosis in millimeter diameter vessels, like the coronary arteries, requires a different modeling approach. Fogelson and Guy’s macroscale continuum model of platelet thrombosis [2, 3] uses density functions to describe different populations of platelets. It is derived from a multiscale model in which both the millimeter vessel scale and the micron platelet scale were explicitly treated. That model tracked continuous distributions of interplatelet and platelet-wall bonds as the bonds formed and broke, and were reoriented and stretched by flow. However, only the stresses generated by these bonds affected

Modeling of Blood Clotting

929

Fibrinogen

Unactivated Platelet

Prothrombin

Va

X

Fibrin

ATIII

APC VIIIa:IXa

Xa

Activated Platelet

ADP

APC

VIIIa

VIII

V

Va

VIIIa

VIII

V

Va

ATIII VII

THROMBIN

Va:Xa

Va

VIIa

IXa

APC

ATIII IX

X

Xa

VII

TFPI TF

TF:VIIa

TF

Subendothelium

Modeling of Blood Clotting, Fig. 3 Schematic of coagulation reactions. Magenta arrows show cellular or chemical activation processes, blue ones indicate chemical transport in the fluid or on a surface. Double headed green arrows depict binding and unbinding from a surface. Rectangles indicate surface-bound

species. Solid black lines with open arrows show enzyme action in a forward direction, while dashed black lines with open arrows show feedback action of enzymes. Red disks indicate chemical inhibitors

two simulations that differ only in whether the rupture into insoluble fibrin monomers. Once formed, the fiboccurred in high shear or low shear regions had very rin monomers spontaneously bind together into thin different outcomes [3]. strands, these strands join side to side into thicker fibers, and a branching network of these fibers grows between and around the platelets in a wall-bound platelet aggregate. Coagulation Modeling In vitro coagulation experiments are often performed under static conditions and without platelets. Coagulation Enzyme Reactions In addition to triggering platelet deposition, exposure A large concentration of phospholipid vesicles is used of the subendothelium brings the passing blood into in order to provide surfaces on which the surfacecontact with Tissue Factor (TF) molecules embedded phase coagulation reactions can occur. Most models in the matrix and initiates the coagulation process (see of the coagulation enzyme system have aimed to Fig. 3). The first coagulation enzymes are produced on describe this type of experiment. These models assume the subendothelial matrix and released into the plasma. that chemical species are well mixed and that there If they make their way through the fluid to the surface is an excess of appropriate surfaces on which the of an activated platelet, they can participate in the surface-phase reactions take place. The models do not formation of enzyme complexes on the platelet surface explicitly treat binding reactions between coagulation that continue and accelerate the pathway to thrombin proteins and these surfaces. The Hockin-Mann model production. Thrombin released from the platelet sur- [6] is a prime example and has been fit to experimental face feeds back on the enzyme network to accelerate data from Mann’s lab and used to infer, for example, its own production, activates additional platelets, and the effect of different concentrations of TF on the converts soluble fibrinogen molecules in the plasma timing and extent of thrombin production, and to

M

930

characterize the influence of chemical inhibitors in the response of the system. More recently, models that account for interactions between platelet events and coagulation biochemistry and which include treatment of flow have been introduced. The Kuharsky–Fogelson (KF) model [8] was the first such model. It looks at coagulation and platelet deposition in a thin reaction zone above a small injury and treats as well-mixed the concentration of each species in this zone. Reactions are distinguished by whether they occur on the subendothelium, in the fluid, or on the surface of activated platelets. Transport is described by a mass transfer coefficient for each fluid-phase species. Reactions on the subendothelial and platelet surfaces are limited by the availability of binding sites for the coagulation factors on these surfaces. The model consists of approximately sixty ODEs for the concentrations of coagulation proteins and platelets. The availability of subendothelial TF is a control parameter, while that of platelet binding sites depends on the number of activated platelets in the reaction zone, which in turn depends in part on the extent of thrombin production. Studies with this model and its extensions showed (1) that thrombin production depends in a threshold manner on the exposure of TF, thus providing a “switch” for turning the system on only when needed, (2) that platelets covering the subendothelium play an inhibiting role by covering subendothelial enzymes at the same time as they provide the surfaces on which other coagulation reactions occur, (3) that the flow speed and the coverage of the subendothelium by platelets have big roles in establishing the TF-threshold, (4) that the bleeding tendencies seen in hemophilias A and B and thrombocytopenia have kinetic explanations, and (5) that flow-mediated dilution may be the most important regulator of thrombin production (rather than chemical inhibitors of coagulation reactions) at least for responses to small injuries. Several of these predictions have been subsequently confirmed experimentally. The KF model was recently extended by Leiderman and Fogelson [9] to account for spatial variations and to give a much more comprehensive treatment of fluid dynamics and fluid–platelet interactions. Although studies of this model are ongoing, it has already confirmed predictions of the simpler KF model, and has given new information and insights about the spatial organization of the coagulation reactions in a growing thrombus including strong indications that transport within the

Modeling of Blood Clotting

growing thrombus is important to its eventual structure. For another spatial-temporal model that builds on the KF treatment of platelet–coagulation interactions, see [15]. Fibrin Polymerization Several modeling studies have looked at different aspects of fibrin polymerization. Weisel and Nagaswami [13] built kinetic models of fibrin strand initiation, elongation, and thickening, and drew conclusions about the relative rates at which these happen. Guy et al. [5] coupled a simple model of thrombin production to formulas derived from a kinetic gelation model to examine what limits the growth of a fibrin gel at different flow shear rates. This study gave the first (partial) explanation of the reduced fibrin deposition seen at high shear rates. Fogelson and Keener [4] developed a kinetic gelation model that allowed them to examine a possible mechanism for fibrin branch formation. They showed that branching by this mechanism results in gel structures that are sensitive to the rate at which fibrin monomer is supplied. This is in accord with observations of fibrin gels formed in vitro in which the density of branch points, pore sizes, and fiber thicknesses varied with the concentration of exogenous thrombin used.

Conclusion Mathematical models and computer simulations based on these models have contributed significant insights into the blood clotting process. These modeling efforts are just a beginning, and much remains to be done to understand how the dynamic interplay of biochemistry and physics dictates the behavior of this system. In addition to the processes described in this entry, other aspects of clotting, including the regulation of platelet responses by intraplatelet signaling pathways, the dissolution of fibrin clots by the fibrinolytic system, and the interactions between the clotting and immune systems are interesting and challenging subjects for modeling.

References 1. Filipovic, N., Kojic, M., Tsuda, A.: Modeling thrombosis using dissipative particle dynamics method. Philos. Trans. Ser. A, Math. Phys. Eng. Sci. 366, 3265–3279 (2008)

Molecular Dynamics 2. Fogelson, A.L., Guy, R.D.: Platelet-wall interactions in continuum models of platelet aggregation: formulation and numerical solution. Math. Biol. Med. 21, 293–334 (2004) 3. Fogelson, A.L., Guy, R.D.: Immersed-boundary-type models of intravascular platelet aggregation. Comput. Methods Appl. Mech. Eng. 197, 2087–2104 (2008) 4. Fogelson, A.L., Keener, J.P.: Toward an understanding of fibrin branching structure. Phys. Rev. E 81, 051922 (2010) 5. Guy, R.D., Fogelson, A.L., Keener, J.P.: Modeling fibrin gel formation in a shear flow. Math. Med. Biol. 24, 111–130 (2007) 6. Hockin, M.F., Jones, K.C., Everse, S.J., Mann, K.G.: A model for the stoichiometric regulation of blood coagulation. J. Biol. Chem. 277, 18322–18333 (2002) 7. Kamada, H., Tsubota, K., Nakamura, M., Wada, S., Ishikawa, T., Yamaguch, T.: A three-dimensional particle simulation of the formation and collapse of a primary thrombus. Int. J. Numer. Methods Biomed. Eng. 26, 488– 500 (2010) 8. Kuharsky, A.L., Fogelson, A.L.: Surface-mediated control of blood coagulation: the role of binding site densities and platelet deposition. Biophys. J. 80, 1050–1074 (2001) 9. Leiderman, K.M., Fogelson, A.L.: Grow with the flow: a spatial-temporal model of platelet deposition and blood coagulation under flow. Math. Med. Biol. 28, 47–84 (2011) 10. Mody, N.A., King, M.R.: Platelet adhesive dynamics. Part I: characterization of platelet hydrodynamic collisions and wall effects. Biophys. J. 95, 2539–2555 (2008) 11. Peskin, C.S.: The immersed boundary method. Acta Numer. 11, 479–517 (2002) 12. Pivkin, I.V., Richardson, P.D., Karniadakis, G.: Blood flow velocity effects and role of activation delay time on growth and form of platelet thrombi. Proc. Natl. Acad. Sci. 103, 17164–17169 (2006) 13. Weisel, J.W., Nagaswami, C.: Computer modeling of fibrin polymerization kinetics correlated with electron microscope and turbidity observations: clot structure and assembly are kinetically controlled. Biophys. J. 63, 111–128 (1992) 14. Xu, Z., Chen, N., Kamocka, M.M., Rosen, E.D., Alber, M.: A multiscale model of thrombus development. J. R. Soc. Interface 5, 705–722 (2008) 15. Xu, Z., Lioi, J., Mu, J., Kamocka, M.M., Liu, X., Chen, D.Z., Rosen, E.D., Alber, M.: A multiscale model of venous thrombus formation with surface-mediated control of blood coagulation cascade. Biophys. J. 98, 1723–1732 (2010)

Molecular Dynamics Benedict Leimkuhler Edinburgh University School of Mathematics, Edinburgh, Scotland, UK

931

dynamics is modeled by quantum mechanics, for example, using the Schr¨odinger equation for the nuclei and the electrons of all the atoms (a partial differential equation). Because of computational difficulties inherent in treating the quantum mechanical system, it is often replaced by a classical model. The BornOppenheimer approximation is obtained by assuming that the nuclear degrees of freedom, being much heavier than the electrons, move substantially more slowly. Averaging over the electronic wave function then results in a classical Newtonian description of the motion of N nuclei, a system of point particles with positions q1 ; q2 ; : : : ; qN 2 R3 . In practice, the Born-Oppenheimer potential energy is replaced by a semiempirical function U which is constructed by solving small quantum systems or by reference to experimental data. Denoting the coordinates of the i th atom by qi;x ; qi;y ; qi;z , and its mass by mi , the equations of motion for the i th atom are then mi

d2 qi;y @U @U d2 qi;x D  ; m D ; i 2 2 dt @qi;x dt @qi;y

mi

d2 qi;z @U D : 2 dt @qi;z

The equations need to be extended for effective treatment of boundary and environmental conditions, sometimes modeled by stochastic perturbations. Molecular dynamics is a widely used tool which in some sense interpolates between theory and experiment. It is one of the most effective general tools for understanding processes at the atomic level. The focus in this article is on classical molecular dynamics models based on semiempirical potential energy functions. For details of quantum mechanical models and related issues, see  Schr¨odinger Equation for Chemistry,  Fast Methods for Large Eigenvalues Problems for Chemistry,  Born–Oppenheimer Approximation, Adiabatic Limit, and Related Math. Issues, and  Density Functional Theory. Molecular simulation (including molecular dynamics) is treated in detail in [2, 7, 10, 18].

Background, Scope, and Application

The term molecular dynamics is used to refer to a Molecular dynamics in its current form stems from broad collection of models of systems of atoms in mo- work on hard-sphere fluid models of Alder and tion. In its most fundamental formulation, molecular Wainwright [1] dating to 1957. An article of Rahman

M

932

Molecular Dynamics

[16] described the use of smooth potentials. In 1967, Verlet [23] gave a detailed description of a general procedure, including the popular Verlet integrator and a procedure for reducing the calculation of forces by the use of neighbor lists. This article became the template for molecular dynamics studies. The use of molecular dynamics rapidly expanded in the 1970s, with the first simulations of large biological molecules, and exploded toward the end of the 1980s. As the algorithms matured, they were increasingly implemented in general-purpose software packages; many of these packages are of a high standard and are available in the public domain [6]. Molecular dynamics simulations range from just a few atoms to extremely large systems. At the time of this writing, the largest atomistically detailed molecular dynamics simulation involved 320 billion atoms and was performed using the Blue Gene/L computer at Lawrence Livermore National Laboratory, using more than 131,000 processors. The vast majority of molecular dynamics simulations are much smaller than this. In biological applications, a common size would be between 104 and 105 atoms, which allows the modeling of a protein together with a sizeable bath of water molecules. For discussion of the treatment of largescale models, refer to  Large-Scale Computing for Molecular Dynamics Simulation. Perspectives on applications of molecular modeling and simulation, in particular molecular dynamics, are discussed in many review articles, see, for example, [19] for a discussion of its use in biology.

The Potential Energy Function

Length Bond

r

s

0

Lennard−Jones

Coulomb

Molecular Dynamics, Fig. 1 Example potential energy contributions p

where r D rij D .qi;x  qj;x /2 C .qi;y  qj;y /2 C .qi;z  qj;z /2 is the distance between the two atoms. Representative graphs of the three potential energy contributions mentioned above are shown in Fig. 1. The various coefficients appearing in these formulas are determined by painstaking analysis of experimental and/or quantum mechanical simulation data; they depend not only on the types of atoms involved but also, often, on their function or specific location within the molecule and the state, i.e., the conditions of temperature, pressure, etc. In addition to two-atom potentials, there may be three or four atom terms present. For example, in a carbohydrate chain, the carbon and hydrogen atoms appear in sequence, e.g., CH3 CH2 CH2 : : : . Besides the bonds and other pair potentials, proximate triples also induce an angle-bond modeled by an energy function of the form

The complexity of molecular dynamics stems from the variety of nonlinear functional terms incorporated 'ija:bk .qi ; qj ; qk / D Bij k .†.qi ; qj ; qk /  Nij k /2 ; (additively) into U and the potentially large number of atoms needed to achieve adequate model realism. Of particular note are potential energy contri- where butions depending on the pairwise interaction of the

atoms, including Lennard-Jones, Coulombic, and co1 .qj  qi /  .qj  qk / ; †.qi ; qj ; qk / D cos valent length-bond contributions with respective defikqj  qi kkqj  qk k nitions as follows, for a particular pair of atoms labeled i; j : h while a torsional dihredral-bond potential on the angle 12  6 i ;  r=ij 'ijLJ .r/ D 4"ij r=ij between planes formed by successive triples is also incorporated. Higher-body contributions (5-, 6-, etc.) 'ijC .r/ D Cij =r; are only occasionally present. In materials science, l:b: 2 complex multibody potentials are often used, such as ' .r/ D Aij .r  rNij / ; ij

Molecular Dynamics

bond-order potentials which include a model for the local electron density [20]. One of the main limitations in current practice is the quality of the potential energy surface. While it is common practice to assume a fitted functional form for U for reasons of simplicity and efficiency, there are some popular methods which attempt to determine this “on the fly,” e.g., the Car-Parinello method models changes in the electronic structure during simulation, and other schemes may further blur the boundaries between quantum and classical approaches, such as the use of the reaxFF forcefield [22]. In addition, quantum statistical mechanics methods such as Feynman path integrals introduce a classical model that closely resembles the molecular model described above. Molecular dynamics-like models also arise in the so-called mesoscale modeling regime wherein multiatom groups are replaced by point particles or rigid bodies through a process known as coarse-graining. For example, the dissipative particle dynamics method [12] involves a conservative component that resembles molecular dynamics, together with additional stochastic and dissipative terms which are designed to conserve net momentum (and hence hydrodynamics). The molecular model may be subject to external driving perturbation forces which are time dependent and which do not have any of the functional forms mentioned above.

933

mi

X @gj d2 qi;x @U D  j ; 2 dt @qi;x j D1 @qi;x

(1)

mi

X @gj d2 qi;y @U D   j ; dt 2 @qi;y j D1 @qi;y

(2)

mi

X @gj d2 qi;z @U D  j : 2 dt @qi;z j D1 @qi;z

(3)

m

m

m

The j may be determined analytically by differentiating the constraint relations gj .q.t// D 0 twice with respect to time and making use of the second derivatives from the equations of motion. In practice, for numerical simulation, this approach is not found to be as effective as treating the constrained equations as a combined differential-algebraic system of special type (See “Construction of Numerical Methods,” below). In many cases, molecular dynamics is reduced to a system of rigid bodies interacting in a force field; then there are various options regarding the form of the equations which may be based on particle models, Euler parameters or Euler angles, quaternions, or rotation matrices. For details, refer, for example, to the book on classical mechanics of Goldstein [8].

Particle Density Controlled by Periodic Boundary Conditions

Constraints The basic molecular dynamics model often appears in modified forms that are motivated by modeling considerations. Types of systems that arise in practice include constrained systems (or systems with rigid bodies) which are introduced to coarse-grain the system or simply to remove some of the most rapid vibrational modes. An important aspect of the constraints used in molecular modeling is that they are, in most cases, holonomic. Specifically, they are usually functions of the positions only and of the form g.q/ D 0, where g is a smooth mapping from R3N to R. The constraints may be incorporated into the equations of motion using Lagrange multipliers. If there are m constraints gj .q/ D 0, j D 1; : : : ; m, then we may write the differential equations as (for i D 1; : : : ; N )

In most simulations, it is necessary to prescribe the volume V of simulation or to control the fluctuations of volume (in the case of constant pressure simulation). Since the number of atoms treated in simulation is normally held fixed (N ), the control of volume also provides control of the particle density (N=V ). Although other mechanisms are occasionally suggested, by far the most common method of controlling the volume of simulation is the use of periodic boundary conditions. Let us suppose we wish to confine our simulation to a simulation cell consisting of a cubic box with side length L and volume V D L3 . We begin by surrounding the system with a collection of 26 periodic replicas of our basic cell. In each cell copy, we assume the atoms have identical relative positions as in the basic cell and we augment the total potential energy with interaction potentials for pairs consisting of an atom of the basic cell and one in each of the neighboring cells.

M

934

Molecular Dynamics

If 'ij is the total potential energy function for inter- may include the body-centered cubic (BCC), faceactions between atoms i and j , then periodic boundary centered cubic (FCC), and hexagonal close-packed conditions with short-ranged interactions involves an (HCP) structures. extended potential energy of the form In the case of biological molecules, the initial positions are most often found by experimental techniques N 1 X (e.g., nuclear magnetic resonance imaging or x-ray X NX U pbc .q/ D 'ij .qi ; qj Ck1 Cl2 Cm3 /; crystallography). Because these methods impose artifiklm i D1 j Di C1 cial assumptions (isolation of the molecule in vacuum or frozen conditions), molecular dynamics often plays where k; l; m run over 1; 0; 1, and 1 D Le1 ; 2 D a crucial role in refining such structural information Le2 ; 3 D Le3 , where ei is the i th Euclidean basis so that the structures reported are more relevant for vector in R3 . During simulation, atoms leaving the box the liquid state conditions in which the molecule is are assumed to reenter on the opposite face; thus the found in the laboratory (in vitro) or in a living organism coordinates must be checked and possibly shifted after (in vivo). each positional step. For systems with only short-ranged potentials, the cell size is chosen large enough so that atoms do Properties of the Model not “feel” their own image; the potentials are subject to a cutoff which reduces their influence to the box For compactness, we let q be a vector of all 3N and the nearest replicas. For systems with Coulombic position coordinates of the atoms of the system, and potentials, this is not possible, and indeed it is typically we take v to be the corresponding vector of velocnecessary to calculate the forces of interaction for ities. The set of all allowed positions and velocinot just the adjacent cells but also for the distant ties is called the phase space of the system. U D periodic replicas; fortunately, the latter calculation can U.q/ is the potential energy function (assumed time be greatly simplified using a technique known as Ewald independent for this discussion), F .q/ D rU.q/ summation (see below). is the force (the gradient of potential energy), and M D diag.m1 ; m1 ; m1 ; m2 ; : : : ; mN ; mN ; mN / is the mass matrix. The molecular dynamics equations of motion may be written compactly as a first order Molecular Structure system of dimension 6N : The molecular potential energy function will have vast qP D v; M vP D F .q/: numbers of local minima corresponding to specific organizations of the atoms of the system relative to one another. The design of the energy function is typically (The notation xP refers to the time derivative of the performed in such a way as to stabilize the most quantity x.) likely structures (perhaps identified from experiment) The motion of the system beginning from prewhen the system is at mechanical equilibrium. It is scribed initial conditions (q.0/ D , v.0/ D , for impossible, using current algorithms, to identify the given vectors ;  2 R3N ) is a trajectory .q.t/; v.t//. global minimum of even a modest molecular energy The state of the system at any given time is completely landscape from an arbitrary starting point. Therefore characterized by the state at any previous time; thus the specification of appropriate initial data may be of there is a well-defined flow map ˆ of the phase space, defined for any , such that .q.t/; v.t// D importance. In the case of a system in solid state, simulations ˆ .q.t /, p.t //. Because of the form of the force typically begin from the vicinity of a textbook crystal laws, involving as they typically do an overwhelming structure, often a regular lattice in the case of a homo- repulsive component at short range (due to avoidgeneous system which describes the close-packed con- ance of overlap of the electron clouds and normally figurations of a collection of spheres. Certain lattices modeled as part of a Lennard-Jones potential), the are found to be most appropriate for given chemical separation distance between pairs of atoms at constant constituents at given environmental conditions. These energy is uniformly bounded away from zero. This

Molecular Dynamics

935

is an important distinction from gravitational N -body Flow Map dynamics where the close approaches of atoms may The energy of the system is E D E.q; v/ D dominate the evolution of the system. v T M v=2 CU.q/. It is easy to see that E is a conserved quantity (first integral) of the system since Equilibria and Normal Modes The equilibrium points of the molecular model satisfy dE D rq E  qP C rv E  vP qP D vP D 0; hence dt v D 0;

F .q/ D rq U.q/ D 0:

Thus the equilibria q  are the critical points of the potential energy function. It is possible to linearize the system at such an equilibrium point by computing the Hessian matrix W  whose ij entry is the mixed second partial derivative of U with respect to qi and qj evaluated at the equilibrium point. The equations of motion describing the motion of a material point near to such an equilibrium point are of the form dıq D ıv; dt

dıv D W  ıq; dt

D v  rU C .M v/  .M 1 rU / D 0; implying that it is a constant function of time as it is evaluated along a trajectory. Energy conservation has important consequences for the motion of the system and is often used as a simple check on the implementation of numerical methods. The flow map may possess additional invariants that depend on the model under study. For example, if the N atoms of a system interact only with each other through a pairwise potential, it is easy to see that the sum of all forces will be zero and the total momentum is therefore a conserved quantity. Likewise, for such a closed system of particles, the total angular momentum is a conserved quantity. When the positions of all the atoms are shifted uniformly by a fixed vector offset, the central forces, based only on the relative positions, are clearly invariant. We say that a closed molecular system is invariant under translation. Such a system is also invariant under rotation of all atoms about the center of mass. The symmetries and invariants are linked, as a consequence of Noether’s theorem. When periodic boundary conditions are used, the angular momentum conservation is easily seen to be P destroyed, but the total momentum i pi remains a conserved quantity, and the system is still invariant under translation. Reflecting the fact that the equations of motion qP D M 1 p, pP D rU are invariant under the simultaneous change of sign of time and momenta, we say that the system is time reversible. The equations of motion of a Hamiltonian system are also divergence free, so the volume in phase space is also preserved. The latter property can be related to a more fundamental geometric principle: Hamiltonian systems have flow maps which are symplectic, meaning that they conserve the canonical differential twoform defined by

where ıq qq  , ıv vv  . The motion of this linear system may be understood completely in terms of its eigenvalues and eigenvectors. When the equilibrium point corresponds to an isolated local minimum of the potential energy, the eigenvalues of W  are positive, and their square roots !1 ; !2 ; : : : ; !3N are proportional to the frequencies of the normal modes of oscillation of the molecule; the normal modes themselves are the corresponding eigenvectors of W  . Depending on the symmetries of the system, some of the characteristic frequencies vanish, and the number of normal modes is correspondingly reduced. The normal modes provide a useful perspective on the local dynamics of the system near an equilibrium point. As an illustration, a linear triatomic molecule consists of three atoms subject to pairwise and angle bond potentials. The energetically favored configuration is an arrangement of the atoms in a straight line. The normal modes may be viewed as directions in which the atomic configuration is deformed from the linear configuration. The triatomic molecule has six symmetries and a total of 3  3  6 D 3 normal modes, including symmetrical and asymmetrical stretches and a bending mode. More complicated systems have a wide range of normal modes which may be obtained numerically using eigenvector solvers.  Eigenvalues and Eigenvectors: Computation.  D dq1;x ^dp1;x Cdq2;x ^dp2;x C: : :CdqN;z ^dpN;z ;

M

936

Molecular Dynamics

i.e., ˆ  D , where ˆ  represents the pullback of the differential form . Another way to state this  of the flow map property is that the Jacobian matrix @ˆ @z satisfies

This follows from the Liouville equation @%=@t D LH %. Invariant distributions (i.e., those % such that L% D 0) of the equations of motion are associated to the long-term evolution of the system. Due to the chaotic nature of molecular dynamics, these invariant     @ˆ @ˆ T 0 I 0 I distributions may have a complicated structure (e.g., : D I 0 I 0 @z @z their support is often a fractal set). In some cases, the invariant distribution may appear to be dense in The various properties mentioned above have im- the phase space (or on a large region in phase space), portant ramifications for numerical method develop- although rigorous results are not available for comment. plicated models, unless stochastic perturbations are introduced. Invariant Distribution A crucial aspect of nearly all molecular dynamics mod- Constraints els is that they are chaotic systems. One consequence In the case of a constrained system, the evolution of this is that the solution depends sensitively on the is restricted to the manifold f.q; p/jg.q/ D initial data (small perturbations in the initial data will 0; g 0 .q/M 1 p D 0g. Note that the hidden constraint grow exponentially rapidly in time). The chaotic nature g 0 .q/M 1 p D 0 arises from time differentiation of of the model means that the results obtained from long the configurational constraints which must be satisfied simulations are typically independent of their precise for all points on a given trajectory. The Hamiltonian starting point (although they may depend on the energy structures, invariant properties, and the concept of or momentum). invariant distribution all have natural analogues for Denote by LH u the Lie-derivative operator for the the system with holonomic constraints. In the compact differential equations defined for any scalar function notation of this section, the constrained system may be u D u.q; p/ by written qP D M 1 p;

LH u D .M 1 p/  rq u  .rq U /  rp u;

pP D rU.q/  g 0 .q/T ;

g.q/ D 0; (6)

which represents the time derivative of u along a so- where  is now a vector of m Lagrange multipliers, lution of the Hamiltonian system. Thus e t LH qi can be g W R3N ! Rm , and g 0 is the 3N  m-dimensional represented using a formal Maclaurin series expansion: Jacobian matrix of g. e t LH qi D qi C tLH qi C D qi C t qPi C

t2 2 L qi C : : : 2 H

(4)

Construction of Numerical Methods

2

t qRi C : : : ; 2

(5) Numerical methods are used in molecular simulation for identifying stable structures, sampling the potential and hence viewing this expression as evaluated at the energy landscape (or computing averages of functions initial point, we may identify this directly with qi .t/. of the positions and momenta) and calculating dynamThus e t LH is a representation for the flow map. As a ical information; molecular dynamics may be involved shorthand, we will contract this slightly and use the no- in all three of these tasks. The term “molecular dynamtation ˆt D e tH to denote the flow map corresponding ics” often refers to the generation of trajectories by use to Hamiltonian H . of timestepping, i.e., the discretized form of the dyGiven a distribution %0 on phase space, the density namical system based on a suitable numerical method associated to the distribution will evolve under the for ordinary differential equations. In this section, we action of the exponential of the Liouvillian operator discuss some of the standard tools of molecular dynamLH D LH , i.e., ics timestepping. The basic idea of molecular dynamics timestepping %.t/ D e t LH %0 : is to define a mapping of phase space ‰h which

Molecular Dynamics

approximates the flow map ˆh on a time interval of length h. A numerical trajectory is a sequence defined by iterative composition of the approximate flow map applied to some initial point .q 0 ; p 0 /, i.e., f.q n ; p n / D ‰hn .q 0 ; p 0 /jn D 0; 1; 2; : : :g. Note the use of a superscript to indicate timesteps, to make the distinction with the components of a vector, which are enumerated using subscripts. Very long trajectories are typically needed in molecular simulation. Even a slow drift of energy (or in some other, less easily monitored physical property) will eventually destroy the usefulness of the technique. For this reason, it is crucial that algorithms be implemented following mathematical principles associated to the classical mechanics of the model and not simply based on traditional convergence analysis (or local error estimates). When very long time trajectories are involved, the standard error bounds do not justify the use of traditional numerical methods. For example, although they are all formally applicable to the problem, traditional favorites like the popular fourth-order RungeKutta method are in most cases entirely inappropriate for the purpose of molecular dynamics timestepping. Instead, molecular dynamics relies on the use of geometric integrators which mimic qualitative features of the underlying dynamical system; see  Symplectic Methods. ¨ Stormer-Verlet Method By far the most popular method for evolving the constant energy (Hamiltonian) form of molecular dynamics is one of a family of methods investigated by Carl St¨ormer in the early 1900s for particle dynamics (it is often referred to as St¨ormer’s rule, although the method was probably in use much earlier) and which was adapted by Verlet in his seminal 1967 paper on molecular dynamics. The method proceeds by the sequence of steps:

937

p nC1=2 D p n1=2  hrU.q n /; q nC1 D q n C hM 1 p nC1=2 : This method is explicit, requiring a single force evaluation per timestep, and second-order accurate, meaning that on a fixed time interval the error may be bounded by a constant times h2 , for sufficiently small h. When applied to the harmonic oscillator qP D pI pP D ! 2 q, it is found to be numerically stable for h!  2. More generally, the maximum usable stepsize is found to be inversely dependent on the frequency of fastest oscillation. Besides this stepsize restriction, by its nature, the method is directly applicable only to systems with a separable Hamiltonian (of the form H.q; p/ D T .p/ C U.q/); this means that it must be modified for use in conjunction with thermostats and other devices; it is also only suited to deterministic systems. Composition Methods The splitting framework of geometric integration is useful for constructing molecular dynamics methods. In fact, the Verlet method can be seen as a splitting method in which the simplified problems

q 0 D v1 WD ; p rU

1

d q M p D v2 WD ; 0 dt p d dt

d dt



q D v1 p

are solved sequentially, the first and last for half a timestep and the middle one for the full timestep. Note that in solving, the first system q is seen to be constant; hence the solution evolved from some given point .q; p/ is .q; p .h=2/rU.q//; this can be seen as an impulse or “kick” applied to the system. The other vector field can be viewed as inducing a “drift” (linear motion along the direction M 1 p). Thus St¨ormerh Verlet can be viewed as “kick-drift-kick.” Using the p nC1=2 D p n  rU.q n /; notation introduced in the first section of this article, 2 we may write, for the St¨ormer-Verlet method, ‰hSV D nC1 n 1 nC1=2 D q C hM p ; q exp. h2 U / exp.hK/ exp. h2 U /, where K D p T M 1 p=2 h is the kinetic energy. p nC1 D p nC1=2  rU.q nC1 /: 2 Higher-order composition methods may be constructed by using Yoshida’s method [24]. In practice, In common practice, the first and last stages are amal- the benefit of this higher-order of accuracy is only seen when sufficiently small timesteps are used (i.e., in a gamated to produce the alternative (equivalent) form

M

938

Molecular Dynamics

relatively high-accuracy regime), and this is normally not required in molecular simulation. Besides one-step methods, one also occasionally encounters the use of multistep methods such as Beeman’s method [7]. As a rule, molecular dynamicists favor explicit integration schemes whenever these are available. Multistep methods should not be confused with multiple timestepping [21]. The latter is a scheme (or rather, a family of schemes, whereby parts of the system are resolved using a smaller timestep than others. This method is very widely used since it can lead to dramatic gains in computational efficiency; however, the method may introduce resonances and instability and so should be used with caution [15]. Numerical Treatment of Constraints An effective scheme for simulating constrained molecular dynamics is the SHAKE method [17], which is a natural generalization of the Verlet method. This method is usually written in a staggered form. In [3], this method was rewritten in the “self-starting” RATTLE formulation that is more natural as a basis for mathematical study. (SHAKE and RATTLE are conjugate methods, meaning that one can be related to the other via a simple reflexive coordinate transformation; see [14].) The RATTLE method for the constrained system (6) is q nC1 D q n C hM 1 p nC1=2 ; p nC1=2 D p n 

h h rU.q n /  g 0 .q n /T n ; 2 2

0 D g.q nC1 /; p nC1 D p nC1=2 

h rU.q nC1 / 2

h  g 0 .q nC1 /T nC1 ; 2 0 D g 0 .q nC1 /M 1 p nC1 ;

In this expression, QN n WD q n C hM 1 p n  .h2 =2/rU.q n/ and G n D g 0 .q n / are both known at start of the step; thus we have m equations for m variables ƒ. This system may be solved by a Newton iteration [4] or by a Gauss-Seidel-Newton iteration [17]. Once ƒ is known, p nC1=2 and q nC1 are easily found. Equations 10 and 11 are then seen to represent a linear system for nC1 . Once this is determined, p nC1 can be found and step is complete. Note that a crucial feature of the RATTLE method is that, while implicit (it involves the solution of nonlinear equations at each step), the method only requires a single unconstrained force evaluation (F .q/ D rU.q/) at each timestep.

Properties of Numerical Methods

As the typical numerical methods used in (microcanonical) molecular dynamics may be viewed as mappings that approximate the flow map, it becomes possible to discuss them formally using the same language as one would use to discuss the flow map of the dynamical system. The global, structural, or geometric properties of the flow map approximation have important consequences in molecular simulation. The general study of numerical methods preserving geometric structures is referred to as geometric integration or sometimes, mimetic discretization. Almost all numerical methods, including all those (7) mentioned above, preserve linear symmetries such as the translation symmetry (or linear invariants like the (8) total momentum). Some numerical methods (e.g., Verlet) preserve angular momentum. The time-reversal (9) symmetry mentioned previously may be expressed in terms of the flow map by the relation

(10)

ˆt ı R D R ı ˆ1 t ;

(11) where R is the involution satisfying R.q; p/ D .q; p/. A time-reversible one-step method is one where additional multipliers n 2 Rm have been that shares this property of the flow map. For example, introduced to satisfy the “hidden constraint” on the the implicit midpoint method and the Verlet method momentum. This method is implemented in two stages. are both time reversible. Thus stepping forward a The first two equations may be inserted into the con- timestep, then changing the sign of p then stepping straint equation (9) and the multipliers  rescaled (ƒ D forward a timestep and changing again the sign of .h2 =2/n ) to obtain p returns us to our starting point. The time-reversal symmetry is often heralded as an important feature of methods, although it is unclear what role it plays g.QN n  G n ƒ/ D 0:

Molecular Dynamics

in simulations of large-scale chaotic systems. It is often used as a check for correct implementation of complicated numerical methods in software (along with energy conservation). The Symplectic Property and Its Implications Some numerical methods share the symplectic property of the flow map. Specifically those derived by Hamiltonian splitting are always symplectic, since the symplectic maps form a group under composition. The Verlet method is a symplectic method since it is constructed by composing the flow maps associated to the Hamiltonians H1 D p T M 1 p=2 and H2 D U.q/. A symplectic numerical method applied to solve a system with Hamiltonian H can be shown to closely approximate the flow map of a modified system with Hamiltonian energy HQ h D H C hr H .r/ C hrC1 H .rC1/ C : : : ; where r is the order of the method. More precisely, the truncated expansion may be used to approximate the dynamics on a bounded set (the global properties of the truncation are not known). As the number of terms is increased, the error in the approximation, on a finite domain, initially drops but eventually may be expected to increase (as more terms are taken). Thus there is an optimal order of truncation. Estimates of this optimal order have been obtained, suggesting that the approximation error can be viewed as exponentially small in h, i.e., of the form O.e 1= h /, as h ! 0. The existence of a perturbed Hamiltonian system whose dynamics mimic those of the original system is regarded as significant in geometric integration theory. One consequence of the perturbative expansion is that the energy error will remain of order O.hp / on a time interval that is long compared to the stepsize. The implication of these formal estimates for molecular dynamics has been examined in some detail [5]; their existence is a strong reason to favor symplectic methods for molecular dynamics simulation. In the case of constraints, it is possible to show that the RATTLE method (7)–(11) defines a symplectic map on the contangent bundle of the configuration manifold, while also being time reversible [14]. Theoretical and practical issues in geometric integration, including methods for constructing symplectic integrators and methods for constraints and rigid bodies, are addressed in [11, 13].

939

The Force Calculation In almost all molecular simulations, the dominant computational cost is the force calculation that must be performed at each timestep. In theory, this calculation (for the interactions of atoms within the simulation cell) requires computation of O.N 2 / square roots, where N is the number of atoms, so if N is more than a few thousand, the time spent in computing forces will dwarf all other costs. (The square roots are the most costly element of the force calculation.) If only Lennard-Jones potentials are involved, then the cost can be easily and dramatically reduced by use of a cutoff, i.e., by smoothly truncating 'L:J: at a prescribed distance, typically 2 or greater. When long-ranged Coulombic forces are involved, the situation is much different and it is necessary to evaluate (or approximate) these for both the simulation cell and neighbor cells and even for more distant cell replicas. One of the most popular schemes for evaluating the Coulombic potentials and forces is the Particle-MeshEwald (PME) method which relies on the decomposition UCoulomb D Us:r: C Ul:r: , where Us:r: and Ul:r: represent short-ranged and long-ranged components respectively; such a decomposition may be obtained by splitting the pair potentials. The long-ranged part is assumed to involve the particles in distant periodic replicas of the simulation cell. The short-ranged part is then evaluated by direct summation, while the longranged part is calculated in the Fourier domain (based P 2 , where UQ .k/ Q on Parceval’s relation) as UQ .k/j.k/j is the Fourier transform of the potential, and Q is the Fourier transform of the charge density in the central simulation cell, the latter calculated by approximation on a regular discrete lattice and use of the fast Fourier transform (FFT). The exact placement of the cutoffs (which determines what part of the computation is done in physical space and what part in reciprocal space) has a strong bearing on efficiency. Alternative approaches to handling the long-ranged forces in molecular modeling include multigrid methods and the fast multipole method (FMM) of Greengard and Rokhlin [9].

Temperature and Pressure Controls Molecular models formulated as conservative (Hamiltonian) systems usually need modification to allow specification of a particular temperature or pressure.

M

940

Thermostats may be viewed as substitution of a simplified model for an extended system in such as way as to correctly reflect energetic exchanges between a modeled system and the unresolved components. Likewise, barostats are the means by which a system is reduced while maintaining the correct exchange of momentum. The typical approach is to incorporate auxiliary variables and possibly stochastic perturbations into the equations of motion in order that the canonical ensemble, for example (in the case of a thermostat), rather than the microcanonical ensemble is preserved. For details of these methods, refer to  Sampling Techniques for Computational Statistical Physics for more details.

References 1. Alder, B.J., Wainwright, T.E.: Phase transition for a hard sphere system. J. Chem. Phys. 27, 1208–1209 (1957) 2. Allen, M.P., Tildesley, D.J.: Computer Simulation of Liquids. University Press, Oxford, UK (1988) 3. Andersen, H.: RATTLE: a “Velocity” version of the SHAKE algorithm for molecular dynamics calculations. J. Comput. Phys. 52, 2434 (1983) 4. Barth, E., Kuczera, K., Leimkuhler, B., Skeel, R.: Algorithms for constrained molecular dynamics. J. Comput. Chem 16, 1192–1209 (1995) 5. Engle, R.D., Skeel, R.D., Drees, M.: Monitoring energy drift with shadow Hamiltonians. J. Comput. Phys. 206, 432–452 (2005) 6. Examples of Popular Molecular Dynamics Software Packages Include: AMBER (http://en.wikipedia.org/ wiki/AMBER), CHARMM (http://en.wikipedia.org/ wiki/CHARMM), GROMACS (http://en.wikipedia.org/ wiki/Gromacs) and NAMD (http://en.wikipedia.org/wiki/ NAMD) 7. Frenkel, D., Smit, B.: Understanding Molecular Simulation: From Algorithms to Applications, 2nd edn. Academic Press, San Diego (2001) 8. Goldstein, H.: Classical Mechanics, 3rd edn. AddisonWesley, Lebanon (2001) 9. Greengard, L., Rokhlin, V.: A fast algorithm for particle simulations. J. Comput. Phys. 73, 325–348 (1987) 10. Haile, J.M.: Molecular Dynamics Simulation: Elementary Methods. Wiley, Chichester (1997) 11. Hairer, E., Lubich, C., Wanner, G.: Geometric Numerical Integration: Structure-Preserving Algorithms for Ordinary Differential Equations. Springer, Berlin/New York (2006) 12. Hoogerbrugge, P.J., Koelman, J.M.V.A.: Simulating microscopic hydrodynamic phenomena with dissipative particle dynamics. Europhys. Lett. 19, 155–160 (1992) 13. Leimkuhler, B., Reich, S.: Simulating Hamiltonian Dynamics. Cambridge University Press, Cambridge, UK/New York (2004)

Molecular Dynamics Simulations 14. Leimkuhler, B., Skeel, R.: Symplectic numerical integrators in constrained Hamiltonian systems. J. Comput. Phys. 112, 117125 (1994) 15. Ma, Q., Izaguirre, J., Skeel, R.D.: Verlet-I/r-RESPA is limited by nonlinear instability. SIAM J. Sci. Comput. 24, 1951–1973 (2003) 16. Rahman, A.: Correlations in the motion of atoms in liquid argon. Phys. Rev. 136, A405–A411 (1964) 17. Ryckaert, J.-P., Ciccotti, G., Berendsen, H.: Numerical integration of the Cartesian equations of motion of a system with constraints: molecular dynamics of n-Alkanes. J. Comput. Phys. 23, 327–341 (1977) 18. Schlick, T.: Molecular Modeling and Simulation. Springer, New York (2002) 19. Schlick, T., Collepardo-Guevara, R., Halvorsen, L.A., Jung, S., Xiao, X.: Biomolecular modeling and simulation: a field coming of age. Q. Rev. Biophys. 44, 191–228 (2011) 20. Tersoff, J.: New empirical approach for the structure and energy of covalent systems. Phys. Rev. B 37, 6991–7000 (1988) 21. Tuckerman, M., Berne, B.J., Martyna, G.J.: Reversible multiple time scale molecular dynamics. J. Chem. Phys. 97, 1990–2001 (1992) 22. van Duin, A.C.T., Dasgupta, S., Lorant, F., Goddard, III, W.A.: J. Phys. Chem. A 105, 9396–9409 (2001) 23. Verlet, L.: Computer “experiments” on classical fluids. I. thermodynamical properties of Lennard-Jones molecules. Phys. Rev. 159, 98–103 (1967) 24. Yoshida, H.: Construction of higher order symplectic integrators. Phys. Lett. A 150, 262–268 (1990)

Molecular Dynamics Simulations Tamar Schlick Department of Chemistry, New York University, New York, NY, USA

Overview Despite inherent limitations and approximations, molecular dynamics (MD) is considered today the gold standard computational technique by which to explore molecular motion on the atomic level. Essentially, MD can be considered statistical mechanics by numbers, or Laplace’s vision [1] of Newtonian physics on modern supercomputers [2]. The impressive progress in the development of biomolecular force fields, coupled to spectacular computer technology advances, has now made it possible to transform this vision into a reality, by overcoming the difficulty noted by Dirac of solving the equations of motion for multi-body systems [3].

Molecular Dynamics Simulations

MD’s esteemed stature stems from many reasons. Fundamentally, MD is well grounded in theory, namely, Newtonian physics: the classical equations of motion are solved repeatedly and numerically at small time increments. Moreover, MD simulations can, in theory, sample molecular systems on both spatial and temporal domains and thus address equilibrium, kinetic, and thermodynamic questions that span problems from radial distribution functions of water, to protein-folding pathways, to ion transport mechanisms across membranes. (Natural extensions to bond breaking/forming events using quantum/classicalmechanics hybrid formulations are possible.) With steady improvements in molecular force fields, careful treatment of numerical integration issues, adequate statistical analyses of the trajectories, and increasing computer speed, MD simulations are likely to improve in both quality and scope and be applicable to important molecular processes that hold many practical applications to medicine, technology, and engineering. Since successful applications were reported in the 1970s to protein dynamics, MD has now become a popular and universal tool, “as if it were the differential calculus” [4]. In fact, Fig. 1 shows that, among the modeling and simulation literature citations, MD leads as a technique. Moreover, open source MD programs have made its usage more attractive (Fig. 1b). MD is in fact one of the few tools available, by both experiment and theory, to probe molecular motion on the atomic scale. By following the equations of motion as dictated by a classical molecular mechanics force field, complex relationships among biomolecular structure, flexibility, and function can be investigated, as illustrated in the examples of Fig. 2. Today’s sophisticated dynamics programs, like NAMD or GROMACS, adapted to parallel and massively parallel computer architectures, as well as specialized hardware, have made simulations of biomolecular systems in the microsecond range routinely feasible in several weeks of computing. Special hardware/software codesign is pushing the envelope to long time frames (see separate discussion below). Though the well-recognized limitations of sampling in atomistic dynamics, as well as in the governing force fields, have led to many innovative sampling alternatives to enhance coverage of the thermally accessible conformational space, many approaches still rely on MD for local sampling.

941

a 7000 Papers by method

MD MC ab initio other methods coarse graining QM/MM

6000

5000

4000

3000

2000

1000

M 0 1976

b

1980

1985

1990 1995

2000

2005 2009

2000

2005 2009

MD packages citations

600 400 200 0 1976 1980

CHARMM Amber Gromacs NAMD

1985

1990 1995 Year

Molecular Dynamics Simulations, Fig. 1 Metrics for the rise in popularity of molecular dynamics. The number of molecular modeling and simulation papers is shown, grouped by simulation technique in (a) and by reference to an MD package in (b). Numbers are obtained from a search in the ISI Web of Science using the query words molecular dynamics, biomolecular simulation, molecular modeling, molecular simulation, and/or biomolecular modeling

Overall, MD simulations and related modeling techniques have been used by experimental and computational scientists alike for numerous applications, including to refine experimental data, shed further insights on structural and dynamical phenomena, and

942

Molecular Dynamics Simulations

Molecular Dynamics Simulations, Fig. 2 MD application examples. The illustrations show the ranges of motion or structural information that can be captured by dynamics simulations: (a) protein (DNA polymerase ) motions [110], (b) active site details gleaned from a polymerase  system complexed to

misaligned DNA [111], (c) differing protein/DNA flexibility for eight single-variant mutants of DNA polymerase  [112], and (d) DNA simulations. In all cases, solvent and salt are included in the simulation but not shown in the graphics for clarity

help resolve experimental ambiguities. See [5,6] for recent assessment studies. Specifically, applications extend to refinement of X-ray diffraction and NMR structures; interpretation of single-molecule force-extension curves (e.g., [5]) or NMR spin-relaxation in proteins (e.g., [7–9]); improvement of structure-based function predictions, for example, by predicting calcium binding sites [10]; linking of static experimental structures to implied pathways (e.g., [11, 12]); estimating the importance of quantum effects in lowering free-energy barriers of biomolecular reactions [13]; presenting structural predictions; deducing reaction mechanisms; proposing free energy pathways and associated mechanisms (e.g., [14–16]); resolving or shedding light on experimental ambiguities, for example, involving chromatin fiber structure (zigzag or solenoid) [17] or Gquadruplex architecture (parallel or antiparallel backbone arrangements) [18]; and designing new folds and

compounds, including drugs and enzymes (e.g., [19– 22]). Challenging applications to complex systems like membranes, to probe associated structures, motions, and interactions (e.g., [23–25]), further demonstrate the utility of MD for large and highly charged systems.

Historical Perspective Several selected simulations that exemplify the field’s growth are illustrated in Fig. 3 (see full details in [26]). The first MD simulation of a biological process was for the small protein BPTI (Bovine Pancreatic Trypsin Inhibitor) in vacuum [27], which revealed substantial atomic fluctuations on the picosecond timescale. DNA simulations of 12- and 24-base-pair (bp) systems in 1983 [28], in vacuum without electrostatics (of length about 90 ps), and of a DNA pentamer in 1985,

Molecular Dynamics Simulations

with 830 water molecules and 8 sodium ions and full electrostatics (of length 500 ps) [29], revealed stability problems for nucleic acids and the importance of considering long-range electrostatics interactions; in the olden days, DNA strands untwisted and separated in some cases [28]. Stability became possible with the introduction of scaled phosphate charges in other pioneering nucleic-acid simulations [30–32] and the presentation a decade later of more advanced treatments for solvation and long-range electrostatics [33]. The field developed in dazzling pace in the 1980s with the advent of supercomputers. For example, a 300 ps dynamics simulation of the protein myoglobin in 1985 [34] was considered three times longer than the longest previous MD simulation of a protein; the work indicated a slow convergence of many thermodynamic properties. System complexity also increased, as demonstrated by the ambitious, large-scale phospholipid aggregate simulation in 1989 of length 100 ps [35]. In the late 1990s, long-range electrostatics and parallel processing for speedup were widely exploited [36]. For example, a 100 ps simulation in 1997 of an estrogen/DNA system [37] sought to explain the mechanism underlying DNA sequence recognition by the protein; it used the multipole electrostatic treatment and parallel computer architecture. The dramatic effect of fast electrostatics on stability was further demonstrated by the Beveridge group [38], whose 1998 DNA simulation employing the alternative, Particle Mesh Ewald (PME), treatment uncovered interesting properties of A-tract sequences. The protein community soon jumped on the MD bandwagon with the exciting realization that proteins might be folded using the MD technique. In 1998, simulations captured reversible, temperaturedependent folding of peptides within 200 ns [39], and a landmark simulation by the late Peter Kollman made headlines by approaching the folded structure of a villin-headpiece within 1 &s [40]. This villin simulation was considered longer by three orders of magnitude than prior simulations and required 4 months of dedicated supercomputing. MD triumphs for systems that challenged practitioners due to large system sizes and stability issues soon followed, for example, the bc1 protein embedded in a phospholipid bilayer [41] for over 1 ns, and an aquaporin membrane channel protein in a lipid membrane for 5 ns [42]; both suggested mechanisms and pathways for transport.

943

The usage of many short trajectories to simulate the microsecond timescale on a new distributed computing paradigm, instead of one long simulation, was alternatively applied to protein folding using folding@home a few years later (e.g., protein G hairpin for 38 &s aggregate dynamics) [43,44]. Soon after, many long folding simulations have been reported, with specialized programs that exploit high-speed multiple-processor systems and/or specialized computing resources, such as a 1.2 &s simulation of a DNA dodecamer with a MareNostrum supercomputer [45], 1.2 &s simulation for ubiquitin with program NAMD [46], a 20 &s simulation for ˇ2 AR protein with the Desmond program [47], and small proteins like villin and a WW domain for over 1 &s [48]. Simulations of large systems, such as viruses containing one million atoms, are also noteworthy [49]. Indeed, the well-recognized timestep problem in MD integration – the requirement for small timesteps to ensure numerical stability – has limited the biological time frames that can be sampled and thus has motivated computer scientists, engineers, and biophysical scientists alike to design special-purpose hardware for MD. Examples include a transputer computer by Schulten and colleagues in the early 1990s [50], a Japanese MD product engine [51], IBM’s petaflop Blue Gene Supercomputer for protein folding [52, 53], and D. E. Shaw Research’s Anton machine [54]. A milestone of 1 ms simulations was reached with Anton in 2010 for two small proteins studied previously (BPTI and WW domain of Fip35) [55]. An extrapolation of the trends in Fig. 3 suggests that we will attain the milestone of 1-s simulations in 2015! At the same time as these longer timescales and more complex molecular systems are being simulated by atomistic MD, coarse-grained models and combinations of enhanced configurational sampling methods are emerging in tandem as viable approaches for simulating large macromolecular assemblies [56– 59]. This is because computer power alone is not likely to solve the sampling problem in general, and noted force field imperfections [60] argue for consideration of alternative states besides lowest energy forms. Long simulations also invite more careful examination of long-time trajectory stability and other numerical issues which have thus far not been possible to study. Indeed, even with quality integrators, energy drifts, resonance artifacts, and chaotic events are expected over millions and more integration steps.

M

944

Molecular Dynamics Simulations 2015, s

106

ms

105

Simulation Length (ns)

104

ms

103

102

101

100

ns

10−1

10−2

ps 1976

1980

1984

1988

1992

1996

2000

2004

2008 2010

Year

Molecular Dynamics Simulations, Fig. 3 MD evolution examples. The field’s evolution in simulation scope (system and simulation length) is illustrated through representative systems

discussed in the text. See [26] for more details. Extrapolation of the trends suggests that we will reach a second-length simulation in 2015

Molecular Dynamics Simulations

Early on, it was recognized that MD has the potential application to drug design, through the identification of motions that can be suppressed to affect function, and through estimates of binding free energies. More generally, modeling molecular structures and dynamics can help define molecular specificity and clarify functional aspects that are important for drug development [61]. Already in 1984, the hormoneproducing linear decapeptide GnRH (gonadotropinreleasing hormone) was simulated for 150 ps to explore its pharmaceutical potential [62]. Soon after the HIV protease was solved experimentally, 100 ps MD simulations suggested that a fully open conformation of the protease “flaps” may be favorable for drug access to the active site [63–67]. Recent simulations have also led to design proposals [68] and other insights into the HIV protease/drug interactions [21]. MD simulations of the HIV integrase have further suggested that inhibitors could bind in more than one orientation [69, 70], that binding modes can be selected to exploit stronger interactions in specific regions and orientations [69, 71, 72], and that different divalent-ion arrangements are associated with these binding sites and fluctuations [70]. Molecular modeling and simulation continue to play a role in structure-based drug discovery, though modern challenges in the development of new drug entities argues for a broader systems-biology multidisciplinary approach [73, 74].

Algorithmic Issues: Integration, Resonance, Fast Electrostatics, and Enhanced Sampling When solving the equations of motion numerically, the discretization timesteps must be sufficiently small to ensure reasonable accuracy as well as stability [26]. The errors (in energies and other ensemble averages) grow rapidly with timestep size, and the stability is limited by the inherent periods of the motion components, which range from 10 fs for light-atom bond stretching to milliseconds and longer for slower collective motions [75]. Moreover, the crowded frequency spectrum that spans this large range of six or more orders of magnitude is intricately coupled (see Fig. 4). For example, bond vibrations lead to angular displacements which in turn trigger side-chain motions and collective motions. Thus, integrators that have worked in other applications that utilize timescale separation, mode filtering, or mode neglect are not possible for

945

biomolecules in general. For this reason, analysis of MD integrators has focused on establishing reliable integrators that are simple to implement and as low as possible in computational requirements (i.e., dominated by one force evaluation per timestep). When mathematicians began analysis of MD integrators in the late 1980s, it was a pleasant surprise to discover that the Verlet method, long used for MD [76], was symplectic, that is, volume preserving of Hamiltonian invariants [77]. Further rigorous studies of symplectic integrators, including Verlet variants such as leap frog and velocity Verlet and constrained dynamics formulations (e.g., [26, 77]), have provided guidelines for researchers to correctly generate MD trajectories and analyze the stability of a simulation in terms of energy conservation and the robustness of the simulation with respect to the timestep size. For example, the Verlet stability limit for characteristic motions is shown in Fig. 4. Resonance artifacts in MD simulations were also later described as more general numerical stability issues that occur when timesteps are related to the natural frequency of the system as certain fractions (e.g., one third the period, see Fig. 4) [78, 79]. Highlighting resonance artifacts in MD simulations, predicting resonant timesteps, and establishing stochastic solution to resonances [80–82] have all led to an improved understanding and quality of MD simulations, including effective multiple-timestep methods [26, 83]. Note that because the value of the inner timestep in multipletimestep methods is limited by stability and resonance limits, even these methods do not produce dramatic computational advantages. The advent of efficient and particle mesh Ewald (PME) [84] and related methods [85–87] for evaluation of the long-range electrostatic interactions, which constitute the most time-consuming part of a biomolecular simulation, has made possible more realistic MD simulations without nonbonded cutoffs, as discussed above. A problem that in part remains unsolved involves the optimal integration of PME methods with multiple-timestep methods and parallelization of PME implementations. The presence of fast terms in the reciprocal Ewald component limits the outer timestep and hence the speedup [83,88–91]. Moreover, memory requirements create a bottleneck in typical PME implementations in MD simulations longer than a microsecond. This is because the contribution of the long-range electrostatic forces imposes a global data dependency on all the system charges; in practice, this implies

M

Low Frequency Motions Time (sec)

102

21

21. Reverse translocation of tRNA in ribosome 75 s

101

100 19

19. Site juxtaposition in supercoiled DNA ~5 ⫻ 10−1s

Protein folding

20 20. MerP folding 0.8 s

10−1

10−2

1.6 ms

1.8 ms

2.0 ms

2.2 ms

2.4 ms

2.6 ms

2.8 ms

3.0 ms

3.2 ms

3.4 ms

3.6 ms

3.8 ms

4.0 ms

4.2 ms

10−3

18 18. NTL9(1-39) folding −3 1.5 ⫻ 10 s

10−4

10−5 15 −6

10

17 16 16. Protein G β-hairpin folding 4.7 ⫻ 10−6s

14 13

12 −8

15. Villin folding −6 4.3 ⫻ 10 s 14. Translocation of ion across a channel −6 ~10 s

13. BPTI side-chain rotations −7 3 ⫻ 10 s

10−7

12. Global DNA bending ~5 ⫻ 10−8s

17. Fip35 folding 10−5s

Collective subgroup motion

10

11

10−9

10 9

11. β-heptapeptide folding 1.44 ⫻ 10−8s

10. Pol β open-to-closed motion 10−9s

9. Ubiquitin diffusion time −10 7.4 ⫻ 10 s

28 Å

34 Å

10−10

10−11

7. Adenine out-of-plane butterfly motion 1.34 ⫻ 10−13s

C H

H

C

HN H

N

N

N H

N

10−12

8

8. Global DNA twisting ~10−12s

10−13

7

H

H H

C

O

3 3. Double bond O 2. Triple bond 2 stretch −14 −14 stretch 1.96 ⫻ 10 s 10 1. Light atom stretch 1.59 ⫻ 10−14s −15 (5.41 fs/6.24 fs) 9.26 ⫻ 10 s (4.38 fs/5.05 fs) (2.55 fs/2.95 fs)

6 45 1

C

H C

C C

H

H

4. CH2 bend −14 2.28 ⫻ 10 s (6.29 fs/7.26 fs)

H

H H

5. CH2 twist −14 2.56 ⫻ 10 s (7.06 fs/8.15 fs)

C

H H

H

C

6. CH2 wag −14 2.61 ⫻ 10 s (7.19 fs/8.31 fs)

High Frequency Motions

Molecular Dynamics Simulations, Fig. 4 Biomolecular Motion Ranges. Representative motions with associated periods are shown, along with associated timestep limits for third-order resonance and linear stability for high-frequency modes (Time periods for the highest frequency motions are derived from frequency data in [113]. Values for adenine butterfly motion are obtained from [114]. Timescales for DNA twisting, bending, and site juxtaposition are taken from [26]. Pol ˇ’s estimated 1 ns open-to-closed hinge motion is taken from [115]. The diffusion

time for ubiquitin is from [116]. The transition-path time for local conformational changes in BPTI is obtained from [55]. The folding timescales are taken as follows: ˇ-heptapeptide [39], C-terminal ˇ-hairpin of protein G [117], villin headpiece subdomain [118], Fip 35 [55], N-terminal domain of ribosomal protein L9 [NTL9(1–39)] [119], and mercury binding protein (MerP) [120]. The approximate time for a single ion to traverse a channel is from [121]. The reverse translocation time of tRNA within the ribosome is from [122])

Molecular Dynamics Simulations

communication problems [92]. Thus, much work goes into optimizing associated mesh sizes, precision, and sizes of the real and inverse spaces to delay the communication bottleneck as possible (e.g., [54]), but overall errors in long simulations are far from trivial [83, 93]. In addition to MD integration and electrostatic calculations, sampling the vast configurational space has also triggered many innovative approaches to capture “rare events.” The many innovative enhanced sampling methods are either independent of MD or based on MD. In the former class, as recently surveyed [58, 94, 95], are various Monte Carlo approaches, harmonic approximations, and coarse-grained models. These can yield valuable conformational insights into biomolecular structure and flexibility, despite altered kinetics. Although Monte Carlo methods are not always satisfactory for large systems on their own right, they form essential components of more sophisticated methods [59] like transition path sampling [96] and Markov chain Monte Carlo sampling [97]. More generally, MD-based methods for enhanced sampling of biomolecules can involve modification of the potential (like accelerated MD [98]), the simulation protocol (like replica-exchange MD or REMD [99]), or the algorithm. However, global formulations such as transition path sampling [96, 100], forward flux simulation [101], and Markov state models [102] are needed more generally not only to generate more configurations or to suggest mechanistic pathways but also to compute free energy profiles for the reaction and describe detailed kinetics profiles including reaction rates. There are many successful reports of using tailored enhanced sampling methods (e.g., [11, 103–109]), but applications at large to biomolecules, especially in the presence of incomplete experimental endpoints, remain a challenge.

Conclusion When executed with vigilance in terms of problem formulation, implementational details, and force field choice, atomic-level MD simulations present an attractive technique to visualize molecular motion and estimate many properties of interest in the thermally accessible conformational space, from equilibrium distributions to configurational transitions and pathways. The application scope can be as creative as the scientist

947

performing the simulation: from structure prediction to drug design to new mechanistic hypotheses about a variety of biological processes, for a single molecule or a biomolecular complex. Extensions of MD to enhanced sampling protocols and coarse-graining simulations are further enriching the tool kit that modelers possess, and dramatic advances in computer speed, including specialized computer architecture, are driving the field through exciting milestones. Perhaps more than any other modeling technique, proper technical details in simulation implementation and analysis are crucial for the reliability of the biological interpretations obtained from MD trajectories. Thus, both expertise and intuition are needed to dissect correct from nonsensical behavior within the voluminous data that can be generated quickly. In the best cases, MD can help sift through conflicting experimental information and provide new biological interpretations, which can in turn be subjected to further experimentation. Still, MD should never be confused for reality! As larger biomolecular systems and longer simulation times become possible, new interesting questions also arise and need to be explored. These concern the adequacy of force fields as well as long-time stability and error propagation of the simulation algorithms. For example, a 10 &s simulation of the ˇ-protein Fip35 [46] did not provide the anticipated folded conformation nor the folding trajectory from the extended state, as expected from experimental measurements; it was determined subsequently that force field inaccuracies for ˇ-protein interactions affect the results, and not incorrect sampling [60]. In addition, the effects of shortcuts often taken (e.g., relatively large, 2.5 fs, timesteps, which imply corruption by third-order resonances as shown in Fig. 4, and rescaling of velocities to retain ensemble averages) will have to be examined in detail over very long trajectories. The rarity of large-scale conformational transitions and the stochastic and chaotic nature of MD simulations also raise the question as to whether long simulations of one biomolecular system rather than many shorter simulations provide more cost-effective, statistically sound, and scientifically relevant information. Given the many barriers we have already crossed in addressing the fundamental sampling problem in MD, it is likely that new innovative approaches will be invented by scientists in allied fields to render MD

M

948

simulations better and faster for an ever-growing level of biological system sophistication. Acknowledgements I thank Rosana Collepardo, Meredith Foley, and Shereef Elmetwaly for assistance with the figures.

References 1. de Laplace, P.S.: Oeuvres Compl`etes de Laplace. Th´eorie Analytique des Probabilit´es, vol. VII, third edn. GauthierVillars, Paris (1820) 2. Schlick, T.: Pursuing Laplace’s vision on modern computers. In: Mesirov, J.P., Schulten, K., Sumners, D.W. (eds.) Mathematical Applications to Biomolecular Structure and Dynamics. IMA Volumes in Mathematics and Its Applications, vol. 82, pp. 219–247. Springer, New York (1996) 3. Dirac, P.A.M.: Quantum mechanics of many-electron systems. Proc. R Soc. Lond. A123, 714–733 (1929) 4. Maddox, J.: Statistical mechanics by numbers. Nature 334, 561 (1989) 5. Lee, E.H., Hsin, J., Sotomayor, M., Comellas, G., Schulten, K.: Discovery through the computational microscope. Structure 17, 1295–1306 (2009) 6. Schlick, T., Collepardo-Guevara, R., Halvorsen, L.A., Jung, S., Xiao, X.: Biomolecular modeling and simulation: a field coming of age. Q. Rev. Biophys. 44, 191–228 (2011) 7. Tsui, V., Radhakrishnan, I., Wright, P.E., Case, D.A.: NMR and molecular dynamics studies of the hydration of a zinc finger-DNA complex. J. Mol. Biol. 302, 1101–1117 (2000) 8. Case, D.A.: Molecular dynamics and NMR spin relaxation in proteins. Acc. Chem. Res. 35, 325–331 (2002) 9. Henzler-Wildman, K.A., Thai, V., Lei, M., Ott, M., WolfWatz, M., Fenn, T., Pozharski, E., Wilson, M.A., Petsko, G.A., Karplus, M.: Intrinsic motions along an enzymatic reaction trajectory. Nature 450, 838–844 (2007) 10. Altman, R., Radmer, R., Glazer, D.: Improving structurebased function prediction using molecular dynamics. Structure 17, 919–929 (2009) 11. Radhakrishnan, R., Schlick, T.: Orchestration of cooperative events in DNA synthesis and repair mechanism unraveled by transition path sampling of DNA polymerase ˇ’s closing. Proc. Natl. Acad. Sci. U.S.A. 101, 5970–5975 (2004) 12. Golosov, A.A., Warren, J.J., Beese, L.S., Karplus, M.: The mechanism of the translocation step in DNA replication by DNA polymerase I: a computer simulation. Structure 18, 83–93 (2010) 13. Hu, H., Elstner, M., Hermans, J.: Comparison of a QM/MM force field and molecular mechanics force fields in simulations of alanine and glycine “dipeptides” (AceAla-Nme and Ace-Gly-Nme) in water in relation to the problem of modeling the unfolded peptide backbone in solution. Proteins Struct. Funct. Genet. 50, 451–463 (2003) 14. Radhakrishnan, R., Schlick, T.: Fidelity discrimination in DNA polymerase ˇ: differing closing profiles for a mismatched G:A versus matched G:C base pair. J. Am. Chem. Soc. 127, 13245–13252 (2005) 15. Karplus, M., Kuriyan, J.: Molecular dynamics and protein function. Proc. Natl. Acad. Sci. U.S.A. 102, 6679–6685 (2005)

Molecular Dynamics Simulations 16. Faraldo-Gomez, J., Roux, B.: On the importance of a funneled energy landscape for the assembly and regulation of multidomain Src tyrosine kinases. Proc. Natl. Acad. Sci. U.S.A. 104, 13643–13648 (2007) 17. Grigoryev, S.A., Arya, G., Correll, S., Woodcock, C.L., Schlick, T.: Evidence for heteromorphic chromatin fibers from analysis of nucleosome interactions. Proc. Natl. Acad. Sci. U.S.A. 106, 13317–13322 (2009) 18. Campbell, H., Parkinson, G.N., Reszka, A.P., Neidle, S.: Structural basis of DNA quadruplex recognition by an acridine drug. J. Am. Chem. Soc. 130, 6722–6724 (2008) 19. Neidle, S., Read, M., Harrison, J., Romagnoli, B., Tanious, F., Gowan, S., Reszka, A., Wilson, D., Kelland, L.: Structure-based design of selective and potent G quadruplex-mediated telomerase inhibitors. Proc. Natl. Acad. Sci. USA 98, 4844–4849 (2001) 20. Baker, D., Kuhlman, B., Dantas, G., Ireton, G., Varani, G., Stoddard, B.: Design of a novel globular protein fold with atomic-level accuracy. Science 302, 1364–1368 (2003) 21. Hornak, V., Simmerling, C.: Targeting structural flexibility in HIV-1 protease inhibitor binding. Drug Discov. Today 12, 132–138 (2007) 22. Jiang, L., Althoff, E.A., Clemente, F.R., Doyle, L., R¨othlisberger, D., Zanghellini, A., Gallaher, J.L., Betker, J.L., Tanaka, F., Barbas, C.F., III, Hilvert, D., Houk, K.N., Stoddard, B.L., Baker, D.: De novo computational design of retro-aldol enzymes. Science 319, 1387–1391 (2008) 23. Grossfield, A., Pitman, M.C., Feller, S.E., Soubias, O., Gawrisch, K.: Internal hydration increases during activation of the G-protein-coupled receptor rhodopsin. J. Mol. Biol. 381, 478–486 (2008) 24. Khelashvili, G., Grossfield, A., Feller, S.E., Pitman, M.C., Weinstein, H.: Structural and dynamic effects of cholesterol at preferred sites of interaction with rhodopsin identified from microsecond length molecular dynamics simulations. Proteins 76, 403–417 (2009) 25. Vasquez, V., Sotomayor, M., Cordero-Morales, J., Schulten, K., Perozo, E.: A structural mechanism for MscS gating in lipid bilayers. Science 321, 1210–1214 (2008) 26. Schlick, T.: Molecular Modeling: An Interdisciplinary Guide, second edn. Springer, New York (2010) 27. McCammon, J.A., Gelin, B.R., Karplus, M.: Dynamics of folded proteins. Nature 267, 585–590 (1977) 28. Levitt, M.: Computer simulation of DNA double-helix dynamics. Cold Spring Harb. Symp. Quant. Biol. 47, 251– 275 (1983) 29. Seibel, G.L., Singh, U.C., Kollman, P.A.: A molecular dynamics simulation of double-helical B-DNA including counterions and water. Proc. Natl. Acad. Sci. U.S.A. 82, 6537–6540 (1985) 30. Prabhakaran, M., Harvey, S.C., Mao, B., McCammon, J.A.: Molecular dynamics of phenylanlanine transfer RNA. J. Biomol. Struct. Dyn. 1, 357–369 (1983) 31. Harvey, S.C., Prabhakaran, M., Mao, B., McCammon, J.A.: Phenylanine transfer RNA: molecular dynamics simulation. Science 223, 1189–1191 (1984) 32. Tidor, B., Irikura, K.K., Brooks, B.R., Karplus, M.: Dynamics of DNA oligomers. J. Biomol. Struct. Dyn. 1, 231–252 (1983) 33. Cheatham, T.E., III, Miller, J.L., Fox, T., Darden, T.A., Kollman, P.A.: Molecular dynamics simulations

Molecular Dynamics Simulations

34.

35.

36.

37.

38.

39.

40.

41.

42.

43.

44.

45.

46.

47.

48.

of solvated biomolecular systems: the particle mesh Ewald method leads to stable trajectories of DNA, RNA, and proteins. J. Am. Chem. Soc. 117, 4193–4194 (1995) Levy, R.M., Sheridan, R.P., Keepers, J.W., Dubey, G.S., Swaminathan, S., Karplus, M.: Molecular dynamics of myoglobin at 298K. Results from a 300-ps computer simulation. Biophys. J. 48, 509–518 (1985) Wendoloski, J.J., Kimatian, S.J., Schutt, C.E., Salemme, F.R.: Molecular dynamics simulation of a phospholipid micelle. Science 243, 636–638 (1989) Schlick, T., Skeel, R.D., Br¨unger, A.T., Kal´e, L.V., Board, J.A., Jr., Hermans, J., Schulten, K.: Algorithmic challenges in computational molecular biophysics. J. Comput. Phys. 151, 9–48 (1999) (Special Volume on Computational Biophysics) Kosztin, D., Bishop, T.C., Schulten, K.: Binding of the estrogen receptor to DNA: the role of waters. Biophys. J. 73, 557–570 (1997) Young, M.A., Beveridge, D.L.: Molecular dynamics simulations of an oligonucleotide duplex with adenine tracts phased by a full helix turn. J. Mol. Biol. 281, 675–687 (1998) Daura, X., Jaun, B., Seebach, D., Van Gunsteren, W.F., Mark, A.: Reversible peptide folding in solution by molecular dynamics simulation. J. Mol. Biol. 280, 925– 932 (1998) Duan, Y., Kollman, P.A.: Pathways to a protein folding intermediate observed in a 1-microsecond simulation in aqueous solution. Science 282, 740–744 (1998) Izrailev, S., Crofts, A.R., Berry, E.A., Schulten, K.: Steered molecular dynamics simulation of the Rieske subunit motion in the cytochrome bc1 complex. Biophys. J. 77, 1753– 1768 (1999) Tajkhorshid, E., Nollert, P., Ø Jensen, M., Miercke, L.J.W., O’Connell, J., Stroud, R.M., Schulten, K.: Control of the selectivity of the aquaporin water channel family by global orientational tuning. Science 296, 525–530 (2002) Snow, C.D., Nguyen, H., Pande, V.S., Gruebele, M.: Absolute comparison of simulated and experimental protein folding dynamics. Nature 420, 102–106 (2002) Ensign, D.L., Kasson, P.M., Pande, V.S.: Heterogeneity even at the speed limit of folding: large-scale molecular dynamics study of a fast-folding variant of the villin headpiece. J. Mol. Biol. 374, 806–816 (2007) P´erez, A., Luque, J., Orozco, M.: Dynamics of B-DNA on the microsecond time scale. J. Am. Chem. Soc. 129, 14739–14745 (2007) Freddolino, P.L., Liu, F., Gruebele, M., Schulten, K.: Tenmicrosecond molecular dynamics simulation of a fastfolding WW domain. Biophys. J. 94, L75–L77 (2008) Dror, R.O., Arlow, D.H., Borhani, D.W., Ø Jensen, M., Piana, S., Shaw, D.E.: Identification of two distinct inactive conformations of the 2-adrenergic receptor reconciles structural and biochemical observations. Proc. Natl. Acad. Sci. U.S.A. 106, 4689–4694 (2009) Mittal, J., Best, R.B.: Tackling force-field bias in protein folding simulations: folding of villin HP35 and Pin WW domains in explicit water. Biophys. J. 99, L26–L28 (2010)

949 49. Freddolino, P.L., Arkhipov, A.S., Larson, S.B., McPherson, A., Schulten, K.: Molecular dynamics simulations of the complete satellite tobacco mosaic virus. Structure 14, 437–449 (2006) 50. Heller, H., Schulten, K.: Parallel distributed computing for molecular dynamics: simulation of large heterogeneous systems on a systolic ring of transputers. Chem. Des. Autom. News 7, 11–22 (1992) 51. Toyoda, S., Miyagawa, H., Kitamura, K., Amisaki, T., Hashimoto, E., Ikeda, H., Kusumi, A., Miyakawa, N.: Development of MD engine: high-speed accelerator with parallel processor design for molecular dynamics simulations. J. Comput. Chem. 20, 185–199 (1999) 52. Butler, D.: IBM promises scientists 500-fold leap in supercomputing power. . . . . . and a chance to tackle protein structure. Nature 402, 705–706 (1999) 53. Zhou, R., Eleftheriou, M., Hon, C.-C., Germain, R.S., Royyuru, A.K., Berne, B.J.: Massively parallel molecular dynamics simulations of lysozyme unfolding. IBM J. Res. Dev. 52, 19–30 (2008) 54. Shaw, D.E., Dror, R.O., Salmon, J.K., Grossman, J.P., Mackenzie, K.M., Bank, J.A., Young, C., Deneroff, M.M., Batson, B., Bowers, K.J., Chow, E., Eastwood, M.P., Ierardi, D.J., Klepeis, J.L., Kuskin, J.S., Larson, R.H., Lindorff-Larsen, K., Maragakis, P., Moraes, M.A., Piana, S., Shan, Y., Towles, B.: Millisecond-scale molecular dynamics simulations on Anton. In: SC ’09: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, San Diego, pp. 1–11. ACM (2009) 55. Shaw, D.E., Maragakis, P., Lindorff-Larsen, K., Piana, S., Dror, R.O., Eastwood, M.P., Bank, J.A., Jumper, J.M., Salmon, J.K., Shan, Y., Wriggers, W.: Atomic-level characterization of the structural dynamics of proteins. Science 330, 341–346 (2010) 56. Lei, H., Duan, Y.: Improved sampling methods for molecular simulation. Curr. Opin. Struct. Biol. 17, 187–191 (2007) 57. Klein, M.L., Shinoda, W.: Large-scale molecular dynamics simulations of self-assembling systems. Science 321, 798–800 (2008) 58. Schlick, T.: Monte Carlo, harmonic approximation, and coarse-graining approaches for enhanced sampling of biomolecular structure. F1000 Biol. Rep. 1, 48 (2009) 59. Schlick, T.: Molecular-dynamics based approaches for enhanced sampling of long-time, large-scale conformational changes in biomolecules. F1000 Biol. Rep. 1, 51 (2009) 60. Freddolino, P.L., Park, S., Roux, B., Schulten, K.: Force field bias in protein folding simulations. Biophys. J. 96, 3772–3780 (2009) 61. Schwede, T., Sali, A., Honig, B., Levitt, M., Berman, H.M., Jones, D., Brenner, S.E., Burley, S.K., Das, R., Dokholyan, N.V., Dunbrack, R.L., Jr., Fidelis, K., Fiser, A., Godzik, A., Huang, Y.J., Humblet, C., Jacobson, M.P., Joachimiak, A., Krystek, S.R., Jr., Kortemme, T., Kryshtafovych, A., Montelione, G.T., Moult, J., Murray, D., Sanchez, R., Sosnick, T.R., Standley, D.M., Stouch, T., Vajda, S., Vasquez, M., Westbrook, J.D., Wilson, I.A.: Outcome of a workshop on applications of protein models in biomedical research. Structure 17, 151–159 (2009)

M

950 62. Struthers, R.S., Rivier, J., Hagler, A.T.: Theoretical simulation of conformation, energetics, and dynamics in the design of GnRH analogs. Trans. Am. Crystallogr. Assoc. 20, 83–96 (1984). Proceedings of the Symposium on Molecules in Motion, University of Kentucky, Lexington, Kentucky, May 20–21, (1984) 63. Harte, W.E., Jr., Swaminathan, S., Beveridge, D.L.: Molecular dynamics of HIV-1 protease. Proteins Struct. Funct. Genet. 13, 175–194 (1992) 64. Collins, J.R., Burt, S.K., Erickson, J.W.: Flap opening in HIV-1 protease simulated by activated’ molecular dynamics. Nat. Struct. Mol. Biol. 2, 334–338 (1995) 65. Hamelberg, D., McCammon, J.A.: Fast peptidyl cistrans isomerization within the flexible Gly-rich flaps of HIV-1 protease. J. Am. Chem. Soc. 127, 13778–13779 (2005) 66. Tozzini, V., McCammon, J.A.: A coarse grained model for the dynamics of flap opening in HIV-1 protease. Chem. Phys. Lett. 413, 123–128 (2005) 67. Hornak, V., Okur, A., Rizzo, R.C., Simmerling, C.: HIV-1 protease flaps spontaneously open and reclose in molecular dynamics simulations. Proc. Natl. Acad. Sci. U.S.A. 103, 915–920 (2006) 68. Scott, W.R., Schiffer, C.A.: Curling of flap tips in HIV-1 protease as a mechanism for substrate entry and tolerance of drug resistance. Structure 8, 1259–1265 (2000) 69. Schames, J.R., Henchman, R.H., Siegel, J.S., Sotriffer, C.A., Ni, H., McCammon, J.A.: Discovery of a novel binding trench in HIV integrase. J. Med. Chem. 47, 1879– 1881 (2004) 70. Perryman, A.L., Forli, S., Morris, G.M., Burt, C., Cheng, Y., Palmer, M.J., Whitby, K., McCammon, J.A., Phillips, C., Olson, A.J.: A dynamic model of HIV integrase inhibition and drug resistance. J. Mol. Biol. 397, 600–615 (2010) 71. Lin, J.H., Perryman, A.L., Schames, J.R., McCammon, J.A.: Computational drug design accommodating receptor flexibility: the relaxed complex scheme. J. Am. Chem. Soc. 124, 5632–5633 (2002) 72. Hazuda, D.J., Anthony, N.J., Gomez, R.P., Jolly, S.M., Wai, J.S., Zhuang, L., Fisher, T.E., Embrey, M., Guare, J.P., Jr., Egbertson, M.S., Vacca, J.P., Huff, J.R., Felock, P.J., Witmer, M.V., Stillmock, K.A., Danovich, R., Grobler, J., Miller, M.D., Espeseth, A.S., Jin, L., Chen, I.W., Lin, J.H., Kassahun, K., Ellis, J.D., Wong, B.K., Xu, W., Pearson, P.G., Schleif, W.A., Cortese, R., Emini, E., Summa, V., Holloway, M.K., Young, S.D.: A naphthyridine carboxamide provides evidence for discordant resistance between mechanistically identical inhibitors of HIV-1 integrase. Proc. Natl. Acad. Sci. U.S.A. 101, 11233–11238 (2004) 73. Kitano, H.: A robustness-based approach to systemsoriented drug design. Nat. Rev. Drug Discov. 6, 202–210 (2007) 74. Munos, B.: Lessons from 60 years of pharmaceutical innovation. Nat. Rev. 8, 959–968 (2009) 75. Schlick, T.: Some failures and successes of long-timestep approaches for biomolecular simulations. In: Deuflhard, P., Hermans, J., Leimkuhler, B., Mark, A.E., Reich, S., Skeel, R.D. (eds.) Computational Molecular Dynamics: Challenges, Methods, Ideas – Proceedings of the 2nd

Molecular Dynamics Simulations

76.

77.

78.

79.

80.

81.

82.

83.

84.

85.

86.

87. 88.

89.

90.

91.

92.

International Symposium on Algorithms for Macromolecular Modelling, Berlin, May 21–24, 1997. Lecture Notes in Computational Science and Engineering (Series Eds. Griebel, M., Keyes, D.E., Nieminen, R.M., Roose, D., Schlick, T.), vol. 4, pp. 227–262. Springer, Berlin (1999) Verlet, L.: Computer ‘experiments’ on classical fluids: I. Thermodynamical properties of Lennard-Jones molecules. Phys. Rev. 159(1), 98–103 (1967) Leimkuhler, B., Reich, S.: Simulating Hamiltonian Dynamics. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, Cambridge (2004) Mandziuk, M., Schlick, T.: Resonance in the dynamics of chemical systems simulated by the implicit-midpoint scheme. Chem. Phys. Lett. 237, 525–535 (1995) Schlick, T., Mandziuk, M., Skeel, R.D., Srinivas, K.: Nonlinear resonance artifacts in molecular dynamics simulations. J. Comput. Phys. 139, 1–29 (1998) Schlick, T., Barth, E., Mandziuk, M.: Biomolecular dynamics at long timesteps: bridging the timescale gap between simulation and experimentation. Annu. Rev. Biophys. Biomol. Struct. 26, 179–220 (1997) Barth, E., Schlick, T.: Overcoming stability limitations in biomolecular dynamics: I. combining force splitting via extrapolation with Langevin dynamics in LN . J. Chem. Phys. 109, 1617–1632 (1998) Sweet, C.R., Petrine, P., Pande, V.S., Izaguirre, J.A.: Normal mode partitioning of Langevin dynamics for biomolecules. J. Chem. Phys. 128, 145101 (2008) Morrone, J.A., Zhou, R., Berne, B.J.: Molecular dynamics with multiple time scales: how to avoid pitfalls. J. Chem. Theory Comput. 6, 1798–1804 (2010) Essmann, U., Perera, L., Berkowitz, M.L., Darden, T., Lee, H., Pedersen, L.G.: A smooth particle mesh Ewald method. J. Chem. Phys. 103, 8577–8593 (1995) Greengard, L., Rokhlin, V.: A new version of the fast multipole method for the Laplace equation in three dimensions. Acta Numer. 6, 229–269 (1997) Skeel, R.D., Tezcan, I., Hardy, D.J.: Multiple grid methods for classical molecular dynamics. J. Comput. Chem. 23, 673–684 (2002) Duan, Z.-H., Krasny, R.: An Ewald summation based multipole method. J. Chem. Phys. 113, 3492–3495 (2000) Stuart, S.J., Zhou, R., Berne, B.J.: Molecular dynamics with multiple time scales: the selection of efficient reference system propagators. J. Chem. Phys. 105, 1426–1436 (1996) Procacci, P., Marchi, M., Martyna, G.J.: Electrostatic calculations and multiple time scales in molecular dynamics simulation of flexible molecular systems. J. Chem. Phys. 108, 8799–8803 (1998) Zhou, R., Harder, E., Xu, H., Berne, B.J.: Efficient multiple time step method for use with Ewald and particle mesh Ewald for large biomolecular systems. J. Chem. Phys. 115, 2348–2358 (2001) Qian, X., Schlick, T.: Efficient multiple-timestep integrators with distance-based force splitting for particle-meshEwald molecular dynamics simulations. J. Chem. Phys. 116, 5971–5983 (2002) Fitch, B.G., Rayshubskiy, A., Eleftheriou, M., Ward, T.J.C., Giampapa, M., Pitman, M.C., Germain, R.S.: Blue

Molecular Geometry Optimization, Models

93. 94. 95.

96.

97.

98.

99.

100.

101.

102.

103.

104.

105.

106.

107.

108.

109.

matter: approaching the limits of concurrency for classical molecular dynamics. In: Supercomputing, 2006. SC’06. Proceedings of the ACM/IEEE SC 2006 Conference, pp 44. ACM (2006) Snir, M.: A note on N-body computations with cutoffs. Theory Comput. Syst. 37, 295–318 (2004) Earl, D.J., Deem, M.W.: Monte Carlo simulations. Methods Mol. Biol. 443, 25–36 (2008) Liwo, A., Czaplewski, C., Oldziej, S., Scheraga, H.A.: Computational techniques for efficient conformational sampling of proteins. Curr. Opin. Struct. Biol. 18, 134– 139 (2008) Dellago, C., Bolhuis, P.G.: Transition path sampling simulations of biological systems. Top. Curr. Chem. 268, 291–317 (2007) Pan, A.C., Roux, B.: Building Markov state models along pathways to determine free energies and rates of transitions. J. Chem. Phys. 129, 064107 (2008) Grant, B.J., Gorfe, A.A., McCammon, J.A.: Large conformational changes in proteins: signaling and other functions. Curr. Opin. Struct. Biol. 20, 142–147 (2010) Sugita, Y., Okamoto, Y.: Replica-exchange molecular dynamics methods for protein folding. Chem. Phys. Lett. 314, 141–151 (1999) Bolhuis, P.G., Chandler, D., Dellago, C., Geissler, P.L.: Transition path sampling: throwing ropes over rough mountain passes, in the dark. Annu. Rev. Phys. Chem. 53, 291–318 (2002) Borrero, E.E., Escobedo, F.A.: Optimizing the sampling and staging for simulations of rare events via forward flux sampling schemes. J. Chem. Phys. 129, 024115 (2008) No´e, F., Fischer, S.: Transition networks for modeling the kinetics of conformational change in macromolecules. Curr. Opin. Struct. Biol. 8, 154–162 (2008) No´e, F., Horenko, I., Sch¨utte, C., Smith, J.C.: Hierarchical analysis of conformational dynamics in biomolecules: transition networks of metastable states. J. Chem. Phys. 126, 155102 (2007) Ozkan, S.B., Wu, G.A., Chodera, J.D., Dill, K.A.: Protein folding by zipping and assembly. Proc. Natl. Acad. Sci. U.S.A. 104, 11987–11992 (2007) No´e, F., Schutte, C., Vanden-Eijnden, E., Reich, L., Weikl, T.R.: Constructing the equilibrium ensemble of folding pathways from short off-equilibrium simulations. Proc. Natl. Acad. Sci. U.S.A. 106, 19011–19016 (2009) Berezhkovskii, A., Hummer, G., Szabo, A.: Reactive flux and folding pathways in network models of coarse-grained protein dynamics. J. Chem. Phys. 130, 205102 (2009) Chennamsetty, N., Voynov, V., Kayser, V., Helk, B., Trout, B.L.: Design of therapeutic proteins with enhanced stability. Proc. Natl. Acad. Sci. U.S.A. 106, 11937–11942 (2009) Abrams, C.F., Vanden-Eijnden, E.: Large-scale conformational sampling of proteins using temperature-accelerated molecular dynamics. Proc. Natl. Acad. Sci. U.S.A. 107, 4961–4966 (2010) Voelz, V.A., Bowman, G.R., Beauchamp, K., Pande, V.S.: Molecular simulation of ab initio protein folding for a millisecond folder NTL9(1–39). J. Am. Chem. Soc. 132, 1526–1528 (2010)

951 110. Li, Y., Schlick, T.: Modeling DNA polymerase  motions: subtle transitions before chemistry. Biophys. J. 99, 3463– 3472 (2010) 111. Foley, M.C., Padow, V., Schlick, T.: The extraordinary ability of DNA pol  to stabilize misaligned DNA. J. Am. Chem. Soc. 132, 13403–13416 (2010) 112. Foley, M.C., Schlick, T.: Simulations of DNA pol  R517 mutants indicate 517’s crucial role in ternary complex stability and suggest DNA slippage origin. J. Am. Chem. Soc. 130, 3967–3977 (2008) 113. Colthup, N.B., Daly, L.H., Wiberley, S.E.: Introduction to Infrared and Raman Spectroscopy. Academic Press, Boston (1990) 114. Weiner, S.J., Kollman, P.A., Nguyen, D.T., Case, D.A.: An all atom force field for simulations of proteins and nucleic acids. J. Comput. Chem. 7, 230–252 (1986) 115. Kim, S.V.J., Beard, W.A., Harvey, J., Shock, D.D., Knutson, J.R., Wilson, S.H.: Rapid segmental and subdomain motions of DNA polymerase ˇ. J. Biol. Chem. 278, 5072– 5081 (2003) 116. Nederveen, A.J., Bonvin, A.M.J.J.: NMR relaxation and internal dynamics of ubiquitin from a 0.2 &s MD simulation. J. Chem. Theory Comput. 1, 363–374 (2005) 117. Zagrovic, B., Sorin, E.J., Pande V.: ˇ-hairpin folding simulations in atomistic detail using an implicit solvent model. J. Mol. Biol. 313, 151–169 (2001) 118. Kubelka, J., Eaton, W.A., Hofrichter, J.: Experimental tests of villin subdomain folding simulations. J. Mol. Biol. 329, 625–630 (2003) 119. Horng, J.V.C., Moroz, V., Raleigh, D.P.: Rapid cooperative two-state folding of a miniature ˛–ˇ protein and design of a thermostable variant. J. Mol. Biol. 326, 1261–1270 (2003) 120. Aronsson, G., Brorsson, A.V.C., Sahlman, L., Jonsson, B.V.H.: Remarkably slow folding of a small protein. FEBS Lett. 411, 359–364 (1997) 121. Daiguji, H.: Ion transport in nanofluidic channels. Chem. Soc. Rev. 39, 901–911 (2010) 122. Fischer, N., Konevega, A.L., Wintermeyer, W., Rodnina, M.V., Stark, H.: Ribosome dynamics and tRNA movement by time-resolved electron cryomicroscopy. Nature 466, 329–333 (2010)

Molecular Geometry Optimization, Models Gero Friesecke1 and Florian Theil2 1 TU M¨unchen, Zentrum Mathematik, Garching, M¨unich, Germany 2 Mathematics Institute, University of Warwick, Coventry, UK

Mathematics Subject Classification 81V55, 70Cxx, 92C40

M

952

Short Definition Geometry optimization is a method to predict the threedimensional arrangement of the atoms in a molecule by means of minimization of a model energy. The phenomenon of binding, that is to say the tendency of atoms and molecules to conglomerate into stable larger structures, as well as the emergence of specific structures depending on the constituting elements, can be explained, at least in principle, as a result of geometry optimization.

Pheonomena Two atoms are said to be linked together by a bond if there is an opposing force against pulling them apart. Associated with a bond is a binding energy, which is the total energy required to separate the atoms. Except at very high temperature, atoms form bonds between each other and conglomerate into molecules and larger aggregates such as atomic or molecular chains, clusters, and crystals. The ensuing molecular geometries, that is to say the 3D arrangements of the atoms, and the binding energies of the different bonds, crucially influence physical and chemical behavior. Therefore, theoretically predicting them forms a large and important part of contemporary research in chemistry, materials science, and molecular biology. A major difficulty is that binding energies, preferred partners, and local geometries are highly chemically specific, that is to say they depend on the elements involved. For instance, the experimental binding energies of the diatomic molecules Li2 , Be2 , and N2 (i.e., the dimers of element number 3, 4, 7 in the periodic table) are roughly in the ratio 10:1:100. And CH2 is bent, whereas CO2 is straight. When atoms form bonds, their electronic structure, that is to say the probability cloud of electrons around their atomic nucleus, rearranges. Chemists distinguish phenomenologically between different types of bonds, depending on this type of rearrangement: covalent, ionic, and metallic bonds, as well as weak bonds such as hydrogen or van der Waals bonds. A covalent bond corresponds to a substantial rearrangement of the electron cloud into the space between the atoms

Molecular Geometry Optimization, Models

while each atom maintains a net charge neutrality, as in the C–C bond. In a ionic bond, one electron migrates almost fully to the other atom, as in the dimer Na–Cl. The metallic bond between atoms in a solid metal is pictured as the formation of a “sea” of free electrons, no longer associated to any particular atom, surrounding a lattice of ionic cores. The above distinctions, albeit a helpful guide, should not be taken too literally and are often not so clear-cut in practice. A unifying theoretical viewpoint of the 3D molecular structures resulting from interatomic bonding, regardless of the type of bonds, is to view them as geometry optimizers, i.e., as locally or globally optimal spatial arrangements of the atoms which minimize overall energy. For a mathematical formulation, see section “Geometry Optimization and Binding Energy Prediction.” If the number of atoms or molecules is large (&100), then the system will start behaving in a thermodynamic way. At sufficiently low temperature, identical atoms or molecules typically arrange themselves into a crystal, that is to say the positions of the atomic nuclei are given approximately by a subset of a crystal lattice. A crystal lattice L is a finite union of discrete subsets of R3 of form fi e C jf C kg j i; j; k 2 Zg, where e; f; g are linearly independent vectors in R3 . Near the boundaries of crystals, the underlying lattice is often distorted. Closely related effects are the emergence of defects such as vacancies, interstitial atoms, dislocations, and continuum deformations. Vacancies and interstitial atoms are missing respectively additional atoms. Dislocations are topological crystallographic defects which can sometimes be visualized as being caused by the termination of a plane of atoms in the middle of a crystal. Continuum deformations are small long-wavelength distortions of the underlying lattice arising from external loads, as in an elastically bent macroscopic piece of metal. A unifying interpretation of the above structures arises by extending the term “geometry optimization,” which is typically used in connection with single molecules, to large scale systems as well. The spatial arrangements of the atoms can again be understood, at least locally and subject to holding the atomic positions in an outer region fixed, as geometry optimizers, i.e., minimizers of energy.

Molecular Geometry Optimization, Models

Geometry Optimization and Binding Energy Prediction Geometry optimization, in its basic all-atom form, makes a prediction for the 3D spatial arrangement of the atoms in a molecule, by a two-step procedure. Suppose the system consists of M atoms, with atomic numbers Z1 ; : : : ; ZM . Step A: Specify a model energy or potential energy surface (PES), that is to say a function ˚ W R3M ! R [ fC1g which gives the system’s potential energy as a function of the vector X D .X1 ; : : : ; XM / 2 R3M of the atomic positions Xj 2 R3 . Step B: Compute (local or global) minimizers .X1 ; : : : ; XM / of ˚. Basic physical quantities of the molecule correspond to mathematical quantities of the energy surface as follows:

953

Model Energies A wide range of model energies are in use, depending on the type of system and the desired level of understanding. To obtain quantitatively accurate and chemically specific predictions, one uses ab initio energy surfaces, that is to say surfaces obtained from a quantum mechanical model for the system’s electronic structure which requires as input only atomic numbers. For large systems, one often uses classical potentials. The latter are particularly useful for predicting the 3D structure of systems composed from many identical copies of just a few basic units, such as crystalline clusters, carbon nanotubes, or nucleic acids.

Stable configuration

Difference between minimum energy and sum of energies of subsystems Local minimizer

Born-Oppenheimer Potential Energy Surface The gold standard model energy of a system of M atoms, which in principle contains the whole range of phenomena described in section “Pheonomena,” is the ground state Born-Oppenheimer PES of nonrelativistic quantum mechanics. With X D .X1 ; : : : ; XM / 2 R3M denoting the vector of nuclear positions, it has the general mathematical form

Transition state Bond length/angle

Saddle point Parameter in minimizing configuration

˚ BO .X / D min E.X;  /;

Binding energy

 2AN

(1)

where E is an energy functional depending on an infinite-dimensional field  , the electronic wave function. For a molecule with N electrons, the latter is a function on the configuration space .R3  Z2 /N of the electron positions and spins. More precisely AN D f 2 L2 ..R3  Z2 /N / ! C j jj jjL2 D 1; r 2 L2 ;  antisymmetricg, where antisymmet E D min ˚.X /  lim minf˚.X / W R!1 ric means, with xi ; si denoting the position and spin dist.fX1 ; : : : ; XK g; fXKC1; : : : ; XM g/  Rg: of the i th electron,  .: : : ; xi ; si ; : : : ; xj ; sj ; : : :/ D  .: : : ; xj ; sj ; : : : ; xi ; si ; : : :/ for all i < j . The functional E is given, in atomic units, by E.X;  / D Potential energy surfaces have the general property of R  3 N  H  where Galilean invariance, that is to say ˚.X1 ; : : : ; XM / D .R Z2 / ˚.RX1 C a; : : : ; RXM C a/, for any translation vector N X X a 2 R3 and any rotation matrix R 2 SO.3/. Thus, a H D v .x /C rx2i C Wee .xi xj /CWnn .X / X 1 one-atom surface ˚.X1 / is independent of X1 , and a j D1 1i 0 are empirical parameters. For a variety of monatomic systems, more sophisticated classical potentials containing three-body and higher interactions, ˚ classical .X1 ; : : : ; XM / D C

X

X

V2 .Xi ; Xj /

i 0 be such In this paragraph we confine our attention to a C r (r  1) map f W U  E ! F between two Euclidean that B. ; r/  U and

Newton-Raphson Method

1025

˚  spaces with dim E  dim F . Let V D f 1 .0/ be Mf1 . / D x 2 W W Mf .x/ D the zero set of f . We suppose that Df .x/ is onto for any x 2 U . In that case V is a C r submanifold, its is a submanifold of class C r1 . This submanifold is dimension is equal to dim E  dim F , and its tangent invariant by Nf and contains . The tangent space at space at 2 V is T V D ker Df . /: In this context, to M 1 . / is f we define the Newton operator by Nf .x/ D x  Df .x/ f .x/ with

T Mf1 . / D .ker Df . //? :

Df .x/ D Df .x/ .Df .x/Df .x/ /1

Overdetermined Systems that is the minimal norm solution of the linearized equation at x 2 U : f .x/ C Df .x/.y  x/ D 0: Notice that V is equal to the set of fixed points of Nf . As usual, to this Newton operator we associate an iterative process defined by

We consider here the case of an overdetermined system f W U  E ! F , where E and F are two Euclidean spaces with dim F > dim E, U is an open subset in E and f 2 C 1 .U /. In general, such a system has no solution. Let us denote by

xkC1 D Nf .xk /: F .x/ D This concept has been introduced for the first time by Ben-Israel [7], then studied by Allgower-Georg [3], Beyn [8], Shub-Smale [33], Dedieu-Shub [12], Dedieu-Kim [11], and Dedieu [10]. The convergence of Newton’s sequences is quadratic like in the usual case (see Shub-Smale [33] or Dedieu [10, Theorems 130 and 137]).

1 kf .x/k2 2

the residue function associated with f . Our objective is to minimize F , that is, to find its global minimum (least squares method). But this goal is, in general, difficult to satisfy. Thus, we reduce our ambition to local minima or even to stationnary points of F . For this reason we call least-squares solution of the system f .x/ D 0 Theorem 3 There exists an open neighborhood V of at any stationary point of the residue function, i.e., V contained in U such that for any x 2 V, the Newton DF .x/ D 0. To find such least-squares solutions, following sequence xkC1 D Nf .xk /, x0 D x, converges to a Gauss 1809, we linearize the equation f .y/ D 0 point of V denoted Mf .x/. Moreover in the neighborhood of a given x 2 U . We obtain  2k 1 f .x/ C Df .x/.y  x/ D 0; and we consider the least1 d.x; V /: kxk  Mf .x/k  2 squares solution of this linear system. When Df .x/ is 2 one to one, we get In fact, the limit operator Mf acts asymptotically y D x  Df .x/ f .x/; with like a projection onto V as shown by the following  1 (Dedieu [10, Corollary 138]). This result may also be Df .x/ : Df .x/ D Df .x/ Df .x/ considered as an instance of the invariant manifold theorem (Hirsch-Pugh-Shub [22]). This defines an operator y D Nf .x/ and an iterative Theorem 4 Suppose that r  2. Then, Mf has class method xkC1 D Nf .xk / called the Newton-Gauss method. C r1 and its derivative at 2 V is The properties of Newton-Gauss method are very different from the classical well-determined case DMf . / D …ker Df . / (dim E D dim F and Df .x/ invertible). We resume these properties in the following: the orthogonal projection onto ker Df . /. Moreover, there is an open neighborhood W of V contained in U Theorem 5 1. Fixed points for Nf correspond to stationary points for F . such that, for any 2 V , the set

N

1026

Newton-Raphson Method

2. When f 2 C 2 .U /, let 2 U be such that DF . / D 0 and Df . / are one to one. Then: (a) If is an attractive fixed point for Nf , then it is also a strict local minimum for the residue function F . (b) If is a strict local maximum for F , then it is a repulsive fixed point for Nf . 3. The convergence of Newton-Gauss sequences to an attractive fixed point is linear. This convergence is quadratic when f . / D 0. See Dennis-Schnabel [15], Dedieu-Shub [13], Adler-Dedieu-Martens-Shub [2], Dedieu-Kim [11], and Dedieu [10].

Newton Method on Manifolds Let us consider a nonlinear system f W V ! F where V is a smooth manifold, F a Euclidean space, and f 2 C 1 .V /. Let us denote by Tx V the tangent space at x to V . In this context Df .x/ W Tx V ! F and, when this derivative is invertible, Df .x/1 W F ! Tx V . To define a Newton operator associated with f , we introduce a retraction R W T V ! V where T V , the disjoint union of the tangent spaces Tx V , x 2 V , is the tangent bundle of V . R is assumed to be a smooth map defined in a neighborhood of V in T V (x 2 V is indentified with the zero tangent vector 0x 2 Tx V ), taking its values in V and satisfying the following properties. Let us denote by Rx the restriction of R to Tx V ; then: 1. Rx is defined in an open neighborhood of 0x 2 Tx V . 2. Rx .u/ D x if and only if u D 0x . 3. DRx .0x / D idTx V . In this context, the Newton operator is defined by Nf .x/ D Rx .Df .x/1 f .x// so that Nf W V ! V . Examples of retractions are given by Rx .u/ D x C u when V D E, a Euclidean space. In that case Tx E, the tangent space at x to E, may be identified with E itself and Nf .x/ D x  Df .x/1 f .x/. A second example of retraction is given by the exponential map of a Riemannian manifold exp W T V ! V . This gives Nf .x/ D expx .Df .x/1 f .x//. Other examples (sphere, projective space, orthogonal group) are given

in Adler et al. [2]. The properties of this method are similar to the classical case: fixed points for Nf correspond to nonsingular zeros for f and, at such a zero, DNf .x/ D 0.

Zeros of Vector Fields By a vector field on manifold V , we mean a smooth map X which assigns to each x 2 V a tangent vector X.x/ 2 Tx V . We are interested in using Newton’s method to find zeros of X , i.e., points x 2 V such that X.x/ D 0x the zero vector in Tx V: An important example is X D grad (the gradient vector field) where  W V ! R is a smooth real-valued function, so the zeros of X are the critical points of : In order to define Newton’s method for vector fields, we resort to an object studied in differential geometry, namely, the covariant derivative of vector fields. The covariant derivative of a vector field X defines a linear map rX.x/ W Tx V ! Tx V for any x 2 V . For example, if X D grad for  a real-valued function on V and X.x/ D 0; then rX.x/ D Hess.x/ the Hessian of  at x: Let R W T V ! V be a retraction (see the previous section). We define the Newton operator for the vector field X by NX .x/ D Rx .rX.x/1 X.x// as long as rX.x/ is invertible and rX.x/1 X.x/ is contained is the domain of Rx : Newton’s method has the usual property of quadratic convergence for simple zeros of vector fields [30]. Proposition 1 If x 2 V is a fixed point for NX .x/, then X.x/ D 0x and DNX .x/ D 0: The Rayleigh-quotient iteration, introduced by Lord Rayleigh one century ago for the eigenvalue problem, may be seen in this context. Newton method on manifolds for nonlinear systems or vector fields appears in Shub [30], Udris¸te [37], and Smith [35] in a general setting. The case of overdetermined and underdetermined systems is studied in Adler-Dedieu-Martens-Shub [2]. See also Edelman-Arias-Smith [16] and the book by AbsilMahony-Sepulchre [1] more oriented to matrix manifolds. In the context of projective spaces, see

Newton-Raphson Method

Shub [31], Malajovich [26], and two important papers by Shub-Smale related to the complexity of B´ezout’s theorem [32] and [33]. The multihomogeneous Newton method is studied by Dedieu-Shub [12]; this iteration is defined in a product of projective spaces. Many papers study the metric properties of Newton method in the context of Riemannian manifolds: Ferreira-Svaiter [20] (Kantorovich theory), Dedieu-Priouret-Malajovich [14] (alpha-theory), and Alvarez-Bolte-Munier [4]. See also Li-Wang [24] and Li-Wang [25]. The more specific case of Newton method on Lie groups is studied by Owren-Welfert [29].

References 1. Absil, P.-A., Mahony, R., Sepulchre, R.: Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton/Woodstock (2008) 2. Adler, R., Dedieu, J.-P., Martens, M., Shub, M.: Newton’s method on Riemannian manifolds with an application to a human spine model. IMA J. Numer. Anal. 22, 1–32 (2002) 3. Allgower, E., Georg, K.: Numerical Continuation Methods. Springer, Berlin/New York (1990) 4. Alvarez, F., Bolte, J., Munier, J.: A unifying local convergence result for Newton’s method in Riemannian manifolds. Found. Comput. Math. 8, 197–226 (2008) 5. Argyros, I.: Convergence and Applications of Newton-Type Iterations. Springer, New York/London (2008) 6. Argyros, I., Guti´errez, J.: A unified approach for enlarging the radius of convergence for Newton’s method and applications. Nonlinear Funct. Anal. Appl. 10, 555–563 (2005) 7. Ben-Israel, A.: A Newton-Raphson method for the solution of systems of equations. J. Math. Anal. Appl. 15, 243–252 (1966) 8. Beyn, W.-J.: On smoothness and invariance properties of the Gauss-Newton method. Numer. Funct. Anal. Optim. 14, 243–252 (1993) 9. Blum, L., Cucker, F., Shub, M., Smale, S.: Complexity and Real Computation. Springer, New York (1997) 10. Dedieu, J.-P.: Points Fixes, Z´eros et la M´ethode de Newton. Springer, Berlin/New York (2006) 11. Dedieu, J.-P., Kim, M.-H.: Newton’s Method for Analytic Systems of Equations with Constant Rank Derivatives. J. Complex. 18, 187–209 (2002) 12. Dedieu, J.-P., Shub, M.: Multihomogeneous Newton’s method. Math. Comput. 69, 1071–1098 (2000) 13. Dedieu, J.-P., Shub, M.: Newton’s method for overdetermined systems of equations. Math. Comput. 69, 1099–1115 (2000) 14. Dedieu, J.-P., Priouret, P., Malajovich, G.: Newton’s method on Riemannian manifolds: covariant alpha theory. IMA J. Numer. Anal. 23, 395–419 (2003) 15. Dennis, J., Schnabel, R.: Numerical Methods for Unconstrained Optimization and Nonlinear Equation. Prentice Hall, Englewood Cliffs (1983)

1027 16. Edelman, A., Arias, T., Smith, S.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20, 303–353 (1998) 17. Ezquerro, J., Guti´errez, J., Hernandez, M., Romero, N., Rubio, M.-J.: El metodo de Newton: de Newton a Kantorovich. La Gaceta de la RSME 13, 53–76 (2010) 18. Ferreira, O.P.: Local convergence of Newton’s method in Banach space from the viewpoint of the majorant principle. IMA J. Numer. Anal. 29, 746–759 (2009) 19. Ferreira, O.P.: Local convergence of Newton’s method under majorant condition. J. Comput. Appl. Math. 235, 1515–1522 (2011) 20. Ferreira, O.P., Svaiter, B.F.: Kantorovich’s theorem on Newton’s method on Riemannian manifolds. J. Complex. 18, 304–329 (2002) 21. Goldstine, H.: A History of Numerical Analysis from the 16th Through the 19th Century. Springer, New York (1977) 22. Hirsch, M., Pugh, C., Shub, M.: Invariant Manifolds. Lecture Notes in Mathematics, vol. 583. Springer, Berlin/New York (1977) 23. Kantorovich, L.: Sur la m´ethode de Newton. Travaux de l’Institut des Math´ematiques Steklov XXVIII, 104–144 (1949) 24. Li, C., Wang, J.H.: Newton’s method on Riemannian manifolds: Smale’s point estimate theory under the r-condition. IMA Numer. Anal. 26, 228–251 (2006) 25. Li, C., Wang, J.H.: Newton’s method for sections on Riemannian manifolds: generalized covariant alpha-theory. J. Complex. 24, 423–451 (2008) 26. Malajovich, G.: On generalized Newton’s methods. Theor. Comput. Sci. 133, 65–84 (1994) 27. Ortega, J., Rheinboldt, V.: Numerical Solutions of Nonlinear Problems. SIAM, Philadelphia (1968) 28. Ostrowski, A.: Solutions of Equations in Euclidean and Banach Spaces. Academic, New York (1976) 29. Owren, B., Welfert, B.: The Newton iteration on Lie groups. BIT 40, 121–145 (2000) 30. Shub, M.: Some remarks on dynamical systems and numerical analysis. In: Lara-Carrero, L., Lewowicz, J. (eds.) Dynamical Systems and Partial Differential Equations, Proceedings of VII ELAM. Equinoccio, Universidad Simon Bolivar, Caracas (1986) 31. Shub, M.: Some remarks on B´ezout’s theorem and complexity. In: Hirsch, M.V., Marsden, J.E., Shub, M. (eds.) Proceedings of the Smalefest, pp. 443–455. Springer, New York (1993) 32. Shub, M., Smale, S.: Complexity of B´ezout’s theorem I: geometric aspects. J. Am. Math. Soc. 6, 459–501 (1993) 33. Shub, M., Smale, S.: Complexity of B´ezout’s theorem IV: probability of success, extensions. SIAM J. Numer. Anal. 33, 128–148 (1996) 34. Smale, S.: Newton’s method estimates from data at one point. In: Ewing, R., Gross, K., Martin, C. (eds.) The Merging of Disciplines: New Directions in Pure, Applied and Computational Mathematics. Springer, New York (1986) 35. Smith, S.: Optimization Techniques on Riemannian Manifolds. Fields Institute Communications, vol. 3, pp. 113–146. AMS, Providence (1994) 36. Stoer, J., Bulirsch, R.: Introduction to Numerical Analysis. Springer, New York (2002)

N

1028 37. Udris¸te, C.: Convex Functions and Optimization Methods on Riemannian Manifolds. Kluwer, Dordrecht/Boston (1994) 38. Wang, X.H.: Convergence of Newton’s method and uniqueness of the solution of equations in Banach spaces. IMA J. Numer. Anal. 20, 123–134 (2000) 39. Wang, X.H., Han, D.F.: On the dominating sequence method in the point estimates and Smale’s theorem. Sci. Sin. Ser. A 33, 135–144 (1990) 40. Ypma, T.: Historical development of the Newton-Raphson method. SIAM Rev. 37, 531–551 (1995)

Nuclear Modeling Bernard Ducomet Departement de Physique Theorique et Appliquee, CEA/DAM Ile De France, Arpajon, France

Nuclear Modeling

(analogous to pairing in super- conductivity) clearly beyond mean field theories. The region 1012 g=cm3 < < 1016 g=cm3 where relativistic effects become important is finally followed by the ultradense matter > 1016 g=cm3 where intermediate (mesonic and baryonic) degrees of freedom begin to appear, and then internal structure of nucleons ultimately dominates (quark matter). To these various density regions correspond various theoretical models leading to fruitful comparisons with experiments in recent years. To maintain this overview within reasonable bounds, we focus on the non-relativistic modeling, and just say a few words and quote important references concerning the relativistic problem in the last section.

The Hamiltonian for Nuclear Matter: A Short Review Short Definition Several mean-field type schemes are presented in order to describe accurately the physics of atomic nuclei, depending on the kind of information requested (static properties of light or heavy nuclei, individual particle excitations, collective motions).

Description The spirit of this review consists in a brief presentation of nuclear modeling as a description of dense matter. One knows [1] that in states with densities typically between 5:102 and 108 g=cm3 (“subferrous matter”), matter is a mixture of electrons and nuclei interacting through Coulomb two-body interaction. The domain of nuclear physics schematically begins in the following density region where 108 g=cm3 < < 1012 g=cm3 which corresponds to the socalled subnuclear matter, where nucleons (particles composing nuclei: positively charged protons and neutral neutrons) play an essential role. In the sector 1012 g=cm3 < < 1015 g=cm3 of “nuclear” and “transnuclear matter,” the description of subtle couplings between individual motions as independent particles and collective ones (vibrations, rotations) often needs theories including various correlations

Low-energy nuclear physics considers nucleons (neutrons and protons) as the elementary constituents of nuclei. In fact nucleons may be considered as bound states of quarks, themselves strongly interacting through gluons, but one (presently) does not know how to derive quantitatively the nucleon–nucleon interaction from quantum chromodynamics (QCD), the corresponding gauge field theory of strong interaction. As far as low-energy nuclear physics is concerned, the quarkgluons degrees of freedom are not directly observed and nucleons are the physically relevant objects. However, even in this context, properties of nuclei cannot be directly derived from a possible “bare” interaction between nucleons, which is too singular to be treated through perturbative methods, even if recent studies show that it is possible [2] to use renormalization group method to construct low-momentum (the so-called Vlow k ) interactions which are supposed to parametrize a high-order chiral effective field theory for a twonucleon force. So one is forced to build “dressed” (effective) interactions, modeling the “nuclear medium” in a phenomenological way, the so-called effective phenomenological interactions, for which mean field methods such as Hartree-Fock approximation may be used [3]. These effective forces built from a few general symmetry principles, include a number of parameters which have to be adjusted in order to fit experimental data.

Nuclear Modeling

1029

“Bare” Versus “Effective” Nucleon–Nucleon Interaction From low-energy nucleon–nucleon scattering experiments, some basic features emerge [4]: the interaction is short range (about 1 fm = 1015 m), within this range it is attractive at “large distance” and strongly repulsive at “short distance” (0.5 fm), and it depends both on the spin and isospin of the two nucleons. Starting from the idea that the nucleon–nucleon interaction is mediated by pions as Coulomb interaction is mediated by photons, one gets various expressions [5]. One of them is the OPEP (one pion exchange potential) VOPEP .r1  r2 ; 1 ; 2 ; 1 ; 2 / D

f2 e jr1 r2 j .T1  T2 /.S1  r1 /.S2  r2 / ; (1) 4 jr1  r2 j

where S1;2 (resp. T1;2 ) are the spin (resp. isospin) operators of the two particles, f is a coupling constant and  the mass of the pion. As it appears that in nuclei the two-body interaction is strongly modified by complicated many-body effects, it becomes more profitable to replace (1) by effective interactions taking into accounts medium effects, more tractable for meanfield calculations. The most widely used effective interactions in Hartree-Fock calculations are the Skyrme forces [6] which include two-body and three-body contributions. The total interaction is then V D

N X

Vij C

X

at its right (resp. left). The three-body part is taken as a simple zero-range expression Vij k D t3 ı.ri  rj /ı.rj  rk / I: Finally, the six parameters t0 ; t1 ; t2 ; t3 ; x0 , and W0 are chosen in order to reproduce a number of properties of some well-known finite nuclei (see [7] for various choices of Skyrme interactions). One observes that despite its simple form, from a mathematical point of view, the previous Skyrme interaction does not lead to a well-behaved hamiltonian due to the presence of Dirac distributions. Moreover it leads to “physical divergences” where pairing properties (illustrating superfluid properties of the main part of nuclei) are involved, because of its zero range. In order to avoid these drawbacks, a finite-range interaction has been proposed by Decharg´e and Gogny in the 1970s [8], which is free of these divergences and may be considered as a smeared version of the Skyrme interaction. For the two-body operator Vij , they consider the short-range model

Vij D

2 X

e

jri rj j2 n

.wn C bn P  hn P  mn P P /

nD1

Ci w0 0 .Si C Sj /  k0  ı.ri  rj / k   ri C rj Ct 0 3 .1 C P /ı.ri  rj / 1=3 ; 2

(3)

Vij k ;

where the sum involves the operator P which exchanges isospins i and j . In these expressions where the two-body term contains momentum depen- wn ; bn ; hn ; mn ; n are the so-called Wigner, Bartlett, dence, spin-exchange contributions, and a spin-orbit Heisenberg, Majorana, and range coefficients, and the last density-dependent term simulates a threeterm body contribution. As for Skyrme forces, all of these parameters are fitted to reproduce experimental values Vij D t0 .1 C x0 P / ı.ri  rj / of selected observables measured on a few stable  1 2 nuclei. C t2 ı.ri  rj /k2 C k0 ı.ri  rj / 2 Ct2 k0  ı.ri  rj /k i is replaced by row number i of A, then we will denote the modified elementary weights by ˚i .t/. The elementary weights can be defined recursively

˚i . / D

s X

aij D ci ;

j D1

˚. / D ˚i .Œt1 t2    tm / D

s X i D1 s X

bi ; aij ˚j .t1 /˚j .t2 /    ˚j .tm /;

j D1

˚ .Œt1 t2    tm / D

s X i D1

bi ˚i .t1 /˚i .t2 /    ˚i .tm /:

Taylor Expansion of y.x/ Theorem 1 The Taylor expansion of y.x0 Ch/ is given by X hr.t / F .t/.y0 /: (6) y.x0 C h/ D y0 C .t/.t/ t 2T Proof. Let Tn denote the subset of trees whose order is limited to n. We will show that X  r.t / hr.t / F .t/.y0 / C O.hnC1 /; .t/.t/ t 2Tn (7) for  2 Œ0; 1, which is true when n D 0, follows from the same statement with n replaced by n  1 if n > 0. First note that the formal Taylor series for f .y0 C a/, P when a is replaced by m i D1 ai given by

y.x0 C h/ D y0 C

f .y0 C a1 C a2 C    C am / For the trees up to order 4, the elementary weights are included in Table 1. km X a1k1 a2k2    am f .k1 Ck2 CCkm / D Our aim now is to find the Taylor expansions of the k1 Šk2 Š    km Š k1 ;k2 ;:::;km 0 solution to the differential equation y.x0 C h/ after a unit time step and, for comparison, the Taylor expan.a1 ; a1 ; : : : ; am ; am ; : : :/; (8) sion of y1 , the approximation to the same quantity as computed using a Runge–Kutta method. A particular where the operands of the .k1 C k2 C    C km /method will have order p if and only if the two series linear operator consist of ki repetitions of ai , i D 1; 2; : : : ; m. Evaluate the series, up to hn terms, for agree to within O.hpC1 /.

Order Conditions and Order Barriers

1105

y.x0 C h/ D y0 Z    X r.t / hr.t / F .t/.y0 / d ; Ch f y0 C .t/.t/ 0 t 2T

The theory leading up to the order conditions was given in [1] and modern formulations are presented in [6] and [8].

n1

First-Order Equations If the analysis of order had been carried out using the first-order model problem (1), because of the coinciTaylor Expansion of y1 dence between certain of the elementary differentials Theorem 2 The Taylor expansion of y1 given by (4) is in this special case, the number of order conditions is reduced. However, this does not have any effect until given by order 5, where the number of conditions is reduced from 17 to 16, and order 6 where there are now 31 r.t / X h ˚.t/ F .t/.y0 /: y1 D y0 C (9) instead of 37 conditions, and higher orders where the .t/ t 2T reduction in conditions is even more drastic. These questions are considered in [3] and [4]. Proof. We will prove that using (8), and the result agrees with (7).

X hr.t / ˚i .t/ F .t/.y0 / C O.hmC1 /; (10) Low-Order Explicit Methods .t/ t 2Tm For orders up to p D 3, it is an easy matter to derive specific methods with s D p stages. which is true for m D 0, follows from the same In the case of s D p D 1, the only available method statement with m replaced by m  1. To verify this, substitute (10), with i replaced by j and m replaced by is the Euler method 0 m  1 into (3), where Fj has been replaced by f .Yj /, 1 and expand using (8). The coefficient of F .t/.y0 / agrees with the corresponding coefficient in (10). To complete the proof replace ˚i .t/ by ˚.t/ so that Yi s D p D 2 The equations for this case were investigated by becomes y1 . Runge [10] Yi D y0 C

Order Conditions b1 C b2 D 1; To obtain order p, the two Taylor series for the approx1 imate solution given by (9) must agree with the Taylor b2 c2 D ; 2 expansion for the exact solution, given by (6), up to coefficients of hm for m D 1; 2; : : : ; p. That is, the two leading to the following tableau where c2 is an arbitrary series nonzero parameter r.t / X h F .t/.y0 / 0 .t/.t/ r.t /p c2 c2 X hr.t / ˚.t/ 1 1 1 F .t/.y0 /; and 2c2 2c2 .t/ r.t /p

must be identical. Hence, we have Theorem 3 A Runge–Kutta method .A; b > ; c/ has order p if and only if 1 ˚.t/ D ; .t/

r.t/  p:

The special cases c2 D simple coefficients 0 1 2

1 2 0

1 2

and c2 D 1 give particularly

0 1 1

1 1 2

1 2

O

1106

Order Conditions and Order Barriers

sDpD3 For this case, we obtain four order conditions b1 C b2 C b3 D 1;

out, substitute the solution values into (22). This leads to the simple consistency condition c4 D 1. We will prove this result in the more general case s D p  4, (11) assuming that methods with s D p > 4 actually exist.

1 ; 2 1 b2 c22 C b3 c32 D ; 3 1 b3 a32 c2 D : 6

(12) Value of c4 when s D p  4 We will prove the following result (13) Theorem 4 Let .A; b > ; c/ be the coefficient arrays for an explicit Runge–Kutta method with s D p  4. Then (14) c4 D 1.

b2 c2 C b3 c3 D

Let u> v > and x; y be s dimensional vectors deIt is convenient to choose c2 ¤ 0 and c3 and to then fined by solve (11), (12), and (13) for b1 ; b2 ; b3 . As long as u> D b > As4 .C  c4 I /; b3 ¤ 0, a32 can then be found from (14). v > D b > As3 ; The analysis of the s D p D 3 case was completed in [7]. x D Ac; y D .C  c2 I /c: sDpD4 The complete classification of methods with s D p D 4 where C D diag.c/. Because of the special structure was published in [9]. of these vectors they have the forms There are now eight order conditions: b1 C b2 C b3 C b4 D 1;

2 3 2 3 2 3 3 0 v1 0 u1 60 7 6 v2 7 607 6 u2 7 6 7 6 7 6 7 6 7 6 y3 7 6 v3 7 6 x3 7 6 u3 7 (16) 6 7 6 7 6 7 6 7 u D 6 0 7 ; v D 6 0 7 ; x D 6 x4 7 ; y D 6 y4 7 : 6 7 6 7 6 7 6 7 6 : 7 6 : 7 6 : 7 6 : 7 (17) 4 :: 5 4 :: 5 4 :: 5 4 :: 5 0 0 xs ys (18) It now follows that (19) .u> x/.v > y/ D .u> y/.v > x/ (23) (15)

2

1 ; 2 1 b2 c22 C b3 c32 C b4 c42 D ; 3 1 b3 a32 c2 C b4 a42 c2 C b4 a43 c3 D ; 6 1 b2 c23 C b3 c33 C b4 c43 D ; 4 1 b3 c3 a32 c2 C b4 c4 a42 c2 C b4 c4 a43 c3 D ; (20) 8 because each of these equals u3 v3 x3 y3 . The factors 1 appearing in (23) are each linear combinations of 2 2 2 b3 a32 c2 C b4 a42 c2 C b4 a43 c3 D ; (21) 12 elementary weights which can be evaluated by the order conditions as follows 1 b4 a43 a32 c2 D : (22) 24 c4 3 ; u> x D b > As4 .C  c4 I /Ac D  sŠ .s  1/Š It is natural to attempt to find solutions to these equac2 2 tions in three steps as for order 3. Step one is to v > y D b > As3 .C  c2 I /c D  ; sŠ .s  1/Š choose suitable values of c2 ; c3 ; c4 ; step two is to solve for b1 ; b2 ; b3 ; b4 from (15), (16), (17), (19); u> y D b > As4 .C  c4 I /.C  c2 I /c and the third step would be to solve for a32 ; a42 ; a43 2.c2 C c4 / c2 c4 6 from (18), (20), (21). But this solution method is  C ; D sŠ .s  1/Š .s  2/Š incomplete because no attempt has been made to sat1 isfy (22). However, if we had chosen c2 ; c3 ; c4 as (24) v > x D b > As3 Ac D : sŠ parameters and, after the third step has been carried b2 c2 C b3 c3 C b4 c4 D

Order Conditions and Order Barriers

1107

Substitute into (23) and simplify the result. It is If such a method did exist, it would follow from found that c2 .c4  1/ D 0. It is not possible that Theorem 4, that c4 D 1. Now, assuming s  5, modify c2 D 0, because this would contradict (24), and hence the argument by replacing u> by c4 D 1. u> D b > As5 .C  c5 I /A:

Order Barriers for Explicit Methods

The values of u> x and u> y now get replaced by

As p increases, the number of conditions that must be satisfied to achieve order p increases in accordance with the number of rooted trees up to this order. Denote this by M.p/. If this order is to be achieved using an s stage Runge–Kutta method, the number of available parameters also increases and is equal to N.s/ D s.s C 1/=2. The growth of these quantities can be seen up to p D 8 and s D 11 in Table 2. The matching lines in the two parts of this table are intended to indicate the number of stages necessary to obtain a particular order. It is not surprising that order p D s is possible for s  4, because N.s/  M.s/, and that p  s C 1 is necessary for s > 4 because N.s/ < M.s/. But for the matching orders and stage numbers for p  6 and s  7, it is remarkable that it becomes possible to satisfy M.p/ conditions with fewer parameters. We will only prove the simplest of the order barriers suggested by Table 2: Theorem 5 An explicit Runge–Kutta method with s stages cannot have order p D s if s > 4.

Order Conditions and Order Barriers, Table 2 The number of order conditions M.p/ compared with the number of parameters N.s/ p

M.p/

s

N.s/

1 2 3 4

1 2 4 8

5 6

17 37

7

85

8

200

1 2 3 4 5 6 7 8 9 10 11

1 3 6 10 15 21 28 36 45 55 66

u> x D b > As5 .C  c5 I /A2 c D

c5 4  ; sŠ .s  1/Š

u> y D b > As5 .C  c5 I /A.C  c2 I /c D

3c2 C 2c4 c2 c4 8  C ; sŠ .s  1/Š .s  2/Š

and again the two sides of (23) are each equal to u3 v3 x3 y3 . Simplifying the new form of (23) leads to c2 .c5  1/ D 0 and to the conclusion that c5 D c4 D 1. We obtain a contradiction by evaluating b > As5 .C  I /A2 c D .4  s/=sŠ. However, because of the strictly lower triangular structure of A, all terms in the product are zero. The fifth-order barrier and higher order barriers were presented in [3] and [4].

O References 1. Butcher, J.C.: Coefficients for the study of Runge–Kutta integration processes. J. Aust. Math. Soc. 3, 185–201 (1963) 2. Butcher, J.C.: On the integration processes of A. Huˇta. J. Aust. Math. Soc. 3, 202–206 (1963) 3. Butcher, J.C.: On the attainable order of Runge–Kutta methods. Math. Comp. 19, 408–417 (1965) 4. Butcher, J.C.: The nonexistence of ten-stage eighth order explicit Runge–Kutta methods. BIT 25, 521–540 (1985) 5. Butcher, J.C.: On fifth order Runge–Kutta methods. BIT 35, 202–209 (1995) 6. Butcher, J.C.: Numerical methods for ordinary differential equations, 2nd edn. Wiley, Chichester (2008) 7. Heun, K.: Neue Methoden zur approximativen Integration der Differentialgleichungen einer unabh¨angigen Ver¨anderlichen. Z. Math. Phys. 45, 23–38 (1900) 8. Hairer, E., Nørsett, S.P., Wanner, G.: Solving ordinary differential equations I: Nonstiff problems, 2nd edn. Springer, Berlin/Heidelberg/New York (1993) 9. Kutta, W.: Beitrag zur n¨aherungsweisen Integration totaler Differentialgleichungen. Z. Math. Phys. 46, 435–453 (1901) ¨ 10. Runge, C.: Uber die numerische Aufl¨osung von Differentialgleichungen. Math. Ann. 46, 167–178 (1895)

1108

Order Stars and Stability Domains

Order Stars and Stability Domains Ernst Hairer and Gerhard Wanner Section de Math´ematiques, Universit´e de Gen`eve, Gen`eve, Switzerland

Throughout the twentieth century, and before, scientists were struggling to find numerical methods for initial value problems satisfying the following requirements • High computational speed • High precision • High stability Order stars turn these analytical properties into geometrical quantities. They are thus not only attractive for the importance of their theoretical results, but also for aesthetics and beauty.

Stability Function While the first two of the above requirements are more or less trivial claims, the importance of the third one became clear, to many researchers, only after several numerical disasters. Stability theory for numerical methods started with Courant–Friedrichs–Lewy [3]; the classical papers Dahlquist [4] and Guillou–Lago [7] initiated stability theory for stiff equations. The principal tool is Dahlquist’s test equation yP D y

its stability domain. The criterion for stability requires that all eigenvalues , multiplied by the step size h, must lie in S . For hyperbolic PDEs this is known as the CFL-condition. Examples of stability functions and stability domains are presented in Fig. 1 for the explicit Euler method (see entry  One-Step Methods, Order, Convergence), Runge’s method of order 2 (see entry  Runge–Kutta Methods, Explicit, Implicit), the implicit Euler method, and the trapezoidal rule. We observe that for explicit methods, with k function evaluations per step, R.z/ is a polynomial of degree k, which leads to the existence of k zeros (marked by ) lying inside of S . The corresponding stability domain is a bounded set, so that severe step size restrictions can occur for stiff equations. On the contrary, implicit stages lead to poles (marked by ?), so that R.z/ becomes rational which allows both methods to be A-stable, that is, the stability domain covers the entire left half plane of C. Here, for stable problems, there are no stability restrictions, independent of how stiff the equation is. Implicit Runge–Kutta Methods It was discovered by Ehle [5] that the stability functions of high order implicit Runge–Kutta methods are Pad´e approximations to the exponential function Rk;l .z/ D

Pk;l .z/ Qk;l .z/

where ! k X k .k C l  j /Š j z Pk;l .z/ D .k C l/Š j j D0

which models the behavior of stiff systems in the neighborhood of a stationary point. Here,  represents one of the (possibly complex) eigenvalues of the Jacoand Qk;l .z/ D Pl;k .z/: bian matrix of the vector field. The equation is stable “in the sense of Lyapunov” if jez jg domain S . So, if we know that the order star does (see Fig. 2 to the right), which we call order star. It not contain part of the imaginary axis and if, in adtransforms numerical quantities into geometric properdition, all poles stay in the right half plane, we have ties as follows: A-stability by the maximum principle. Inversely, if • Computational speed: This is determined by the the order star covers a part of the imaginary axis, we number of stages, which the method uses at each have no A-stability.

O

1110

Some elementary complex analysis (the argument principle) allows the conclusion that all bounded black fingers must contain a pole, and that every bounded subset of A, whose boundary returns j times to the origin, must contain j poles (see, e.g., “Lemma 4.5” of [8]). Ehle’s results and Ehle’s conjecture can now be directly seen from the right picture of Fig. 2, because only for k  l  k C 2 the imaginary axis can pass between the white and black fingers without touching the order star (for details see Theorems 4.6–4.11 of [8]).

Order Stars and Stability Domains

and we have stability where both roots are bounded by 1. This determines the stability domain (see the first two pictures of Fig. 3). At certain “branching points” these roots merge and we observe a discontinuity in their neighborhood. Thus, we place the two roots one above the other (third picture of Fig. 3) and obtain one analytic function R.z/ on the Riemann surface M .

Order Stars for Multistep Methods The definition (2) of order stars carries over to Riemann surfaces without changes. There will be a star on the principal sheet with p C 1 sectors, an explicit stage leads to a zero on one of the sheets (see Fig. 4), each Multistep Methods implicit stage gives rise to a pole on one of the sheets and for A-stability the order star must stay away from We choose as example the two-step explicit Adams the imaginary axis on all sheets. method   Daniel–Moore Conjecture 1 3 fnC1  fn : ynC2 D ynC1 C h An A-stable multistep method with k implicit stages 2 2 has order p  2k. The proof of this statement is illustrated in Fig. 5. This order barrier was originally For Dahlquist’s test equation we obtain ynC2  .1 C 3 1 n n conjectured for multistep-Taylor methods in 1970 and 2 z/ynC1 C 2 zyn D 0, that is, yn D c1 R1 C c2 R2 where became a theorem in 1978. The special case for linear R1 and R2 are the roots of the characteristic equation multistep methods, where k D 1 and hence p  2, is   Dahlquist’s second order barrier from 1963. 1 3 R2  1 C z R C z D 0: 2 2 Jeltsch–Nevanlinna Theorem For z D 0 this equation becomes R2 R D 0 with roots During the first half of the twentieth century it was R1 D 1 (the principal root) and R2 D 0 (the parasitic widely believed that multistep methods, which, due to root). Both roots continue as complex functions of z the use of several consecutive solution approximations,

Order Stars and Stability Domains, Fig. 3 Stability domain for Adams2; principal root jR1 .z/j (left); parasitic root jR2 .z/j (middle); both roots on the Riemann surface (right)

Order Stars and Stability Domains

1111

Order Stars and Stability Domains, Fig. 4 Order star for Adams2; principal root and parasitic root (left and middle); on Riemann surface (right)

O

Order Stars and Stability Domains, Fig. 5 Two-step method with three poles, order 5 (left; A-stable), order 6 (middle; A-stable), order 7 (right; not A-stable); R.z/ given by Butcher–Chipman approximations to ez

can be so easily extended to high orders, were in any respect superior to one-step methods. Slowly, during the 1960s and 1970s, it became apparent that this high order must be paid with lack of stability. Several papers of Jeltsch and Nevanlinna then brought more and more light into this question. Together with order star theory this then led to many important results in [10] and [11]. We illustrate the idea by comparing the stability domain of Runge–Kutta2 with that of Adams2 (see first picture of Fig. 6): RK2 possesses a larger stability domain than Adams2, but RK2 requires two derivative evaluations per step and Adams2 only one. So, for a fair comparison we must scalepthe stability domain, that is, we replace RRK2 .z/ by RRK2 .2z/ (second picture

of Fig. 6). We get a nice surprise: Neither method is always more stable than the other. In order to explain this, we compare the stability function of a method, not to jez j as in (2), but to the stability function of the other method, that is, we replace (2) by

jR1 .z/j >1 : B WD z 2 CI jR2 .z/j

(3)

We choose for R1 .z/ the principal root of Adams2 and for R2 .z/ the scaled stability function of RK2 (third picture of Fig. 6). The zeros of R2 .z/ turn into poles of the ratio and the order star must cross the scaled

1112

Orthogonal Polynomials: Computation SRK2

−2

2

1

scaled

0

−1

0

−2

−1

1

−1

0

−1

Order Stars and Stability Domains, Fig. 6 Comparing stability domains for RK2 versus Adams2 (left), scaled RK2 versus Adams2 (middle), proof of the Jeltsch–Nevanlinna theorem (right)

stability boundary of RK2 (both methods are explicit and have the same behavior at infinity).

Notes

8.

9.

Further results concern the Nørsett–Wolfbrandt conjecture for approximations with real poles (DIRK and 10. SIRK methods), Abdulle’s proof [1] of the Lebedev conjecture for Chebyshev approximations of high or- 11. der, and an application to delay differential equations [6]. For a thorough treatment of order stars see Sects. IV.4 and V.4 of [8] as well as the monograph 12. [9]. A variant of order stars – order arrows – has been introduced by Butcher [2].

References 1. Abdulle, A.: On roots and error constants of optimal stability polynomials. BIT 40(1), 177–182 (2000) 2. Butcher, J.C.: Numerical Methods for Ordinary Differential Equations. Wiley, Chichester (2003) ¨ 3. Courant, R., Friedrichs, K., Lewy, H.: Uber die partiellen Differenzengleichungen der mathematischen. Phys. Math. Ann. 100(1), 32–74 (1928) 4. Dahlquist, G.: A special stability problem for linear multistep methods. BIT 3, 27–43 (1963) 5. Ehle, B.L.: On Pad´e approximations to the exponential function and A-stable methods for the numerical solution of initial value problems. Tech. Report CSRR 2010, Department of AACS, University of Waterloo, Waterloo (1969) 6. Guglielmi, N., Hairer, E.: Order stars and stability for delay differential equations. Numer. Math. 83(3), 371–383 (1999) 7. Guillou, A., Lago, B.: Domaine de stabilit´e associ´e aux formules d’int´egration num´erique d’´equations diff´erentielles a`

pas s´epar´es et a` pas li´es, pp. 43–56. 1er Congress association France Calcul, AFCAL, Grenoble, Sept 1960 (1961) Hairer, E., Wanner, G.: Solving Ordinary Differential Equations II. Stiff and Differential-Algebraic Problems. Springer Series in Computational Mathematics, vol. 14, 2nd edn. Springer, Berlin (1996) Iserles, A., Nørsett, S.P.: Order Stars. Applied Mathematics and Mathematical Computation, vol. 2. Chapman, London (1991) Jeltsch, R., Nevanlinna, O.: Stability of explicit time discretizations for solving initial value problems. Numer. Math. 37(1), 61–91 (1981) Jeltsch, R., Nevanlinna, O.: Stability and accuracy of time discretizations for initial value problems. Numer. Math. 40(2), 245–296 (1982) Wanner, G., Hairer, E., Nørsett, S.P.: Order stars and stability theorems. BIT 18(4), 475–489 (1978)

Orthogonal Polynomials: Computation Francisco Marcell´an Departamento de Matem´aticas, Universidad Carlos III de Madrid, Legan´es, Spain

Orthogonality and Polynomials Orthogonality is defined in the linear space of polynomials with complex coefficients with respect to an inner product which involves a measure of integration supported on some subset E of the complex plane. If this subset is finite, then the discrete orthogonality appears. Sequences of orthogonal

Orthogonal Polynomials: Computation

polynomials (OP) are built when you apply the standard Gram-Schmidt process to the canonical basis .zn /n0 . This is the natural way to generate them. Indeed, if we denote by cj;k D< zj ; zk >; j; k  0; the moments of the inner product, a very well-known determinantal expression due to E. Heine gives the explicit representation of OP in terms of the moments, but in general it is not useful from a computational point of view. As an alternative way, they can be obtained as solutions of a linear system associated with the moment matrix, that is, an Hermitian matrix. The complexity of the computation of such polynomials is strongly related to the structure of such a matrix. Nevertheless, if the multiplication operator, i.e., the moment matrix, has some special structure with respect to the inner product, then you can compute these OP in a more simple way. Orthogonal Polynomials on the Real Line When the inner product is associated with an integration measure supported on a subset E of the real line, then < zp.z/; q.z/ >D< p.z/; zq.z/ > and the corresponding orthonormal polynomials .Pn /n0 satisfy a three-term recurrence relation zPn .z/ D anC1 PnC1 .z/ C bn Pn .z/ C an Pn1 .z/. This relation plays a central role in the computation of such orthogonal polynomials. Notice that the above recurrence relation yields a matrix representation zvnC1 .z/ D JnC1 vnC1 .z/ C anC1 PnC1 .z/enC1 . Here JnC1 is a tridiagonal and symmetric (Jacobi) matrix of size .nC1/.nC1/, vnC1 D .P0 .z/; : : : ; Pn .z//t , and enC1 D .0; : : : ; 0; 1/t . As a simple consequence of this fact, the zeros of PnC1 are real and simple and interlace with the zeros of Pn . Thus, you can compute them using the standard algorithms for the eigenvalue problem. On the other hand, the entries of the Jacobi matrix JnC1 can be obtained using an algorithm based on the modified moments of the measure of integration in order to have a better conditioned problem [3, 4]. As a nice application, the classical Gaussian quadrature rule for n C 1 nodes (the zeros of the orthonormal polynomial PnC1 ) reads P nC1;k f .xnC1;k / D 0 e1t f .JnC1 /e1 where as nC1 1 0 D< 1; 1 >; and e1 D .1; 0; : : : ; 0/t [2]. Orthogonal Polynomials on the Unit Circle When the inner product is associated with an integration measure supported on the unit circle, then

1113

< zp.z/; zq.z/ >D< p.z/; q.z/ >. The Gram matrix of this product in terms of the canonical basis is a Toeplitz matrix. Furthermore, the corresponding monic orthogonal polynomials .˚n /n0 satisfy a forward recurrence relation ˚nC1 .z/ D z˚n .z/ C ˚nC1 .0/˚n .z/, where j˚n .0/j < 1; n  1; are said to be the reflection parameters of the measure and ˚n .z/ denotes the reversed polynomial of ˚n .z/ [5]. This relation is related to the Levinson algorithm that provides the solution of a positive-definite Toeplitz system in a fast and robust way in O.n2 / flops instead of O.n3 / flops as in the case of classical algorithms to solve general (positive definite) systems of linear equations. Notice that the above recurrence relation, when orthonormal polynomials 'n are considered, yields a matrix representation zunC1 .z/ D HnC1 unC1 .z/ C nC1 'nC1 .z/enC1 ; nC1 D .1  j˚nC1 .0/j2 /1=2 , where HnC1 is a lower Hessenberg matrix of size .n C 1/  .n C 1/ and unC1 D .'0 .z/; : : : ; 'n .z//t . HnC1 is “almost unitary,” i.e., the first n rows form an orthonormal set and the last row is orthogonal to this set, but is not normalized and their eigenvalues are the zeros of the polynomial ˚nC1 . Taking into account these zeros belong to the unit disk, the analog of the Gaussian quadrature rule for n C 1 nodes on the unit circle must be reformulated in terms of the zeros of para-orthogonal polynomials BnC1 .z/ D ˚nC1 .z/ C  .z/ where j n j D 1; [1]. Indeed, they are the

n ˚nC1 eigenvalues of a unitary matrix given in terms of a perturbation of the last row of HnC1 .

References 1. Garza, L., Marcell´an, F.: Quadrature rules on the unit circle: a survey. In: Arves´u, J., Marcell´an, F., Mart´ınez Finkelshtein, A. (eds.) Recent Trends in Orthogonal Polynomials and Approximation Theory. Contemporary Mathematics, vol. 507, pp. 113–139. American Mathematical Society, Providence (2010) 2. Gautschi, W.: The interplay between classical analysis and (numerical) linear algebra-A tribute to Gene H. Golub. Electron. Trans. Numer. Anal. 13, 119–147 (2002) 3. Gautschi, W.: Orthogonal Polynomials: Computation and Approximation. Numerical Mathematics and Scientific Computation. Oxford Science Publications/Oxford University Press, New York (2004) 4. Gautschi, W.: Orthogonal polynomials, quadrature, and approximation: computational methods and software (in Matlab). In: Marcell´an, F., Van Assche, W. (eds.) Orthogonal

O

1114 Polynomials and Special Functions: Computation and Applications. Lecture Notes in Mathematics, vol. 1883, pp. 1–77. Springer, Berlin/Heidelberg (2006) 5. Simon, B.: Orthogonal Polynomials on the Unit Circle. American Mathematical Society Colloquium Publications, vol. 54 (2 volumes). American Mathematical Society, Providence (2005)

Oscillatory Problems

Examples and Types of Oscillatory Differential Equations One may roughly distinguish between problems with extrinsic oscillations, such as yR D y C ! 2 sin.!t/;

Oscillatory Problems Christian Lubich Mathematisches Institut, Universit¨at T¨ubingen, T¨ubingen, Germany

Synonyms Differential equations with high oscillations; Oscillatory differential equations The main challenges in the numerical treatment of ordinary differential equations with highly oscillatory solution behavior are twofold: • To use step sizes that are large compared to the fastest quasi-periods of the solution, so that the vector field, or the computationally expensive part of it, is not evaluated repeatedly within a quasiperiod • To preserve qualitatively and quantitatively correct solution behavior over times scales that are much longer than quasi-periods Standard numerical integrators, such as Runge– Kutta or multistep methods, typically fail in both respects, and hence special numerical integrators are needed, whose construction depends on the particular type of oscillatory problem at hand (Fig. 1).

Oscillatory Problems, Fig. 1 Oscillations and long time steps

!  1;

where the highly oscillatory behavior stems from explicitly time-dependent oscillatory source terms, and problems with intrinsic oscillations, such as yR D ! 2 y C sin.t/;

!  1;

where the oscillations are created by the differential equation itself. This occurs typically when the Jacobian matrix in the first-order formulation of the differential equation has large imaginary eigenvalues, in the above example ˙i!. An important class of problems are oscillatory Hamiltonian systems qP D rp H.q; p/;

pP D rq H.q; p/

with a Hamilton function H.q; p/, possibly depending in addition on the independent variable t, which has a positive semi-definite Hessian of large norm. The simplest example is the harmonic oscillator given by the Hamiltonian function H.p; q/ D 12 p 2 C 12 ! 2 q 2 , with the equations of motion qP D p, pP D ! 2 q, which combine to the second-order differential equation qR D ! 2 q. This is trivially solved exactly, a fact that can be exploited for constructing methods for problems with Hamiltonian H.p; q/ D

1 1 T 1 p M p C q T Aq C U.q/ 2 2

with a positive semi-definite constant stiffness matrix A of large norm, with a positive definite constant mass matrix M (subsequently taken as the identity matrix for convenience), and with a smooth potential U having moderately bounded derivatives. The chain of particles illustrated in Fig. 2 with equal harmonic stiff springs is an example of a system with a single constant high frequency 1=". With the midpoints and elongations of the stiff springs as position coordinates, we have

Oscillatory Problems

stiff harmonic

1115

soft nonlinear

Oscillatory Problems, Fig. 2 Chain with alternating soft nonlinear and stiff linear springs

AD

  1 0 0 ; "2 0 I

0 < "  1:

Other systems have several high frequencies as in AD

1 diag.0; !1 ; : : : ; !m /; "2

0 < "  1;

with 1  !1      !m , or a wide range of low to high frequencies without gap as in spatial discretizations of semilinear wave equations. The prototype model with explicitly time-dependent high frequencies is the harmonic oscillator with timevarying frequency, H.p; q; t/ D 12 p 2 C 12 "2 !.t/2 q 2 , with !.t/ and !.t/ P of magnitude 1 and "  1. Solutions of the equation of motion qR D "2 !.t/2 q oscillate with a quasi-period ", but the frequencies change on the slower time scale 1. The action (energy divided by frequency) I.t/ D H.p.t/; q.t//=!.t/ is an almost-conserved quantity, called an adiabatic invariant. Numerical methods designed for problems with nearly constant frequencies (and, more importantly, nearly constant eigenspaces) behave poorly on this problem, or on its higher-dimensional extension H.p; q; t/ D

1 1 T p M.t/1 p C 2 q T A.t/q C U.q; t/; 2 2"

which describes oscillations in a mechanical system undergoing a slow driven motion when M.t/ is a positive definite mass matrix, A.t/ is a positive semidefinite stiffness matrix, and U.q; t/ is a potential, all of which are assumed to be smooth with derivatives bounded independently of the small parameter ". This problem again has adiabatic invariants associated with each of its high frequencies as long as the frequencies remain separated. However, on small time intervals where eigenvalues almost cross, rapid non-adiabatic transitions may occur, leading to further numerical challenges.

Oscillatory Problems, Fig. 3 Triple pendulum with stiff springs

Similar difficulties are present, and related numerical approaches have been developed, for problems with state-dependent high frequencies such as H.p; q/ D

1 1 T p M.q/1 p C 2 V .q/ C U.q/; 2 "

with a constraining potential V .q/ that takes its minimum on a manifold and grows quadratically in nontangential directions, thus penalizing motions away from the manifold. In appropriate coordinates, we have V .q/ D 12 q1T A.q0 /q1 for q D .q0 ; q1 / with a positive definite matrix A.q0 /. A multiple spring pendulum with stiff springs as illustrated in Fig. 3 is a simple example, with angles as slow variables q0 and elongations of stiff springs as fast variables q1 . Here the frequencies of the high oscillations depend on the angles which change during the motion. As in the case of time-dependent frequencies, numerical and analytical difficulties arise when eigenfrequencies cross or come close, which here can lead to an indeterminacy of the slow motion in the limit " ! 0 (Takens chaos).

Building-Blocks of Long-Time-Step Methods: Averaging, Splitting, Linearizing, Corotating, Ansatzing Many numerical methods proposed for oscillatory differential equations are based on a handful of construction principles, which may possibly appear combined in various ways. Averages A basic principle underlying all long-time-step methods for oscillatory differential equations is the

O

1116

Oscillatory Problems

requirement to avoid isolated pointwise evaluations of oscillatory functions, but instead to rely on averaged quantities. We illustrate this for a method for second-order differential equations qR D f .q/;

The above method is efficient if solving the reduced equation over the whole interval Œh; h is computationally less expensive than evaluating the slow force f Œslow . Otherwise, to reduce the number of function evaluations we can replace the above average by an average with smaller support,

f .q/ D f Œslow .q/ C f Œfast .q/:

The classical St¨ormer–Verlet method with step size h uses a pointwise evaluation of f ,

Z qnC1  2qn C qn1 D h2

ı ı

Cf

 K./ f Œslow .qn /

Œfast



 u.h/ d

qnC1  2qn C qn1 D h2 f .qn /;

with ı  1 and an averaging kernel K./ with integral equal to 1. This is further approximated by a quadrawhereas the exact solution satisfies   ture sum involving the values f Œfast uN .mh=N / with jmj  M and 1  M  N . The resulting method q.t C h/  2q.t/ C q.t  h/ is an example of a heterogeneous multiscale method, Z 1   2 with macro-step h and micro-step h=N . .1  jj/ f q.t C h/ d: Dh 1 In the above methods, the slow force is evaluated, somewhat arbitrarily, at the particular value qn approxiThe integral on the right-hand side represents a mating the oscillatory solution q.t/. Instead, one might weighted average of the force along the solution, which evaluate f Œslow at an averaged position qN , defined by n will now be approximated. At t D tn , we replace solving approximately an approximate equation     f q.tn C h/  f Œslow .qn / C f Œfast u.h/ uR D f Œfast .u/; u.0/ D qn ; uP .0/ D 0; Z ı where u. / is a solution of the reduced differential e K./ u.h/ d; and setting qNn D equation ı uR D f Œslow .qn / C f Œfast .u/: We then have Z

1

h2 1

   .1  jj/ f Œslow .qn / C f Œfast u.h/ d

e with another averaging kernel K./ having integral 1. Such an approach can reduce the sensitivity to step size resonances in the numerical solution if ı D 1.

Splitting The St¨ormer–Verlet method can be interpreted as approximating the flow 'hH of a system with Hamiltonian For the reduced differential equation, we assume the H.p; q/ D T .p/ C V .q/ with T .p/ D 12 p T p by the initial values u.0/ D qn and uP .0/ D qPn or simply symmetric splitting uP .0/ D 0. This initial value problem is solved nuV V merically, e.g., by the St¨ormer–Verlet method with a ı 'hT ı 'h=2 : 'h=2 micro-step size ˙h=N with N  1 on the interval Œh; h, yielding numerical approximations uN .˙h/ In the situation of a potential V D V Œfast C V Œslow , and uP N .˙h/ to u.˙h/ and uP .˙h/, respectively. No we may instead use a different splitting of H D .T C further evaluations of f Œslow are needed for the comŒfast V / C V Œslow and approximate the flow 'hH of the putation of uN .˙h/ and uP N .˙h/. This finally gives the system by symmetric method D u.h/  2u.0/ C u.h/:

qnC1  2qn C qn1 D uN .h/  2uN .0/ C uN .h/:

Œslow

V 'h=2

ı 'hT CV

Œfast

Œslow

V ı 'h=2 :

Oscillatory Problems

1117

This is the impulse method: 1. Kick: set pnC D pn 

1 Œslow .qn / 2 h rV Œfast

2. Oscillate: solve qR D rV .q/ with initial values .qn ; pnC / over a time step h  / to obtain .qnC1 ; pnC1 1  pnC1 D pnC1  2 hrV Œslow .qnC1 /.

We illustrate the basic procedure for Schr¨odingertype equations 1 i P .t/ D H.t/ .t/; "

"  1;

with a time-dependent real symmetric matrix H.t/ changing on a time scale 1, for which the soluStep 2 must in general be computed approximately by tions are oscillatory with quasi-period ". A timea numerical integrator with a smaller time step. The dependent linear transformation .t/ D T" .t/ .t/ impulse method can be mollified by replacing the slow takes the system to the form Œslow Œslow .q/ by a modified potential V .q/, N potential V where qN represents a local average as considered i above. .t/ P D S" .t/ .t/ with S" D TP" T"1  T" H T"1 : " 3. Kick: set

Variation of Constants Formula A first approach is to freeze H.t/  H over a time A particular situation arises when the fast forces are step and to choose the transformation linear, as in   qR D Ax C g.q/ (1) it H T" .t/ D exp " with a symmetric positive semi-definite matrix A of large norm. With ˝ D A1=2 , the exact solution satisfies yielding a matrix function S" .t/ that is highly oscilla     q0 cos t˝ ˝ 1 sin t˝ q.t/ tory and bounded in norm by O.h="/ for jt  t0 j  h, (2) D qP0 ˝ sin t˝ cos t˝ q.t/ P if H D H.t0 C h=2/. Step sizes are still restricted  Z t  1 by h D O."/ in general, but can be chosen larger  ˝ sin.t  s/˝  in the special case when the derivatives of 1" H.t/ are g q.s/ ds: C cos.t  s/˝ 0 moderately bounded. A uniformly bounded matrix S" .t/ is obtained by Discretizing the integral in different ways gives rise diagonalizing to various numerical schemes. We mention a class of trigonometric integrators that reduces to the St¨ormer– H.t/ D Q.t/ .t/ Q.t/T ; Verlet method for A D 0 and gives the exact solution for g D 0. In its two-step version, the method reads with a real diagonal matrix .t/ D diag .j .t// and 2 an orthogonal matrix Q.t/ of eigenvectors depending qnC1  2 cos.h˝/ qn C qn1 D h g.˚ qn /: smoothly on t (possibly except where eigenvalues cross). The unitary adiabatic transformation Here  D .h˝/ and ˚ D .h˝/, where the filter functions and  are smooth, bounded, real-valued   i functions with .0/ D .0/ D 1. The choice of ˚.t/ Q.t/T .t/ .t/ D exp " the filter functions has a substantial influence on the Z t long-time properties of the method. The computation with ˚.t/ D diag . .t// D .s/ ds; j of the matrix functions times a vector can be done by 0 diagonalization of A or by Krylov subspace methods. represents the solution in a rotating frame of eigenvectors. Figure 4 illustrates the effect of this transformaTransformation to Corotating Variables For problems where the high frequencies and the cor- tion, showing solution components in the original and responding eigenspaces depend on time or on the in the adiabatic variables. The transformation to adiabatic variables yields solution, it is useful to transform to corotating variables a differential equation where the "-independent in the numerical treatment.

O

1118

Oscillatory Problems real part of η1 (t)

q1 (t) 1

real part of η1 (t) (zoomed)

0.01

0

1

0

x 10−3

0 −1 −1

0 time t

1

−0.01 −1

0 time t

−1 0.95

1

1 time t

Oscillatory Problems, Fig. 4 Oscillatory solution component and adiabatic variable as functions of time

P T Q.t/ is framed skew-symmetric matrix W .t/ D Q.t/ by oscillatory diagonal matrices:

yj .t/ 

X

zkj .t/e i.k!/t =" ;

kkkK

Pm with k  ! D `D1 k` !` and integers k` and with slowly varying modulation functions zkj .t/. For some problems with a time- or state-dependent high freThis differential equation is easier to solve numerically quency, a useful approach may come from a WKB than the original one. The simplest of methods freezes ansatz y.t/ D A.t/e i .t /=" the slow variables .t/ and W .t/ at the mid-point of the time step, makes a piecewise linear approximation to with slowly varying functions A.t/ and .t/, which the phase ˚.t/, and then integrates the resulting system may be further expanded in asymptotic series in powers exactly over the time step. This gives the following of ". In either case, the solution ansatz is inserted in the adiabatic integrator: differential equation and the coefficient and parameter functions are determined such that the approximation 1 gives a small defect in the differential equation. This nC1 D n C hB.tnC1=2 / . n C nC1 / with yields differential equations for the parameters which 2    are hopefully easier to solve.  i This entry is essentially an abridged version of the B.t/ D exp  j .t/  k .t/ " review article by Cohen, Jahnke et al. [1], which in turn     h is largely based on Chaps. XIII and XIV of the book by j .t/  k .t/ wj k .t/  sinc : Hairer et al. [2]. Both contain detailed references to the 2" j;k literature. The book by Leimkuhler and Reich [3] also treats numerical methods for highly oscillatory difNumerical challenges arise near almost-crossings of ferential equations. Analytical averaging techniques, eigenvalues, where .t/ remains no longer nearly useful also for the design of numerical methods, are constant and a careful step size selection strategy is described in the book by Sanders et al. [4]. 

i ˚.t/ .t/ P D exp "



  i W .t/ exp  ˚.t/ .t/: "

required in order to follow the rapid non-adiabatic transitions.

References

Problem-Adapted Solution Ansatz 1. Cohen, D., Jahnke, T., Lorenz, K., Lubich, C.: Numerical In some problems one may guess, or know from theory, integrators for highly oscillatory Hamiltonian systems: a review. In: Mielke, A. (ed.) Analysis, Modeling and Simulation an approximation ansatz for the solution with slowly of Multiscale Problems, pp. 553–576. Springer, Berlin (2006) varying parameters. For problems with m constant high 2. Hairer, E., Lubich, C., Wanner, G.: Geometric Numerical frequencies !` =", .` D 1; : : : ; m/, this ansatz may be Integration. Structure-Preserving Algorithms for Ordinary a modulated Fourier expansion Differential Equations, 2nd edn. Springer, Berlin (2006)

Overview of Inverse Problems 3. Leimkuhler, B., Reich, S.: Simulating Hamiltonian Dynamics. Cambridge Monographs on Applied and Computational Mathematics, vol. 14. Cambridge University Press, Cambridge (2004) 4. Sanders, J.A., Verhulst, F., Murdoch, J.: Averaging Methods in Nonlinear Dynamical Systems, 2nd edn. Springer, New York (2007)

Overview of Inverse Problems Joyce R. McLaughlin Department of Mathematical Sciences, Rensselaer Polytechnic Institute, Troy, NY, USA

Introduction In inverse problems, the goal is to find objects, sources, or changes in medium properties from indirectly related data. The solution is usually given as an image, and as such the word imaging is often a descriptor for an inverse problem. What distinguishes an inverse problem from, e.g., an image deblurring problem is that the mathematical model, for the process that generates the data, plays an integral role in the solution of the problem. Inverse problems are ill-posed. This means that (1) there may be more than one solution; (2) small changes in the data may yield large changes in the solution; or (3) a solution may not always exist, especially when the data is noisy. To improve the likelihood of a unique stable solution, several approaches are taken: (1) One is to add more data by doing a sequence of very similar experiments or by measuring additional properties of the output of an experiment; this improves the likelihood of a unique, sometimes stable, solution; when the data is noisy, a solution may not exist; however, an approximate solution may exist. (2) A second possibility is to treat the problem as an optimization problem adding a regularizing term to both improve the mathematical properties of the cost functional, which is to be optimized, and to add mathematical structure to find the approximation properties of the solution. (3) A third possibility is to reduce the sought-after properties of a solution; e.g., instead of looking for all the changes in a medium, one might seek only the support of the region where changes occur or one might look only for the discontinuities of the medium. And (4) a fourth

1119

possibility is to use coupled physics, that is, to utilize two physical properties simultaneously; an example of this is elastography where one mechanically creates shear motion in the tissue while simultaneously taking a sequence of RF/IQ (ultrasound) or a sequence of MR (magnetic resonance) data sets; see [88] or, for more examples, the Coupled Physics special issue of Inverse Problems, 28(8), 2012. An important feature in inverse problems is to utilize a realistic mathematical model whose numerical or exact solution can be shown to be consistent with measured data and to use the model to make the correct physical interpretation of the inverse problems solution. The mathematical structure utilized to obtain the solution is also related to the targeted solution feature. For example, (1) in the viscoelastic and wave equation models [55], the arrival time of waves together with the Eikonal equation can be utilized to find the fastest compression, shear, or acoustic wave speed; (2) microlocal analysis can be applied to a broad set of models and is well suited for finding discontinuities with essentially far-field data; (3) linear sampling and factorization methods yield the support of inhomogeneities in a constant background, again with far-field data; and (4) in media with well-distributed small-scale fluctuations and most accurately modeled as random media, the inverse problem is a source location problem and cannot be an inhomogeneity identification problem.

1D Inverse Spectral Problems A first question to ask is: Given (1) the form of a mathematical model, that is, a second-order eigenvalue problem with square integrable potential and Dirichlet boundary conditions on a finite interval, and (2) the eigenvalues for this problem, does there exist a unique potential corresponding to the mathematical model and the eigenvalues? The answer is yes provided that the eigenvalues satisfy a suitably general asymptotic form and the potential is symmetric on the interval [33, 40, 78]. If one relaxes the condition on the potential, that is, it is no longer symmetric, then one must add additional data to obtain existence and uniqueness of the solution. Here there are essentially three choices: (1) for each eigenfunction, add the ratio of the first derivative at an endpoint to the L2 norm of the eigenfunction [33]; (2) for each eigenfunction, add the ratio of the first

O

1120

derivatives at the endpoints for each of the eigenfunctions [78]; or (3) add a second sequence of eigenvalues where one of the boundary conditions is changed and has an unknown coefficient to be determined by the data [53]. Each of these additional sequences must satisfy a suitably general asymptotic form, as before. If one relaxes smoothness on the potential or equivalently changes the second-order differential equation to have a positive impedance with square integrable derivative; see [25, 26]; similar statements as in (2) above apply with the exception that the data sequences have weaker asymptotic forms, and with Dirichlet boundary conditions, one additional data must be added, e.g., the integral of the impedance over the interval where the problem is defined. If the smoothness is further relaxed so that, e.g., the positive impedance is of bounded variation, the asymptotics that can be established for the eigenvalues has very limited structure; see [35,38]. Here some progress has been made with the boundary control method; see [87], but there are still many open problems. See also [87] for more discussion of 1D inverse spectral problems and also [60].

Inverse Nodal Problems In 1D a related but very different problem to that described above is to use the nodes, or zeros, of eigenfunctions as data for the inverse problem. These are points in a vibrating system where there is no vibration. This inverse problem was first defined in [59] for a square integrable potential and was later extended to an impedance with integrable first derivative and densities in BV [36, 38]. Algorithms based on this idea have remarkable convergence properties, even under the weakest conditions on the unknown coefficients [36,38]. Extensive stability estimates in the case where the potential is square integrable or smoother are described in [51]. Inverse nodal problems in 2D are significantly more difficult partly due to the fact that the detailed asymptotics that can be obtained in 1D is much more difficult to obtain in 2D. Nevertheless, for a rectangular domain and with square integrable potential, a fundamental result is obtained in [37,61] that established uniqueness and an algorithm for finding the potential from nodal lines.

Overview of Inverse Problems

Elastography: A Coupled Physics Imaging Modality The goal in elastography is to create images of biomechanical properties of tissue. It is inspired by (1) the doctors’ palpation exam where the doctor presses against the skin to feel abnormalities in the body and (2) compression and vibration experiments that show cancerous tumors have shear wave speed with more than twice the value in normal tissue [90] or similar experiments that show fibrotic and cirrhotic liver can have shear moduli at least double that of normal liver tissue [95]. This capability, i.e., the imaging of biomechanical properties of tissue, is created by combining two physical properties of tissue. The basic idea is that the tissue is mechanically moved in a shearing motion, and while it is moving, a sequence of RF/IQ ultrasound data sets are acquired or a sequence of magnetic resonance, MR, data sets are acquired. The sequences are processed to produce a movie of the tissue moving within the targeted area of the body. The moving displacement data sets are the data for the inverse problem. The success of these experiments is based on the fact that tissue is mostly water. This means that the ultrasound data acquisition utilizes the fact that compression waves in the body travel at nearly 1,500 m/s ,while shear waves, which are the mechanically induced motion, travel at approximately 3 m/s in normal tissue. So the shear wave can be considered as fixed during a single ultrasound sequence data acquisition. The compression and the shear waves are the two physical properties that are coupled for this type of experiment. The MR acquired data sets are based on the spin of water molecules, a property distinct from the shear wave propagation; MR and shear wave propagation are thus the two physical properties that are coupled for this experiment. These coupled physics imaging data acquisition modalities are also referred to as hybrid imaging modalities. The mechanical motion is induced in one of three ways: (1) The tissue is moved with a sinusoidal motion at a rate from 60–300 Hz; both MR and RF/IQ data sets can produce the targeted movie; e.g., this is done with RF/IQ data sets in sonoelasticity [76] or with MR data sets in MRE [69, 70]. (2) Motion is induced by an interior radiation force push which is a pulse induced within the tissue by focused ultrasound [81]; excellent examples of the use of this method are supersonic

Overview of Inverse Problems

imaging [13] and ARFI and SWEI [73, 74]. And (3) the tissue is compressed in very small increments and is allowed to relax between compressions; see [75] for the initial compression experimental results. When the tissue is mechanically excited by a pulse, a wave propagates away from the pulse location. This wave has a front, a moving surface where ahead of the front there is very little motion and at the front there is large amplitude. The front has a finite propagation speed. The time, T, at which the front arrives at a given location, the arrival time, is the richest data subset in the movie data set. Under suitable hypotheses, it can be shown that T is the viscosity solution of an Eikonal equation, jgrad Tj2 D .1=c/2 , where c is the shear wave speed; this property is utilized by the arrival time algorithm [62–64]. The quantity, c, is the imaging functional. Another algorithm, the time-offlight algorithm utilizes a one-dimensional version of this Eikonal equation; see [13, 73], for applications in supersonic imaging or ARFI or SWEI. Note that the presence of viscoelasticity can be observed in the time trace at each pixel in the tissue as the pulse spreads in the direction behind the front as the pixel location is taken further and further from the initial pulse location. This occurs because the speed of the frequency content in the pulse propagates at a frequency-dependent wave speed with the fastest speeds at the highest frequencies. A mathematical model that contains both the viscoelastic effect and the finite propagation speed property is the generalized linear solid model. See [65] for theoretical results for the forward problem, including the finite propagation speed property, and [55] for the use of this model to create shear wave speed images with acoustic radiation force-induced (CAWE) crawling wave data. When the tissue is mechanically moved in a sinusoidal motion, a time harmonic wave is induced. The amplitude of this wave satisfies the time harmonic form of the linear elastic system and is sometimes simplified to the Helmholtz equation. A viscoelastic model is required, and in the time harmonic case, this yields complex-valued coefficients. The Helmholtz or elastic model, as opposed to the Eikonal equation, for the frequency-dependent displacement is utilized. The most often used imaging functional is the complexvalued shear modulus. The Fourier transform of the general linearized solid model is appropriate here. For this model, the complex-valued shear modulus

1121

approaches a finite limit as the frequency goes to infinity. Other viscoelastic models have been utilized; see [14, 43], [96]; in [43] and [14], the complex-valued shear modulus becomes unbounded as the frequency becomes large; at the same time, the application of the model in [43] is usually for fixed frequency, but not in [14]. For fixed frequency, the shear modulus satisfies a first-order partial differential equation system. The difficulties here are that (1) the solution, that is, the shear modulus, is complex valued so some known methods, e.g., computing along characteristic curves, cannot be employed, and (2) the coefficients of the first derivative terms can be zero on a large set of points, lines, or surfaces (but not in open subsets of the region of interest). This makes stability and uniqueness results difficult to obtain; however, uniqueness and stability results are contained in [41] where solutions and coefficients are assumed to be subanalytic; an earlier paper [2] achieved results when (1) the frequency is zero, (2) the dimension is 2, and (3) the shear modulus is real: under more general smoothness conditions. A number of approaches are utilized to overcome this difficulty of having possible zero values of the coefficients of the first derivative terms: (1) Set all derivatives of the shear modulus to zero and solve the resulting problem [88]; a bound on the error that is made when this is done, under the assumptions that the coefficients of the first derivative terms are nonzero, is contained in [54] when the coefficients are real; the same proof will yield a similar result when coefficients are complex valued. (2) Another is to first linearize the problem about a base problem, and then use multiple data sets to eliminate the problem of having zero coefficients of the derivatives of the sought-after shear modulus; see, for example, [8]. And (3) employ optimization and iterative [43] methods.

Inverse Scattering Problems The first type of inverse scattering problem is defined as follows. Suppose: (1) a constant, infinite in extent, background surrounds a bounded object or a bounded region containing an inhomogeneous medium; (2) one initiates an incoming plane harmonic wave that scatters from the object or region; and (3) the scattered wave is measured in the far field. Then the classic acoustic, electromagnetic, or elastic inverse scattering

O

1122

problem is to recover the object or inhomogeneity from the scattered data. Even when the forward problem is linear, the inverse problem is nonlinear. First the mathematical structure for the forward problem must be established, and then one can address the inverse problem. Considerable mathematical structure has been developed toward solving the inverse problem and the literature is vast. It has been shown, in the acoustic and electromagnetic case, that if one has scattering data at all outgoing angles from an incoming plane wave from a single direction and oscillating at a single frequency, then only one polygonal object can correspond to that data [56–58]; for the inhomogeneous medium problem, it is known that if one has scattering data at all outgoing angles, for incoming waves from all incoming directions and oscillating at a single frequency, then an inhomogeneous medium in a bounded region is unique [18, 21, 34, 71]. Nevertheless, the problem is severely ill-posed. The illposedness arises for several reasons, one being that the information content lies deep in the data and another being that some operators developed in the solution structure have only dense range. The latter means that solution methods can be unstable in the presence of noise in the data or when approximations of the data are used. To cope with the ill-posedness, several approaches are taken; here are a few examples: (1) The problem can be linearized; this is called the Born approximation; this doesn’t remove the ill-posedness, but the illposedness of the linearized problem is somewhat easier to analyze. (2) One can add data by including scattered waves from incoming waves, both from a full set of possible directions, and in addition, there are incoming waves, together with their scattered waves, with many frequencies of oscillation; this can be accomplished, as in [10] for the inverse medium problem where the Born approximation is used for each frequency of oscillation, and one iterates by solving the linearized problem for one frequency, using the output from the solution of that linearized problem as input for solving the linearized problem for another frequency, and so on, a similar method is applied in [11] for the inverse source problem. And/or (3) one can target a simpler property, e.g., find only the support of a bounded inhomogeneous region; this approach is taken in the linear sampling method [21, 22, 77] and in the factorization method [47, 52], where the problem is reformulated so that the boundary of the region or object is identified

Overview of Inverse Problems

as the points where an imaging functional becomes large. The second type of inverse scattering we consider is scattering of a wave in a half-space (e.g., a section of the earth relatively near the surface) from a source, for example, a buried explosive or an earthquake. Here it is often assumed that a smooth, slowly varying in space, background medium is known, and what is sought are the abrupt changes, usually referred to as discontinuities and also described as the highly oscillatory part of the medium. What is remarkable here is that in [15] it was shown, under certain assumptions, that the measured pressure field at locations on the surface could be expressed as integrals, referred to as transforms, over space-time surfaces. The integrand of these integrals contains the sought-after unknown in the half-space plus possibly some known quantities. When this transform is a Fourier integral operator (FIO), using concepts from microlocal analysis (see [74,83,89]), an inversion, containing two steps referred to as a migration step together with a microlocal filter, to recover the highly oscillatory part is possible (see [74] and the references therein). A third type of inverse scattering is more related to a method for solution than a specific physical setting and is referred to as an adjoint method [72]. The main feature is that the method makes use of the adjoint of the derivative of the forward map. An example is the iterative method known as the Kaczmarz method. We describe the method where there are a finite number of sources and a corresponding number of responses, measured, for example, on the boundary of the medium to be imaged. An initial guess is made for the medium. At each update, the adjoint of the derivative of the forward operator is computed for the current value of the medium. The new approximation of the medium is obtained by having that adjoint operate on the difference between the simulated “data” computed with the current iterate minus the measured data due to the next source in the iteration. See [72] for further discussion. The method has been applied to a wide range of problems including optical, impedance, ultrasound tomography, and computerized tomography (CT).

Computerized Tomography The inversion of the second inverse scattering problem described above has similarities with the inversion

Overview of Inverse Problems

of the Radon transform utilized in X-ray tomography. X-ray CT has a rich history with Cormack [27] and Hounsfield sharing a 1979 Nobel Prize for the discovery and initial practical implementation. The mathematical problem is the recovery of the X-ray attenuation (often referred to as density) from line integrals of the attenuation. This problem was first solved by Radon [79]; the integrals over lines or planes of an integrable function is now referred to as the Radon transform. There is a vast literature on inversion formulas to find the pointwise attenuation from line integrals of the attenuation. Of particular interest are as follows: (1) Inversion formulas that utilize Fourier multiplication operators, such as are derived from the Hilbert transform, and also the formal adjoint of the Radon transform; regularization can be applied to avoid mild ill-posedness that can occur with the use of such formulas; see [30]. (2) Formulas for data taken in 3D on helical curves; this is motivated by the use of a circular scanner that takes data as the patient is moved linearly through the scanner; remarkably Katsevich [46] discovered an exact formula when the attenuation is smooth and when it is assumed that the helix lies on a cylinder; additional very useful formulas were obtained in [32, 97] which relate the derivative of the line integral transform to the Hilbert transform. And (3) inversion formulas that utilize an expansion of the unknown attenuation in terms of a set of basis functions; of particular interest are basis functions known as “blobs” which have representations in terms of Bessel functions; once a representation is chosen, then the inversion is formulated as an iteration of algebraic reconstructions; see the discussion of ART in [39].

1123

When the medium contains very well-distributed scatterers of varying small sizes, it is not possible to image the inhomogeneities. What is remarkable is that, in this case, when those inhomogeneities can be thought of as being randomly distributed with a not too large variance, then the multiple scattering from the inhomogeneities provides a significant advantage in locating the source. Indeed what occurs is that (1) limited aperture measured signals, when back propagated, can have excellent refocusing properties; (2) the refocusing breaks the diffraction limit; (3) even in unbounded domains, the back propagated signals focus; and (4) the length of the measured received signals can be quite short. All of these properties have been well demonstrated experimentally by Mathias Fink’s group; see [31]. This is very much related to coherent interferometric imaging (see [16]), where the fundamental tool is local cross correlations of the data traces at nearby receivers, estimates of the frequency range for which wave fields are statistically correlated, and favorable resolution limits can be obtained. See also [9] for additional descriptions of the advantages of cross correlations when dealing with random media. As multiple scattering is the main physical property that is yielding the advantage here, additional work has been done in waveguides containing random media when additional multiple scattering from the boundaries provides even more advantage; see [1]. In contrast, see [28] for recovery of inhomogeneities in waveguides that do not contain random media.

Optical Tomography Time Reversal and Random Media Time reversal invariance in acoustic, elastic, and electromagnetic wave propagation is the basic concept behind time reversal mirrors. In free space, in an enclosed region, if a source emits a signal and (1) the signal is measured for a very long time, at all points on the boundary, and (2) the received signal on the boundary is time reversed and back propagated into the region, then the back propagated signal will focus at the original source location.

Optical tomography is an imaging modality that uses the transmission and reception of light at collectors to image properties of tissue. Light scatters significantly in tissue making the imaging problem quite difficult and ill-posed. Furthermore, the mathematical formulation of the inverse problem depends on the scale of the scattering that is taken into account; the mathematical formulation depends fundamentally on the experimental setup. The latter is well described in [5]. Mathematical formulations can utilize the radiative transport equation or the diffusion equation; both formulations are discussed in [85].

O

1124

Hybrid- or Coupled Physics-Based Imaging Modalities As medical imaging and nondestructive testing have become widely used, a broad set of experiments to provide useful data have been defined. Initially, a medium, e.g., tissue, was probed by a signal from outside the medium and the scattering from that signal, which is also measured outside of the medium, was utilized to obtain the image or the information needed for nondestructive testing. Typically, the edges of changes in the medium were the property that could most easily be imaged. In many cases, these edges could be imaged with high resolution. What was needed were new methods to determine changes within the regions which are defined by the edges or surfaces. This need is being addressed by coupled physics, or also called hybrid, modalities. In these modalities, two physical properties of the medium are utilized; e.g., (1) in elastography (see above paragraph), one uses sequences of RF/IQ ultrasound data, or sequences of MR data, the latter acquired while making a shearing motion in the tissue so that compression wave and water molecule spin properties of the medium are utilized; these ideas are incorporated in ultrasound and MR machines; (2) in photoacoustic imaging [49], where lowfrequency electromagnetic (EM) waves expose small regions of the medium to a short pulse, an acoustic wave is emitted and recorded outside the object, and from this data the electromagnetic absorption for the small region is determined; (3) ultrasound-modulated optical tomography or ultrasound-modulated electrical impedance tomography combine ultrasound which is used to perturb the medium with optical tomography or electrical impedance measurements (see [68]); and (4) magnetic resonance electric impedance tomography where MR is combined with electric impedance tomography (see [86]). See also [6] for additional coupled physics problems and results.

Inverse Boundary Problems and Inverse Source Problems An example of the inverse boundary problem is (1) posed on a bounded domain, (2) has an electromagnetic model for the forward problem, and (3) has application where one applies, e.g., all possible voltages at the boundary and, for each voltage distribution, one

Overview of Inverse Problems

measures the current on the boundary. This set of experiments gives data pairs (voltage and current) that define the Dirichlet to Neumann (DtN) map. A similar problem can be defined when the model is the acoustic or elastic model. The goal then is to determine unknown electric, acoustic, or elastic properties from the DtN map. A significant literature has built up studying uniqueness, stability, isotropic and anisotropic models, and the partial data problem. We begin with those mathematical models that are elliptic partial differential equations that are defined in the (time) frequency domain. Historically the problem was posed by Calder´on [23] in the isotropic electrostatic case, that is, the frequency is zero. A powerful tool that has been used to study, even the more general frequency not equal to zero problem and including acoustic, electrogmagnetic, and elastic models, is microlocal analysis. We address the uniqueness problem in the acoustic and electromagnetic problem. For this case, uniqueness for dimension n  3 has been established for positive conductivities that are somewhat less smooth than twice continuously differentiable but not as rough as Lipschitz continuous. For dimension two, uniqueness is established for essentially bounded positive conductivities. See [92] for a discussion of uniqueness and a method for finding the support of an obstacle for the electromagnetic case. One of the difficulties in using the DtN map in applications is that while the stability for recovering the conductivity from the data has been established, that stability is logarithmic and is quite weak; see [3]. Nevertheless, partial data, that is, knowledge of the DtN map on only part of the boundary, can also yield uniqueness for n  3. The unique recovery of anisotropic properties, properties that depend on direction, from the DtN is not possible as a change of variables that leaves the boundary and the boundary data fixed produces a counterexample to uniqueness. A major question is whether or not this change of variables property is the only obstruction to uniqueness. Toward this end, it has been shown that in two dimension, this is the only obstruction to uniqueness under very general smoothness conditions [7]. A related problem is an inverse source problem where Cauchy data, that is, a single Dirichlet and Neumann boundary data pair, is known for a secondorder, static, linear problem with known coefficients and an unknown source in a bounded domain in n

Overview of Inverse Problems

dimensions. From that data, one seeks to determine the source. There are a number of examples to show that the solution is not unique so a typical redefinition of the problem is to look for the minimum, square integrable source. Alternatively one can consider finding sources of constant value but confined to a subregion. Here to obtain uniqueness, one makes assumptions on the geometry of the region. The problem is very ill-posed with, as in the inverse DTN map problem, logarithmic stability. If the mathematical model for the physical problem contains a frequency term, as in the Helmholtz equation, or models time-dependent waves, with timedependent data, this substantively changes the problem; see [42]. In this case, use of continuation principles has played a significant role in uniqueness results.

Distributions, Fourier Transforms, and Microlocal Methods Microlocal methods are an important mathematical tool. The description of these methods requires an excellent understanding of distributions and Fourier transforms ; see [83] for this much needed background. As explained in [89], microlocal analysis is, roughly speaking, the local study of functions near points and including directions. These methods have been important in the study of inverse problems, particularly in the location of sharp changes of an unknown coefficient. The reason for this latter property is that microlocal methods enable the identification of the wave front set – a generalization of the notion of singular support of a function, which is the complement of the largest open set where a function is smooth. See [89] for more description and a related description of geometric optics.

Regularization A well-posed problem has a unique solution that depends continuously on data. Inverse problems are characteristically ill-posed. To deal with this difficulty, regularization can be employed. It is an optimization method whereby the original inverse problem is changed to another well-posed problem to obtain what is mathematically characterized as the best possible approximate solution to the given inverse problem

1125

given the characteristics of the data. A typical example of a regularization method is Tikhonov regularization; see [29]. Regularization methods are applied when (1) there is more than one solution to the problem (typically the minimum norm solution is sought) and/or (2) the forward operator has only dense range making the inverse operator unbounded; when data is noisy or approximate, it may not be in the range of the forward operator; filtering and/or projection and/or mollifying can be applied to find an approximate solution. For a large set of linear problems, the theory yields direct methods and these methods have been extensively applied. For linear problems, the regularization parameters can be chosen so that (1) when the inverse problem has a unique solution when exact data is given, (2) when the regularization parameters approach a given value, and (3)when the approximate data approaches the exact data, then the approximate inverse problems solution approaches the true solution. Tikhonov regularization, which seeks an approximate solution while minimizing the norm of the solution, can be applied in the nonlinear case. For Tikhonov regularized nonlinear problems, as well as other regularized nonlinear problems, iterative methods are often applied to get approximate solutions. These methods require a stopping criteria and nonlinearity conditions to establish convergence (see [29] and for example [45] and [84]). There is a wide range of additional considerations that must be taken into account when defining the spaces that define the norms for the regularization. If the mathematical setting for the regularization is set in Hilbert space, the inverse problems solution, or image, is typically smoothed. To avoid this, for problems where discontinuities or sharp changes in the recovered parameter, or image, are expected, the nonreflexive Banach space with the BV norm may be used. This changes the methods for establishing convergence; one of the tools is Bregman distance; see, e.g., [20] and [84]. Bregman distance is used to establish convergence when the derivative of the cost functional is a subdifferential, which is a set rather than a single operator. Other choices for Tikhonov regularization can include sparsity constraints; this choice is used if it is expected that the exact solution, or an excellent approximate solution, can be represented by the sum of a small number of terms. A typical norm in the regularization term is L1 in this case.

O

1126

Photonic Crystals and Waveguides and Inverse Optical Design

Overview of Inverse Problems

term parameter, œ, so there is reason to want to select it optimally. Often this is done by making several numerical calculations of approximate solutions. As an alternative, one can use statistical methods to make an optimal choice of the regularization parameter; one such method is cross validation [93, 94]. See [91] for more discussion about uncertainty quantification methods. See also [44] for discussion of statistical methods in inverse problems.

For inverse optical design, the goal is to determine a structure that has certain properties when electromagnetic waves are oscillating in the optical range and propagating through or along the surface of a dielectric material. One example is a photonic crystal where one starts with a periodic structure that has a bandgap that is an interval of oscillation frequencies, for which the wave decays exponentially. Then one introduces defects in the material that have the effect of enlarging the bandgap, and the aim is to place the defects so as to References maximize the bandgap. The goal is to have a large set of oscillating waves that have very little amplitude after 1. Acosta, S., Alonso, R., Borcea, L.: Source estimation with incoherent waves in random waveguides. Inverse Probl. 31, passing through the structure. This problem ultimately 35 (2015) results in maximizing the difference of two eigenval- 2. Alessandrini, G.: An identification problem for an elliptic equation in two variables. Ann. Mat. Pura Appl. 145, ues, a nonlinear optimization problem. 265–296 (1986) Another problem is a shape optimization problem 3. Alessandrini, G.: Stable determination of conductivity by where one seeks to design a rough surface grating boundary measurements. Appl. Anal. 27, 153–172 (1988) coupler that will focus light to nanoscale wave lengths 4. Ammari, H., Garapon, P., Kang, H., Lee, H.: A method of biological tissues elasticity reconstruction using magnetic while maximizing the power output of the structure resonance elastography measurements. Q. Appl. Math. 66, (see [14]). An overview of an adjoint method that has 139–175 (2008) been utilized successfully in optimal design, together 5. Arridge, S.: Optical Tomography: Applications. This Encywith an explanation that motivates the method steps, clopedia (2015) and some examples where the method successfully 6. Arridge, S.R., Scherzer, O. (eds.): Special section: imaging from coupled physics. Inverse Probl. 28(8), 080201–084009 produced a useful design are in [67].

Statistical Methods for Uncertainty Quantification In inverse problems, one often encounters the following scenario. We are given noisy data, y C ©, where © represents the noise. The exact data y D Kf, where K is an operator with unbounded inverse. The goal is to recover f. An approximation to f is determined using the noisy data, by solving an optimization problem with a regularizing term (see [29]) containing a parameter, œ. The regularizing term introduces a bias in the approximate, fa of f. When the statistics of the noise, ©, are known, that is, the mean and variance of the noise are known, under certain hypotheses, one can recover the statistics for the approximate, fa , including the statistics of the bias. This, then, is a method for quantifying the uncertainty in the approximate solution. This can be a significant aid in the interpretation of the approximate, fa . We note that all of this analysis includes the regularizing

(2012) 7. Astala, K., Lassas, M., P¨aiv¨arinta, L.: Calder´on inverse problem for anisotropic conductivity in the plane. Commun. Partial Differ. Equ. 30, 207–224 (2005) 8. Bal, G., Uhlmann, G.: Reconstruction of coefficients in scalar second-order elliptic equations from knowledge of their solutions. Commun. Pure Appl. Math. 66, 1629–1652 (2013) 9. Bal, G., Pinaud, O., Ryzhik, L.: Random Media in Inverse Problems, Theoretical Aspects. This Encyclopedia (2015) 10. Bao, G., Triki, F.: Error estimates for the recursive linearization for solving inverse medium problems. J. Comput. Math. 28, 725–744 (2010) 11. Bao, G., Lin, J., Triki, F.: A multi-frequency inverse source problem. J. Differ. Equ. 249, 3443–3465 (2010) 12. Belishev, M.I.: Boundary Control Method. This Encyclopedia (2015) 13. Bercoff, J., Tanter, M., Fink, M.: Supersonic shear imaging: a new technique for soft tissue elasticity mapping. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 19, 396–40 (2004) 14. Bercoff, J., Tanter, M., Muller, M., Fink, M.: The role of viscosity in the impulse diffraction field of elastic waves induced by the acoustic radiation force. IEEE Trans Ultrason. Ferroelectr. Freq. Control 51(11), 1523–1536 (2004) 15. Beylkin, G.: Imaging of discontinuities in the inverse scattering problem by inversion of a causal generalized Radon transform. J. Math. Phys. 26, 99–108 (1985)

Overview of Inverse Problems 16. Borcea, L.: Interferometric Imaging and Time Reversal in Random Media. This Encyclopedia (2015) 17. Borg, B.: Eine Umkerung der Sturm Liouville Eigenwertaufgabe. Acta Math. 78, 1–96 (1946) 18. Bukhgeim, A.: Recovering a potential from Cauchy data in the two-dimensional case. J. Ill-Posed Probl. 16, 19–33 (2008) 19. Burger, M.: Photonic Crystals and Waveguides. Simulation and Design. This Encyclopedia (2015) 20. Burger, M., Osher, S.: Convergence rates of convex variational regularization. Inverse Probl. 20(5), 1411–1421 (2004) 21. Cakoni, F.: Inhomogeneous Media Identification. This Encyclopedia (2015) 22. Cakoni, F., Colton, D., Monk, P.: The Linear Sampling Method in Inverse Electromagnetic Scattering. CBMS-NSF, vol. 80. SIAM Publications, Philadelphia (2010) 23. Calder´on, A.P.: On an inverse boundary value problem. In: Seminar on Numerical Analysis and Its Applications to Continuum Physics, Rio de Janeiro pp. 65–73. Sociedade Brasileira de Matematica, Rio De Janeiro (1080) 24. Cheney, M., Borden, B.: Radar Imaging. This Encyclopedia (2015) 25. Coleman, C.F., McLaughlin, J.R.: Solution of the inverse spectral problem for an impedance with integrable derivative, Part I. CPAM XLVI, 145–184 (1993) 26. Coleman, C.F., McLaughlin, J.R.: Solution of the inverse spectral problem for an impedance with integrable derivative, Part II. CPAM XLVI, 185–212 (1993) 27. Cormack, A.: Representation of a function by its line integrals, with some radiological applications. J. Appl. Phys. 34(9), 2722–2727 (1963) 28. Dediu, S., McLaughlin, J.: Recovering inhomogeneities in a waveguide using eigensystem decomposition. Inverse Probl. 22, 1227–1246 (2006) 29. Engl, H.W., Ramlau, R.: Regularization of Inverse Problems. This Encyclopedia (2015) 30. Finch, D.V., Faridani, A.: X-Ray Transmission Tomography. This Encyclopedia (2015) 31. Fink, M.: Time Reversal Experiments in Acoustics. This Encyclopedia (2015) 32. Gel’fand, I.M., Graev, J.: Crofton’s function and the inversion formulas in real integral geometry. Funct. Anal. Appl. 25, 1–5 (1991) 33. Gel’fand, I.M., Levitan, B.M.: On the determination of a differential equation from its special function. Izv. Akad. Nauk SSR. Ser. Mat. 15, 309–360 (1951) (Russian): English transl. in Am. Math. Soc. Transl. Ser. 2(1), 253–304 (1955) 34. H¨ahner, P.: Electromagnetic wave scattering. In: Pike, R., Sabatier, P. (eds.) Scattering. Academic, New York (2002) 35. Hald, O.: Discontinuous inverse eigenvalue problems. CPAM 37, 539–577 (1984) 36. Hald, O.H., McLaughlin, J.R.: Solutions of inverse nodal problems. Inverse Probl. 5, 307–347 (1989) 37. Hald, O.H., McLaughlin, J.R.: Inverse nodal problems: finding the potential from nodal lines. Mem. Am. Math. Soc. 119(572), 146 (1996) 38. Hald, O.H., McLaughlin, J.R.: Inverse problems: recovery of BV coefficients from nodes. Inverse Probl. 14(2), 245– 273 (1998)

1127 39. Herman, G.: Computerized Tomography. ART, This Encyclopedia (2015) 40. Hochstadt, H.: Asymptotic estimates for the Sturm– Liouville spectrum. CPAM 14, 749–764 (1961) 41. Honda, N., McLaughlin, J., Nakamura, G.: Conditional stability for a single interior measurement. Inverse Probl. 30, 19 (2014) 42. Isakov, V.: Locating a Source. This Encyclopedia (2015) 43. Jiang, Y., Fujiwara, H., Nakamura, G.: Approximate steady state models for magnetic resonance elastography. SIAM J. Appl. Math. 71(6), 1965–1989 (2011) 44. Kaipio, J., Somersalo, E.: Statistical and Computational Inverse Problems. Springer, Berlin/Heidelberg/New York (2004) 45. Kaltenbacher, B., Hofmann, B.: Convergence rates for the iteratively regularized Gauss-Newton method in Banach spaces. Inverse Probl. 26, 035007 (2010) 46. Katsevich, A.: An improved exact filtered backprojection inversion algorithm for spiral cone-beam CT. Adv. Appl. Math. 32, 681–697 (2004) 47. Kirsch, A., Grinbert, N.: The Factorization Method for Inverse Problems. Oxford Lecture Series in Mathematics and Its Applications, vol. 36. Oxford University Press, Oxford (2008) 48. Klein, J., McLaughlin, J., Renzi, D.: Improving arrival time identification in transient elastography. Phys. Med. Biol. 57(8), 2151–2168 (2012) 49. Kuchment, P., Scherzer, O.: Mathematical Methods in Photo- and Thermoacoustic Imaging. This Encyclopedia (2015) 50. Lassas, M., Milton, G.: Invisibility Cloaking. This Encyclopedia (2015) 51. Law, C.-K.: Inverse Nodal Problems 1D. This Encyclopedia (2015) 52. Lechleiter, A.: Factorization Method in Inverse Scattering. This Encyclopedia (2015) 53. Levitan, B.M.: Inverse Sturm-Liouville Problems. VNU Science Press, Ultrecht (1997) 54. Lin, K., McLaughlin, J.: An error estimate on the direct inversion model in shear stiffness imaging. Inverse Probl. 25(7), 19 (2009) 55. Lin, K., Mclaughlin, J., Thomas, A., Parker, K., Castaneda, B., Rubens, D.: Two-dimensional shear wave speed and crawling wave speed recoveries from in vitro prostate data. J. Acoust. Soc. Am. 130(1), 585–98 (2011) 56. Liu, H.: A global uniqueness for formally determined inverse electromagnetic obstacle scattering. Inverse Probl. 24, 13 (2008) 57. Liu, H., Zou, J.: On uniqueness in inverse acoustic and electromagnetic obstacle scattering problems. 4th AIP International Conference and the 1st Congress of the IPIA. J. Phys.: Conf. Ser. 124, 12 (2006) 58. Liu, H., Zou, J.: Uniqueness in an inverse acoustic obstacle scattering problem for both sound-hard and soundsoft polyhedral scatterers. Inverse Probl. 23, 515–524 (2006) 59. McLaughlin, J.R.: Inverse spectral theory using nodal points as data – a uniqueness result. J. Differ. Equ. 73, 354–362 (1988) 60. McLaughlin, J.R.: Solving inverse problems with spectral data. In: Colton, D., Engl, H., Louis, A., McLaughlin, J.,

O

1128

61. 62.

63.

64. 65.

66.

67. 68.

69. 70.

71. 72. 73.

74. 75.

76.

77. 78.

Rundell, W. (eds.) Surveys on Solution Methods for Inverse Problems, pp. 169–194. Springer, New York (2000) McLaughlin, J., Hald, H.: A formula for finding a potential from nodal lines. Bull. Am. Math. Soc. 32, 241–247 (1995) McLaughlin, J., Renzi, D.: Shear wave speed recovery in transient elastography and supersonic imaging using propagating fronts. Inverse Probl. 22, 681–706 (2006) McLaughlin, J., Renzi, D.: Using level set based inversion of arrival times to recover shear wavespeed in transient elastography and supersonic imaging. Inverse Probl. 22, 707–725 (2006) McLaughlin, J., Yoon, J.-R.: Arrival times for the wave equation. Commun. Pure Appl. Math. 64(3), 313–327 (2011) Mclaughlin, J., Thomas, A., Yoon, J.R.: Basic theory for generalized linear solid viscoelastic models. In: Bal, G., Finch, D., Kuchment, P., Schotland, J., Stefanov, P., Uhlmann, G. (eds.) AMS Contemporary Mathematics Volume: Tomography and Inverse Transport Theory, pp. 101– 134. American Mathematical Society, Providence (2011) McLaughlin, J.R., Oberai, A.A., Yoon, J.R.: Formulas for detecting a spherical stiff inclusion from interior data: a sensitivity analysis for the Helmholtz equation. Inverse Probl. 28(8, Special Issue on Coupled Physics), 21 (2012) Miller, O.D., Yablonovitch, E.: Inverse Optical Design. This Encyclopedia (2015) Monard, F., Bal, G.: Inverse anisotropic diffusion from power density measurements in two dimensions. Inverse Probl. 28, 20 (2012) Muthupillai, R., Ehman, R.: Magnetic resonance elastography. Nat. Med. 2, 601–603 (1996) Muthupillari, R., Lomas, D.J., Rossman, P.J., Greenleaf, J.F., Manduca, A., Ehman R.L. (1995) Magnetic resonance elastography by direct visualization of propagating acoustic strain wave. Science. 269, 1854–1857 Nachman, A.: Reconstructions from boundary measurements. Ann. Math. 128, 531–576 (1988) Natterer, F.: Adjoint Methods as Applied to Inverse Problems. This Encyclopedia (2015) Nightingale, K.R., Palmeri, M.L., Nightingale, R.W., Trahey, G.E.: On the feasibility of remote palpation using acoustic radiation force. J. Acoust. Soc. Am. 110, 625–634 (2001) Nolan, C.: Inversion Formula in Inverse Scattering. This Encyclopedia (2015) Ophir, J., Cespede, I., Ponnekanti, H., Yazdi, Y., Li, X.: Elastography: a quantitative method for imaging the elasticity of biological tissues. Ultrason. Imaging 13, 111–134 (1991) Parker, K., Fu, D., Gracewski, S., Yeung, F., Levinson, S.: Vibration sonoelastography and the detectability of lesions. Ultrasound Med. Biol. 24, 1937–1947 (1998) Piana, M.: The Linear Sampling Method. This Encyclopedia P¨oschel, J., Trubowitz, E.: Inverse Spectral Theory. Academic, Boston (1986)

Overview of Inverse Problems ¨ 79. Radon, J.: Uber die Bestimmung von Funktionen durch ihre Integralwerte l¨angs gewisser Mannigfaltigkeiten. Ber Verh S¨achs Adad Wiss Leipzig Math-phys Kl 69, 262–277 (1917) 80. Rouviere, O., Yin, M., Dresner, M., Rossman, P., Burgart, L., Fidler, J., Ehman, R.: MR elastography of the liver: preliminary results. Radiology 240:440–448 (2006) 81. Saarvazyan, A., Emelinnov, S., O’Donnell, M.: Tissue elasticity reconstruction based on ultrasonic displacement and strain imaging. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 42, 747–65 (1995) 82. Sacks, P.: Inverse Spectral Problems, 1-D. This Encyclopedia (2015) 83. Salo, M.: Distributions and Fourier Transform. This Encyclopedia (2015) 84. Scherzer, O., Grasmair, M., Grossauer, H., Haltmeier, M., Lenzen, F.: Variational Methods in Imaging. Springer, Berlin (2008) 85. Schotland, J.C.: Optical Tomography: Theory. This Encyclopedia (2015) 86. Seo, J., Kim, D.-H., Lee, J., Kwon, O.I., Sajib, S.Z.K., Woo, E.J.: Electrical tissue property imaging using MRI and dc and Larmor frequency. 28, 26 (2012) 87. Sini, M.: Inverse Spectral Problems. 1-D, Theoretical Results, This Encyclopedia (2015) 88. Sinkus, R.: MR-Elastography. This Encyclopedia (2015) 89. Stefanov, P.: Microlocal Analysis Methods. This Encyclopedia (2015) 90. Tanter, M., Bercoff, J., Athanasiou, A., Deffleux, T., Gennisson, J.L., Montaido, G., et al.: Quantitative assessment of breast lesion viscoelasticity; initial clinical results using supersonic imaging. Ultrasound Med. Biol. 34(9), 1373–1386 (2008) 91. Tenorio, L.: Inverse Problems: Statistical Methods for Uncertainty Quantification. This Encyclopedia (2015) 92. Uhlmann, G., Zhou, T.: Inverse Electromagnetic Problems. This Encyclopedia (2015) 93. Wahba, G.: Baysian “confidence intervals’ for the crossvalidated smoothing spline. J. R. Stat. Soc. B 45:133 (1983) 94. Want, Y., Wahba, G.: Bootstrap confidence intervals for smoothing splines and their comparision to Bayesian confidence intervals, J. Stat. Comput. Simul. 51, 263 (1995) 95. Yin, M., Talwalkar, J.A., Glaser, K.J., Manduca, A., Grimm, R.D., Rossman, P.J., Fidler, J.L., Ehman, R.L.: Assessment of hepatic fibrosis with magnetic resonance elastography. Clin. Gastroenterol. Hepatol. 5(10), 1207–1213 (2007) 96. Zhang, M., Nigwekar, P., Casstaneda, B., Hoyt, K., Joseph, J.V., Di Sant’Agnese, A., Messing, E.M., Strang, J.G., Rubens, D.J., Parker, K.J.: Quantitative characterization of viscoelastic properties of human prostate correlated with histology. Ultrasound Med. Biol. 34(7), 1033–1042 (2008) 97. Zou, Y., Pan, X.: Exact image reconstruction on PI-lines from minimum data in helical cone-beam CT. Phys. Med. Biol. 49, 941–959 (2004)

P

Parallel Computing Xing Cai Simula Research Laboratory, Center for Biomedical Computing, Fornebu, Norway University of Oslo, Oslo, Norway

Introduction Parallel computing can be understood as solving a computational problem through collaborative use of multiple resources that belong to a parallel computer system. Here, a parallel system can be anything between a single multiprocessor machine and an Internet-connected cluster that is made up of hybrid compute nodes. There are two main motivations for adopting parallel computations. The first motivation is about reducing the computational time, because employing more computational units for solving a same problem usually results in lower wall-time usage. The second – and perhaps more important – motivation is the wish of obtaining more details, which can arise from higher temporal and spatial resolutions, more advanced mathematical and numerical models, and more realizations. In this latter situation, parallel computing enables us to handle a larger amount of computation under the same amount of wall time. Very often, it also gives us access to more computer memory, which is essential for many large computational problems. The most important issues for understanding parallel computing are finding parallelism, implementing parallel code, and evaluating the performance. These will be briefly explained in the following text, with simple supporting examples.

Identifying Parallelism Parallelism roughly means that some work of a computational problem can be divided into a number of simultaneously computable pieces. The applicability of parallel computing to a computational problem relies on the existence of inherent parallelism in some form. Otherwise, a parallel computer will not help at all. Let us take, for example, the standard axpy operation, which updates a vector y by adding it to another vector x as follows: y

’x C y;

where ’ is a scalar constant. If we look at the entries of the y vector, y1 ; y2 ; : : : ; yn , we notice that computing yi is totally independent of yj , thus making each entry of y a simultaneously computable piece. For instance, we can employ n workers, each calculating a single entry of y. The above example is extremely simple, because the n pieces of computation are completely independent of each other. Such a computational problem is often termed embarrassingly parallel. For other problems, however, parallelism may be in disguise. This can be exemplified by the dot product between two vectors x and y:

d D x  y WD

n X

xi yi D x1 y1 C x2 y2 C : : : C xn yn :

i D1

At a first glance, parallelism is not obvious within a dot product. However, if an intermediate vector d is introduced, such that di D xi yi , then parallelism

© Springer-Verlag Berlin Heidelberg 2015 B. Engquist (ed.), Encyclopedia of Applied and Computational Mathematics, DOI 10.1007/978-3-540-70529-1

1130

Parallel Computing

immediately becomes evident because the n entries of The above examples are only meant for illustrathe d vector can be computed simultaneously. Never- tion. Parallelism in practical computational problems theless, the remaining computational task exists in many more different forms. An incomplete list of frequently encountered types of parallelizable d D 0; d d C di for i D 1; 2; : : : ; n computations involves dense linear-algebra operations, requires collaboration and coordination among n work- sparse linear-algebra operations, explicit and implicit ers to extract parallelism. The idea is as follows. First, computations associated with regular meshes, implicit each worker with an odd-number ID adds its value computations associated with irregular meshes, fast with the value from the neighboring worker with an Fourier transforms, and many-body computations. ID of one higher. Thereafter, all the even-numbered workers retire and the remaining workers repeat the Parallelization same process until there is only one worker left. The solely surviving worker possesses the desired final Finding parallelism in a computational problem is value of d . Actually, this is how a parallel reduction op- only the start. A formal approach to designing parallel eration is typically implemented. We can also see that algorithms is Foster’s Methodology [2, 7], which is a the parallelized summation has dlog2 ne stages, each four-step process. The first step is partitioning, which involving simultaneous additions between two and two cuts up the concurrent computational work and/or workers. Although it may seem that the parallel version the accompanying data into as many small pieces as should be dramatically faster than the original serial possible. The second step of Foster’s Methodology version of summation, which has n stages, we have to is about finding out what data should be exchanged remember that each stage in the parallel counterpart between which pieces. The third step is about agglomrequires data transfer between two and two workers, erating the many small pieces into a few larger tasks, to causing the so-called communication overhead. obtain an appropriate level of granularity with respect What is more intriguing is that parallelism can to the hardware resources on a target parallel comexist on different levels. Let us revisit the example of puter. The last step of Foster’s Methodology is about summing up the d vector, but assume now that the mapping the tasks to the actual hardware resources, so number of workers, m, is smaller than the vector length that load balance is achieved and that the resulting data n. In such a case, each worker becomes responsible communication cost is low. A rule of thumb regarding for several entries of the d vector, and here are several communication is that two consecutive data transfers, issues that require our attention: between the same sender and receiver, are more costly 1. The n entries of the d vector should be divided than one merged data transfer. This is because each among the m workers as evenly as possible. This is data transfer typically incurs a constant start-up cost, called load balancing. For this particular example, termed latency, which is independent of the amount of even when n is not a multiple of m, a fair work data to be transferred. division makes the heaviest and lightest loaded To make a parallel algorithm run efficiently on a workers only differ by one entry. parallel computer, the underlying hardware architec2. Suppose each worker prefers a contiguous segment ture has to be considered. Although parallel hardware of d, then worker k, 1  k  m, should be responsi- architectures can be categorized in many ways, the ble for entry indices from ..k  1/  n/=m C 1 until most widely adopted consideration is about whether .k  n/=m. Here, we let / denote the conventional the compute units of a parallel system share the same integer division in computer science. memory address space. If yes, the parallel system is 3. The local summations by the m workers over their termed shared memory, whereas the other scenario is assigned entries can be done simultaneously, and called distributed memory. It should be mentioned that each worker stores its local summation result in a many parallel systems nowadays have a hybrid design temporary scalar value dks . with respect to the memory organization, having a 4. Finally, the m local summation results dks , 1  distributed-memory layout on the top level, whereas k  m, can be added up using a parallel reduction each compute node is itself a small shared-memory operation as described above. system.

Parallel Computing

Luckily, the styles of parallel programming are less diverse than the different parallel architectures produced by different hardware vendors. The MPI programming standard [3, 6] is currently the most widely used. Although designed for distributed memory, MPI programming can also be applied on shared-memory systems. An MPI program operates a number of MPI processes, each with its own private memory. Computational data should be decomposed and distributed among the processes, and duplication of (global) data should be avoided. Necessary data transfers are enabled in MPI by invoking specific MPI functions at appropriate places of a parallel program. Data, which are called messages in MPI terminology, can be either passed from one sender process to a receiver process or exchanged collectively among a group of processes. Parallel reduction operations are namely implemented as collective communications in MPI. OpenMP [1] is a main alternative programming standard to MPI. The advantage of OpenMP is its simplicity and minimally intrusive programming style, whereas the performance of an OpenMP program is most often inferior to that of an equivalent MPI implementation. Moreover, OpenMP programs can only work on shared-memory systems. With the advent of GPUs as main accelerators for CPUs, two new programming standards have emerged as well. The CUDA [4] hardware abstraction and programming language extension are tied to the hardware vendor NVIDIA, whereas the OpenCL framework [5] targets heterogeneous platforms that consist of both GPUs and CPUs and possibly other processors. In comparison with MPI/OpenMP programming, there are considerably more details involved with both CUDA and OpenCL. The programmer is responsible for host-device data transfers, mapping computational work to the numerous computational units – threads – of a GPU, plus implementing the computations to be executed by each thread. In order to use modern GPU-enhanced clusters, MPI programming is typically combined with CUDA or OpenCL.

1131

as strong and weak scalability. The former investigates speedup, i.e., how quickly the wall-time usage can be reduced when more compute units are used to solve a fixed-size computational problem. The latter focuses on whether the wall-time usage remains as constant when the problem size increases linearly proportional to the number of compute units used. The blame for not achieving good scalability has traditionally been put too much on the nonparallelizable fraction of a computational problem, giving rise to the famous laws of Amdahl and GustafsonBarsis. However, for large enough computational problems, the amount of inherently serial work is often negligible. The obstacle to perfect scalability thus lies with different forms of parallelization overhead. In addition to the already mentioned overhead due to data transfers, there are other types of overhead that can be associated with parallel computations: • Parallel algorithms may incur extra calculations that are not relevant for the original serial computational problems. Data decomposition, such as finding out the index range of a decomposed segment of a vector, typically requires such extra calculations. • Synchronization is often needed between computational tasks. A simple example can be found in the parallel reduction operation, where all pairs of workers have to complete, before proceeding to the next stage. • Sometimes, in order to avoid data transfers, duplicated computations may be adopted between neighbors. • In case of load imbalance, because either the target computational problem is impossible to be decomposed evenly or an ideal decomposition is too costly to compute, some hardware units may from time to time stay idle while waiting for the others. It should be mentioned that there are also factors that may be scalability friendly. First, many parallel systems have the capability of carrying out communications at the same time of computations. This gives the possibility of hiding the communication overhead. However, to enable communication-computation overlap can be a challenging programming task. Second, it sometimes happens that by using many nodes of a distributed-memory system, the subproblem per node Performance of Parallel Programs falls beneath a certain threshold size, thus suddenly It is common to check the quality of parallel programs giving rise to a much better utilization of the local by looking at their scalability, which is further divided caches.

P

1132

References 1. Chapman, B., Jost, G., van der Pas, R.: Using OpenMP: Portable Shared Memory Parallel Programming. MIT, Cambridge (2007) 2. Foster, I.: Designing and Building Parallel Programs. Addison-Wesley, Reading (1995) 3. Gropp, W., Lusk, E., Skjellum, A.: Using MPI, 2nd edn. MIT, Cambridge (1999) 4. Kirk, D.B., Hwu, W.-m.W.: Programming Massively Parallel Processors: A Hands-on Approach. Morgan Kaufmann, Burlington (2010) 5. Munshi, A., Gaster, B., Mattson, T.G., Fung, J., Ginsburg, D.: OpenCL Programming Guide. AddisonWesley, Upper Saddle River (2011) 6. Pacheco, P.S.: Parallel Programming With MPI. Morgan Kaufmann, San Francisco (1997) 7. Quinn, M.J.: Parallel Programming in C with MPI and OpenMP. McGrawHill, Boston (2003)

Parallel Computing Architectures Petter E. Bjørstad Department of Informatics, University of Bergen, Bergen, Norway

Introduction Parallel computing architectures are today the only means for doing computational science. It was not always so. Computers have been used for scientific purposes since they emerged shortly after the Second World War. The concept of a supercomputer, designed to be faster than other computers and specifically targeting computational science applications, took a leap forward in 1976 with the Cray-1. Seymour Cray understood that a fast computer was more than a fast CPU; in fact, all data movement within the machine had to be fast. He did not believe in parallel architectures; his quote “If you were plowing a field, which would you rather use: Two strong oxen or 1024 chickens?” can serve as a testimony from his time. However, his pioneering vector architecture helped define an important concept, the single instruction, multiple data (SIMD) mode of computation. A vector was a possibly large set of data elements where each element would be subject to the same computer instruction. An elementary example could be the SAXPY, y.i / WD y.i / C a  x.i /; i D 1; : : : n.

Parallel Computing Architectures

In a vector machine, these operations became efficient by pipelining. The operation was broken into many elementary stages, and the elements were processed like an assembly line, i.e., after a start-up time, the computer was able to deliver a new result every cycle. Already in the mid-1980s, it was understood and projected that parallel computers would change computational science and provide a dramatic improvement in price/performance. The technology trend made the difference between an ox and a chicken get smaller and smaller. An early pioneer was the famous Intel hypercube machine. This architecture was fundamentally different, employing a more general MIMD (multiple instruction, multiple data) computing paradigm coupled with message passing between computing nodes interconnected in a hypercube network. The two paradigms, SIMD and MIMD, remain the two principle parallel architectures today, and state-ofthe-art computers do often appear as MIMD machines with SIMD features at a second layer in the architecture.

What Can a Parallel Machine Do for You? For science and engineering, a parallel computer can solve a larger problem in less time than the alternative. A fixed-size problem will be best solved on a fixedsize parallel computer. You will need to find the best trade-off between efficient use of the computer and the time it takes to solve your problem. Speedup, S.N /, on a parallel computer is defined as the time it would take a single processor to solve a problem divided by the time the same problem would require when using a parallel computer with N processors. If we have a computer program with single processor running time s C P where s is sequential time and P can be reduced by using N > 1 processors, then with ˛ D s=.s C P /, the speedup that one may achieve is limited by S.N / D .s C P /=.s C P =N / D

1 : ˛ C .1  ˛/=N

For a fixed-size problem, this shows that there is a fixed number of processors N beyond which the benefit of employing a greater number of processors will fall below what one would be willing to pay. (This relation is also known as Amdahl’s law.) However, there are many scientific and engineering problems where one is interested in increasing the size of the computational

Parallel Computing Architectures

problem if a larger computer becomes available. In this way, one may increase the accuracy, improve the resolution of the simulation, etc. A simple model for this case is letting P D Np and model the parallel execution time as s C p. Defining ˇ D s=.s C p/ to be the sequential fraction of the overall (parallel) execution time, then yields

1133

In this case, ˛ D s=.s CNp/ decreases with increasing N and acceptable speedup can be maintained. That this works in practice has been verified by actual computations with N in excess of 100,000. This is good news, since it tells us that computer efficiency can scale with N for large-scale problems in science and engineering. Thus, one must decide what kind of computation that needs to be done, then find a best possible computing environment for the task at hand.

the cluster as a parallel architecture for computational science and engineering depends largely on the quality of its interconnecting network. The technology trend is also here standardization. InfiniBand has emerged as a leading standard. Vendor-specific technologies are shrinking, and in 2012, Cray sold its interconnection technology to Intel. There are a few different topologies, hypercube, fat trees, 3-D torus, etc., but largely, these details are no longer of concern to the user of a parallel machine. The interconnect must scale well and have low latency coupled with a very large aggregate bandwidth for data exchange between the nodes. MPI (Message Passing Interface) is the dominating programming model. The development of this standard has been very important for the successful development of advanced software for a large range of parallel architecture machines. It has protected investment in software and made it possible to maintain this software across multiple generations of hardware.

Multicore Machines

GPUs and Accelerators

From about 2005 and onwards, “the chickens” have stopped getting stronger, the clock rate has stalled in the 2.0–2.5 GHz range. However, Moores Law has not quite ended, and the result is a new development in computer architecture, the multicore. This is many identical processing elements on a single chip. These units will share memory and cache and favor computations with limited data movement and very Lowlatency internal communication. In the near future, one can expect more than 100 cores on a single chip. This trend has important software and programming consequences. With a bit of simplification, many characteristic features of previous SMP machines (symmetric multiprocessing) can now be implemented at the single chip level. Many programming models are possible, but multithreading and the adoption of OpenMP were very natural first steps. Many new developments are on their way as this architecture will be the basic building block of parallel computer systems.

The temptation to replace a computational node in full or partially by custom-designed processing elements that have superior performance with respect to floating point operations has always been part of high performance, parallel architecture computers. We broadly may call these architectures for accelerators. Largely, this trend has met with limited success because of two factors: first, the steady improvement of standard processors and, second, the added complexity of a design that was not mainstream. Combined, these two effects have largely limited the lifetime and cost-effectiveness of special purpose hardware. Recently, two new trends may change this picture: first, the fact that our “chickens” have reached a mature size and, second, the commercial marked for computer games and high-performance graphics cards has driven the development of very special parallel “vector units” that now achieve superior performance with respect to cost and power consumption when compared with standard (general purpose) processors. Thus, the standard processor is not coming from behind and the custom hardware has a mass market. The use of this technology, called GPU (graphics processing unit), comes with the cost of increased complexity at the programming level. Much work is currently going on to address this issue. The longer time scale that now seems available to cover this

S.N / D .s C Np/=.s C p/ D N  ˇ.N  1/

Interconnecting Networks One or a low number of processor chips typically define a node having its own (local) memory. These units are interconnected in a network to form a larger parallel machine that we will call a cluster. The quality of

P

1134

investment may, for the first time, make accelerators a more permanent and stable component of future parallel computer architectures for computational science and engineering. SIMD programming is attractive at this level, since a large number of data elements that are subject to the same floating point operations can be treated with low overhead and a relatively simple programming model. This is also consistent with large-scale computational science, since these problems typically have data that must scale up in size and be subject to similar computational procedures, in order to allow for proper scaling of the overall computer work.

Exaflop Computing The largest computer systems are expected to achieve exaflop performance (i.e., 1018 floating point operations per second) around 2020. This is about 1,000 times the performance of the fastest systems available in 2010. Such systems can be expected to have (at least) two levels of parallel architectures as defined above. The second level will consist of multicore chips and/or GPU accelerators. Data locality, low overhead multithreading, and a large degree of SIMD like processing will be required in order to achieve good parallel scaling at this level. The first level will be a cluster where each node is a second level system. Smaller systems that will be more widespread can be expected to share the same basic two-level parallel architecture but with fewer nodes. The very largest systems are breaking trail for the mid-scale systems that will serve the majority of scientists and engineers for their computational needs. Two-level parallel architectures make the overall programming model significantly more complex, and major advances in programming languages as well as in compiler technology are needed in order to keep software applications portable but also efficient. Costeffective computing in science and engineering has two components: the system cost, including electricity, plus the application development cost, including the porting of such software to newer parallel architectures.

References In order to read and learn more in depth about this topic, there are many references that may be consulted. A good, general

Parameter Identification reference on computer architecture is “Computer Architecture: A Quantitative Approach,” by Hennessy and Patterson (5th Edition, Elsevier, 2011). A quick, but pretty nice overview of specific parallel computers by Dave Turner (Ames Laboratory) can be found at http://www.scl.ameslab.gov/Projects/ parallel computing/. The (online) book “Designing and Building Parallel Programs,” by Ian Foster, http://www.mcs.anl.gov/ itf/dbpp/, is a very comprehensive and well-organized source of information. As regards message passing, there are good resources available on the Web (e.g., http://www.mpitutorial. com/), a classic book on MPI is “Parallel Programming with MPI,” by Peter Pacheco (Morgan Kaufmann, 1997). Web-based resources with information about OpenMP and GPU computing are http://www.openmp.org and http://www.gpucomputing.net/, respectively. The early references to the two concepts of speedup that has been discussed are as follows: 1. Amdahl, G.: Validity of the single processor approach to achieving large-scale computing capabilities (PDF). In: AFIPS Conference Proceedings, Atlantic City, 1967, vol. 30, pp. 483–485 2. Gustafson, J.L.: Reevaluating Amdahl’s law. Commun. ACM 31(5), 532–533 (1988)

Parameter Identification Daniela Calvetti and Erkki Somersalo Department of Mathematics, Applied Mathematics and Statistics, Case Western Reserve University, Cleveland, OH, USA

Synonyms Data fitting; Inference; Inverse problems; Parameter fitting

Definition Parameter identification refers to numerous different methods of determining unknown parameter values in a mathematical model based on equations that relate the parameters to measured data or express consistency conditions that the parameters need to satisfy to make the model meaningful. The parameters are often subject to inequality constraints such as non-negativity. Parametric models may comprise algebraic relations between quantities; systems of linear equations are an example. Often, parametric models involve differential equations, and the unknown parameters appear as

Parameter Identification

1135

coefficients in the equation. Parameter identification methods can be roughly divided in two classes: (a) deterministic methods and (b) probabilistic methods. The former methods encompass different optimization algorithms, and the latter leads to statistical methods of inference.

where b 2 Rm is a vector containing measured data and e 2 Rm is an unknown noise vector. A common starting point for the estimation of x is to write a weighted output error functional, f .x/ D

m X

wj .bj  Fj .x//2 ;

j D1

Overview and Model Problems

where the weights wj define how much weight, or importance, we give to each component in the equation In this section, we denote by x 2 R a parameter and define the parameter identification problem as a vector with entries xj , 1  j  n. For consistency, weighted output least squares problem: Find x such we assume always that x is a column vector. that   x D argmin f .x/ : Deterministic Methods We identify in this section a number of model prob- A particularly important case is when F .x/ is a linear lems, indicating the methodology that is usually em- function of x, F .x/ D Ax. Assuming that the weights wj are all equal, the minimizer x is the solution of the ployed for solving it. Consider a mathematical model involving a param- linear least squares problem, eter vector x 2 Rn , written in the form x D argminkb  Axk2 : n f .x/ D 0; f W R ! R differentiable: If the matrix A has full rank, the solution is found by Assuming that a solution to this problem exists, the solving the normal equations, that is, standard approach is to use any of the numerous vari 1 ants of Newton method [2]: Iteratively, let xc denote xLS D AT A Ab: the current value and w 2 Rn a unit vector pointing to the direction of the gradient of f at xc , If A is not full rank, or numerically of ill-determined n

rf .xc / ; wD krf .xc /k

rf .xc / ¤ 0:

By using the Taylor approximation at t D 0, we write def

g.t/ D f .xc C tw/  f .xc / C wT rf .xc /t D f .xc / C krf .xc /kt; and the value of t that makes the right-hand side vanish gives the next approximate value of x, xC D xc C tw;

f .xc / : t D krf .xc /k

Another common problem in parameter identification involves noisy data. Consider the problem of finding a parameter vector x 2 Rn satisfying b D F .x/ C e;

rank, the problem requires regularization, a topic that is extensively studied in the context of inverse problems [1, 3]. The linear model gives an idea how to treat the nonlinear problem in an iterative manner. Let xc denote the current approximation of the solution, and write x D xc C z. By linearization, we may approximate f .x/ D f .xc C z/ D kb  F .xc C z/k2  kb F .xc /  DF .xc /zk2 ; and using the least squares solution for the linear problem as a model, we find an update for x as xC D xc C z;

 1 z D DF .xc /T DF .xc /

DF .xc /T .b  f .xc //:

This algorithm is the Gauss-Newton iteration, and it F W Rn ! Rm differentiable, (1) requires that the Jacobian of F is of full rank. If this

P

1136

Parameter Identification

is not a case, a regularization is needed. Above, 0 < The Bayesian equivalent for the maximum likelihood   1 is an adjustable relaxation parameter. Choosing estimator is the maximum a posteriori (MAP) a value of  that minimizes the residual norm along the estimator: z-direction is referred to as backtracking. If the parameters need to satisfy bound constraints, xMAP D argmin.P .x j b//;   e.g., of the form P .x j b/ D  log .b j x/prior .x/ : `j  xj  hj ;

1  j  n;

constrained optimization methods need to be used; see [5]. Probabilistic Methods Deterministic parameter identification has an interpretation in the statistical framework. This is best understood by looking at the model (1): The noise vector can be thought of as a random variable with probability density noise W Rm ! RC , and therefore, the data b, given the value of the parameter vector x, has the same probability density but shifted around f .x/, that is,

This estimator can be interpreted as the most probable value for the parameter in the light of data and a priori information. For a general reference about the connection of the deterministic and statistical methods, and for the differences in interpretation between frequentist and Bayesian statistics, we refer to [1]. The MAP estimate is only one possible estimator that can be extracted from the posterior density. Another popular estimate is the posterior mean or conditional mean (CM) estimate: Z xCM D

x.x j b/dx;

the integral being extended over the whole state space. This integration, e.g., by quadrature methods, is often unfeasible, and moreover, the posterior density is ofwhere .b j x/ is referred to as the likelihood density. tenknown only up to a multiplicative constant. Both of In the frequentist statistics, a commonly used estimate these problems can be overcome by using Monte Carlo for x is the maximum likelihood (ML) estimator, integration: By using, e.g., Markov Chain Monte Carlo (MCMC) methods, an ensemble fx 1 ; x 2 ; : : : ; x N g of xML D argmin.L.x j b//; parameter vectors is randomly drawn from the poste  rior density, and the CM estimator is approximated by L.x j b/ D  log .b j x/ ; .b j x/ / noise .b  f .x//;

assuming that a minimizer exists. One can interpret the ML estimator as the parameter value that makes the observation at hand most probable. Obviously, from this point on, the problem is reduced to an optimization problem. In contrast, in Bayesian statistics, all unknowns such as the parameter vector itself are modeled as random variables. All possible information concerning x that is independent of the measurement b, such as bound constraints, is encoded in the probability density prior .x/, called the prior probability density. Bayes’ formula states that the posterior probability density of x that integrates all the information about x coming either from the measurement or being known a priori is given, up to a scaling factor, as a product: .x j b/ / prior .x/.b j x/:

xCM 

N 1 X n x : N nD1

The connection between the statistical and deterministic approaches is discussed in [1, 3]. For a topical review of MCMC methods and relevant literature, see [4].

References 1. Calvetti, D., Somersalo, E.: Introduction to Bayesian Scientific Computing – Ten Lectures on Subjective Computing. Springer, New York (2007) 2. Dennis, J.E., Jr., Schnabel, R.B.: Numerical Methods for Unconstrained Optimization and Nonlinear Equations. SIAM, Philadelphia (1996) 3. Kaipio, J., Somersalo, E.: Statistical and Computational Inverse Problems. Springer, New York (2004)

Particulate Composite Media 4. Mira, A.: MCMC methods to estimate Bayesian parametric models. In: Dey, D.K., Rao, C.R. (eds.) Handbook of Statistics, vol. 25, pp. 415–436. Elsevier, Amsterdam (2005) 5. Nocedal, J., Wright, S.J.: Numerical Optimization. Springer, New York (1999)

1137 ENGINEERING DEVICE

Particulate Composite Media Tarek I. Zohdi Department of Mechanical Engineering, University of California, Berkeley, CA, USA

Introduction During the development of new particulate composite materials, experiments to determine the appropriate combinations of particulate and matrix phases are time-consuming and expensive. Therefore, elementary “microstructure-macroproperty” methods have been generated over the last century in order to analyze and guide new material development. The overall properties of such materials is the aggregate response of the collection of interacting components (Fig. 1). The macroscopic properties can be tailored to the specific application, for example, in structural engineering applications, by choosing a harder particulate phase that serves as a stiffening agent for a ductile, easy-to-form, base matrix material. “Microstructure-macroproperty” (micromacro) methods are referred to by many different terms, such as “homogenization,” “regularization,” “mean-field theory,” and “upscaling,” in various scientific communities to compute effective properties of heterogeneous materials. We will use these terms interchangeably in this chapter. The usual approach is to compute a constitutive “relation between averages,” relating volume-averaged field variables, resulting in effective properties. Thereafter, the effective properties can be used in a macroscopic analysis. The volume averaging takes place over a statistically representative sample of material, referred to in the literature as a representative volume element (RVE). The internal fields, which are to be volumetrically averaged, must be computed by solving a series of boundary value problems with test loadings. There is a vast literature on methods, dating back to Maxwell [14, 15]

MATERIAL SAMPLE

Particulate Composite Media, Fig. 1 An engineering structure comprised of a matrix binder and particulate additives

and Lord Rayleigh [18], for estimating the overall macroscopic properties of heterogeneous materials. For an authoritative review of the general theory of random heterogeneous media, see Torquato [23]; for more mathematical homogenization aspects, see Jikov et al. [12]; for solid-mechanics inclined accounts of the subject, see Hashin [5], Mura [16], Nemat-Nasser and Hori [17], and Huet [10, 11]; for analyses of cracked media, see Sevostianov et al. [21]; and for computational aspects, see Zohdi and Wriggers [26], Ghosh [3], and Ghosh and Dimiduk [4]. Our objective in this chapter is to provide some very basic concepts in this area, illustrated by a linear elasticity framework, where the mechanical properties of microheterogeneous materials are characterized by a spatially variable elasticity tensor IE. In order to characterize the effective (homogenized) macroscopic response of such materials, a relation between averages, h i˝ D IE W h"i˝ ;

(1)

is sought, where def

hi˝ D

1 j˝j

Z  d˝ ;

(2)

˝

and where  and " are the stress and strain tensor fields within a statistically representative volume element (RVE) of volume j˝j. The quantity IE is known as the effective property. It is the elasticity tensor used in usual structural analyses. Similarly, one can describe other effective quantities such as conductivity or diffusivity, in virtually the same manner, relating

P

1138

Particulate Composite Media

other volumetrically averaged field variables. However, Testing Procedures for the sake of brevity, we restrict ourselves to linear To determine IE , one specifies six linearly indepenelastostatics problems. dent loadings of the form, 1. uj@˝ D E .1!6/  x or 2. tj@˝ D L.1!6/  n; where E .1!6/ and L.1!6/ are symmetric secondBasic Micro-Macro Concepts order strain and stress tensors, with spatially constant (nonzero) components. This loading is applied to For a relation between averages to be useful, it a sample of microheterogeneous material. Each must be computed over a sample containing a independent loading yields six different averaged stress statistically representative amount of material. This is a components, and hence, provides six equations to requirement that can be formulated in a concise mathdetermine the constitutive constants in IE . In order for ematical form. A commonly accepted micro-macro such an analysis to be valid, i.e., to make the material criterion used in effective property calculations is the data reliable, the sample of material must be small so-called Hill’s condition, h W i˝ D h i˝ W hi˝ . enough that it can be considered as a material point Hill’s condition [9] dictates the size requirements on with respect to the size of the domain under analysis the RVE. The classical argument is as follows. For but large enough to be a statistically representative any perfectly bonded heterogeneous body, in the sample of the microstructure. absence of body forces, two physically important If the effective response is assumed to be isotropic, loading states satisfy Hill’s condition: (1) linear then only one test loading (instead of usually  six), con displacements of the form uj@˝ D E  x ) hi˝ D E taining nonzero dilatational t r3 and t r3 and devi  and (2) pure tractions in the form tj@˝ D L  n ) tr tr 0 def 0 def h i˝ D L; where E and L are constant strain atoric components  D  3 I and  D  3 I , and stress tensors, respectively. Applying (1)- or is necessary to determine the effective bulk and shear (2)-type boundary conditions to a large sample is moduli: a way of reproducing approximately what may be s tr h i occurring in a statistically representative microscopic h 0 i˝ W h 0 i˝ ˝ def def and 2 D : (3) 3  D t r3 sample of material in a macroscopic body. The h0 i˝ W h0 i˝ h 3 i˝ requirement is that the sample must be large enough to have relatively small boundary field fluctuations In general, in order to determine the material properties relative to its size and small enough relative to the of a microheterogeneous material, one computes 36 macroscopic engineering structure. These restrictions constitutive constants (There are, of course, only 21 force us to choose boundary conditions that are constants, since IE is symmetric.) Eij kl in the followuniform. ing relation between averages,

8 9 2  h11 i˝ > E1111 ˆ ˆ > ˆ >  6 ˆ > h i E ˆ 22 ˝ > 2211 ˆ > < = 6 6 E h33 i˝ 3311 D6 6 E h12 i˝ > ˆ ˆ > 6 ˆ > 4 1211  ˆ ˆ > h23 i˝ > E2311 ˆ > : ;  h13 i˝ E1311

 E1122  E2222  E3322  E1222  E2322  E1322

 E1133  E2233  E3333  E1233  E2333  E1333

As mentioned before, each independent loading leads to six equations, and hence, in total 36 equations are generated by the independent loadings, which are used to determine the tensor relation between average

 E1112  E2212  E3312  E1212  E2312  E1312

 E1123  E2223  E3323  E1223  E2323  E1323

9 38  h11 i˝ > E1113 ˆ > ˆ  > 7ˆ ˆ E2213 h22 i˝ > > ˆ > 7 ˆ  7 < h33 i˝ = E3313 7  7 ˆ 2h12 i˝ > : E1213 > 7ˆ >  ˆ 5ˆ > ˆ E2313 2h23 i˝ > > ˆ ; :  E1313 2h13 i˝

(4)

stress and strain; IE . IE is exactly what appears in engineering literature as the “property” of a material. The usual choices for the six independent load cases are

Particulate Composite Media

1139

2

3 2 3 2 3 2 3 2 3 2 3 ˇ00 0 00 000 0 ˇ0 00 0 0 0ˇ 6 7 6 7 6 7 6 7 6 7 6 7 E or L D 4 0 0 0 5 ; 4 0 ˇ 0 5 ; 4 0 0 0 5 ; 4 ˇ 0 0 5 ; 4 0 0 ˇ 5 ; 4 0 0 0 5 ; 0 00 0 00 00ˇ 0 0 0 0ˇ 0 ˇ00 (5)

where ˇ is a load parameter. For completeness, we The Average Strain Theorem record a few related fundamental results, which are If a heterogeneous body (see Fig. 2) has the following useful in micro-macro mechanical analysis. uniform loading on its surface: uj@˝ = E  x, then

hi˝ D

1 2j˝j

1 D 2j˝j 1 D 2j˝j 1 D 2j˝j 1 D 2j˝j

Z .ru C .ru/T / d˝ ˝

Z



Z .ru C .ru/T / d˝ C

Z

˝1

.ru C .ru/T / d˝ ˝2

 .u ˝ n C n ˝ u/ dA

Z

.u ˝ n C n ˝ u/ dA C Z

@˝1

@˝2

 .juŒj ˝ n C n ˝ juŒj/ dA

Z

..E  x/ ˝ n C n ˝ .E  x// dA C Z

@˝

@˝1 \@˝2

 .juŒj ˝ n C n ˝ juŒj/ dA

Z T

.r.E  x/ C r.E  x/ / d˝ C ˝

1 DEC 2j˝j

@˝1 \@˝2

Z

.juŒj ˝ n C n ˝ juŒj/ dA;

(6)

@˝1 \@˝2

P def

Ω1

Ω2

where .u ˝ nDui nj / is a tensor product of the vector u and vector n. juŒj describes the displacement jumps at the interfaces between ˝1 and ˝2 . Therefore, only if the material is perfectly bonded, then hi˝ D E. Note that the presence of finite body forces does not affect this result. Also note that the third line in (6) is not an outcome of the divergence theorem, but of a generalization that can be found in a variety of books, for example, Chandrasekharaiah and Debnath [2].

The Average Stress Theorem Again we consider a body (in static equilibrium) with tj@˝ D L  n, where L is a constant tensor. We make Ω use of the identity r  . ˝ x/ D .r   / ˝ x C   rx D f ˝ x C  , where f represents the body forces. Particulate Composite Media, Fig. 2 Nomenclature for the Substituting this into the definition of the average stress averaging theorems yields

1140

Particulate Composite Media

Z 1 .f ˝ x/ d˝ j˝j ˝ ˝ Z Z 1 1 D . ˝ x/  n dA C .f ˝ x/ d˝ j˝j @˝ j˝j ˝

h i˝ D

D

1 j˝j

1 j˝j

Z

r  . ˝ x/ d˝ C

Z .L ˝ x/  n dA C @˝

1 D LC j˝j

Z

1 j˝j

Z

Z

Z

.f ˝ x/ d˝:

sought to approximate or bound the effective material responses. Many classical approaches start by splitting the stress field within a sample into a volume average and a purely fluctuating part,  D hi˝ C Q , and we directly obtain Q W IE W Q d˝ D

0 .f ˝ x/ d˝

˝

. W IE W   2hi˝ ˝

W IE W  C hi˝ W IE W hi˝ / d˝

˝

(7)

˝

If there are no body forces, f D 0, then h i˝ D L. Note that debonding (interface separation) does not change this result.

D .hi˝ W IE W hi˝  2hi˝ W h i˝ C hi˝ W hIEi˝ W hi˝ /j˝j D hi˝ W .hIEi˝  IE / W hi˝ j˝j:

(10)

Similarly, for the complementary case, with  D h i˝ C Q , and the following assumption (microscopic Satisfaction of Hill’s Energy Condition energy equals the macroscopic energy) Consider a body (in static equilibrium) with a perfectly bonded microstructure and f D 0. This condition yields h W IE1 W  i˝ D h i˝ W IE1 W h i˝ ; „ ƒ‚ … „ ƒ‚ … Z Z Z macro energy micro energy u  t dA D u    n dA D r  .u   / d˝: @˝ @˝ ˝ where hi˝ D IE1 W h i˝ ; (11) (8) Z r  .u   / d˝ D With r   D 0, it follows that we have ˝ Z Z Z ru W  d˝ D  W  d˝. If uj@˝ D E  x and f D ˝ ˝ Q W IE1 W Q d˝ 0 0, then ˝ Z Z Z D . W IE1 W   2h i˝ W IE1 W  C h i˝ u  t dA D E  x    n dA ˝ @˝

Z

@˝

r  .E  x   / d˝

D Z

W IE1 W h i˝ / d˝ (9)

˝

r.E  x/ W  d˝ D E W h i˝ j˝j:

D ˝

Noting that hi˝ D E, we have hi˝ WR h i˝ D h W  R i˝ . If tj@˝ D L  n Rand f D 0, then @˝ u R t dA D dA D ˝ r  .u  L/ d˝ D ˝ ru W @˝ u  L  n R L d˝ D L W ˝  d˝. Therefore, since h i˝ D L, as before we have hi˝ W h i˝ D h W  i˝ . Satisfaction of Hill’s condition guarantees that the microscopic and macroscopic energies will be the same, and it implies the use of the two mentioned test boundary conditions on sufficiently large samples of material.

D .h i˝ W IE1 W h i˝  2hi˝ W h i˝ C h i˝ W hIE1 i˝ W h i˝ /j˝j D h i˝ W .hIE1 i˝  IE1 / W h i˝ j˝j:

(12)

Invoking Hill’s condition, which is loading-independent in this form, we have  IE  hIEi˝ : hIE1 i1 „ƒ‚… „ ƒ‚ ˝… Reuss

(13)

Voigt

This inequality means that the eigenvalues of the ten sors IE  hIE1 i1 ˝ and hIEi˝  IE are nonnegative. The practical outcome of the analysis is that bounds The Hill-Reuss-Voigt Bounds on effective properties are obtained. These bounds Until recently, the direct computation of micromaterial are commonly known as the Hill-Reuss-Voigt bounds, responses was very difficult. Classical approaches have for historical reasons. Voigt [24], in 1889, assumed

Particulate Composite Media

that the strain field within a sample of aggregate of polycrystalline material was uniform (constant), under uniform strain exterior loading. If the constant strain Voigt field is assumed within the RVE,  D 0 , then h i˝ D hIE W i˝ D hIEi˝ W 0 , which implies IE D hIEi˝ . The dual assumption was made by Reuss [19], in 1929, who approximated the stress fields within the aggregate of polycrystalline material as uniform (constant),  D  0 , leading to hi˝ D hIE1 W  i˝ D hIE1 i˝ W  0 and thus to IE D hIE1 i1 ˝ . Remark 1 Different boundary conditions (compared to the standard ones specified earlier) are often used in computational homogenization analysis. For example, periodic boundary conditions are sometimes employed. Although periodicity conditions are really only appropriate for perfectly periodic media for many cases, it has been shown that, in some cases, their use can provide better effective responses than either linear displacement or uniform traction boundary conditions (e.g., see Terada et al. [22] or Segurado and Llorca [20]). Periodic boundary conditions also satisfy Hill’s condition a priori. Another related type of boundary conditions are so-called “uniform-mixed” types, whereby tractions are applied on some parts of the boundary and displacements on other parts, generating, in some cases, effective properties that match those produced with uniform boundary conditions, but with smaller sample sizes (e.g., see Hazanov and Huet [8]). Another approach is “framing,” whereby the traction or displacement boundary conditions are applied to a large sample of material, with the averaging being computed on an interior subsample to avoid possible boundary-layer effects. This method is similar to exploiting a St. Venant-type of effect, commonly used in solid mechanics, to avoid boundary layers. The approach provides a way of determining what the microstructure really experiences, without “bias” from the boundary loading. However, generally, the advantages of one boundary condition over another diminish as the sample increases in size. Improved Estimates Over the last half-century, improved estimates have been pursued, with a notable contribution being the Hashin-Shtrikman bounds [5–7]. The HashinShtrikman bounds are the tightest possible bounds on isotropic effective responses, with isotropic microstructures, where the volumetric data and

1141

phase contrasts of the constituents are the only data known. For isotropic materials with isotropic effective (mechanical) responses, the Hashin-Shtrikman bounds (for a two-phase material) are as follows for the bulk modulus def

 ; D 1 C C

v2 3.1v2 / C 3 C4 1 1 def 1v2

1 2 1

1 1 2

3v2 2 C42

C 3

    2

D  ;C ;

and for the shear modulus def

; D 1 C C

1 2 1

C

v2 6.1v2 /.1 C21 / 51 .31 C41 /

.1v2 / 6v2 .2 C22 / 1 C 1 2 52 .32 C42 /

   2

def

D ;C ;

where 2 and 1 are the bulk moduli and 2 and 1 are the shear moduli of the respective phases (2  1 and 2  1 ), and where v2 is the second-phase volume fraction. Note that no geometric or other microstructural information is required for the bounds. Remark 2 There exist a multitude of other approaches which seek to estimate or bound the aggregate responses of microheterogeneous materials. A complete survey is outside the scope of the present work. We refer the reader to the works of Hashin [5], Mura [16], Aboudi [1], Nemat-Nasser and Hori [17], and recently Torquato [23] for such reviews. Remark 3 Numerical methods have become a valuable tool in determining micro-macro relations, with the caveat being that local fields in the microstructure are resolved, which is important in being able to quantify the intensity of the loads experienced by the microstructure. This is important for ascertaining failure of the material. In particular, finite elementbased methods are extremely popular for micro-macro calculations. Applying such methods entails generating a sample of material microstructure, meshing it to sufficient resolution for tolerable numerical accuracy and solving a series of boundary value problems with different test loadings. The effective properties can be determined by post processing (averaging over the RVE). For an extensive review of this topic, see Zohdi and Wriggers [26]. We also refer the reader to that work for more extensive mathematical details and background information.

P

1142

References 1. Aboudi, J.: Mechanics of Composite Materials – A Unified Micromechanical Approach, vol. 29. Elsevier, Amsterdam (1992) 2. Chandrasekharaiah, D.S., Debnath, L.: Continuum Mechanics. Academic, Boston (1994) 3. Ghosh, S.: Micromechanical Analysis and Multi-Scale Modeling Using the Voronoi Cell Finite Element Method. CRC Press/Taylor & Francis (2011) 4. Ghosh, S., Dimiduk, D.: Computational Methods for Microstructure-Property Relations. Springer, New York (2011) 5. Hashin, Z.: Analysis of composite materials: a survey. ASME J. Appl. Mech. 50, 481–505 (1983) 6. Hashin, Z., Shtrikman, S.: On some variational principles in anisotropic and nonhomogeneous elasticity. J. Mech. Phys. Solids 10, 335–342 (1962) 7. Hashin, Z., Shtrikman, S.: A variational approach to the theory of the elastic behaviour of multiphase materials. J. Mech. Phys. Solids 11, 127–140 (1963) 8. Hazanov, S., Huet, C.: Order relationships for boundary conditions effect in heterogeneous bodies smaller than the representative volume. J. Mech. Phys. Solids 42, 1995–2011 (1994) 9. Hill, R.: The elastic behaviour of a crystalline aggregate. Proc. Phys. Soc. Lond. A65, 349–354 (1952) 10. Huet, C.: Universal conditions for assimilation of a heterogeneous material to an effective medium. Mech. Res. Commun. 9(3), 165–170 (1982) 11. Huet, C.: On the definition and experimental determination of effective constitutive equations for heterogeneous materials. Mech. Res. Commun. 11(3), 195–200 (1984) 12. Jikov, V.V., Kozlov, S.M., Olenik, O.A.: Homogenization of Differential Operators and Integral Functionals. Springer, Berlin/New York (1994) 13. Kr¨oner, E.: Statistical Continuum Mechanics. CISM Lecture Notes, vol. 92. Springer, Wien (1972) 14. Maxwell, J.C.: On the dynamical theory of gases. Philos. Trans. Soc. Lond. 157, 49 (1867) 15. Maxwell, J.C.: A Treatise on Electricity and Magnetism, 3rd edn. Clarendon, Oxford (1873) 16. Mura, T.: Micromechanics of Defects in Solids, 2nd edn. Kluwer Academic, Dordrecht (1993) 17. Nemat-Nasser, S., Hori, M.: Micromechanics: Overall Properties of Heterogeneous Solids, 2nd edn. Elsevier, Amsterdam (1999) 18. Rayleigh, J.W.: On the influence of obstacles arranged in rectangular order upon properties of a medium. Philos. Mag. 32, 481–491 (1892) 19. Reuss, A.: Berechnung der Fliessgrenze von Mischkristallen auf Grund der Plastizit¨atsbedingung f¨ur Einkristalle. Z. angew. Math. Mech. 9, 49–58 (1929) 20. Segurado, J., Llorca, J.: A numerical approximation to the elastic properties of sphere-reinforced composites. J. Mech. Phys. Solids 50(10), 2107–2121 (2002) 21. Sevostianov, I., Gorbatikh, L., Kachanov, M.: Recovery of information of porous/microcracked materials from the effective elastic/conductive properties. Mater. Sci. Eng. A 318, 1–14 (2001)

Particulate Flows (Fluid Mechanics) 22. Terada, K., Hori, M., Kyoya, T., Kikuchi, N.: Simulation of the multi-scale convergence in computational homogenization approaches. Int. J. Solids Struct. 37, 2229–2361 (2000) 23. Torquato, S.: Random Heterogeneous Materials: Microstructure and Macroscopic Properties. Springer, New York (2002) ¨ 24. Voigt, W.: Uber die Beziehung zwischen den beiden Elastizit¨atskonstanten isotroper K¨orper. Wied. Ann. 38, 573–587 (1889) 25. Zohdi, T.I.: Electromagnetic Properties of Multiphase Dielectrics. A Primer on Modeling, Theory and Computation. Springer, Berlin/New York (2012) 26. Zohdi, T.I., Wriggers, P.: Introduction to Computational Micromechanics. Springer, Berlin (2008)

Particulate Flows (Fluid Mechanics) Venkat Raman Aerospace Engineering, University of Michigan, Ann Arbor, MI, USA

Synonyms Dispersed-phase flows; Particle laden flows

Introduction The transport of liquid or solid particles in a background fluid is of importance in a number of practical applications including aircraft and automobile engines, fluidized beds, pneumatic converting systems, and material synthesis in flames. Particle-laden flows are defined as a subclass of two-phase flows in which one of the phases does not exhibit a connected continuum [1]. Here, the term particles refers broadly to solid particles, liquid droplets, or bubbles. The particles form the dispersed phase while the background flow is referred to as the continuous phase. The mathematical modeling of particle-laden flows is classified based on the nature of interaction between the particles and the interaction between the particles and the fluid phase [3]. A characteristic length scale ratio based on mean particle separation (S ) and particle diameter (d ) is used to denote the flow regime. If S=d > 100, the particles are sufficiently separated that interaction among the dispersed phase is not important.

Particulate Flows (Fluid Mechanics)

Further, the impact of the particles on the continuous phase is also minimal and could be disregarded. In this one-way coupling, only the fluid flow affects particle motion. For 10 < S=D < 100, the volume occupied by the particle phase is sufficiently large to alter the continuous phase flow dynamics. This is especially important in turbulent flows, where small-scale dissipation is affected by the presence of particles. Here, two-way coupling between the dispersed and continuous phases exists, where each phase is affected by the other. Flows in these two regimes are collectively called dilute suspensions. If S=d < 10, the particles are close enough to undergo collisions. Here, in addition to the particle-continuous phase interactions, particle-particle interactions have to be accounted for and are termed as four-way coupling. The mathematical description of these flows requires transport equations for the evolution of the continuous phase as well as the dispersed phase. From a microscopic point of view, it is possible to develop transport equations for the mass, momentum, and energy in the two phases separately. However, a direct solution of this system will be computationally intractable for any practical flow configuration due to the large number of particles in the dispersed phase as well as the multiscale nature of the problem. From an engineering standpoint, we only seek the evolution of phase-averaged properties, which will be referred to as macroscopic properties of the system. In general, there are two approaches available for developing macroscopic evolution equations [8]. In the two-fluid approach [2, 9], ensemble averaging is used to arrive at transport equations for the phase-averaged properties. In the kinetic-theory-based technique [4, 5], an intermediate mesoscopic description of the dispersed phase is used to derive the final macroscopic equations.

Ensemble-Averaging Approach

1143

Z f D

f .x; tI /dm./;

(1)

"

where dm is the density for the measure on the set " that contains all realizations. Further, an indicator function Xk that picks out a phase k is defined such that ( Xk .x; tI / D

1 x 2 k in realization ; 0 otherwise:

(2)

Based on these definitions, volume fraction of phase k, the ensemble-averaged density k , and velocity uk are obtained as ˛k D Xk k D

Xk

Xk u uk D ; ˛k ˛k k

(3)

where and u are the microscopic density and velocity, respectively, at a given point in the domain. The governing equations for the ensemble-averaged variables are obtained by applying the indicator-function weighted ensemble-averaging procedure to the transport equation for the microscopic variables. The mass balance equation for phase k is @˛k k C r  ˛k k uk D k ; @t

(4)

where k denotes mass transfer due to phase change. The momentum balance equation is written as   @˛k k uk Re C r  ˛k k uk uk Dr  ˛k k Tk C Tk @t C Mk C uki k ;

(5)

where the first term on the right-hand side is related to the stress tensor, the second term describes the interfacial momentum exchange, and the last term arises from the momentum imparted in the mass transfer process. The effect of molecular fluxes, leading to pressure and viscous stresses, is described by Tk . The additional Re stress term, Tk , arises from the ensemble averaging of the nonlinear convection term in the microscopic transport equation:

Consider a flow domain, , with a spatial distribution of the particles evolving in time. Following [2], a single realization is defined as all the values of flowrelated quantities such as velocity and interface locations within the domain over the duration of the flow. Denoting this realization as , and any function that Re depends on the realization-specific values as f .x; tI /, Xk uu D ˛k k uk uk CXk u0 u0 D ˛k k uk uk ˛k T k ; (6) the ensemble average is defined as

P

1144

Particulate Flows (Fluid Mechanics)

where u0 D u  u. This term is often compared to the Reynolds stress that arises in Reynolds averaging of Navier-Stokes equations [6], which represents the correlation of the velocity fluctuations. However, it should be noted that in dispersed-phase flows, this term is nonzero even in the laminar flow regime and represents the variations in the local velocity field due to the differences in the spatial distribution of the dispersed phase between different realizations. Hence, models for this stress term based on a turbulence analogy are not strictly valid [4]. The interfacial momentum exchange Mk D T  rXk

point for developing the governing equations [4, 8]. Here, the distributed dispersed-phase objects is treated statistically using a point-process description [7]. The mesoscopic treatment is based on the number density function (NDF), f .x; v; ; t/, where x; v, and denote the location, velocity, and particle volume, respectively. Note that the choice of phase-space variables is not unique, and other dispersed-phase attributes could be added. The NDF has been shown to be linked to the Liouville description of the dispersed phase [7]. By integrating the moments of the NDF over phase space, key characteristics of the dispersed phase could be obtained. For instance, the number density of particles (7) is given by

Z Z 1 denotes the effect of the stress tensor at the interfaces f .x; v; ; t/d d v (9) N.x; t/ D and provides the coupling between the phases. R3 0 Developing closures for this term requires extensive modeling and requires a characterization of the and the volume fraction of the dispersed phase is microstructure of the dispersed phase. For instance, obtained as one approach is based on models for the ensembleZ Z 1 averaged interfacial stress leading to f .x; v; ; t/d d v: (10) ˛d .x; t/ D Mk D Tki  r˛k C M0k

R3

0

(8)

where Tki is the interfacial stress model and M0k is a residual force term. The momentum imparted by phase transfer also requires modeling of the interface velocity (uki ) and appears in the last term in Eq. 5. These transport equations could be solved using any grid-based discretization schemes. Although widely used, the ensemble-averaging approach has two limitations. First, the modeling of the unclosed terms is very challenging due to the lack of a natural physics-based hierarchy. Consequently, closure models often invoke restrictive assumptions regarding particle behavior. Second, the use of the averaging process couples turbulence-related phase-specific property fluctuations with the fluctuations caused due to the variations introduced by the dispersed-phase microstructure between different realizations. As noted by [4], it is difficult to decouple the two sources of fluctuations which leads to additional modeling challenges.

The evolution equation for the NDF is given by

@f C rx  vf C rv  Af C r f D fPcoll C fPcoal C fPbu ; @t (11) where A is the particle acceleration and is the rate of change of particle volume. The right-hand side includes contributions due to particle collisions (fPcoll ), coalescence (fPcoal ), and breakup (fPbu ). In this NDF description, the particle acceleration and mass transfer rate terms have to be modeled, apart from the rate terms on the right-hand side. However, it has been noted [4] that this formulation provides a better starting point for incorporating particle phase microstructure information. The NDF transport equation could be solved in a number of ways. The Lagrangian method, commonly used in droplet-laden flows, employs a particle-based numerical method [8]. It is also possible to directly compute transport equations for the moments of the NDF and solve the resulting Eulerian transport equations. A third approach relies on quadrature-based Mesoscopic Equation-Based Approach approximation, where the NDF is reconstructed by solving a set of lower-order moment equations [4]. An alternative approach is based on using a meso- The governing equations for the continuous phase scopic description of the dispersed phase as a starting are similar in structure to those derived using the

Pattern Formation and Development

1145

ensemble-averaging approach [5, 8], except that the Morphogenesis The generation of structure and momentum exchange and mass transfer source terms form in an embryo. are directly obtained from the NDF description of the Morphogen A chemical that influences the differenparticle phase. tiation of cells during embryogenesis.

References 1. Crowe, C.T., Troutt, T.R., Chung, J.N.: Numerical models for two-phase turbulent flows. Annu. Rev. Fluid Mech. 28, 11–43 (1996) 2. Drew, D.A., Passman, S.L.: Theory of Multicomponent Fluids. Springer, New York (1999) 3. Elghobashi, S.: Particle-laden turbulent flows: diret simulation and closure models. Appl. Sci. Res. 48, 301–314 (1991) 4. Fox, R.O.: Large-eddy-simulation tools for multiphase flows. Annu. Rev. Fluid Mech. 44, 47–76 (2012) 5. Garzo, V., Tenneti, S., Subramaniam, S., Hrenya, C.M.: Enskog kinetic theory for monodisperse gas-solid flows. J. Fluid Mech. 712, 129–168 (2012) 6. Pope, S.B.: Turbulent Flows. Cambridge University Press, Cambridge/New York (2000) 7. Subramaniam, S.: Statistical representation of a spray as a point process. Phys. Fluids 12(10), 2413–2431 (2000) 8. Subramaniam, S.: Lagrangian-Eulerian methods for multiphase flows. Prog. Energy Combust. Sci. 39(2–3), 215–245 (2013) 9. Zhang, D.Z., Prosperetti, A.: Averaged equations for inviscid dispersed two-phase flow. J. Fluid Mech. 267, 185–219 (1994)

Short Definition The emergence of global spatiotemporal order from local interactions during embryonic development.

Description Development is overflowing with examples of selforganization, where local rules give rise to complex structures and patterns which, ultimately, bring about the final body structure in multicellular organisms. Understanding the mechanisms governing and regulating the emergence of structure and heterogeneity within cellular systems, such as the developing embryo, represents a multiscale challenge typifying current mathematical biology research.

Classical Models Classical models in the field consist mainly of systems of partial differential equations (PDEs) describing concentrations of signaling molecules and densities of Pattern Formation and Development various cell species [8]. Spatial variation, arising from diffusion/random motion and a variety of different Ruth E. Baker and Philip K. Maini types of directed motion (for example, due to chemical Centre for Mathematical Biology, Mathematical or adhesion gradients), is represented through the use Institute, University of Oxford, Oxford, UK of different types of flux terms, and chemical reactions, cell proliferation and cell death through source terms which are polynomial and/or rational functions. The Mathematics Subject Classification major advantages of using such types of models lie in the wealth of analytical and numerical tools available 35K57; 92-00; 92BXX for the analysis of PDEs. For simple systems, exact analytical solutions may be possible and, where they Synonyms are not, separation of space and time scales or the exploitation of some other small parameter enables the Morphogenesis; Self-organization use of multiscale asymptotic approaches which give excellent insight into system behavior under different Glossary parameter regimes [3]. As the number of model components becomes too unwieldy or the interactions too Diffusion-driven instability The mechanism via complex for such approaches, increasingly sophistiwhich chemical patterns are created from an cated computational methods allow accurate numerical initially uniform field due to the destabilizing action approximations to be calculated over a wide range of of diffusion. parameter space.

P

concentration

1146

Pattern Formation and Development, Fig. 1 An illustration of Wolpert’s “French Flag” model [15]. A concentration gradient of a morphogen induces subsequent cell differentiation according to thresholds in concentration with cells experiencing concentrations above the highest threshold becoming red, cells between thresholds becoming white, and cells below the lower threshold becoming blue

Morphogen Gradient Models Wolpert [15] proposed one of the first mechanisms for providing positional information by a morphogen gradient with his “French Flag” model. In the model, each cell in a field has potential to be either blue, white, or red. When exposed to a concentration gradient of morphogen, arising from the combination of production at a localized source, diffusion, and decay, each cell interprets the information from the concentration profile by varying its response to different concentration thresholds of morphogen: Cells become blue, white, or red according to their interpretation of the information – see Fig. 1 for an illustration. Applications of Wolpert’s model are still used in a number of fields, including whole organism scale modeling of Drosophila patterning [13]. Turing Reaction-Diffusion Models Turing’s seminal work [12] proposed a mechanism via which a field could organize without any external cue from the environment. Given a system consisting of two or more chemicals (morphogens), which react according to certain rules, and diffuse at different rates throughout a field, spontaneous patterns in chemical concentration may arise as diffusion destabilizes the

Pattern Formation and Development

spatially uniform steady state of the system. This is known as a diffusion-driven instability and subsequent cell differentiation is then assumed to arise much in the same way postulated by Wolpert except that here, typically, cells respond to one threshold instead of multiple thresholds. Applications of the Turing model to patterning during development abound, and potential candidates for Turing morphogens include: (1) Nodal and Lefty in the amplification of an initial signal of left–right axis formation and zebrafish mesoderm cell fates; (2) Wnt and Dkk in hair follicle formation; (3) TGF-ˇ as the activator, plus an unknown inhibitor, in limb bud morphogenesis [1]. General, necessary and sufficient, conditions for a Turing instability on an n-dimensional spatial domain are presented in the literature for the two-component system and can be found in most textbooks, see for example [9], but the analysis for more than two chemicals is still an open question. Methods for the analysis of Turing systems on finite domains start by linearizing around a spatially homogeneous steady state and examining the behavior of the discrete spatial Fourier modes as one of the model parameters is varied. Asymptotic techniques, such as the method of multiple scales, and the Fredholm alternative are used to examine the exchange of stability of bifurcating solution branches in a small neighborhood of the bifurcation point, and may be used to distinguish the types of patterns that arise. Figure 2 illustrates the patterns that may arise in such a model in two spatial dimensions. Turing’s postulation has stimulated vast amounts of theoretical research into examining the finer detail of the Turing model for patterning, for example: (1) characterization of the amplitude equation and possible bifurcations in terms of group symmetries of the underlying problem being offered as an alternative approach to the weakly nonlinear analysis; (2) many results have been derived on the existence and uniqueness of localized patterns, such as spikes, that arise in certain Turing models; (3) the development of sophisticated numerical methods for solving Turing models on a variety of surfaces and investigating bifurcation behavior. For a comprehensive guide to the analytical and numerical methods used to investigate Turing models, see [14] and the references therein. The model has been shown to be consistent with many observed pattern formation processes and to also yield predictions that agree with experimental manipulations of the system.

Pattern Formation and Development

80

1147 inhibitor concentration

activator concentration

80 60 y position

y position

60 40 20 0 0

40 20

20

40 x position

60

80

0 0

20

40 x position

60

80

Pattern Formation and Development, Fig. 2 Results from numerical simulation of a Turing reaction-diffusion model in two dimensions. Gierer–Meinhardt kinetics [9, page 77] were used

with parameters b D 0:35 and D D 30 (see [2] for more details) and red (blue) indicates high (low) chemical concentration

However, the model can produce many more patterns than those observed in nature and this leads to the intriguing question of why patterning in biology is rather restricted. It is observed that in many cases in biology, patterning occurs behind an advancing front, either of a permissive signaling cue, or of domain growth. Analysis of the model shows that precisely these constraints are sufficient to select in a robust manner certain (observed) patterns at the expense of other (unobserved) patterns. It is important to note that such a propagating front can also serve to move a bistable system from one state to another, and this is an alternative mechanism to the Turing model for pattern formation.

stiffness (mechanotaxis), light (phototaxis), to name but a few [1, 11]. Whereas the Turing model gives rise to a parabolic system, taxis models can be of mixed parabolic/hyperbolic type, although the parabolic part is usually taken to dominate. Mathematically, therefore, taxis models are similar to the Turing model, with linear and nonlinear analyses demonstrating the existence of bifurcations and predicting the emergence of steady state patterns. In addition, a large body of work has been devoted to considering the potential for certain formulations of the chemotaxis model to exhibit “blow up,” where solutions become infinite in finite time, and showing existence and uniqueness of solutions [5]. The unifying mechanistic theme behind many of these models – Turing, chemotaxis, and mechanochemical – is that of short-range activation and long-range inhibition [9]. From a mathematical viewpoint, the patterns exhibited by all these models at bifurcation are eigenfunctions of the Laplacian.

Cell Chemotaxis Models Whereas Turing’s model assumes no explicit interaction between cells underlying the field and evolution of the chemical pattern, cell chemotaxis models assume that cells move preferentially up chemical gradients, and at the same time amplify the gradient by producing the chemical themselves. Examples of chemotaxis during development include the formation of the gut (gastrulation), lung morphogenesis, and feather bud formation [1]. In addition to modeling chemotaxis in development, such models are also commonly considered for coat marking patterns and swarming microbe motility. We note that there are numerous types of “taxis” that can be observed during development, including those up/down gradients in cellular adhesion sites (haptotaxis), substrate

Growing Domains Throughout development the embryo undergoes enormous changes in size and shape, and as a result biologically accurate patterning models must take these variations into account if they are to be capable of validating hypotheses and making predictions. The inclusion of growth in reaction-diffusion models was first considered sytematically by Crampin and coworkers [4] who derived a general formulation by considering conservation of mass and the application

P

1148

Pattern Formation and Development

8

deterministic activator concentration

8

6 position

position

6

4

2

0 0.0

8

0.5

1.0 1.5 rescaled time

0 0.0

2.0

stochastic activator concentration

8

0.5

1.0 1.5 rescaled time

2.0

stochastic inhibitor concentration

6 position

position

4

2

6

4

4

2

2

0 0.0

deterministic inhibitor concentration

0.5

1.0 1.5 rescaled time

2.0

Pattern Formation and Development, Fig. 3 The Turing model on a growing domain using both deterministic (PDE) and stochastic (Monte Carlo) formulations. Schnakenberg kinetics [9, page 76] were used with parameters k1 D 1:0, k2 D 0:02, k3 D 106 , k4 D 3:0, DA D 105 , DB D 103 , and uniform

of Reynold’s transport theorem. The extra terms arising in the reaction-diffusion system as a result of growth occur as material is both transported around the domain and diluted during growth. Key to applications of Turing’s model to development was the discovery that domain growth increases the reliability of pattern selection, giving rise to consistent patterns without such tight control of the reaction parameters and, more recently, to the discovery that patterns may form in systems that do not satisfy Turing conditions under certain types of domain growth [7]. Figure 3 shows the results of numerical simulation of the Turing model on a growing domain, and illustrates the changing patterns that arise as the domain grows.

0 0.0

0.5

1.0 1.5 rescaled time

2.0

growth rate r D 104 . (See [4] for more details of the growing domain formulation.) Black shading indicates where the system is above the spatially uniform steady state, and the red line the edge of the domain

terms in the conservation formulation generally employed are often phenomenological, without derivation from universal or fundamental principles. In addition, as the material density becomes low, stochastic effects can become significant. (Compare, for example, the results of stochastic and deterministic simulations of patterning on a growing domain in Fig. 3.) Finally, tortuous cellular level geometry complicates the investigation of spatial fluctuations at the cellular scale and the possibility of large variations among neighboring cells prevents straightforward use of a continuum limit. Moreover, the parameters within the kinetic terms themselves arise due to dynamics at a lower scale level. As such, many of the recent developments in modeling pattern formation have explored the derivation of these classical models More Recent Developments However, one should be aware of the limitations of from individual considerations where cell-level these classical models. The flux and/or production behavior may be taken into account [10] and the

Petrov-Galerkin Methods

1149

role of noise explicitly studied. However, with careful Petrov-Galerkin Methods consideration of these pitfalls and awareness of when and where techniques can successfully be applied, Christian Wieners PDEs remain one of the most useful and insightful Karlsruhe Institute of Technology, Institute for tools for modeling self-organization in developmental Applied and Numerical Mathematics, Karlsruhe, biology [1]. Germany

Mathematics Subject Classification References 65N30 1. Baker, R.E., Gaffney, E.A., Maini, P.K.: Partial differential equations for self-organization in cellular and developmental biology. Nonlinearity 21, R251–R290 (2009) 2. Baker, R.E., Schnell, S., Maini, P.K.: Waves and patterning in developmental biology: vertebrate segmentation and feather bud formation as case studies. Int. J. Dev. Biol. 53, 783–794 (2009) 3. Britton, N.F.: Reaction-Diffusion Equations and Their Applications to Biology. Academic, London (1986) 4. Crampin, E.J., Hackborn, W.W., Maini, P.K.: Pattern formation in reaction-diffusion models with nonuniform domain growth. Bull. Math. Biol. 64, 747–769 (2002) 5. Hillen, T., Painter, K.J.: A user’s guide to PDE models for chemotaxis. J. Math. Biol. 58, 183–217 (2009) 6. Keller, E.F., Segel, L.A.: Initiation of slime mold aggregation viewed as an instability. J. Theor. Biol. 26, 399–415 (1970) 7. Madzvamuse, A., Gaffney, E.A., Maini, P.K.: Stability analysis of non-autonomous reaction-diffusion systems: the effects of growing domains. J. Math. Biol. 61, 133–164 (2010) 8. Murray, J.D.: Mathematical Biology I: An Introduction. Springer, New York (2002) 9. Murray, J.D.: Mathematical Biology II: Spatial Models and Biomedical Applications. Springer, New York (2002) 10. Othmer, H.G., Stevens, A.: Aggregation, blowup, and collapse: the ABC’s of taxis in reinforced random walks. SIAM J. Appl. Math. 57, 1044–1081 (1997) 11. Othmer, H.G., Painter, K.J., Umulis, D., Xue, C.: The intersection of theory and application in elucidating pattern formation in developmental biology. Math. Model. Nat. Phenom. 4, 3–82 (2009) 12. Turing, A.M.: The chemical basis of morphogenesis. R. Soc. Lond. Philos. Trans. B 237, 37–72 (1952) 13. Umulis, D.M., Shimmi, O., O’Connor, M.B., Othmer, H.G.: Organism-scale modeling of early Drosophila patterning via bone morphogenetic proteins. Dev. Cell 18, 260–274 (2010) 14. Ward, M.J.: Asymptotic methods for reaction-diffusion systems: past and present. Bull. Math. Biol. 68, 1151–1167 (2006) 15. Wolpert, L., Smith, J., Jessel, T., Lawrence, P., Robertson, E., Meyerowitz, E.: Principles of Development, 3rd edn. Oxford University Press, Oxford (2006)

Petrov-Galerkin Methods for Variational Problems For the solution of partial differential equations, a corresponding variational problem can be derived, and the variational solution can be approximated by ! Galerkin methods. Petrov-Galerkin methods extend the Galerkin idea using different spaces for the approximate solution and the test functions. This is now introduced for abstract variational problems. Let U and V be Hilbert spaces, let aW U  V ! R be a bilinear form, and for a given functional f 2 V 0 let u 2 U be the solution of the variational problem a.u; v/ D hf; vi for all v 2 V . Let UN U and VN V be discrete subspaces of finite dimension N D dim UN D dim VN . The PetrovGalerkin approximation uN 2 UN is a solution of the discrete variational problem a.uN ; vN / D hf; vN i for all vN 2 VN . It is not a priori clear that the continuous and the discrete variational problems have unique solutions. For the well-posedness of the continuous problem, we assume that positive constants C  ˛ > 0 exist such that ja.u; v/j  C kukU kvkV and a.u; v/  ˛ kukU ; v2V nf0g kvkV sup

and that for every v 2 V n f0g some uv 2 U exists such that a.uv ; v/ ¤ 0. The variational problem corresponds to the equation Au D f , where A 2 L.U; V 0 / is the

P

1150

Petrov-Galerkin Methods

linear operator defined by hAu; vi D a.u; v/ for all u 2 U and v 2 V . The assumptions on the bilinear form yield ˛ kukU  kAukV 0  C kukU , which shows that the operator A is injective and that the range A.U / D A.U / is closed in V 0 . Moreover hA0 v; uv i D a.uv ; v/ ¤ 0 shows that the adjoint operator A0 2 L.V; U 0 / is injective, and thus, the operator A is surjective. This proves that the continuous variational problem has a unique solution u 2 U satisfying kukU  ˛ 1 kf kV 0 . For the well-posedness and the stability of the discrete problem, we assume in addition that a constant N ;vN / ˛0 > 0 exists such that supvN 2VN nf0g a.u  kvN kV ˛0 kuN kU holds for all uN 2 UN . Then, the PetrovGalerkin approximation AN 2 L.UN ; VN0 / defined by hAN uN ; vN i D a.uN ; vN / for uN 2 UN and vN 2 VN is injective and, since dim UN D dim VN D N < 1, also surjective. Let fN 2 VN be defined by hfN ; vN i D hf; vN i for vN 2 VN . The unique solution uN 2 UN of AN uN D fN solves the discrete variational problem and is bounded by kuN kU  ˛01 kf kV 0 . For a dense family .UN  VN /N 2N in U  V , the Petrov-Galerkin approximations .uN /N 2N converge to the continuous solution u 2 U , if the constant ˛0 can be chosen independent of N 2 N . Then, for any suitable interpolation IN W U ! UN , the approximation error can be bounded by the interpolation error u  IN u in two steps. The orthogonality relation a.u  uN ; vN / D 0 for vN 2 VV gives

˛0 kuN  IN ukU  sup

vN 2VN

D sup

vN 2VN

a.uN  IN u; vN / kvN kV

a.u  IN u; vN /  C ku  IN ukU kvN kV

and ku  uN kU  ku  IN ukU C kIN u  uN kU gives  C ku  uN kU  1 C ku  IN ukU ˛0 (for an improved estimate see [4]). Depending on regularity properties of the solution u and suitable interpolation estimates, this also provides a priori estimates for the convergence rate. In the special case U D V it also may be advantageous to use Petrov-Galerkin methods with UN ¤ VN , e.g., to improve the approximation properties for nonsymmetric or indefinite problems.

Applications The most well-known family of Petrov-Galerkin methods are streamline-diffusion methods for convectiondominated problems introduced in [2]. Here, a standard finite element space UN is combined with a test space VN where the finite element basis functions are modified depending on the differential operator. These methods allow for robust convergence estimates in the case of vanishing diffusion and are often applied to flow problems. A simple example for a robust nonconforming Petrov-Galerkin method for the model S N D  RD problem "u C b  ru C cu D f in is defined by Z 

 "ru  rv C .b  ru C cu/v dx



C

X 

Z D

Z ˇ

."u C b  ru C cu/.b  rv/ dx 

f v dx C

X 

Z ˇ

f b  rv dx : 

using test functions of the form v C ˇ b  rv with a mesh-dependent stabilization parameter ˇ > 0 depending on the element . Recently, a class of discontinuous Petrov-Galerkin methods was proposed [3]. In this method the solution is approximated by its traces on the element faces and discontinuous element contributions in the interior of the elements. A special basis in the test space VN is constructed locally by solving the adjoint problem in a larger space VOM V : for every basis function n 2 UN , the optimal test function n 2 VOM is determined by the variational problem . n ; vO /V D a.n ; vO / for vO 2 VOM . Then, the discrete solution P uN D N nD1 un n is obtained from the linear system A u D f with the symmetric positive definite matrix     A D a.n ; m / D . n ; m /V 2 RN N m;n   m;n and right-hand side f D hf; n i 2 RN . It can n be shown that this method is robust, e.g., for the Helmholtz problem with large wave numbers. Another class with broad applications is the meshless local Petrov-Galerkin method (MLPG) introduced in [1]. Several concepts for the local construction of trial functions exist, e.g., moving least squares, the partition of unity method, or radial basis functions.

Phase Plane: Computation

1151

Cross-References  Galerkin Methods

References 1. Atluri, S., Zhu, T.: A new meshless local Petrov-Galerkin (MLPG) approach in computational mechanics. Comput. Mech. 22(2), 117–127 (1998) 2. Brooks, A.N., Hughes, T.J.: Streamline upwind/PetrovGalerkin formulations for convection dominated flows with particular emphasis on the incompressible Navier-Stokes equations. Comput. Methods Appl. Mech. Eng. 32, 199–259 (1982) 3. Demkowicz, L., Gopalakrishnan, J.: Analysis of the DPG method for the Poisson equation. SIAM J. Numer. Anal. 49(5), 1788–1809 (2011) 4. Xu, J., Zikatanov, L.: Some observations on Babuˇska and Brezzi theories. Numer. Math. 94, 195–202 (2003)

Phase Plane: Computation H¨useyin Koc¸ak Department of Computer Science, University of Miami, Coral Gables, FL, USA

Curves Defined by Differential Equations

great many problems in pure mathematics and mechanics. Unfortunately, in the vast majority of cases which we encounter, we cannot integrate these equations using already-known functions – for example, using functions defined by quadrature. If we therefore limit ourselves to those cases which can be studied using definite or indefinite integrals, the scope of our research will be strikingly narrowed, and the vast majority of problems which occur in applications will remain unsolvable. It is thus imperative to study functions defined by differential equations in themselves, without trying to reduce them to simpler functions, as we have done for algebraic functions, which we tried to reduce to radicals and which we now study directly, or as we have done for the integrals of algebraic differential equations, which we have long tried to express in finite terms. We must naturally approach the theory of each and every function by the qualitative part; that is why the first problem we encounter is the following: to construct curves defined by differential equations.

Presently, using numerical methods, one can compute remarkably accurate approximations to solutions of almost any differential equation for finite time. However, such approximation methods may not reliably predict the longtime behavior of orbits, which is the main concern in the study of dynamical systems. Longtime behavior of orbits is determined by their limit sets, and the identification of limit sets in a specific set of differential equations on the computer can be a challenging task.

Consider the system of autonomous ordinary differential equations dx D f .x; y/; dt

dy D g.x; y/ dt

P

(1)

where the given functions f and g are sufficiently smooth. The phase portrait of such a system is the collection of the parametrized curves, also called orbits, on the .x; y/-plane defined by the solutions .x.t/; y.t// of the differential equations for all initial conditions .x.0/ D x0 ; y.0/ D y0 /. Early investigators in differential equations, with few exceptions, occupied themselves with local properties of functions as solutions of special equations. In a series of remarkable memoirs [16, 17], Poincar´e initiated the global qualitative study of curves defined by solutions of differential equations: A comprehensive theory of functions defined by differential equations would be enormously useful in solving a

Phase Plane: Computation, Fig. 1 Poincare’s original handdrawn phase portrait of (2) in his paper [17]

1152

Phase Plane: Computation

Poincar´e produced the following example exhibitA Phase Portrait by Poincare´ Possible longtime qualitative behavior of orbits of ing the possible limit sets listed above as a parameter, planar differential equations was determined by K, is varied: Poincar´e [16, 17], with additional details supplied by Bendixson [2]: Suppose that the planar system Eq. (1) dy dx D AC  B; D BC C A (2) has isolated equilibrium points. If an orbit remains dt dt bounded for all forward time then the limit set (!limit set) of the forward orbit (similarly, if an orbit where remains bounded for all backward time then the limit set (˛-limit set) of the backward orbit) is either (1) an A D x.2x 2 C 2y 2 C 1/; equilibrium point, (2) a periodic orbit, or (3) equilibrium points and orbits connecting them. A noteworthy B D y.2x 2 C 2y 2  1/; implication of this result is the absence of chaotic beC D .x 2 C y 2 /2 C x 2 C y 2  K: havior in the dynamics of planar differential equations. Phase Plane: Computation, Fig. 2 Vector field of (2) for K D 0:4 and 250 randomly chosen initial conditions in a box near the origin integrated backward and forward in time. The limit sets are the three equilibria; the origin is a saddle, and the other two equilibria are sources. The locations of the equilibria persist for all values of K

Phase Plane: Computation, Fig. 3 Orbits of ten initial conditions, marked with circles, of (2) for K D 0:25. The arrows indicate the forward time direction. The two source equilibria have become nonhyperbolic with pure imaginary eigenvalues and are about to undergo a Poincar´e–Andronov–Hopf bifurcation

Phase Plane: Computation

1153

Phase Plane: Computation, Fig. 4 For K D 0:1, the two source equilibria became asymptotically stable. This change in stability type gave rise two unstable limit cycles, repelling periodic orbits. The ˛-limit set of an orbit starting inside one of the limit cycles is the limit cycle; the !-limit set is the equilibrium inside

Phase Plane: Computation, Fig. 5 The limit sets of the previous phase portrait for K D 0:1

P

Phase Plane: Computation, Fig. 6 As K is increased, the two periodic orbits grow, and for K D 0:0, they touch at the origin forming a “figure 8” consisting of two homoclinic orbits to the saddle equilibrium at the origin. An orbit starting inside a loop has its ˛-limit set as a homoclinic orbit, while an orbit starting on the outside of the figure 8 has the entire figure 8 as its ˛-limit set

1154 Phase Plane: Computation, Fig. 7 The limit sets of the previous phase portrait for K D 0:0

Phase Plane: Computation, Fig. 8 For K D 0:2, the figure 8 loses its pinch at the origin and gives rise to a large hyperbolic periodic orbit which is the ˛-limit set of most orbits

Phase Plane: Computation, Fig. 9 The limit sets of the previous phase portrait for K D 0:2

Phase Plane: Computation

Phase Plane: Computation

1155

A hand-drawn phase portrait of these equations (on • Bifurcations: Like the parameter K in the exthe so-called Poincar´e sphere) by Poincar´e himself is ample of Poincar´e, most model equations in apshown in Fig. 1. Following the computational strategies plications may contain parameters. The study of outlined below, several representative computerqualitative changes in the phase portraits of dygenerated phase portraits on the plane are illustrated in namical systems as parameters are varied is called Figs. 2–9 using [15]. bifurcation theory. A slight change in a parameter can result in, for example, the disappearance of equilibrium points through a saddle-node bifurcation or the appearance of a periodic orbit through Computational Tools and Strategies a Poincar´e–Andronov–Hopf bifurcation. A list of generic bifurcations of planar systems depending on • Vector field: Equation (1) defines a vector a parameter is in [12]; it is important to be aware of .f .x; y/; g.x; y// at each point .x; y/ of the such a list. At a parameter value where a bifurcaplane. One can populate the plane with vectors, tion occurs, orbits usually approach nonhyperbolic preferably normalized, at some grid points. The equilibria, or periodic orbits, at a painfully slow resulting collection of vectors is called a vector field rate. of (1). Any orbit must be tangent to the vector at • Specialized tools: Distinguished orbits like stable each point the curve passes through. The relatively and unstable manifolds, connecting homoclinic safe and inexpensive computation of a vector field and heteroclinic or periodic orbits, can be can yield a rough idea of the interesting regions of approximated more accurately using specialized a phase portrait. numerical techniques involving boundary value • Algorithm: A general-purpose variable step-size problems rather than by simple integration of algorithm like Dormand–Prince 5(4) [9] suffices orbits through select initial conditions. Also, these for most calculations. However, certain differential orbits can be followed, even through bifurcations, equations may require more specialized algorithms. as parameters are varied using continuation For stiff equations, a good choice is an implicit algomethods [1, 3, 7, 14]. rithm, e.g., SEULEX [10]. For Hamiltonian systems, • Existence from numerics: Most numerical symplectic algorithms [11], e.g., symplectic Euler, methods mentioned above are designed for might yield better results for longtime integrations. approximating an orbit whose existence is • Time: To compute the full orbit through an initial already known or presumed. Rigorously escondition, one must integrate in both backward and tablishing the existence of a periodic orbit forward time. To identify the limit set of an orbit, from a numerically computed orbit that may it may be necessary to omit the plotting of transient appear nearly periodic, or the existence of behavior. an infinite homoclinic orbit from finite time • Equilibria: A necessary task is to locate the equicomputations, requires careful mathematical libria, that is, to find the solutions of the equations analysis [5, 8]. f .x; y/ D 0 and g.x; y/ D 0. This can be accomplished using a root finder like the Newton– What to Do in Practice? Raphson method with a number of randomly chosen It is difficult to ascertain the correctness of a numerically computed phase portrait. On a set of initial constarting points. • Invariant manifolds: Orbits eventually follow un- ditions, one should, in the least, use several algorithms stable manifolds in forward time and the stable with various settings of step size, tolerance, order, etc. manifolds in backward time. Therefore, in the vicin- If the resulting phase portraits appear to be qualitaity of an equilibrium point, one can take some tively equivalent (preserving the number and stability initial conditions and follow their orbits in both types of equilibria, periodic and connecting orbits), backward and forward time to see these invariant one can be reasonably confident of the resulting phase manifolds, which are the candidates for connecting portrait. If the differences are too great, the problem may not be tractable numerically, e.g., Hilbert’s 16th orbits.

P

1156

Photonic Crystals and Waveguides: Simulation and Design

problem of determining the number of limit cycles even in the simplest case when the functions f and g in (1) are quadratic polynomials [4, 6, 13, 18].

Photonic Crystals and Waveguides: Simulation and Design

References

Martin Burger Institute for Computational and Applied Mathematics, Westf¨alische Wilhelms-Universit¨at (WWU) M¨unster, M¨unster, Germany

1. AUTO: Software for continuation and bifurcation problems in ordinary differential equations. http://cmvl.cs.concordia. ca/auto/ (2007) 2. Bendixson, I.: Sur les courbes d´efinies par une e´ quation diff´erentielle. Acta Math. 24, 1–88 (1901) 3. Beyn, W.-J.: The numerical computation of connecting orbits in dynamical systems. IMA J. Numer. Anal. 9, 379–405 (1990) 4. Chicone, C., Tian, J.: On general properties of quadratic systems. Am. Math. Mon. 89, 167–178 (1982) 5. Coomes, B., Koc¸ak, H., Palmer, K.: Shadowing in ordinary differential equations. Rend. Sem. Mat. Univ. Pol. Torino 65, 89–114 (2007) 6. De Maesschalck, P., Dumortier, F.: Classical Li´enard equa C 2 limit cycles. tions of degree n  6 can have Œ n1 2 J. Differ. Equ. 250, 2162–2176 (2011) 7. Doedel, E.: Lecture Notes on Numerical Analysis of Nonlinear Equations. http://cmvl.cs.concordia.ca/auto/notes.pdf (2010) 8. Franke, J., Selgrade, J.: A computer method for verification of asymptotically stable periodic orbits. SIAM J. Math. Anal. 10, 614–628 (1979) 9. Hairer, E., Norsett, S., Wanner, G.: Solving Ordinary Differential Equations I. Nonstiff Problems. Springer Series in Computational Mathematics, vol. 8. Springer, Berlin/New York (1993) 10. Hairer, E., Norsett, S., Wanner, G.: Solving Ordinary Differential Equations II. Stiff and Differential-Algebraic Problems. Springer Series in Computational Mathematics, vol. 14. Springer, Berlin (1996) 11. Hairer, E., Lubich, C., Wanner, G.: Geometric Numerical Integration: Structure-Preserving Algorithms for Ordinary Differential Equations. Springer Series in Computational Mathematics, vol. 31. Springer, Berlin/New York (2002) 12. Hale, J., Koc¸ak, H.: Dynamics and Bifurcations. Springer, Berlin/Heidelberg/New York (1992) 13. Ilyashenko, Y.: Centennial history of Hilbert’s 16th problem. Bull. A.M.S. 39, 301–354 (2002) 14. Kuznetsov, Y.A.: Elements of Applied Bifurcation Theory, 3rd edn. Springer, New York (2004) 15. PHASER: A universal simulator for dynamical systems. http://www.phaser.com (2009) 16. Poincar´e, M.H.: M´emoire sur les courbes d´efinies par une e´ quation diff´erentielle (I). J. Math. Pure et Appl. 7, 375–422 (1881) 17. Poincar´e, M.H.: M´emoire sur les courbes d´efinies par une e´ quation diff´erentielle (II). J. Math. Pure et Appl. 8, 251–296 (1882) 18. Songling, S.: A concrete example of the existence of four limit cycles for plane quadratic systems. Sci. Sinica 23, 153–158 (1980)

Mathematics Subject Classification 35Q60; 35Q61; 65N21; 49N45; 78A50; 78A55

Synonyms Optical waveguides; Photonic bandgaps; Photonic crystals; Plasmonic waveguides

Short Definition Photonic crystals and waveguides are devices to manipulate wave properties (usually in the optical range) by heterogeneous dielectric materials. The term photonic crystals specifically refers to periodic assemblies of heterogeneous structures, which allow to control and manipulate the flow of photons in a similar way as semiconductor crystals do for electrons.

Description Photonic crystals (see, e.g., [1–5]) have received growing attention in engineering and applied mathematics over the last years (cf. [7] and the references therein). In general, the term photonic crystal denotes a structure made of periodic dielectrics (e.g., rods or holes), which is well known to exhibit a certain bandgap, i.e., a range of frequencies (called stop band), where light waves cannot propagate (similar phenomena exist also for sound or elastic waves, called photonic bandgaps in the latter case). In practice, this means that the field obtained from an incoming wave at a frequency in the bandgap decays exponentially with the distance and is usually sufficiently close to zero after 20 periodic cells.

Photonic Crystals and Waveguides: Simulation and Design

Besides producing a bandgap, interesting effects can be achieved in a photonic crystal by introducing point or line defect, i.e., by removing (or changing) single holes or lines of holes. If a line is removed completely, a fill-in of the bandgap occurs, usually by one or two eigenvalues [6]. Even more complex band structures can be produced, when objects with different geometries than the original photonic crystal are inserted into the line defect (still keeping the original periodicity in one direction). This technique allows to filter certain frequencies and to produce smaller bandgaps than the one of the original photonic crystal structure. Mathematical Modelling In the following, we provide a short introduction to the mathematical modelling of photonic crystal structures. For a detailed exposition, we refer to [7]. At a macroscopic level, the light propagation through a photonic crystal structure is determined by Maxwell’s equations, i.e., in the nonmagnetic case, 1 @H ; c @t 1 @E r H D  ; c @t

r E D

r  H D 0; r  .E/ D 0;

1157

 u D ! 2 u

in 0 ;

(1)

where u represents the third component of the electric field. In the case of transverse magnetic (TM) polarized fields, the problem can be reduced to  r 

 1 ru D ! 2 u 

in 0 ;

(2)

where u represents the third component of the magnetic field. In both cases, we can interpret the arising equation as an eigenvalue problem for a second-order partial differential operator, with ! 2 being the eigenvalue. Large Periodic Structures For large periodic structures, a direct solution on the whole domain becomes computationally extremely expensive, and usually Floquet theory [8, 9] is used as an alternative. In this approach one looks for a periodic solution on Rd with rectangular periodic cell , which can be obtained from the Floquet transform

.F u/.x; ˛/ D e i ˛x

X

U.xne/e i ˛n ; x 2 ; ˛ 2 K:

n2Zd

(3) where E and H are the macroscopic electric and magnetic fields,  is the electric permittivity, and c denotes the speed of light. The permittivity  is a function of the material and usually takes two different values within the photonic crystal structure, namely, a high one in the dielectric material and a low one in airholes. By considering monochromatic waves E.:; t/ D O we obtain a stationary e i !t EO and H.:; t/ D e i !t H, system of the form i! O r  EO D  H; c i ! O O D  E; rH c

O D 0; r H O D 0; r  . E/

to be solved in the three-dimensional domain representing the geometry of the photonic crystal structure. Simplifications can be obtained for certain polarizations of the waves, namely, transverse electric (TE) and transverse magnetic (TM). For transverse electric (TE) polarized fields, the problem reduces to a scalar equation of Helmholtz type, i.e.,

d

Here K D .; / is the first Brillouin zone and e the diagonal vector spanning . The differential operator transforms via F .ru/ D .r C i ˛/F .U /, such that the above differential equations can be transformed directly to problems on  K. Denoting u˛ D .F U /.; ˛/, we obtain  .r C i ˛/  .r C i ˛/u˛ D ! 2 u˛ in the TE-polarized case and   1 .r C i ˛/u˛ D ! 2 u˛  .r C i ˛/  

in

(4)

in  K (5)

in the TM-polarized case. Numerical Band Structure Computation Various methods have been proposed for computing band structures numerically. Most approaches are based on finite element or finite difference discretizations of time-harmonic Maxwell’s equations or the equations for TE and TM polarization (we refer,

P

1158

e.g., to [10–12]). Alternative approaches are based on direct integration of time-dependent Maxwell’s equations (TDFE and TDFD) in order to follow the propagation of waves through photonic crystal structures in time. As usual with wave propagation, the modelling of (artificial) boundary conditions is crucial for computational purpose, but a hard task. In this respect absorbing boundary conditions and perfectly matched layers (PML) have evolved to become commonly accepted. Optimal Design Since the fabrication of photonic crystals to reach certain goals is not straightforward and sometimes even slightly counterintuitive, rational design by mathematical modelling and inverse problem techniques has gained increasing importance in the last years. The major design variable is the size and positioning of the air defects in the photonic crystal structures, which can nowadays be built in a high variety of shapes except for a lower bound on the curvature radius. This can be translated into the design of the dielectric coefficient  as a piecewise constant function taking two prescribed values (for the material of the crystal and for air); thus, a standard type of shape and topology optimization problem is obtained, which can be solved numerically by parameterization, level set methods, or relaxation approaches (see [13] for an overview). In the following we detail some important example classes of problems in photonic crystal structures.

Photonic Crystals and Waveguides: Simulation and Design

respect to the topology of the waveguide, however restricted to a fixed length (see [14–16]). For reaching miniaturization of waveguides, one can tackle some kind of dual problem, namely, minimizing the length of the waveguide subject to the constraint of allowed power loss. Bandgap Optimization in Photonic Crystals Optimizing the band structure is a task of central importance for photonic crystals. Referring to the Bloch modes in polarized cases, the band structure is determined by all eigenvalues !k .˛/ such that (4) and (5) have a nontrivial solution. A bandgap is referred to a gap in the band structure, i.e., an interval .! ; !C / such that there is no eigenvalue !k .˛/ inside for all ˛ in the first Brillouin zone. Bandgaps can be used effectively to filter a certain frequency range; usual design goals are: • Maximizing bandgaps: For various applications a large bandgap is of high importance. One thus starts with a setup including a bandgap between the kth and k C 1th eigenvector and tries to maximize this gap by topology optimization (see [17–21]). This is achieved by maximizing functionals like

J D inf !kC1 .˛/  sup !k .˛/: ˛

(7)

˛

A wide bandgap can also be a good starting point for further design tasks related to filtering such as the ones following. • Narrowing bandgaps: In some applications it is Power Transmission Through Waveguides important to filter a range of frequencies above Waveguides are designed to propagate a certain mode or below a certain pass band. This can be tackled through the material, which is however not achieved to by narrowing a given bandgap, i.e., for a given a perfect extent for a guide of finite length. Hence, it frequency !0 inside the bandgap .! ; !C / of the is natural to maximize the power of propagation of the original photonic crystal structure, one wants to desired fundamental mode, i.e., in a polarized case: design the material such that the resulting structure ˇ ˇZ ˇ ˇ has a bandgap .!Q  ; !0 / or .!0 ; !Q C /. This problem (6) u  F d ˇˇ ; P D ˇˇ can be formulated with similar objectives as above O and additional constraints, e.g., minimizing !Q  with a constraint of having a bandgap in .!Q  ; !0 /. where O is the outgoing part of the waveguide boundary. The function F is the fundamental model • Filtering bands: A related problem is to filter a for the further transmission in the outside region, certain band, which can be achieved if there are two which is computed from an eigenvalue problem for the bandgaps below and above the desired filter range. Helmholtz equation in a semi-infinite region (see [14]). Thus, for two given frequencies !1 < !2 , both inside the original bandgap .! ; !C /, one tries to Alternatively one can use a small volume adjacent design the material such that the frequency range to O and a volume integral to compute a power .!1 ; !2 / is filtered, i.e., the arising structure has two functional. In order to obtain an optimal waveguide, bandgaps .! ; !1 / and .!2 ; !C /. a monotone functional of P can be maximized with

Poisson-Nernst-Planck Equation

1159

Depending on the specific type of waves to be 10. Dobson, D.C.: An efficient method for band structure calculations in 2D photonic crystals. J. Comput. Phys. 363, used in the structure, all the design goals above can 363–376 (1999) be tackled for TM polarization, TE polarizations, 11. Dobson, D.C., Gopalakrishnan, J., Pasciak, J.E.: An efficient combinations of both, or even fully three-dimensional method for band structure calculations in 3D photonic crystals. J. Comput. Phys. 668, 668–679 (2000) structures. Several other related design problems for photonic crystals can be found in literature, and 12. Schmidt, F., Friese, T., Zschiedrich, L., Deuflhard, P.: Adaptive multigrid methods for the vectorial maxwell eigenvalue there is certainly a variety of upcoming tasks due to problem for optical waveguide design. In: Jaeger, W., Krebs, technological development. H.J. (eds.) Mathematics. Key Technology for the Future, Plasmon Structures Recently, there appears to be increasing interest in plasmon wave structures. Plasmons are waves at optical frequencies, which propagate along a surface. By the latter the dispersion relations can be changed, and this allows to focus optical light to nanoscale wavelengths. In order to focus a Gaussian beam to the surface, a coupling structure is needed, which can be realized as a grating coupler, i.e., a locally rough structure at the surface. The obvious optimal design problem is to optimize the shape of the grating coupler in order to obtain maximal output power of the created surface plasmons. The objective functionals in such a problem are clearly similar to the ones for waveguides, but the design variable changes to the local shape, usually with strong constraints from the manufacturing abilities. For a piecewise rectangular structure this optimal design problem was investigated in [22], which only seems a first step to various future problems in this area.

References 1. Yablonovitch, E.: Inhibited Spontaneous Emission in SolidState Physics and Electronics. Phys. Rev. Lett. 58, 2059 (1987) 2. John, S.: Strong localization of photons in certain disordered dielectric superlattices. Phys. Rev. Lett. 58, 2486 (1987) 3. Bowden, C.M., Dowling, J.P., Everitt, H.O.: Development and applications of materials exhibiting photonic bandgaps. J. Opt. Soc. Am. B 10, 280–282 (1993). (Special issue) 4. Joannopoulos, J.D., Johnson, S.G., Winn, J.N., Meade, R.D.: Photonic Crystals: Molding the Flow of Light, 2nd edn. Princeton University Press, Princeton (2008) 5. Yablonovitch, E.: Photonic crystals. Sci. Am. 285, 47 (2001) 6. Ammari, H., Santosa, F.: Guided waves in a photonic bandgap structure with a line defect. SIAM J. Appl. Math. 64, 2018 (2004) 7. Kuchment, P.: The mathematics of photonic crystals. In: Bao, G. et al. (eds.) Mathematical Modeling in Optical Science, p. 207. SIAM, Philadelphia (2001) 8. Bensoussan, A., Lions, J.L., Papanicolaou, G.: Asymptotic Methods in Periodic Media. North-Holland, Amsterdam (1978) 9. Kuchment, P.: Floquet Theory for Partial Differential Equations. Birkhaeuser, Boston (1978)

vol. 279. Springer, Heidelberg (2003) 13. Burger, M., Osher, S., Yablonovitch, E.: Inverse problem techniques for the design of photonic crystals. IEICE Trans. E 87, 258 (2004) 14. Felici, T., Engl, H.W.: On shape optimization of optical waveguides using inverse problem techniques. Inverse Probl. 17, 1141 (2001) 15. Gallagher, D.F.G., Felici, T.: Eigenmode Expansion Methods for Simulation of Optical Propagation in Photonics – Pros and Cons. Proc. SPIE 4986, 375 (2003) 16. Frei, W.R., Tortorelli, D.A., Johnson, H.T.: Topology optimization of a photonic crystal waveguide termination to maximize directional emission. Appl. Phys. Lett. 86, 111114 (2005) 17. Cox, S.J., Dobson, D.: Maximizing band gaps in twodimensional photonic crystals. SIAM J. Appl. Math. 59, 2108 (1999) 18. Cox, S.J., Dobson, D.: Band structure optimization of twodimensional photonic crystals in Hpolarization. J. Comput. Phys. 158, 214 (2000) 19. Kao, C.Y., Osher, S.J., Yablonovitch, E.: Maximizing band gaps in two-dimensional photonic crystals by using level set methods. Appl. Phys. B 81, 35 (2005) 20. Halkjaer, S., Sigmund, O., Jensen, J.S.: Inverse design of photonic crystals by topology optimization. Z Kristallographie 220, 895 (2005) 21. Duehring, M.B., Sigmund, O., Feurer, T.: Design of photonic bandgap fibers by topology optimization. J. Opt. Soc. Am. B 27, 51 (2010) 22. Lu, J., Petre, C., Yablonovitch, E., Conway, J.: Numerical optimization of a grating coupler for the efficient excitation of surface plasmons at an Ag-SiO 2 interface. J. Opt. Soc. Am. B 24, 2268 (2007)

Poisson-Nernst-Planck Equation Benzhuo Lu Institute of Computational Mathematics and Scientific/Engineering Computing, Chinese Academy of Sciences, Beijing, China

Mathematics Subject Classification 34A34; 3500; 35J60; 35Q92; 6500; 92XX

P

1160

Poisson-Nernst-Planck Equation

Synonyms

applying mass conservation law to each ion species leads to the drift-diffusion equation or similarly the Nernst-Planck (NP) equation (see (1)). The electric field itself is simultaneously determined from the ion density distribution and environment charges (if existed) through Poisson equation (see (2)),

PNP equation; Poisson-Nernst-Planck equation

Description

@ i @t

Electrodiffusion describes the diffusion process of ions under the influence of an electric field induced by ion charges themselves. The process exists in many apparently different physical objects such as electrolyte solution, microfluidic system, charged porous media, and ion channel (see Fig. 1). Similar transport behavior also apply to the electrons and holes in semiconductor. Continuum theory uses the average ion density and potential representations that can be directly compared with experimental measurements. In a widely known premium continuum theory, the ion flux compromises of a diffusion term due to concentration gradient obeying Fick’s law and a drift term of ions in a potential gradient obeying Ohm’s law:

where  is the dielectric permittivity, and f is the environment permanent charge distribution as doping in transistors and fixed charges in ion channels (typically an ensemble of singular charges inside protein P

f .x/ D j qj ı.x  xj /; see Fig. 1). The permanent charges are explicitly included here, considering that they may play important role in many systems. The coupled (1) and (2) form the Poisson-NernstPlanck (PNP) equation. The equation system was actually studied and applied in above-mentioned areas much earlier than its name was explicitly used in [7]. An equivalent name, NPP (Nernst-Planck-Poisson) equation, was also occasionally used as in [2]. The NP equation was named after its introducers, Nernst and Planck [11, 12], and has the same form as Smoluchowski equation describing motion of a Brownian particle in a prescribed external potential under conditions of high friction. PNP equation is a set of coupled partial differential equations, in which the potential, ion concentrations,

J D D.r C ˇq r/; where is the ion concentration, q and D are the charge and diffusion coefficient of the ion, respectively, and  is the electrostatic potential. The constant ˇD1=.kB T / is the inverse Boltzmann energy where kB is the Boltzmann constant and T is the absolute temperature. Then, for a general n-species system,

a

b

Dr  Ji Dr  Di .r i Cˇqi i r/ ; 1i n; (1) P r  .r/ D ni qi i C f ; (2)

c

Cathode

V

V

+ + + + − − − −

+ + + + − − − −

Membrane

− − + − + − − + + − + − −

I

− + − Protein + − − + + − + − −

V

Anode

Poisson-Nernst-Planck Equation, Fig. 1 (a) Fuel cell; (b) nanofluidic channel with a bias voltage and electrical charged wall inside the channel; (c) ion channel system including bulk

solution, membrane, and protein with fixed charges shown inside. V denotes transmembrane potential

Poisson-Nernst-Planck Equation

1161

as well as the fluxes are determined by the average potential and ion density distributions through a self-consistent solution of (1) and (2). Hence, PNP is a mean-field theory. The equation is used to study ion concentration profiles, diffusion-reaction rate at reactive boundary, and ion conductivity typically represented as current-voltage characteristics (I -V curve). PNP theory, application, and limitations were reviewed in specific area, e.g., ion channel [3, 13].

equation is usually solved numerically (analytical solution is available only in some very special cases, e.g., the classic Goldman-Hodgkin-Katz equation). Effective methods were designed for both steady-state PNP [6] and time-dependent PNP [9] for general 3D systems with permanent singular charges and complex geometry. However, for large practical systems, the numerical efficiency and stability of these numerical methods are subject to further examination. Slotboom transformation is a useful technique in solving the PNP equation. By introducing the SlotRelation to Poisson-Boltzmann Theory Poisson-Boltzmann theory describes the equilibrium boom variables state of ionic solution in which the ion concentration follows Boltzmann distribution, which is a special DN i D Di e ˇqi  ; Ni D i e ˇqi  ; case of PNP theory when flux vanishes everywhere. The PB results can be obtained from the numerical solution of PNP equation by properly setting the the steady-state Nernst-Planck equation is transformed boundary/interface conditions (such as zero-flux to a Laplace equation condition). This property is useful especially in more complicated electro-diffusion models where the   r  DN i r Ni D 0: closed-form equilibrium distribution is not available, but can be implicitly determined by Ji D 0 (see [8]). Ion Selectivity in Nanochannel Ion permeation in ion channel or nanofluidic channel is featured by ionic selectivity. A specific channel is selective to permeation of certain ion species. The permanent environment charges play important role in ion selectivity through affecting the dynamics of ions within a nanochannel. Because the Debye length is usually comparable with the channel width, ions inside the fluid are no longer shielded from the permanent charges inside the channel. The performance of ion selectivity is also largely related to the applied voltage bias, ionic size, and concentration, as well as to the length of channel. Mathematical and Numerical Aspects The PNP equation can be derived from different routes, e.g., from the microscopic model of Langevin trajectories in the limit of large damping and absence of correlations of different ionic trajectories [10]. Mathematical analyses were made for some basic properties such as the existence and stability for the solutions of the steady PNP equations [5], existence and long time behavior of the unsteady PNP equations [1], and the permanent charge effects [4]. Due to the nonlinear nature, and irregular geometries in certain cases as in ion channel, the PNP

These transformations hence give rise to a self-adjoint elliptic operator in case of a fixed potential, and the discretization in solving the transformed equation could produce a symmetric stiffness matrix. At the same time, the Poisson equation will be transformed to a nonlinear equation with similar form of PB equation. When using iterative method to solve the coupled PNP equation system, solution of the transformed one usually converges faster than that of the original one for a variety of practical systems. Model Limitations The mean-field PNP theory treats the diffusive particles with vanishing size and ignores correlations among ions. The assumption is reasonable in case the ionic solution is dilute and the characteristic dimension of space for diffusion is much larger than the ion size. In narrow channel with comparable size, the discrete nature of ion cannot be well represented in a premium continuum description as in PNP theory, and it is necessary to introduce theory beyond PNP (e.g., see [8]). At the same time, in these confined spaces where the ionic solution is far from being homogeneous and dilute, the diffusion coefficient of ions and the dielectric permittivity are not simply constants as frequently used, and become research issues, too.

P

1162

Polynomial Chaos Expansions

-algebra of events, and P is the probability measure. TheR expectation value of u, denoted as EŒu, is defined 2 1. Biler, P., Hebisch, W., Nadzieja, T.: The Debye system: as ˝ u.!/dP .!/. L .˝; F ; P/ is a Hilbert space of existence and large time behavior of solutions. Nonlinear functions with a finite L -norm, kk2 D EŒ   < 1. 2 Anal. 23, 1189–1209 (1994) The chaos expansion, first introduced by Wiener [4], 2. Cooper, K., Jakobsson, E., Wolynes, P.: The theory of ion 2 transport through membrane channels. Prog. Biophys. Mol. is a representation of L .˝; F ; P/ through an orthogBiol. 46(1), 51–96 (1985) onal basis as

References

3. Eisenberg, B.: Computing the field in proteins and channels. J. Membr. Biol. 150, 1–25 (1996) 4. Eisenberg, B., Liu, W.: Poisson-Nernst-Planck systems for ion channels with permanent charges. SIAM J. Math. Anal. 38(6), 1932–1966 (2007) 5. Jerome, J.W.: Analysis of Charge Transport: A Mathematical Study of Semiconductor Devices. Springer, Berlin/New York (1996) 6. Kurnikova, M.G., Coalson, R.D., Graf, P., Nitzan, a.: A lattice relaxation algorithm for three-dimensional PoissonNernst-Planck theory with application to ion transport through the gramicidin A channel. Biophys. J. 76(2), 642–56 (1999) 7. Levie, R.d., Seidah, N.G.: Transport of ions of one kind through thin membranes. III. Current-voltage curves for membrane-soluble ions. J. Membr. Biol. 16, 1–16 (1974) 8. Lu, B.Z., Zhou, Y.C.: Poisson-Nernst-Planck equations for simulating biomolecular diffusion-reaction processes II: Size effects on ionic distributions and diffusion-reaction rates. Biophys. J. 100(10), 2475–2485 (2011) 9. Lu, B.Z., Holst, M.J., McCammon, J.A., Zhou, Y.C.: Poisson-Nernst-Planck equations for simulating biomolecular diffusion-reaction processes I: Finite element solutions. J. Comput. Phys. 229(19), 6979–6994 (2010) 10. Nadler, B., Schuss, Z., Singer, A., Eisenberg, R.S.: Ionic diffusion through confined geometries: from Langevin equations to partial differential equations. J. Phys. Condens. Matter 16(22), S2153–S2165 (2004) 11. Nernst, W.: Die elektromotorische wirksamkeit der ionen. Z Physik Chem. 4, 129 (1889) 12. Planck, M.: u¨ ber die erregung von electricit¨at und w¨arme in electrolyten. Ann. Phys. Chem. 39, 161 (1890) 13. Roux, B., Allen, T., Bernche, S., Im, W.: Theoretical and computational models of biological ion channels. Q. Rev. Biophys. 37(1), 15–103 (2004)

u./ D

1 X

uO k ˚k ./;

kD0

EŒ˚i ./˚j ./ D i ıij ;

i D EŒ˚i ./2 ;

i.e., the basis is orthogonal under the measure associated with the random variables. Once the expansion is known, the orthogonality of the chaos polynomial allows for the computation of the moments by directly manipulating the expansion coefficients. The first two moments are recovered as EŒu D uO 0 ; Var.u/ D EŒ.u  EŒu/2  D

1 X

k uO k :

kD1

Other moments can be recovered in a similar fashion. In practice, one is often concerned with the truncated expansion, M X kD1

Jan S. Hesthaven1 and Dongbin Xiu2 1 Division of Applied Mathematics, Brown University, Providence, RI, USA 2 Department of Mathematics and Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, UT, USA

EŒu./˚k ./ : EŒ˚k2 ./

Cameron and Martin [1] established that any variable u./ 2 L2 .˝; F ; P/ with a finite variance can be expressed in a L2 -convergent series. Here ˚k ./ is a multivariate chaos polynomial with the property that

uM ./ D

Polynomial Chaos Expansions

uO k D

uO k ˚k ./;

M D

.n C d /Š ; nŠd Š

where n reflects the truncation order of the polynomial expansion and M represents the total number of expansion terms. Convergence of the expansion can be established by using results from the classical approximation theory, especially those of spectral expansions [3] that are generally of the form ku  uM ./k2  C nm ku.m/ k2 ;

Consider a function, u./ of d identically distributed where kk2 is the weighted 2-norm. Whenever the func(iid) random variables,  2  Rd where i tion is smooth (with m being large), the convergence L2 .˝; F ; P/. Here ˝ is the event space, F is the rate is fast and achieves exponential rate when the

Polynomial Chaos Expansions

1163

Polynomial Chaos Expansions, Table 1 Wiener-Askey polynomials and associated random variables in polynomial chaos

Continuous distribution

Discrete distribution

Random variable 

Wiener-Askey basis

Support

Gaussian Gamma Beta Uniform Poisson Binomial Negative binomial Hypergeometric

Hermite-chaos Laguerre-chaos Jacobi-chaos Legendre-chaos Charlier-chaos Krawtchouk-chaos Meixner-chaos Hahn-chaos

R RC [a,b] [a,b] f0; 1; 2; : : :g f0; 1; 2; : : : ; N g f0; 1; 2; : : :g f0; 1; 2; : : : ; N g

function is analytic. On the other hand, if one measures many effective algorithms for practical simulations the convergence with respect to the total number of under uncertainty [2]. expansion terms M , the rate becomes less appealing. At high dimensions d 1, M grows rapidly. By approximating M ' nd =d Š, one recovers Generalized Polynomial Chaos ku  uM ./k2  CQ M=d ku.m/ k2 ; indicating a reduction in convergence rate with increasing d or, alternatively, a need for increased smoothness to maintain the rate of convergence with increasing dimension. This deterioration of the effective convergence rate at high dimensions is known as the curse of dimensionality.

Homogeneous Wiener Chaos Under the assumption of Gaussian random variables, Wiener [4] introduced the homogeneous chaos and the associated chaos polynomials, also known as Hermite polynomials ˚k ./ D Hk ./ D e 2   .1/k 1

@k 1 e  2   : @1 ; : : : ; @d

The connection between the orthogonal basis and random variable is made clear by recognizing that Z EŒHi ./Hj ./D

Hi ./Hj ./  1   e 2 d  D i Šıij; .2/d=2 R d

i.e., the Hermite polynomials are orthogonal under the Gaussian measure. The idea of using Hermite polynomials was adopted and has helped to produce

Realizing the close connection between the probability distribution of the random variable and the associated chaos orthogonal polynomial, (generalized) polynomial chaos has been introduced as a generalization of the original homogeneous polynomial chaos. This close connection between random variables and classic orthogonal polynomials has been named the Wiener-Askey scheme, because most of the orthogonal polynomials were chosen from the Askey family. However, the orthogonal basis can employ any suitable polynomials; see [5, 6] for a detailed discussion of this. In this generalized case, the weight under which classic polynomials are orthogonal reflects the nature of the random variables. Examples of some of the wellknown correspondences are listed in Table 1.

References 1. Cameron, R., Martin, W.T.: The orthogonal development of nonlinear functionals in series of Fourier-Hermite functionals. Ann. Math. 48, 385–392 (1947) 2. Ghanem, R.G., Spanos, P.: Stochastic Finite Elements: A Spectral Approach. Springer, New York (1991) 3. Hesthaven, J.S., Gottlieb, S., Gottlieb, D.: Spectral Methods for Time-Dependent Problems. In: Cambridge Monographs on Applied and Computational Mathematics, vol. 21. Cambridge University Press, Cambrigde (2007) 4. Wiener, N.: The homogeneous chaos. Am. J. Math. 60, 897–936 (1938) 5. Xiu, D.: Numerical Methods for Stochastic Computations: A Spectral Method Approach. Princeton University Press, Princeton (2010)

P

1164

Post-Hartree-Fock Methods and Excited States Modeling

6. Xiu, D., Karniadakis, G.: The Wiener-Askey polynomial chaos for stochastic differential equations. SIAM J. Sci. Comput. 24(2), 619–644 (2002)

Post-Hartree-Fock Methods and Excited States Modeling Mathieu Lewin CNRS and D´epartement de Math´ematiques, Universit´e de Cergy-Pontoise/Saint-Martin, Cergy-Pontoise, France

( H WD ' W R3  f"; #g ! C; h'; 'iH

Short Definition

WD

The Hartree-Fock method is one of the most famous theories to approximate the fermionic ground state of a many-body Schr¨odinger operator. The expression “Post-Hartree-Fock” refers to techniques which aim at improving the Hartree-Fock ground state, or at calculating excited states.

j D1

hrj C

X 1k > %k  %k > = a1 D c1 b0 C c0 b1 : .n/ .n/ .n/ :: > > Then, the rational function Rk .t/ D Ak .t/=Bk .t/ > ; .n/ a D c b C c b C    C c b ; p p 0 p1 1 pq q satisfies the interpolation conditions Rk .i / D f .i / for i D n; : : : ; n C k, where, for k D 1; 2; : : :, and with the convention that ci D 0 for i < 0, then it holds n D 0; 1; : : :, .n/

.n/

.n/

.n/

.n/

.n/

.n/

.n/

.n/

.n/

Ak .t/ D .%k  %k2 /Ak1 .t/  .t  k1 /Ak2 .t/ Bk .t/ D .%k  %k2 /Bk1 .t/  .t  k1 /Bk2 .t/ with, for n D 0; 1; : : :, .n/

A1 D 1;

.n/

.n/

A0 D %0

f .t/  R.t/ D O.t pC1 /: Such a rational function is called a Pad´e-type approximant of f , and it is denoted by .p=q/f . Replacing a0 ; : : : ; ap by their expressions in Np , we have Np .t/ D b0 fp .t/ C b1 tfp1 .t/ C    C bq t q fpq .t/;

R

1224

Rational Approximation

Nk .i /  fi Dk .i / D 0;

with fn .t/ D c0 C c1 t C    C cn t n ;

n D 0; 1; : : :

Let us now choose the denominator in order to improve the order of approximation as much as possible. If the coefficients bi satisfy

i D 1; : : : ; k;

where 1 ; : : : ; k are distinct points in the complex plane (none of them being 0). We obtain the system .fk .i /  fi /b0 C i .fk1 .i /  fi /b1 C    C ik .f0 .i /  fi /bk D 0; i D 1; : : : ; k:

9 0 D cpC1 b0 C cp b1 C    C cpqC1 bq > = :: ; : > ; 0 D cpCq b0 C cpCq1 b1 C    C cp bq ;

Setting again b0 D 1, we obtain a system of k linear equations in the k unknowns b1 ; : : : ; bk . Such a rational function is called a Pad´e-type rational interpolant then, solving this system of linear equations after since it interpolates f at the points i and, in addition, kC1 /. setting b0 D 1 (since a rational function is defined up it satisfies f .t/  R.t/ D O.t Consider now the barycentric rational interpolant, to a multiplying factor), we obtain a rational function and let us determine w0 ; : : : ; wk such that R satisfying the approximation condition f .t/  R.t/ D O.t k /:

f .t/  R.t/ D O.t pCqC1 /:

Such a rational function is called a Pad´e approximant In that case, R is a Pad´e-type approximant .k=k/f of of f , it is denoted by Œp=qf , and it holds f , but with a lower order k of approximation instead of ˇq ˇ k C 1. The preceding approximation condition shows ˇt fpq .t / t q1 fpqC1 .t /    fp .t /ˇ that the wi ’s must be a solution of the linear system ˇ ˇ ˇ cpqC1 cpqC2    cpC1 ˇˇ ˇ ˇ :: ˇ :: :: 9 ˇ : ˇˇ : : k ˇ X > wi > > ˇ ˇ > .fi  c0 / D0 cpC1    cpCq cp > >  > i ˇ ˇ Œp=qf .t / D > iD0 > q1 ˇ tq ˇ >   z    1 k > X > ˇ ˇ fi wi c0 = ˇcpqC1 cpqC2    cpC1 ˇ   c1 D0 ˇ ˇ    i i i iD0 > ˇ :: :: ˇ : :: > > ˇ : >  : ˇˇ : > > ˇ ! > k > ˇ cp ˇ X > fi wi c0 c1 cpC1    cpCq > > iD0

There exists recursive formulae for computing any sequence of adjacent Pad´e approximants (i.e., whose degrees only differ by 1) [4]. Pad´e-type and Pad´e approximants have many interesting algebraic and approximation properties, in particular for the analytic continuation of functions outside the region of convergence of the series. See [1, 4, 6]. On new results about their computation, consult [9].

´ Pade-Type Rational and Barycentric Interpolation Let us now consider the Pad´e-type approximant R  .k=k/f and determine b0 ; : : : ; bk such that R.i / D f .i /.DW fi / for i D 1; : : : ; k, that is, such that

ik1



ik1



ik2

     ck1

i

D 0: > ;

Setting w0 D 1, we obtain a system of k equations in the k unknowns w1 ; : : : ; wk . Such a rational function is called a Pad´e-type barycentric interpolant. These procedures can be extended to any degrees in the numerator and in the denominator; see [5].

References 1. Baker, G.A. Jr., Graves–Morris, P.R.: Pad´e Approximants, 2nd edn. Cambridge University Press, Cambridge (1996) 2. Berrut, J.-P.: Rational functions for guaranteed and experimentally well-conditioned global interpolation. Comput. Math. Appl. 15, 1–16 (1988) 3. Berrut, J.-P., Baltensperger, R., Mittelmann, H.D.: Recent developments in barycentric rational interpolation. In: de Bruin, M.G., Mache, D.H., Szabados, J. (eds.) Trends and Applications in Constructive Approximation. International Series

Regression

4. 5.

6.

7.

8.

9.

of Numerical Mathematics, vol. 151, pp. 27–51. Birkh¨auser, Basel (2005) Brezinski, C.: Pad´e–Type Approximation and General Orthogonal Polynomials. Birkh¨auser, Basel (1980) Brezinski, C., Redivo–Zaglia, M.: Pad´e–type rational and barycentric interpolation. Numer. Math., to appear, doi:10.1007/s00211-013-0535-7 Brezinski, C., Van Iseghem, J.: Pad´e approximations. In: Ciarlet, P.G., Lions, J.L. (eds.) Handbook of Numerical Analysis, vol. III, pp. 47–222. North–Holland, Amsterdam (1994) Brezinski, C., Van Iseghem, J.: A taste of Pad´e approximation. In: Iserles, A. (ed.) Acta Numerica 1995, pp. 53–103. Cambridge University Press, Cambridge (1995) Floater, M.S., Hormann, K.: Barycentric rational interpolation with no poles and high rates of approximation. Numer. Math. 107, 315–331 (2007) Gonnet, P., G¨uttel, S., Trefethen, L.N.: Robust Pad´e approximation via SVD. SIAM Rev. (to appear)

Regression

1225

additive models, regression methods for repeated measurements, and regression methods for highdimensional data, to mention some. Most regression models are fitted to data with the purpose of either (1) using the fitted model to predict values of y for new observations of x or (2) understanding the relationships between y and explanatory variables x, determining their significance, and possibly selecting the best subset of explanatory variables to explain the response y. One important objective here is also to study the effect of one explanatory variable while adjusting for the effects of the other explanatory variables. As opposed to deterministic curve fitting, statistical regression allows to quantify uncertainty in estimated model coefficients and thereby provides standard errors and confidence intervals both for the fitted model and for predictions of new observations.

The Multiple Linear Regression Model

Ingrid Kristine Glad and Kukatharmini Tharmaratnam Department of Mathematics, University of Oslo, Oslo, Norway

In multiple linear regression, the continuous response variable y is a scalar. Other terms used for y are dependent variable, regressand, measured variable, explained variable, and outcome or output variable. If y is a vector of responses rather than a scalar for each subject, Synonyms we will have multivariate linear regression, which we present separately later. Linear models; Regression analysis The explanatory variables x1 , x2 ; : : : ; xp are also called independent variables, regressors, covariates, predictors, or input variables. These are for simplicity Short Definition of presentation considered fixed quantities (not random variables). The explanatory variables might be numeriRegression is a statistical approach for estimating the cal, binary, or categorical. The case p D 1 is known as relationships among variables. simple linear regression. Multiple linear model For each of n subjects, we observe a set of variables .yi ; x1i ; x2i ; : : : ; xpi /, i D 1; : : : ; n. Assuming a linear relationship between reRegression is a statistical approach for modelling the sponse and covariates, we have the multiple linear relationship between a response variable y and one regression model: or several explanatory variables x. Various types of regression methods are extensively applied for the yi D ˇ0 C ˇ1 x1i C ˇ2 x2i C    C ˇp xpi C i analysis of data from literarily all fields of quantitative research. For example, multiple linear regression, i D 1; : : : ; n (1) logistic regression, and Cox proportional hazards models have been the main basic statistical tools where ˇ0 is the intercept and the parameters ˇ1 ; : : : ; ˇp in medical research for decades. In the last 20– are the regression coefficients or effects. The i is 30 years, the regression toolbox has been supplied with an error/noise/disturbance term which captures rannumerous extensions, like, for example, generalized dom deviations from the linear relations. Standard

Introduction

R

1226

assumptions are that the stochastic variables i , i D 1; : : : ; n are statistically independent, normally distributed with mean 0 and constant variance  2 . It follows from the model in Eq. (1) that it is the expected value of y conditional on the covariates, E.yjx1 ; x2 ; : : : ; xp /, which is a linear function. Each coefficient ˇj ; j D 1; : : : ; p, is the change in E.yjx1 ; x2 ; : : : ; xp / when covariate xj changes one unit and all other covariates are held fixed. Inference

Regression

fit =lm(y˜factor(x1)+factor(x2)+... +factor(xp)). The vector of predicted values is yO D XˇO D  1 T  T 1 T X y. The matrix H D X XT X X is X X X called the hat-matrix. We find the n-vector of residuals as y  yO D .I  H/y. The residuals are among others used for diagnostics of the aptness of the model; see below. In order to make inference for the regression model used, that is, assessing, for example, whether the various observed effects are significant or not, we need an estimate of the level of noise, the unknown variance  2 . Also here we use the residuals. The error sum of squares, SSE, is the sum of the squared residuals, SSE D .y  yO /T .y  yO /, and the mean squared error MSE D s 2 D SSE=.n  .p C 1// is an unbiased estimator of  2 .

Model fitting – estimation The unknown parameters in a regression model are estimated from data using the principle of maximum likelihood or other estimation methods. For the model in Eq. (1), with independent i N.0;  2 /, i D 1; : : : ; n, maximizing the likelihood of the data is equivalent to the method of least squares. Hence, the parameters ˇ0 ; ˇ1 ; : : : ; ˇp are Testing significance of effects R and other packages estimated as the minimizers of the objective function: will provide estimates ˇOj s with standard errors, that is, the square roots of estimated variances where s 2 n O Furthermore, standard output X substitutes  2 in Var.ˇ/. .yi  E.yi jx1i ; x2i ; : : : ; xpi //2 is a t-test-statistic and P-value for the test H0 W ˇj D 0 i D1 versus alternative Ha W ˇj ¤ 0, j D 0; 1; : : : ; p. n X A small P-value (typically < 0:05) is interpreted D .yi  ˇ0  ˇ1 x1i  ˇ2 x2i      ˇp xpi /2 as covariate xj having a significant effect on the i D1 outcome. The estimated size and sign of this effect (2) (when all other covariates are held fixed) is simply ˇOj . Confidence intervals for the effects can be conIn matrix form, define the n-vector of responses structed around point estimates using standard errors y D .y1 ; : : : ; yn /T , the n .p C1/-matrix of covariates and the t-distribution with .n  .p C 1// degrees of (called the design matrix) X D .1; x1 ; : : : ; xp /, and the freedom. .p C 1/-vector of coefficients ˇ D .ˇ0 ; ˇ1 ; : : : ; ˇp /T . Then, the objective function (2) reads .y  Xˇ/T .y  Confidence intervals and prediction intervals From Xˇ/, and if XT X has an inverse, the solution to the the uncertainty in the estimated coefficients, we can  1 T minimization problem is simply ˇO D XT X X y. find standard errors for the estimate of the mean reThe same solution appears in deterministic curve fitting sponse E.yjx1 ; x2 ; : : : ; xp / for a certain set of prebased on the least square principle. In the stochastic dictors x1 , x2 ; : : : ; xp and hence construct confidence case, ˇO is a normally distributed stochastic vector intervals. If a new set of predictors is given in new, usO D ˇ (unbiased estimator) and variance- ing new=data.frame (x=c(a1,a2,...ap)), a with E.ˇ/  O D XT X 1  2 . Estimates confidence interval for the corresponding mean recovariance matrix Var.ˇ/ are found by statistical software packages. In R, we use sponse E.yja1 ; a2 ; : : : ; ap / can be computed using fit=lm(y˜x1+x2+...+xp) summary(fit)

predict(fit,new,interval ="confidence")

If covariates are categorical, they can be represented Similarly, a prediction interval for a new future by a system of dummy variables. In R, this is response y in the same set of predictors new can taken care of automatically by specifying that these be found by specifying interval="prediction" covariates should be considered as factors, using instead. Note that the interval for a future response is

Regression

1227

wider than the interval for the mean response, as the Constancy of error variance (homoscedasticity) If noise variability comes into play. the model is correctly specified, then there should not be any systematic patterns in the residual plots Coefficient of multiple determination In addition above. If the error variance increases or decreases to the error sum of squares, statistical software will with a covariate, the plot will have a megaphone also calculate the total sum of squares SST D .y  shape. Plots of the absolute values of the residuals yN /T .y  yN / and the regression sum Pn of squares SSR D or of the squared residuals against covariates are .Oy  yN /T .Oy  yN /, where yN D i D1 yi =n. It can be also useful for diagnosing possible heteroscedasticity. shown that SST D SSE C SSR. Standard output Remedies include the use of transformations, of regression software is the coefficient of multiple weighted least squares, or a generalized linear determination R2 , defined as R2 D SSR=SST, which model. is interpreted as the proportion of the total variance in the response that can be explained by the multiple Normality of error terms Tests and confidence inregression model with p covariates. We aim for models tervals are based on the assumption of normal errors. with high R2 . However, as R2 increases when the The residuals can be checked for normality using a number of covariates increases, for model comparison, Q-Q plot (qqnorm(fit$residuals)),where the we need to adjust the measure for the model size in points should be somewhat close to a straight line to some way. In R, we find R2 as “Multiple R-squared,” be coherent with a normal distribution. For large n, while “Adjusted R-squared” is one way of adjusting for all probability statements will be approximately correct the dimension p. even with non-normal errors. Model utility test Furthermore, an F-test, based on the ratio between SSR and SSE, will be standard output of statistical computer packages, testing the hypothesis of a constant mean model, that is, the hypothesis that the explanatory variables are useless for predicting y. This test is usually named the model utility test. A small P-value here means that we reject the hypothesis of a constant mean; hence, at least one of the covariates has a significant effect on the outcome.

Independence of error terms We assume that the errors are uncorrelated, but for temporally or spatially related data, this may well not be true. Plotting the residuals against time or in some other type of sequence can indicate if there is correlation between error terms that are near each other in the sequence. When the error terms are correlated, a direct remedial measure is to turn to suitable models for correlated error terms, like time series models. A simple remedial transformation that is often helpful is to work with first differences [25].

Model Diagnostics and Possible Remedies The residuals e D y  yO are used to check model assumptions as linearity, normally distributed and independent errors, and constant variance.

Presence of outliers and influential observations An outlier is a point that does not fit the current model. An outlier test in Chapter 10 of [25] might be useful because it enables us to distinguish between truly unusual points and residuals which are large but not exceptional. An influential point is one whose removal from the dataset would cause a large change in the fit. The i ’th diagonal element of the hat-matrix H measures the influence of the i ’th observation on its predicted value and is called leverage. An influential point may or may not be an outlier and may not have large leverage but it will tend to have at least one of those two properties. If we find outliers or influential observations in the data, it can be recommended to use robust regression methods; see below.

R

Linearity of the regression function Linearity of the regression function can be studied from residual plots, where the residuals ei are plotted against predictor variables xji or fitted values yOi . The plots should display no systematic tendencies to be positive or negative for a linear regression model to be appropriate. We can use the R-code plot(xj,fit$residuals)to get such plots for xj . In case of nonlinearity, remedies include (log) transformations of response and/or predictors and inclusion of higher-order terms and/or interactions; see below.

1228

Regression

Harrell [15] and Kleinbaum [22] can be recom- in R to fit a model with main effects from x1 and x2 mended as introductory reading on general regression and their pairwise interaction. It is possible to include higher-order interactions as well. techniques and theory. Extensions

Variable selection When there are many possible covariates to include in a multiple linear regression model, it is often actual to do some variable selection, usually facing the trade-off between a parsimonious model and a good empirical fit and prediction power. Classical approaches are forward and backward selection, either starting with an empty model and including one by one covariate or starting with the full model and excluding one by one covariate. In either cases, an appropriate criterion for inclusion and/or exclusion has to be chosen, for instance, the adjusted R2 or alternative measures of model fit, like the Akaike information criterion (AIC), the Bayesian information criterion (BIC), or Mallows Cp; see, for example, Cherkassky and Ma [5] and Kadane and Lazar [21].

Robust regression methods If we have identified potential outliers in the dataset, the least squares estimation method may perform poorly. One approach is to simply eliminate the outliers and proceed with the least squares estimation method. This approach is appropriate when we are convinced that the outliers are truly incorrect observations, but to detect these outliers is not always easy as multiple outliers can mask each other. Sometimes, outliers are actual observations and removing these observations creates other false outliers. A general approach is then to use robust estimation methods which downweight the effect of long-tailed errors; see Maronna et al. [26] and Alma [1]. In R, the Huber estimation (M-estimation) method is the default choice of the rlm() function, which is the robust version of the lm(), to be found in Linear Mixed Effect Models the MASS package of Venables and Ripley [34]. Fox and Weisberg [11] have a separate chapter on robust Multivariate data, where the response y is multivariregression estimation in R. ate for each subject, has to be treated with special care because of the implicit dependencies present. In Nonlinear effects Note that the linear model in a multitude of data structures, such as multivariate Eq. (1) is linear in the coefficients ˇ. It is possible response, but also repeated measurements, longitudito include nonlinear effects of the covariates by nal data, clustered data, and spatially correlated data, introducing polynomials or fractions of these as new dependencies between the observations have to be covariates. The resulting model is still linear in ˇ and taken into account. A simple example of multivariate can be fitted in the same way as above. Expressions response is a study where systolic and diastolic blood like pressures are measured simultaneously for each patient (together with several explanatory variables). If the fit=lm(y˜x1+I(x1ˆ2)+I(x1ˆ(1/3)) diastolic blood pressure is measured for all members of +x2+...) a number of families, the responses will be clustered. are used to fit such types of models in R. See more If for each patient, diastolic blood pressure is recorded details about fractional polynomial regression in, for under several experimental conditions, we have a reexample, Royston and Sauerbrei [32]. peated measures study. In the case that diastolic blood pressure is measured repeatedly over time for each Interactions If the effect on y of one covariate de- subject, we have longitudinal data. In these sets of pends on the level of another covariate, we have an outcomes, the variance-covariance structure is usually interaction. Interactions can be modelled by including complicated. Linear mixed effects models use a mix terms xj  xk in the design matrix, and again use the of fixed effects (like in the standard linear model) linear formulation and estimation by least squares. The and random effects to account for this extra variation coefficients of covariates xj are named main effects, and could be used for analysis of these more complex while coefficients of products xj xk are called pairwise data structures [35]. Mixed models are also named interaction effects. For example, we use multilevel models, hierarchical models, or random cofit=lm(y˜x1+x2+x1*x2) efficient models in specific settings.

Regression

1229

The original R package for fitting mixed effects models is nlme. Bates [2] introduced an improved version in the package lme4; see also Faraway [9] for data examples. More recently, Bates [3] gives a more detailed implementation of the lme4 package. The linear mixed models can be extended to generalized linear mixed models to cover other situations such as logistic and Poisson regression; see below. For further reading on mixed models, see, for example, McCulloch et al. [29] and also Rabe-Hesketh and Skrondal [31] for practical applications based on the Stata software. See Fitzmaurice et al. [10] for further reading on longitudinal data analysis.

used for predicting the probability of the occurrence of an event, based on several covariates that may be numerical and/or categorical. To fit logistic regression models, one can use the glm() function with logit link and the binomial distribution. Also, other link functions like the probit can be used. For example, when y is a binary response and x1 ; x2 ; : : : ; xp are covariates, then the logistic model can be fitted using the following R-code:

Generalized Linear Models

Poisson regression Poisson regression is used when the outcome variable represents counts. Poisson regression assumes that the response variable y has a Poisson distribution and that the logarithm of its expected value can be modelled by a linear combination of covariates. A Poisson regression model is sometimes known as a log-linear model, especially when used to model contingency tables. One can fit the Poisson regression model in R using glm() with the logarithm as the (canonical) link function and the Poisson distribution:

The linear model in Eq. (1) is a special case of a general class of regression models which handles outcomes that are not necessarily continuous and associations between the outcome and covariates that are not necessarily linear. This is the class of generalized linear models (GLM), introduced by Nelder and Wedderburn [30]; see, for example, the books by McCullagh and Nelder [28] or Dobson and Barnett [8] for details. In the GLM formulation, E.yjx1 ; x2 ; : : : ; xp / D and some function g of is linear in the covariates, i.e., g. / D ˇ0 Cˇ1 x1 Cˇ2 x2 C  Cˇp xp . The function g is called the link function, and identity link g. / D gives back the linear model for continuous y. Other link functions can be used to handle binary, categorical, or ordinal outcomes, as well as count data; see below. Maximum likelihood estimation can be used to estimate the parameters of a GLM. Typically, we must resort to numerical optimization (Newton-Raphson). The optimization is equivalent to an iteratively reweighted least squares (IRWLS) estimation method [28]. To fit GLM models in R-software, we use the function glm() with a specification of the distribution of the response and a suitable link function.

fit.logit = glm(y˜x1+x2+...+xp, family = binomial(link="logit")). See Kleinbaum and Klein [23] and Collett [6] for further reading on logistic regression.

fit.poisson = glm(y˜x1+x2+...+xp, family = poisson()).

In the case of over-dispersion (when the residual deviance is much larger than the degrees of freedom), one may want to use the quasipoisson() function instead of poisson() function. See more details in Chapter 24 in Kleinbaum [22]. In addition to the most common binomial, Gaussian and Poisson GLMs, there are several other GLMs which are useful for particular types of data. The gamma and inverse Gaussian families are intended for continuous, skewed response variables. We can use dual GLMs for modelling both the mean and the dispersion of the response variable in some cases [9]. The quasi-GLM is useful for nonstandard response Logistic regression Logistic regression or logit re- variables where we are not able to specify the distribugression is used when the response variable is a cat- tion, but we can state the link function and the variance egorical variable. Usually, logistic regression refers to functions; more details can be found in Faraway [9]. the case when the response variable is binary. If the number of categories of the response is more than two, Cox regression Survival analysis covers a set of techthe model is called multinomial logistic regression, niques for modelling the time to an event. We mention and if the multiple categories are ordered, as ordered some of them here, even if the topic deserves a treatlogistic regression. Logistic regression is commonly ment in itself. A survival regression model relates the

R

1230

Regression

time that passes before some event occurs, to one or more covariates, usually through a hazard function. In a proportional hazards model, like the Cox model, the effect of a unit increase in one covariate is multiplicative with respect to the hazard rate. The Cox proportional hazards model [7] is based on the instantaneous hazard

The generalized version becomes E.yjx1 ; x2 ; : : : ; xp / D and g. / D ˇ0 C f1 .x1 / C f2 .x2 / C    C fp .xp /, where we need to specify a family of distributions and a link function depending on the type of response data, similar to the GLM setting. The smooth, univariate functions can be fitted parametrically or, for increased flexibility, nonparametrically. For nonparametric estimation, it is common to .tjx/ D 0 .t/ exp.ˇ1 x1 C    C ˇp xp / use local polynomials, splines, or wavelets, as briefly described below. To fit a GAM, a backfitting algorithm where 0 .t/ describes how the hazard changes over can be used; see Wood [36] for further reading. The time at baseline levels of covariates and the second gam() function in R uses backfitting and allows for term models how the hazard varies according to p both local polynomials and smoothing splines in the explanatory covariates. Data are typically of the format nonparametric estimation of the fj ./’s. time to event and status (1Devent occurred, 0Devent did not occur). A statusD0 indicates that the observation is right censored. Survival analysis is typically Nonparametric Regression carried out using functions from the survival package in R. For example, When we would like to fit fit.cox=coxph(Surv(y,status)˜ E.yjx1 ; x2 ; : : : ; xp / D f .x1 ; x2 ; : : : ; xp / x1+x2+...+xp)

fits the proportional hazards function on a set of predictor variables, maximizing the partial likelihood of the data (Cox regression). Hosmer et al. [19] and Kleinbaum and Klein [24] are recommended general introductory texts on survival analysis. Martinussen and Scheike [27] provide flexible models with R examples.

Generalized Additive Models Generalized additive models (GAM), introduced by Hastie and Tibshirani [16], are a class of highly general and flexible models handling various types of responses and multiple covariates without assuming linear associations. Using an additive model, the marginal relationships between the predictor variable and the response variable are modelled with interpretable univariate functions. For the simple case with unit link, we have E.yjx1 ; x2 ; : : : ; xp / D ˇ0 C f1 .x1 / C f2 .x2 / C    C fp .xp /; where fj ./, j D 1; : : : ; p are unknown smooth (univariate) functions.

without assuming anything about the shape of the function, other than some degree of smoothness and continuity, we can use nonparametric regression methods. If we would have information about the appropriate parametric family of functions and the model is correctly specified, parametric approaches would be more efficient than nonparametric estimation. Very often, we do not have any information about an appropriate form and it is preferable to let the data determine the shape of f ./. When p > 1, these methods allow to model also interactions between covariates nonparametrically, but in practice, this works only for very low-dimensional problems, allowing p D 2; 3; maximum 4 in some cases. When the number of covariates is higher than this, it is necessary to resort to, for example, additivity assumptions and GAM as above. In such cases, the nonparametric methods are used to estimate the univariate (or low-dimensional) functions fj ./ which added together model the joint effect of the covariates on the response. There are several widely used nonparametric regression approaches, such as kernel estimators and local polynomials, splines, and wavelets. The simplest kernel estimator can be applied using the ksmooth() function in the base package in R and local polynomials using loess() in the stats package. ksmooth() works only for univariate regression, while loess()

Regression

1231

allows up to 4 covariates. The smoothing splines can be estimated using the smooth.spline() function in the stats package in R. The bs() function in the splines package can be used to generate appropriate regression spline bases. The spline solutions are available for univariate and bivariate functions. Wavelet fitting can be implemented using the wavethresh package in R and the function wd() can be used to make the wavelet decomposition. For more details, see Faraway [9].

Regression in (Ultra) High Dimensions (p > n) With high dimensions (or so-called p > n situations) in regression, we refer to the increasingly actual situations in which the number of covariates p is much larger than the sample size n in the various regression models above. With recent technological developments, it has become easy to simultaneously measure ten thousands, or millions, of covariates, on a smaller set of individuals. A typical example is genomics, where, for instance, gene expressions are easily measured for 30,000 genes for a few hundred patients. For the linear regression model Eq. (1), the ordinary least squares estimator will not be uniquely defined when p > n. To handle this problem, one might regularize the optimization by including a penalty in the objective function in (2). Such penalized regression methods shrink the regression coefficients towards zero, introducing some bias to reduce variability. Shrinkage is done by imposing a size constraint on the parameters, equivalent to adding a penalty to the sum of squares, giving 0 O ˇ./ D arg minˇ @jj y  X ˇ jj22 =n C

p X

1 J .jˇj j/A :

j D1

(3) The penalty term J .jˇj j/ depends on a tuning parameter  which controls the amount of shrinkage and can take on various forms, typically involving jˇj jr and some proper value of r distinguishing different methods. Among the most famous are ridge regression [18] with r D 2 and the lasso [33], where r D 1 (also named L1 -penalty). The lasso and its many variants are especially popular because they do not only shrink the coefficients, but put most of them to exactly zero, thus

performing variable selection. This corresponds to an assumption of sparsity, that is, that only some of the covariates are really explaining the outcome. There are several variants with different penalties available in the literature, for example, the adaptive lasso of Huang et al. [20] and the elastic net of Zou and Hastie [38]. Another important extension is the group lasso, where predefined groups of variables are selected together, Yuan and Lin [37]. In applications, the penalty parameter  is most often chosen through k-fold cross-validation which involves minimizing an estimate of the prediction error. Typical choices of k are 5 and 10. The ridge linear regression model can be fitted using the lm.ridge() function in the MASS package in R. The glmnet package fits lasso and elastic net model paths for normal, logistic, Poisson, Cox, and multinomial regression models; see details in Friedman et al. [12]. The glmnet() function is used to fit the model and the cv.glmnet() function is used to do the k-fold cross-validation and returns an optimal value for the penalty parameter . The grplasso() function in the grplasso package in R is used to fit a linear regression model and/or generalized linear regression model with a group lasso penalty. One should specify the model argument inside the function grplasso() as model=LinReg(), model=LogReg(), model=PoissReg() for linear, logistic, and Poisson regression models, respectively. See B¨uhlmann and van de Geer [4] for further reading on methods, theory, and applications for high-dimensional data and penalization methods in particular. Another approach to the dimensionality problem is to use methods like principal components regression (PCR) or partial least squares (PLS). These methods derive a small number of linear combinations of the original explanatory variables and use these as covariates instead of the original variables. This may be very useful for prediction purposes, but models are often difficult to interpret [17].

Bayesian Regression The Bayesian paradigm assumes that all unknowns are random variables, for which information can be expressed a priori, before the actual data are analyzed,

R

1232

with the help of a prior distribution. In the case of regression, the coefficients ˇ are random variables with prior distributions. The posterior distribution of coefficients given the data is used for inference. Summaries of the posterior distribution, including point estimates like the posterior mean, the posterior mode, or the posterior median of the coefficients, are optimal point estimates with respect to appropriate loss functions. Credibility intervals, which cover say 95 % of the posterior probability of the coefficients, describe the a posteriori uncertainty of the estimate. Priors might be conjugate distributions, leading to analytical expressions of the posterior distributions and allowing for explicit posterior estimation, but in practice, we usually have to resort to numerical methods (like Markov chain Monte Carlo) to obtain samples, point estimates, or marginals from the posterior distribution. The penalized regression models like ridge regression and lasso above can be interpreted as Bayesian linear regression models with specific prior distributions (Gaussian and Laplace, respectively) and focus on the posterior mode. Joint credibility intervals, for all parameters of a multiple regression, can be computed. There are several packages in R implementing Bayesian regression. To conduct Bayesian linear models and GLM, use the package arm which contains the bayesglm() function; see Gelman et al. [13] for details. For multivariate response data, the MCMCglmm package can be used to do generalized linear mixed models using Markov chain Monte Carlo techniques, described in Hadfield [14].

References 1. Alma, O.: Comparison of robust regression methods in linear regression. Int. J. Contemp. Math. Sci. 6(9), 409–421 (2011) 2. Bates, D.: Fitting linear mixed models in R. R News 5(1), 27–30 (2005) 3. Bates D, Maechler M, Bolker B and Walker S. lme4: Linear mixed-effects models using Eigen and S4. R package version 1.1-7, ArXiv e-print; submitted to Journal of Statistical Software, (2014) http://CRAN.R-project.org/package=lme4 4. B¨uhlmann, P., van de Geer, S.: Statistics for HighDimensional Data: Methods, Theory and Applications. Springer Series in Statistics. Springer, Heidelberg/Berlin (2011) 5. Cherkassky, V., Ma, Y.: Comparison of model selection for regression. Neural Comput. 15, 1691–1714 (2003) 6. Collett, D.: Modelling Binary Data, 2nd edn. Chapman & Hall/CRC Texts in Statistical Science. Taylor & Francis, Boca Raton, Florida, USA (2002)

Regression 7. Cox, D.R.: Regression models and life-tables. J. R. Stat. Soc. Ser. B (Methodol.) 34(2), 187–220 (1972) 8. Dobson, A., Barnett, A.: An Introduction to Generalized Linear Models, 3rd edn. Chapman and Hall/CRC, Boca Raton, Florida, USA (2008) 9. Faraway, J.J.: Extending the Linear Model with R: Generalized Linear, Mixed Effects and Nonparametric Regression Models. Chapman & Hall/CRC, New York (2006) 10. Fitzmaurice, G., Laird, N., Ware, J.: Applied Longitudinal Analysis, 2nd edn. Wiley Series in Probability and Statistics. John Wiley & Sons, Inc., Hoboken, New Jersey, Wiley, New York (2011) 11. Fox, J., Weisberg, S.: An R Companion to Applied Regression, 2nd edn. Sage, Thousand Oaks (2011) 12. Friedman, J., Hastie, T., Tibshirani, R.: Regularization paths for generalized linear models via coordinate descent. J. Stat. Softw. 33(1), 1–22 (2010) 13. Gelman, A., Su, Y., Yajima, M., Hill, J., Pittau, M.G., Kerman, J., Zheng, T.: Arm: data analysis using regression and multilevel/hierarchical models. R package version 1.3-02, new version 1.6-10 2013 (2010) 14. Hadfield, J.D.: Mcmc methods for multi-response generalised linear mixed models: the mcmcglmm r package. J. Stat. Softw. 33, 1–22 (2010) 15. Harrell, F.E.: Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. Springer Series in Statistics. Springer, New York (2001) 16. Hastie, T., Tibshirani, R.: Generalized Additive Models. Monographs on Statistics and Applied Probability Series. Chapman & Hall/CRC, Boca Raton, London (1990) 17. Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Series in Statistics. Springer, New York (2009) 18. Hoerl, A., Kennard, R.: Ridge regression: biased estimation for nonorthogonal problems. Technometrics 12, 55–67 (1970) 19. Hosmer, D.W.J., Lemeshow, S., May, S.: Applied Survival Analysis: Regression Modeling of Time to Event Data. Wiley Series in Probability and Statistics. Wiley, Hoboken, New Jersey (2011) 20. Huang, J., Ma, S., Zhang, C.H.: Adaptive lasso for sparse high-dimensional regression models. Stat. Sin. 18, 1603–1618 (2008) 21. Kadane, J., Lazar, N.: Methods and criteria for model selection, review article. J. Am. Stat. Assoc. 99(465), 279–290 (2004) 22. Kleinbaum, D.G.: Applied Regression Analysis and Multivariable Methods. Duxbury Applied Series. Thomson Brooks/Cole Publishing, Belmont, California, USA (2007) 23. Kleinbaum, D., Klein, M.: Logistic Regression: A Selflearning Text, 3rd edn. Springer, New York (2010) 24. Kleinbaum, D., Klein, M.: Survival Analysis. Statistics for Biology and Health. Springer, New York (2012) 25. Kutner, M., Nachtcheim, C., Neter, J., Li, W.: Applied Linear Statistical Models, 5th edn. McGraw-Hill Companies, Boston (2005) 26. Maronna, R.A., Martin, R.D., Yohai, V.J.: Robust Statistics: Theory and Methods. Wiley, Chichester, England (2006) 27. Martinussen, T., Scheike, T.: Dynamic Regression Models for Survival Data. Springer, New York (2006)

Regularization of Inverse Problems 28. McCullagh, P., Nelder, J.: Generalized Linear Models, 2nd edn. Chapman & Hall, London (1989) 29. McCulloch, C., Searle, S., Neuhaus, J.: Generalized, Linear and Mixed Models, 2nd edn. Wiley Series in Probability and Statistics. Wiley, New York (2008) 30. Nelder, J., Wedderburn, R. Generalized linear models. J. R. Stat. Soc.: Ser. A 132, 370–384 (1972) 31. Rabe-Hesketh, S., Skrondal, A.: Multilevel and Longitudinal Modeling Using Stata, 3rd edn. Stata Press, College Station (2012) 32. Royston, P., Sauerbrei, W.: Multivariable Model – Building: A Pragmatic Approach to Regression Analysis Based on Fractional Polynomials for Modelling Continuous Variables. Wiley Series in Probability and Statistics. Wiley, Chichester, England (2008) 33. Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. 58, 267–288 (1996) 34. Venables, W., Ripley, B.: Modern Applied Statistics with S, 4th edn. Springer, New York (2002) 35. Verbeke, G., Molenberghs, G.: Linear, Mixed Models for Longitudinal Data. Springer Series in Statistics. Springer, New York (2000) 36. Wood, S.: Generalized Additive Models: An Introduction with R. Texts in Statistical Science Series. Chapman and Hall/CRC, Boca Raton, Florida, USA (2006) 37. Yuan, M., Lin, Y.: Model selection and estimation in regression with grouped variables. J. R. Stat. Soc.: Ser. B 68(1), 49–67 (2006) 38. Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. R. Stat. Soc.: Ser. B (Stat. Methodol.) 67(2), 301–320 (2005)

Regularization of Inverse Problems Heinz W. Engl1 and Ronny Ramlau2 1 Johann Radon Institute for Computational and Applied Mathematics (RICAM), Austrian Academy of Sciences, Linz, Austria 2 Institute for Industrial Mathematics, Kepler University Linz, Linz, Austria

Introduction Inverse problems aim at the determination of a cause x from observations y. Let the mathematical model F W X ! Y describe the connection between the cause x and the observation y. The computation of y 2 Y from x 2 X forms the direct problem. Often the operator F is not directly accessible but given, e.g., via the solution of a differential equation. The inverse problem is to find a solution of the equation

1233

F .x/ D y :

(1)

from given observations. As the data is usually measured, only noisy data with ky  y ı k  ı

(2)

are available. Such problems appear naturally in all kinds of applications. Usually, inverse problems do not fulfill Hadamard’s definition of well-posedness: Definition 1 The problem (1) is well posed, if 1. For all admissible data y exists an x with (1), 2. the solution x is uniquely determined by the data, 3. the solution depends continuously on the data. If one of the above conditions is violated, the problem (1) is ill posed. If F is a linear compact operator with an infinitedimensional range, e.g., a linear integral equation of the first kind, then the problem (1) is ill posed [33, 39, 58]. An important application is the inversion of the Radon transform which models computerized tomography (CT): see, e.g., [41]. Numerically, the biggest problem is a violation of Condition 3 of Definition 1, as numerical algorithms for the inversion will be unstable. This problem can be overcome by regularization methods.

Regularization of Linear Inverse Problems in Hilbert Spaces We treat first linear operator equations Ax D y where A W X ! Y and X; Y are Hilbert spaces. If the range of A, R.A/, is not the whole space Y , then the operator equation Ax D y does not necessarily admit a solution. This can partially be corrected by the concept of best approximate solutions: Definition 2 Let

 U D u 2 X W u D arg min ky  Axk : x2X

The best approximate solution x  2 U is defined as the (unique) element from U with the smallest norm. The generalized inverse is defined as the operator

R

1234

Regularization of Inverse Problems

A assigning to each y 2 R.A/ ˚ R.A/? the best best possible convergence  2  rate under the above source approximate solution x  . condition is of O ı 2C1 ; therefore, a regularization Note that if R.A/ is not closed, e.g., if A is method is called order optimal if for a given parameter ı compact with infinite-dimensional range, then A is not choice rule ˛ D ˛.ı; y / the estimate defined on the whole space Y and is unbounded. In  2  ı kx   x˛.ı;y (4) k D O ı 2C1 order to control the unboundedness of the generalized ı/ inverse, one has to introduce regularization methods. The main idea is to replace the unbounded operator holds for ky ı  yk  ı and A by a family of continuous operators that converge pointwise. x  2 X; WD fx 2 X j x D .A A/ w; and kwk  g (5) Definition 3 A regularization of an operator A is a For convergence rates w.r.t. generalized source confamily of continuous operators .R˛ /˛>0 , R˛ W Y ! X with the following properties: there exists a map ˛ D ditions, we refer to [24, 38]. ˛.ı; y ı / such that for all y 2 D.A / and all y ı 2 Y Filter-Based Regularization Methods with ky  y ı k  ı, ˚ In Hilbert spaces, compact operators A can be decomlim sup kR˛.ı;y ı / y ı  A yk j y ı 2 Y; ky  y ı k  ı posed via their singular system fi ; ui ; vi gi 2N as ı!0

D0 and ˚ lim sup ˛.ı; y ı / j y ı 2 Y; ky  y ı k  ı D 0 :

ı!0

Ax D

1 X

i hx; ui ivi

i D1

with i  0. For y 2 D.A /, the best approximate solution is then given as

The parameter ˛ is called regularization parameter. 1 X   Regularization methods are often defined by a x DA yD i1 hy; vi iui : modification of the (also unbounded) operator A A, i D1 which is due to the fact that the best approximate solution x  can be computed for y 2 D.A / by solving As i ! 0 for i ! 1, the unbounded growth of fi1 g the normal equation has to be compensated by a sufficiently fast decay of the coefficients fhy; vi ig of y, which is another   A Ax D A y (3) manifestation of a source condition. This is usually not in R.A /. The regularization theory is mainly con- the case for noisy data y ı , which causes the instabilities cerned with the analysis of regularization methods, in the reconstruction. Using filter functions F˛ W RC ! R, regularization including their definition, the derivation of suitable parameter choice rules, and convergence analysis. For methods can be defined by a given parameter choice rule, the convergence analX ysis in particular aims at the estimation of the error R˛ y ı WD i1 F˛ .i /hy ı ; vi iui : ı  kR˛.ı/ y  x k. There cannot be a uniform converi 2N gence rate for the regularization error if R.A/ is nonclosed, cf. Schock [57], i.e., any regularization method In order to ensure convergence and convergence rates, can converge arbitrarily slow. In order to obtain a the filter function F˛ has to satisfy certain conditions, convergence rate, a source condition is needed. Here, cf. [16, 36]. Well-known regularization methods generwe restrict ourselves to H¨older-type source conditions, ated by filter functions are: where the solution of the equation can be written 1. Truncated singular value decomposition: as x  D .A A/ w, i.e., x  2 range.A A/ X R˛ y ı WD i1 hy ı ; vi iui D.A/;  > 0. This condition can be understood as an i ˛ abstract smoothness condition. It can be shown that the

Regularization of Inverse Problems

In this case, the filter function is given by

F˛ ./ WD

1 if   ˛ : 0 if  < ˛

1235

lim ˛.ı/ D 0;

ı!0

ı2 D 0: ı!0 ˛.ı/ lim

Morozov’s discrepancy principle where ˛ .ı; y ı / is chosen s.t. ky ı  Ax˛ı  k D ı for fixed  > 1   2 2. Landweber method: For ˇ 2 0; kKk2 and m 2 N, is an a posteriori parameter choice rule. Tikhonov regularization together with the discrepancy principle set is an order optimal regularization method for F1=m ./ D 1  .1  ˇ2 /m : 0 <   1=2. The regularization parameter ˛ D 1=m admits The discrepancy principle can also be used for the discrete values only. The regularized solution due to Landweber method: If the iteration is stopped after m ı the Landweber method, x1=m , can be characterized iterations, where m is the first index with by the mth iterate of the Landweber iteration, xmC1 D xm C ˇA .y ı  Axm /;

ky ı  Axm k  ı < ky ı  Axm1 k

(8)

with 0 < ˇ < 2=kAk2 ; x0 D 0:

then the iteration is an order optimal regularization method for all  > 0. For results concerning different parameter choice The regularization parameter is the reciprocal of the rules, convergence, and convergence rates of related stopping index of the iteration. 3. Tikhonov regularization: Here, the filter function is methods, we refer to [16,36] and the references quoted there. given by 2 : F˛ ./ D 2  C˛ Further Methods for Linear Problems Within this section we will describe regularization The regularized solution due to Tikhonov, methods for linear operators that are not readily charX 2 acterized via filter functions. i x˛ı WD  i1 hy ı ; vi uii ; 2 i C ˛ i Projection Methods is also the unique minimizer of the Tikhonov functional A natural approach of approximating a solution of a linear operator equation in an infinite-dimensional space X is by projection of the operator equation (6) J˛ .x/ D ky ı  Axk2 C ˛kxk2 ; to a finite-dimensional subspace Xn and computing its least-squares approximation in Xn . Given a which is in turn minimized by the unique solution of sequence of finite-dimensional subspaces of Xn the equation with Xn  XnC1 .A A C ˛I /x D A y ı : (7) for all n 2 N, the least-squares approximation of Ax D y in Xn is given as We observe the close connection of (7) to the normal Eq. (3). xn D An y Equipped with a proper parameter choice rule, the above-presented methods regularize ill-posed problems. We distinguish between a priori parameter choice with An D APn , where Pn denotes the orthonormal  rules, where ˛ depends on the noise level ı only, and a projection onto Xn . As An has a finite-dimensional posteriori parameter choice rules, where the regulariza- range, it is bounded and xn is therefore a stable aption parameter depends additionally on the noisy data proximation of x  . The convergence xn ! x  can only y ı . For example, Tikhonov regularization converges be guaranteed under additional conditions, e.g., by the with an a priori parameter choice rule fulfilling condition

R

1236

Regularization of Inverse Problems

  lim sup k An xn k < 1

then the method is order optimal for x  2 X; and all n!1  > 0 [42]. see [37]. For a deeper analysis of cg and related methods in A method that always converges [40] is the dual the context of inverse problems, we refer to [20]. least-squares method, where a sequence of finitedimensional subspaces Mollifier Methods

of R.A/ is chosen. Now xn is defined as the leastsquares solution of the equation

Regularization can also be viewed as the reconstruction of a smoothed version of a solution. Consider a smoothing operator E W X ! X satisfying E x * x for all x 2 X and  ! 0, and assume that E can be represented by a mollifier function e ,

An x D yn

  E x .s/ D he .s; /; xi :

Yn  YnC1

with An D Qn A, yn D Qny , and Qn is the orthogonal projection onto Yn . Again, xn is a stable approximation of x  . The dual least-squares method is a regularization if the parameter n is properly linked to the noise level ı, i.e., the regularization parameter ˛ is given by 1=n. For a rigorous analysis of projection methods, see [40]. Projection methods are frequently combined with other regularization methods as, e.g., Tikhonov regularization or Landweber iteration [43, 44]. Conjugate Gradient Methods The conjugate gradient method (cg) is one of the most powerful methods for the solution of self-adjoint positive semidefinite well-posed problems. It was originally introduced in a finite-dimensional setting by Hestenes and Stiefel [25] but can also be extended to an infinite-dimensional setting. CGNE is the cg method applied to the normal Eq. (3). Its iterates can be characterized as minimizers of the residual over a Krylov subspace, i.e, ˚ ky ı  Axık k D min ky ı  Axk j x  x0 2 Kk .A .y ı  Ax0 /; A A/ ; where the kth Krylov space is defined as Kk .x; A/ D spanfx; Ax; ; A2 x;    ; Ak1 xg and x0 is the initial iterate. Therefore, CGNE requires the fewest iterations among all iterative methods if the discrepancy principle is chosen as a stopping rule for the iteration. Conjugate gradient-type methods depend in a more direct way (nonlinearly) on the data y ı , which requires a more complicated analysis of the method. CGNE with an a priori parameter choice rule is a regularization method. If CGNE is terminated by the discrepancy principle (8),

Instead of reconstructing x  directly, the aim is to reconstruct its mollified version E x  . If e has a representation A vs D e (9) then .E x  /.s/ D hvs ; yi :

(10)

The approximate inverse S W Y ! X is defined as .S y/.s/ WD hvs ; yi : 

With known vs , the evaluation of S is just the evaluation of an inner product. Therefore, the difficult part is solving (9), which can be achieved either analytically or, if this is impossible, numerically. The advantages for the computation of a numerical solution are that the right-hand side of (9) is given exactly, which reduces  the errors in the computation of vs , and that (9) has to be solved only once for many sets of data y. Mollifier methods where introduced in [35] and generalized to nonlinear problems in [34].

Iterative Regularization of Nonlinear Inverse Problems In this section, we consider iterative methods for solving (1) with a nonlinear operator F W X ! Y , X; Y Hilbert spaces. As some of the iterative algorithms display rapid convergence, they are in particular used for solving large-scale inverse problems. Naturally, the analysis of methods for solving nonlinear problems is more complicated than in the linear case. It often needs more or less severe conditions on the operator F .

Regularization of Inverse Problems

1237

Details on the analysis of the presented methods can be and requires regularization. Different methods can be found in [16, 31]. generated by using different approaches for the regularization of the linearized problem. Applying Tikhonov Nonlinear Landweber Iteration regularization to the linearized problem (15) yields the The nonlinear Landweber iteration can be derived as a Levenberg-Marquardt method descent method for the minimization of the functional  1 ı D xkı C F 0 .xkı / F 0 .xkı / C ˛k I xkC1 J.x/ D kF .x/  y ı k2 : (11)   F 0 .xkı / y ı  F .xkı / : (16) The gradient of the functional is given by F 0 .x/ .y ı  In order to prove convergence of the method, the F .x//, leading to the fixed-point iteration nonlinearity condition ı xkC1 D xkı C F 0 .xkı / .y ı  F .xkı // : (12) 0 kF .x/F .x/F Q .x/.xx/k Q  ckxxkkF Q .x/F .x/k Q (17) If the nonlinearity condition and a strategy for the choice of the regularization kF .x/  F .x/ Q  F 0 .x/.x  x/k Q parameters ˛k is needed. In [21], it was proposed to choose ˛k such that  kF .x/  F .x/k; Q  < 1=2 ı ı 0 ı ı ı ı ı is fulfilled and the iteration is stopped with the smallest ky F .xk /F .xk /.xkC1 .˛k /xk /k D qky F .xk /k index k with for some fixed q 2 .0; 1/, and it was shown that ky ı  F .xk /k  ı < ky ı  F .xk1 /k ; (13) the resulting method is a regularization. Assuming the source condition (14) and some additional nonlinearity then the Landweber iteration is a regularization conditions, convergence rates were given in [53]. The iterates of the iteratively regularized Gaussmethod. In order to prove convergence rates, the source Newton method are defined as the minimizers of the condition (5) has to be adapted to the nonlinear setting. functional With the source condition

  x  x0 D F 0 .x  / F 0 .x  / w;

ky ı  F .xkı /  F 0 .xkı /.x  xkı /k2 C ˛k kx  x0 k ;

kwk small enough; 0 <   1=2 (14)

i.e.,

R

and assuming additional nonlinearity conditions, it can  0 ı  0 ı 1 ı ı be shown that the Landweber iteration is of optimal xkC1 D xk C F .xk / F .xk / C ˛k I  0 ı  ı   order; see [23]. F .xk / y  F .xkı / C ˛k .x0  xkı / : Although the Landweber method is very robust, its (18) convergence is rather slow. For accelerated/modified versions of the method, see, e.g., [31, 45]. Convergence and convergence rates with the source condition (14) were shown for   1 in [3] under Newton-Type Methods the assumption of Lipschitz continuity of F 0 . The case In Newton-type methods, the operator F is linearized  < 1 was treated in [4] but needs stronger nonlinearity ı around a current approximation xkı and an update xkC1 conditions. A possible choice for the sequence of is obtained by solving the equation regularization parameters is ı F 0 .xkı /.xkC1  xkı / D y ı  F .xkı / :

(15)

˛k > 0; 1 

If the original nonlinear problem is ill posed, then the linearized problem (15) is in general also ill posed for a fixed r > 1.

˛k  r; lim ˛k D 0 k!1 ˛kC1

1238

A convergence rate analysis for the iteratively regularized Gauss-Newton method under logarithmic source conditions was given in [27] and in a Banach space setting in [30]. For further generalizations of the method, see [31], and for an analysis of Newton-type methods using affinely invariant conditions, see [11]. Another way of solving (15) is by using iterative methods for linear ill-posed problems. As mentioned above, iterative methods have a regularizing effect when stopped early enough. For example, Newton’s method in combination with CGNE for solving (15), the truncated Newton-CG algorithm, is a regularization method if the CGNE iteration is stopped according to the discrepancy principle and the operator F fulfills the nonlinearity condition (17); see [22]. Further Newton-type methods for nonlinear ill-posed problems include also a variant of Broyden’s method [29].

Regularization of Inverse Problems

Under the above assumptions exists a minimizer x˛ı of the Tikhonov functional N 2 J˛ .x/ D ky ı  F .x/k2 C ˛kx  xk which depends in a stable way on the the data y ı . If Tikhonov regularization (19) is combined with a parameter choice rule fulfilling ı2 D0; ı!0 ˛.ı/

lim ˛.ı/ D 0; and lim

ı!0

(20)

then each sequence of minimizers x˛ıkk has for ık ! 0 a subsequence that converges to an x  minimum norm solution. Assuming additionally Lipschitz continuity of the Fr´echet derivative of F , convergence rates can be obtained by requiring the source condition x   x  D F 0 .x  / w

Variational Approaches for Regularization For Tikhonov regularization, the approximations x˛ı to the least-squares solution x  have been characterized for linear problems as the minimizer of the functional (6). This motivates the definition of Tikhonov regularization with a nonlinear operator in Hilbert spaces via ˚ ı ky  F .x/k2 C ˛kx  x  k2 ! minŠ: ;

with norm of w small enough and by using the parameter choice rule ˛ ı. As for linear problems, Morozov’s discrepancy principle can be used as a parameter choice rule. Due to the nonlinearity of F , the existence of a parameter ˛ fulfilling ky ı  F .x˛ı  /k D ı, or, slightly more general, (21) ı  ky ı  F .x˛ı  /k  ı

(19) cannot always be guaranteed. However, if such parameters exist, then a regularization method is obtained.  where xN denotes an a priori guess to a solution x . For convergence and convergence rates, we refer to Based on this definition, several variants of Tikhonov [32, 46, 55]. In the linear case, the Tikhonov functional is strictly regularization have been proposed by changing either convex and therefore has a unique minimizer. For the penalty or the data fit term. nonlinear operators, only local convexity can be expected, with the consequence that standard methods Tikhonov Regularization in Hilbert Spaces for the minimization of the Tikhonov functional might Consider Tikhonov regularization (19) with a nonlinear only recover a local minimizer. For iterative algorithms operator F W D.F /  X ! Y , X; Y Hilbert spaces. and conditions that ensure convergence to a global The following results have been developed in [15]: minimizer, we refer to [47]. Assume that F is continuous and weakly sequentially closed, i.e., weak convergence of a sequence xn * x in X and F .xn / * y in Y implies x 2 D.F / and BV Regularization F .x/ D y. As the nonlinear equation F .x/ D y may Tikhonov regularization with a Hilbert space penalty have several solutions, the concept of an x  minimum term usually results in a smooth reconstruction, which makes a reconstruction of discontinuous functions imnorm solution is chosen, i.e., x  admits possible. A suitable space, in particular for images, that contains functions with discontinuities is the space    F .x / D y and kx  x k of functions with bounded variation over a bounded D minfkx  x  k j F .x/ D yg : region   Rn ,

Regularization of Inverse Problems

1239

BV ./ 8 9 Z < = D x 2 L1 ./ j S.x/ D sup .x div v/ dt < 1 ; : ; x2V 

J˛;p .x/ D ky ı  F .x/k2 C ˛

V D v2

C01 ./

j jv.t/j  1 for all t 2  :

The BV norm is then defined by (22)

If x 2 C 1 ./, then Z jrxj dt :

S.x/ D

wj jhx; j ijp ;

ı D arg min J˛;p .x/ : (24) x˛;p



kxkBV D kxkL1 ./ C S.x/ :

X j 2I

with V being a space of test functions, ˚

some basis. Given an orthonormal system fj gj 2I with index set I , the Tikhonov functional with sparsity constraint is defined as



Note that S.u/ forms a semi-norm for BV./. In order to enable a reconstruction of piecewise constant functions from noisy data, it is therefore natural to consider the Tikhonov functional J˛ .x/ D ky ı  F .x/k2 C ˛P .x/ x˛ı D arg min J˛ .x/ (23) with P being either the BV norm or the BV seminorm. In a slightly different formulation, BV regularization was first considered in [54] for the image denoising problem (where F is the identity) and for linear inverse problems in [1, 8]. In the latter papers also questions concerning the existence of minimizers of the functional (23) and its stable dependence on the regularization parameter as well as on the data have been addressed, and in [1], it was shown that, for linear F , (23) combined with the parameter choice rule (20) is a regularization method. Convergence rates for BV regularization have been obtained for the denoising case in [7] with respect to the Bregman distance (see also section “Regularization in Banach Spaces”) and in [9] with respect to L2 . Several methods have been proposed for the minimization of (23), e.g., a relaxation algorithm in [8] or fixed-point-based algorithms [12, 59].

The sequence fwj g is bounded from below away from zero, and p < 2. The properties of the Tikhonov functional depend crucially on p: if 1 < p, then the functional is strictly convex and differentiable, for p D 1 it is convex but not differentiable, and for p < 1 it is not convex anymore. Nevertheless, the a priori parameter choice rule (20) still makes (24) a regularization method. This was first shown for linear operator equations and 1  p < 2 in [10], for nonlinear operator equations and 1  p < 2 in [49], and for nonlinear equations and p < 1 in [60]. Note that convergence can be considered in different metrics. For example, convergence with respect to L2 has been proven for the above-indicated range of p in [10, 60] and for the stronger metric induced by the penalty term in [49]. Also, (24) combined with the discrepancy principle (21) yields a regularization method [2, 5]. For convergence rates, see [18, 48]. The minimization of the functional (24) causes additional difficulties, in particular if the penalty term is non-differentiable. A common approach is the use of surrogate functionals, which result in an iterative shrinkage algorithm [10, 49]. Other used optimization methods include conditional gradient methods [6] and semismooth Newton methods [19, 28]. For a comprehensive review on regularization with sparsity constraints, we refer to [50], and for its use in systems biology, we refer to [17].

Regularization in Banach Spaces A natural extension of the above-presented approaches toward Tikhonov regularization is to consider the functional (23) with F W U ! Y , U a Banach space and Y a Hilbert space, where the penalty functional P W U ! R [ fC1g is now a general convex functional that is lower semicontinuous in a topology Sparsity T of U with sequentially compact sublevel sets M˛ WD Tikhonov regularization with sparsity constraints is fJ˛  mg 8˛ > 0; m > 0 in the topology T . In used whenever it is assumed that the exact solution this setting, convergence rates are usually derived in can be approximated well with few coefficients w.r.t. the Bregman distance w.r.t. the penalty functional P ,

R

1240

Regularization of Inverse Problems

DP .x; u/ D fP .x/  P .u/  hp; x  ui j p 2 @P .u/g : References (25) In the case where @P .u/ is not a singleton, DP .x; u/ represents a family of distances. For the standard parameter choice rule (20), it can be shown that Tikhonov regularization with the penalty P is a regularization method. In order to give quantitative estimates, the source conditions R.F 0 .x  / / \ @P .x  / ¤ ; 0

 

0

R.F .x / F .x // \ @P .x / ¤ ; 



(26) (27)

can be used, where x  denotes a solution of F .x/ D y with minimal value of P .x/. With ky  y ı k  ı, the source condition (26), and the parameter choice rule ˛ ı, there exists a d 2 DP .x˛ı ; x  / s.t. d D O.ı/ ; see [7]. Assuming that P is twice differentiable with hP 00 .x/.u/; ui  M kuk2 in a neighborhood of x  , then the parameter choice rule ˛ ı 2=3 yields a convergence rate of O.ı 4=3 / [51, 52]. Note that in the nonlinear case the operator F has to fulfill some nonlinearity conditions. For convergence and convergence rates for Tikhonov regularization combined with the discrepancy principle, see [2]. Further extensions of Tikhonov regularization in Banach spaces were proposed in [26], where in particular nonsmooth operators and variational source conditions were considered. The above results can also be partially applied to BV regularization, to regularization with sparsity constraints, and to maximum entropy regularization. The latter was analyzed in [13, 14], the penalty for maximum entropy regularization being given by Z P .x/ D

x log

x dt ; with x; x  > 0; x

where x  is an a priori guess for the solution. The recent book [56] also contains many results on regularization in Banach spaces.

1. Acar, R., Vogel, C.: Analysis of bounded variation penalty methods for ill-posed problems. Inverse Probl. 10, 1217–1229 (1994) 2. Anzengruber, S., Ramlau, R.: Morozovs discrepancy principle for Tikhonov-type functionals with nonlinear operator. Inverse Probl. 26(2), 1–17 (2010) 3. Bakushinskii, A.W.: The problem of the convergence of the iteratively regularized Gauss–Newton method. Comput. Math. Math. Phys. 32, 1353–1359 (1992) 4. Blaschke, B., Neubauer, A., Scherzer, O.: On convergence rates for the iteratively regularized Gauss– Newton method. IMA J. Numer. Anal. 17, 421–436 (1997) 5. Bonesky, T.: Morozov’s discrepancy principle and Tikhonov-type functionals. Inverse Probl. 25, 015,015 (2009) 6. Bredies, K., Lorenz, D., Maass, P.: A generalized conditional gradient method and its connection to an iterative shrinkage method. Comput. Optim. Appl. 42, 173–193 (2008) 7. Burger, M., Osher, S.: Convergence rates of convex variational regularization. Inverse Probl. 20(5), 1411–1421 (2004) 8. Chambolle, A., Lions, P.L.: Image recovery via total variation minimization and related problems. Numer. Math. 76, 167–188 (1997) 9. Chavent, G., Kunisch, K.: Regularization of linear least squares problems by total bounded variation. ESAIM, Control Optim. Calc. Var. 2, 359–376 (1997) 10. Daubechies, I., Defriese, M., DeMol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 51, 1413–1541 (2004) 11. Deuflhard, P., Engl, H., Scherzer, O.: A convergence analysis of iterative methods for the solution of nonlinear illposed problems under affinely invariant conditions. Inverse Probl. 14, 1081–1106 (1998) 12. Dobson, D., Scherzer, O.: Analysis of regularized total variation penalty methods for denoising. Inverse Probl. 12, 601–617 (1996) 13. Eggermont, P.: Maximum entropy regularization for Fredholm integral equations of the first kind. SIAM J. Math. Anal. 24, 1557–1576 (1993) 14. Engl, H., Landl, G.: Convergence rates for maximum entropy regularization. SIAM J. Numer. Anal. 30, 1509–1536 (1993) 15. Engl, H., Kunisch, K., Neubauer, A.: Convergence rates for Tikhonov regularization of nonlinear ill-posed problems. Inverse Probl. 5, 523–540 (1989) 16. Engl, H.W., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Kluwer, Dordrecht (1996) 17. Engl, H., Flamm, C., K¨ugler, P., Lu, J., M¨uller, S., Schuster, P.: Inverse problems in systems biology. Inverse Probl. 25, 123,014 (2009) 18. Grasmair, M., Haltmeier, M., Scherzer, O.: Sparse regularization with `q penalty term. Inverse Probl. 24(5), 1–13 (2008)

Regularization of Inverse Problems 19. Griesse, R., Lorenz, D.: A semismooth Newton method for Tikhonov functionals with sparsity constraints. Inverse Probl. 24(3), 035,007 (2008) 20. Hanke, M.: Conjugate Gradient Type Methods for Ill-Posed Problems. Longman Scientific & Technical, Harlow (1995) 21. Hanke, M.: A regularizing Levenberg–Marquardt scheme, with applications to inverse groundwater filtration problems. Inverse Probl. 13, 79–95 (1997) 22. Hanke, M.: Regularizing properties of a truncated Newton– cg algorithm for nonlinear ill-posed problems. Numer. Funct. Anal. Optim. 18, 971–993 (1997) 23. Hanke, M., Neubauer, A., Scherzer, O.: A convergence analysis of the Landweber iteration for nonlinear ill-posed problems. Numer. Math. 72, 21–37 (1995) 24. Hegland, M.: Variable Hilbert scales and their interpolation inequalities with applications to Tikhonov regularization. Appl. Anal. 59(1–4), 207–223 (1995) 25. Hestenes, M.R., Stiefel, E.: Methods of conjugate gradients for solving linear systems. J. Res. Natl. Bur. Stand. 49, 409– 436 (1952) 26. Hofmann, B., Kaltenbacher, B., P¨oschl, C., Scherzer, O.: A convergence rates result for Tikhonov regularization in Banach spaces with non-smooth operators. Inverse Probl. 23, 987–1010 (2007) 27. Hohage, T.: Logarithmic convergence rates of the iteratively regularized Gauss–Newton method for an inverse potential and an inverse scattering problem. Inverse Probl. 13, 1279– 1299 (1997) 28. Ito, K., Kunisch, K.: Lagrange multiplier approach to variational problems and applications. SIAM, Philadelphia (2008) 29. Kaltenbacher, B.: On Broyden’s method for ill-posed problems. Num. Funct. Anal. Optim. 19, 807–833 (1998) 30. Kaltenbacher, B., Hofmann, B.: Convergence rates for the iteratively regularized Gauss-Newton method in Banach spaces. Inverse Probl. 26, 035,007 (2010) 31. Kaltenbacher, B., Neubauer, A., Scherzer, O.: Iterative Regularization Methods for Nonlinear Ill-Posed Problems. de Gruyter, Berlin (2008) 32. Kravaris, C., Seinfeld, J.H.: Identification of parameters in distributed parameter systems by regularization. SIAM J. Control Optim. 23, 217–241 (1985) 33. Kress, R.: Linear Integral Equations. Springer, New York (1989) 34. Louis, A.: Approximate inverse for linear and some nonlinear problems. Inverse Probl. 12, 175–190 (1996) 35. Louis, A., Maass, P.: A mollifier method for linear operator equations of the first kind. Inverse Probl. 6, 427–440 (1990) 36. Louis, A.K.: Inverse und Schlecht Gestellte Probleme. Teubner, Stuttgart (1989) 37. Luecke, G.R., Hickey, K.R.: Convergence of approximate solutions of an operator equation. Houst. J. Math. 11, 345–353 (1985) 38. Mathe, P., Pereverzev, S.V.: Geometry of linear ill-posed problems in variable Hilbert scales. Inverse Probl. 19(3), 789803 (2003) 39. Morozov, V.A.: Methods for Solving Incorrectly Posed Problems. Springer, New York (1984) 40. Natterer, F.: Regularisierung schlecht gestellter Probleme durch Projektionsverfahren. Numer. Math. 28, 329–341 (1977)

1241 41. Natterer, F.: The Mathematics of Computerized Tomography. Teubner, Stuttgart (1986) 42. Nemirovskii, A.S.: The regularizing properties of the adjoint gradient method in ill posed problems. USSR Comput. Math. Math. Phys. 26(2), 7–16 (1986) 43. Pereverzev, S.V.: Optimization of projection methods for solving ill-posed problems. Computing 55, 113–124 (1995) 44. Plato, R., Vainikko, G.: On the regularization of projection methods for solving ill-posed problems. Numer. Math. 57, 63–79 (1990) 45. Ramlau, R.: A modified Landweber–method for inverse problems. Numer. Funct. Anal. Optim. 20(1&2), 79–98 (1999) 46. Ramlau, R.: Morozov’s discrepancy principle for Tikhonov regularization of nonlinear operators. Numer. Funct. Anal. Optim. 23(1&2), 147–172 (2002) 47. Ramlau, R.: TIGRA–an iterative algorithm for regularizing nonlinear ill–posed problems. Inverse Probl. 19(2), 433–467 (2003) 48. Ramlau, R., Resmerita, E.: Convergence rates for regularization with sparsity constraints. Electron. Trans. Numer. Anal. 37, 87–104 (2010) 49. Ramlau, R., Teschke, G.: A Tikhonov-based projection iteration for non-linear ill-posed problems with sparsity constraints. Numer. Math. 104(2), 177–203 (2006) 50. Ramlau, R., Teschke, G.: Sparse recovery in inverse problems. In: Fornasier, M. (ed.) Theoretical Foundations and Numerical Methods for Sparse Recovery, pp. 201–262. De Gruyter, Berlin/New York (2010) 51. Resmerita, E.: Regularization of ill-posed problems in Banach spaces: convergence rates. Inverse Probl. 21, 1303–1314 (2005) 52. Resmerita, E., Scherzer, O.: Error estimates for nonquadratic regularization and the relation to enhancement. Inverse Probl. 22, 801–814 (2006) 53. Rieder, A.: On convergence rates of inexact Newton regularizations. Numer. Math. 88, 347–365 (2001) 54. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992) 55. Scherzer, O.: The use of Morozov’s discrepancy principle for Tikhonov regularization for solving nonlinear ill–posed problems. Computing 51, 45–60 (1993) 56. Scherzer, O., Grasmair, M., Grossauer, H., Haltmeier, M., Lenzen, F.: Variational Methods in Imaging. Springer, Berlin (2008) 57. Schock, E.: Approximate solution of ill-posed problems: arbitrary slow convergence vs. superconvergence. In: H¨ammerlin, G., Hoffmann, K. (eds.) Constructive Methods for the Practical Treatment of Integral Equation, pp. 234–243. Birkh¨auser, Basel/Boston (1985) 58. Tikhonov, A.N., Arsenin, V.Y.: Solutions of IllPosed Problems. V.H. Winston & Sons, Washington (1977) 59. Vogel, C.R., Oman, M.: Iterative methods for total variation denoising. SIAM J. Sci. Comput. 17, 227–238 (1996) 60. Zarzer, C.A.: On Tikhonov regularization with nonconvex sparsity constraints. Inverse Probl. 25, 025,006 (2009)

R

1242

Relativistic Models for the Electronic Structure of Atoms and Molecules Trond Saue Laboratoire de Chimie et Physique Quantiques, CNRS/Universit´e Toulouse III, Toulouse, France

Relativistic Models for the Electronic Structure of Atoms and Molecules

Relativistic molecular electronic structure calculations are carried out within the Born-Oppenheimer (clamped nuclei) approximation. (See entry  Schr¨odinger Equation for Chemistry.) The electronic Hamiltonian, whether relativistic or not, has the same generic form H D

electrons X i

Definition VNN D

electrons 1 X hO .i / C gO .i; j / C VNN I 2

nuclei 1 X ZA ZB ; 2 RAB

i ¤j

(1) This entry discusses the methodology employed for A¤B calculating electronic structure of atoms and molecules where VNN is the classical repulsion of nuclei. in a relativistic framework. Here and in the following, we will employ SI-based atomic units. The various electronic Hamiltonians are distinguished by the choice of one- and two-electron operators, hO .i / and gO .i; j /, respectively. An important Overview observation is that the derivation of the basic formulas for most electronic structure methods requires A relativistic quantum mechanical wave equation for only the use of the generic form of the electronic the electron was formulated by Dirac in 1928. Dirac Hamiltonian, Eq. (1). This implies that most methods did, however, not consider relativistic effects of any known from nonrelativistic theory (See, for instance, importance in the structure and reactivity of atoms entries  Hartree–Fock Type Methods,  Density and molecules because such effects are associated with Functional Theory,  Post-Hartree-Fock Methods the high speeds attained by the chemically inert core and Excited States Modeling,  Coupled-Cluster electrons in the vicinity of heavy nuclei [4]. Only in the Methods) can be extended to the relativistic domain 1970s did it become clear that relativity propagates out and that a presentation of relativistic electronic into the chemically active valence region of atoms and structure theory should mostly focus on Hamiltonians. may have dramatic effects on the chemistry of heavy However, there are certain features of relativistic elements [5]. Relativistic effects are conveniently diHamiltonians, in particular the unboundedness vided into scalar relativistic effects, associated with the of the Dirac operator, which warrants special relativistic mass increase of electrons, and the spinconsideration and which will be discussed in the orbit interaction, which is the interaction of the spin following. of a reference particle, typically an electron, and the More extensive discussions of relativistic elecmagnetic field induced by other charges in relative tronic structure theory can be found in recent motion. textbooks: [2, 3, 6]. The first computer codes for the relativistic calculation of atomic electronic structure, based on numerical methods (finite differences/elements), appeared toward The Electronic Hamiltonian the end of the 1960s. Corresponding molecular codes are predominantly based on the expansion of one- One-Electron Systems electron functions (orbitals) into a suitable basis, thus The key equation of relativistic quantum mechanics is converting differential equations into matrix algebra. the Lorentz invariant Dirac equation The first such codes appeared in the beginning of the 

1980s but had a difficult beginning since it was at Oi @ Q D0 h (2) first not realized that the basis sets for the large and @t small components of the 4-component orbitals must be constructed such that the correct coupling between 4-component relativistic molecular calculations emthese components can be attained. ploy the Dirac Hamiltonian in the molecular field,

Relativistic Models for the Electronic Structure of Atoms and Molecules

1243

that is, in the electrostatic potential N of clamped Relativistic Models for the Electronic Structure of nuclei hO D hO 0 C VeN I

Atoms and Molecules, Fig. 1 Spectrum of the Dirac Hamiltonian in a molecular field

O I hO 0 D ˇmc 2 C c .˛  p/

VeN D eI4 N .r/

(3)

where e, m, and c refer to the fundamental charge, the electron mass, and the speed of light, respectively. A more complete discussion of this Hamiltonian is found in the entry  Relativistic Theories for Molecular Models. The time independence of the Dirac Hamiltonian, Eq. (3), in the nuclear frame of reference allows the time dependence of the Dirac equation, Eq. (2), to be separated out and leads to the Dirac equation on timeindependent form hO

DE I

Q D e iEt ;

(4)

albeit no longer explicitly Lorentz invariant. The spectrum of the free-particle Hamiltonian hO 0 consists of two branches of continuum states, of positive and negative energy, separated by a large energy gap of 2mc 2   ˝  ˛  hO 0 D 1; mc 2 [ Cmc 2 ; 1 :

From a more mathematical point of view, the timeindependent Dirac equation is a system of first-order partial differential equations coupling the four compoWith the introduction of the molecular field, Eq. (3), nents of the Dirac wave function (denoted 4-spinor) bound solutions appear in the upper part of this energy gap.

L

X˛  X This is illustrated in Fig. 1, where the relativistic D I X D L; S (6) D S Xˇ ; and nonrelativistic energy scales have been aligned by subtracting the electron rest mass mc 2 ; this is formally The upper and lower two components are referred to done by the substitution ˇ ! ˇ 0 D ˇ  I4 in the Dirac as the large and small components, respectively. In Hamiltonian, Eq. (3). The spectrum resembles that of a solutions of positive energy, the small components are nonrelativistic one, with the exception of the presence on average a factor c smaller than the large components of negative-energy solutions. These pose problem in and vanish in the nonrelativistic limit, taken as c ! 1. that quantum theory allows a bound electron to make a This can be seen from the coupling of the large and the transition down to a level of negative energy, liberating small components an infinite amount of energy on its way down the

 1 E  V 1 negative-energy continuum and making matter unstaS L O O O : DX I X D .  p/ 1C ble. Dirac therefore postulated that all negative-energy 2mc 2mc 2 (7) solutions are occupied and therefore not accessible to bound electrons due to the Pauli exclusion principle. The situation is reversed for negative-energy solutions. On the other hand, since the reference vacuum is now the occupied “Dirac sea” of electrons, an electron Many-Electron Systems excited from the negative-energy continuum is observ- In nonrelativistic molecular quantum mechanics, O Eq. (1), is given by the able, as well as the positively charged hole left behind, the two-electron operator g, identified as a positron, the antiparticle of the electron. Coulomb term (5)

R

1244

Relativistic Models for the Electronic Structure of Atoms and Molecules

gO C .1; 2/ D

1 ; r12

(8)

which describes the instantaneous Coulomb interaction. In the relativistic domain, the relative motion of the electrons leads to magnetic induction and thus spin-orbit interaction. The electromagnetic interaction between two charged particles can furthermore be pictured as an exchange of virtual photons and is therefore not instantaneous since photons travel at the finite speed of light. The full relativistic interaction between two electrons accordingly requires a complete specification of the history of the interacting particles and cannot be given on a simple Lorentz invariant closed form. Relativistic molecular quantum mechanics therefore employs an expansion of the full two-electron interaction in powers of c 2 . In Coulomb gauge the zeroth-order term is the Coulomb term, Eq. (8). The first-order term is the Breit term g B .1; 2/ D  „

ec˛1  ec˛2 c 2 r12 ƒ‚ … gG

 „

.ec˛1  r 1 / .ec˛2  r 2 / r12 ; 2c 2 ƒ‚ …

(9)

g gauge

which can be further split into the Gaunt term gG and a gauge-dependent term g gauge . For most chemical purposes, the Coulomb term suffices and defines the Dirac-Coulomb Hamiltonian. In a relativistic framework, the Coulomb term not only describes the instantaneous Coulomb interaction but also the spin-same orbit interaction, due to the relative motion of the reference electron in the nuclear frame. For very precise calculations of molecular spectra, it is recommended to add the Breit term, or at least the Gaunt term, which describes the spin-other orbit interaction, due to the relative motion of the other electron in the nuclear frame.

Relativistic Electronic Structure Methods Relativistic molecular quantum mechanics to a large extent follows the nonrelativistic program of finding approximate solutions to the N -electron problem:

In the first step, the major part of the electronic energy is recovered by writing the wave function as a single Slater determinant and then determining the orbitals which render the electronic energy stationary (the Hartree-Fock method). (See entry  Hartree–Fock Type Methods.) A set of orbitals, both occupied and empty (virtual), is obtained through the solution of effective one-particle equations which describe the motion of a single electron in the mean field of the others. In the second step, electron correlation is captured by writing the wave function as a linear combination of Slater determinants generated from the one-particle basis. The expansion coefficients can be found by perturbation theory, as in the Møller-Plesset method, or by optimization, as in the configuration interaction method. (See entry  Post-Hartree-Fock Methods and Excited States Modeling.) The coupledcluster method provides more efficient electron correlation for a given excitation level than CI, but solutions are obtained by projection rather than optimization. (See  Coupled-Cluster Methods.) In more complicated cases, such as when chemical bonds are broken, the single Hartree-Fock determinant of the initial step may be replaced by a multideterminantal expansion, as in the multi-configuration self-consistent field (MCSCF) method. An alternative approach is density functional theory in which the wave function is replaced by the one-electron density as the central object allowing the calculation of the electronic energy and other properties. (See entry  Density Functional Theory.) There are, however, some distinct differences in relativistic theory due to the fact that the Dirac Hamiltonian is unbounded from below: 1. The spectrum of the effective one-electron equations which are solved to self-consistency in the Hartree-Fock method and in the Kohn-Sham formulation of DFT corresponds to that of the Dirac equation and illustrated in Fig. 1. The large size of the energy gap between the positive- and negativeenergy continuum usually allows straightforward identification of the desired bound solutions for the construction of the mean-field potential. This selection procedure corresponds to the embedding of the electronic Hamiltonian by projection operators, updated in each iteration of the SCF cycle and projecting out the negative-energy solutions of the current iteration. More generally, the Hartree-Fock, MCSCF, or Kohn-Sham energy of the electronic

Relativistic Models for the Electronic Structure of Atoms and Molecules

ground state is not found by minimization of orbital variational parameters, rather by application of a min-max principle. (As discussed in the entry  Relativistic Theories for Molecular Models.) 2. As argued by Brown and Ravenhall (1951), the 4-component relativistic electronic Hamiltonian has no bound solutions. Starting from the oneelectron basis generated by the initial mean-field procedure, it is in principle always possible to generate determinants containing orbitals from both the positive- and negative-energy branch of the continuum which are degenerate with respect to the reference Hartree-Fock determinant, thus “dissolving” the mean-field ground state solution into the continuum. This Brown-Ravenhall disease can be eliminated by restricting the N -particle basis to Slater determinants generated from orbitals of positive energy only and therefore corresponds to the embedding of the relativistic electronic Hamiltonian by projection operators eliminating negative-energy orbitals. The energy of a full CI in such an N -particle basis will depend on the choice of projection operators, that is, the choice of orbitals defining the projectors. Such ambiguity can be avoided by carrying out an MCSCF procedure in which a full CI is carried out in the N -particle basis generated by the positive-energy solutions of an arbitrary one-particle basis but in which rotations between occupied positive-energy orbitals and virtual negative-energy orbitals are maintained, thus allowing complete relaxation of the electronic wave function [9]. 3. The absence of a lower bound or even bound solutions of the 4-component relativistic electronic Hamiltonian in principle invalidates the HohenbergKohn theorem which is the formal foundation of DFT. The situation can be alleviated by formally occupying the negative-energy one-electron solutions and subtract from the resulting vacuum density the density of a reference vacuum, typically generated from the negative-energy solutions of the free-particle Hamiltonian hO 0 of Eq. (3). Such a procedure incorporates vacuum polarization, albeit not renormalized. The effect of vacuum polarization on electronic solutions is minute, and so a pragmatical approach, universally employed in the relativistic DFT community, is to ignore vacuum polarization.

1245

One-Particle Basis For atoms the orbitals obtained from the corresponding Dirac, Hartree-Fock, or Kohn-Sham equations have the general form

.r/ D

RL .r/  ;mj .; / iRS .r/  ;mj .; /

 (10)

The 2-component angular functions  ;mj are O2 O eigenfunctions h  i of the operators j , jz , and O D    Ol C 1 with eigenvalues j.j C 1/, mj , and

, respectively, where Oj D Ol C 12  is the operator of total angular momentum and jOz its component along the z-axis. The angular functions incorporate the effect of spin-orbit coupling, and the eigenvalue

indicates parallel (j D l C 12 I D  .l C 1/) or antiparallel (j D l  12 I D l) coupling of orbital angular momentum l and spin s. In the atomic case, the angular degree of freedom can be handled efficiently, for instance, by Racah algebra, and the first-order coupled differential equations for the radial functions RL and RS by finite difference/element approaches [3]. Solutions are limited to bound ones by imposing the appropriate boundary conditions, notably exponential decay at large radial distance r. A possible limitation of these numerical approaches is that they do not easily generate a sufficient number of virtual orbitals for the inclusion of electron correlation. For molecules the separation of radial and angular degrees of freedom is generally not available, and 4-component relativistic calculations of molecular electronic structure therefore rely on basis set expansions. As in the nonrelativistic case, the choice of the mathematical form of basis functions will be a compromise between the form suggested by relativistic atomic orbitals, Eq. (10), and computational feasibility, typically the ease of generating integrals over the two-electron operator. A crucial feature of relativistic orbitals is the energy-dependent coupling between the large and the small components, shown for the Dirac equation in Eq. (7), and which suggests a separate expansion of the large and small components. In practice the˚ large and small component basis sets, ˚ L and Si , respectively, are related by the i nonrelativistic limit of the exact coupling, Eq. (7), that is ˚ S ˚ O L (11)  D .  p/ i :

R

1246

Relativistic Models for the Electronic Structure of Atoms and Molecules

The above condition is denoted kinetic balance [10] since it assures the correct representation of the kinetic energy operator in the nonrelativistic limit. The final basis must, however, have sufficient flexibility to assure that the exact coupling, Eq. (7), can be obtained. Atomic radial functions display a weak singularity at the origin for j j D 1, contrary to the cusp in the nonrelativistic case. The singularity can be removed by replacing point nuclei by nuclear charge distributions of finite extent which makes the radial functions Gaussian in shape at small radial distance r. This in turn favors the use of Gaussian-type basis functions. One option is to introduce 2-component basis functions using the angular functions  of Eq. (10) combined with radial functions of Gaussian type  RnX D N r n1 exp ˛r 2 ;

X D L; S;

(12)

# 1 XO  I XO 1

" UO D WO 1 WO 2 I

WO 1 D 2 WO 2 D 4

p

1

1CXO  XO

0

0 p

1

3 5 ; (14)

1CXO XO 

where WO 1 decouples the large and small components and WO 2 reestablishes normalization. Due to the energy dependence of the exact coupling XO , Eq. (7), various 2-component Hamiltonians can be defined by approximate decoupling transformations of the Dirac Hamiltonian in the molecular field. 1. The Pauli Hamiltonian is obtained from the approximate coupling XO

1 O .  p/ 2mc

(15)

and retaining terms only to O.c 2 /. The Pauli where N is a normalization factor. Alternatively the Hamiltonian benefits from simplicity and physical large and small components may be expanded in transparency but is not bounded from below and scalar basis functions, such as Cartesian or spherical introduces highly singular terms. Gaussian-type orbitals known from the nonrelativistic 2. These disadvantages are alleviated in the regular domain and which have the advantage that it allows the approximation (RA) based on the approximate rather straightforward use of nonrelativistic integral coupling codes. c O : .  p/ (16) XO 2mc 2  VeN

2-Component Relativistic Hamiltonians The complications introduced by the presence of negative-energy solutions in relativistic theory have motivated the development of 2-component relativistic Hamiltonians with solutions of positive energy only. Such Hamiltonians can be generated by a block diagonalization of the parent 4-component Hamiltonian hO 4c " O  O 4c

U h UO D

hO 2c CC 0 0 hO 2c 

# (13)

retaining the 2-component Hamiltonian hO 2c CC which reproduces the positive-energy spectrum of the parent Hamiltonian. It turns out, however, that the exact decoupling transformation UO is expressed in terms of the exact coupling XO , Eq. (7), between the large and small components of the positive-energy solutions of the parent Hamiltonian, that is

Carrying out only the decoupling transformation WO 1 gives the zeroth order RA (ZORA) Hamiltonian, whereas the infinite-order RA (IORA) is obtained by renormalization WO 2 . 3. Another approach is to first carry out the exact transformation which decouples the free-particle Hamiltonian hO 0 of Eq. (3). This has the advantage of bringing the kinetic energy operator on a square root form which assures variational stability. Further decoupling in terms of the nuclear potential N , Eq. (3), gives the Douglas-Kroll-Hess (DKH) Hamiltonian to various orders. Alternatively, following the freeparticle transformation, the exact coupling relation can be developed to odd orders 2k  1 in c 1 and a single corresponding decoupling transformation carried out, defining Barysz-Sadlej-Snijders (BSS) Hamiltonians to even order 2k in c 1 . More recently it has been realized that the exact decoupling can be achieved in a simple manner by first solving the parent Dirac equation on matrix form,

Relativistic Theories for Molecular Models

1247

keeping in mind that the cost of solving the one- References electron problem is negligible compared to the manyelectron problem usually at hand. The exact coupling 1. Bethe, H.: Termaufspaltung in Kristallen. Ann. Phys. 3, 133–208 (1929) can then be extracted from the eigenvectors which al2. Dyall, K.G., Fægri, K.: Introduction to Relativistic lows the construction of the appropriate coupling transQuantum Chemistry. Oxford University Press, Oxford formation matrix. This leads to the eXact 2-Component (2007) 3. Grant, I.P.: Relativistic Quantum Theory of Atoms and (X2C) Hamiltonian. Molecules. Springer, New York (2006) All operators used in conjunction with a specific 24. Kutzelnigg, W.: Perspective on Quantum mechanics of component relativistic Hamiltonian should be subject many-electron system – Dirac PAM (1929). Proc. R. Soc. to the same decoupling transformation as the oneLond. Ser. A 123, 714. Theor. Chem. Acc. 103, 182–186 (2000) electron Hamiltonian itself. Otherwise picture change errors are introduced, which for operators probing the 5. Pyykk¨o, P.: Relativistic effects in structural chemistry. Chem. Rev. 88, 563 (1988) electron density in the nuclear region may be larger 6. Reiher, M., Wolf, A.: Relativistic Quantum Chemistry: The than the relativistic effects themselves. This also holds Fundamental Theory of Molecular Science. Wiley-VCH, Weinheim (2009) for the two-electron operator, but the forbidding cost 7. Saue, T.: Relativistic Hamiltonians for chemistry: a primer. of such a decoupling transformation has led to the Chem. Phys. Chem. 12, 3077 (2011) extensive use of atomic approximations to the trans- 8. Saue, T., Jensen, H.J.A.: Quaternion symmetry in relativistic formed two-electron operator. Further discussion and molecular calculations: I. the Dirac-Fock method. J. Chem. Phys. 111, 6211 (1999) references are found in [7].

Relativistic Symmetry Symmetry is widely exploited in numerical methods for atomic and molecular models to reduce computational cost. In relativistic models the spin-orbit interaction couples spin and spatial degrees of freedom. Symmetry operations, such as rotations, reflections, and inversion, accordingly act in both spaces conjointly. The introduction of combined spin and spatial symmetry operations leads to the extension of point groups to so-called double groups [1] with extra irreducible representations spanned by (fermion) functions with half-integer spin such as Dirac spinors. Spatial symmetry can be combined with time reversal symmetry [11] which to some extent replaces spin symmetry of the nonrelativistic domain. Based on the relation between basis functions spanning a given irreducible representation and their time-reversed (Kramers) partners, irreducible representations can be classified by the Frobenius-Schur test as real, complex, or pseudoreal. It can furthermore be shown that matrix representation of operators in such a Kramers basis is expressed in terms of quaternion, complex, and real algebras, respectively, providing an illustration of the Frobenius theorem restricting associative real division algebras to the real, complex, and quaternion numbers [8].

9. Saue, T., Visscher, L.: Four-component electronic structure methods for molecules. In: Wilson, S., Kaldor, U. (eds.) Theoretical Chemistry and Physics of Heavy and Superheavy Elements, p. 211. Kluwer, Dordrecht (2003) 10. Stanton, R.E., Havriliak, S.: Kinetic balance: a partial solution to the problem of variational safety in Dirac calculations. J. Chem. Phys. 81, 1910 (1984) ¨ 11. Wigner, E.: Uber die operation der zeitumkehr in der quantenmechanik. Nachr. der Akad. der Wiss. zu G¨ottingen II, 546–559 (1932)

Relativistic Theories for Molecular Models ´ S´er´e Eric CEREMADE, Universit´e Paris-Dauphine, Paris, France

Definition Relativistic effects play an important role in the chemistry of heavy atoms. Indeed, when the number Z of protons in a nucleus is high, the core electrons can no longer be described by the nonrelativistic Schr¨odinger equation. Instead, one must use the Dirac operator, which acts on four-components spinors and is not bounded from below.

R

1248

Relativistic Theories for Molecular Models

The Dirac Operator The Dirac operator for a free electron is a constantcoefficients first-order differential operator which takes the form (see e.g., [26]):

D 0 D i ˛  r C ˇ D i

3 X

˛k @k C ˇ;

(1)

kD1

where ˛1 , ˛2 , ˛3 , and ˇ are hermitian matrices which have to satisfy the following anticommutation relations: 8 < ˛k ˛` C ˛` ˛k D 2 ık` ; (2) ˛k ˇ C ˇ˛k D 0; : ˇ 2 D 1: These relations ensure that .D 0 /2 D  C 1. This identity is the quantum-mechanical analogue of the classical relation between momentum and energy in special relativity: E 2 D c 2 jpj2 C m2 c 4 . Note that we have chosen physical units such that c D 1, „ D 1, and the mass of the electron is m D 1. The smallest dimension in which (2) can take place is 4 (i.e., ˛1 , ˛2 , ˛3 , and ˇ should be 4 4 hermitian matrices), meaning that D 0 has to act on L2 .R3 ; C4 /. The usual representation in 2 2 blocks is given by:   I2 0 ˇD ; 0 I2



0 k ˛k D k 0

 .k D 1; 2; 3/ ;

      01 0 i 1 0 ; 2 D ; 3 D : 10 i 0 0 1

 1  / : 1 D min Œ0; 1/ \ .D 0  ˛  jj

The spectrum of the Dirac operator is not bounded from below: .D 0 / D .1; 1 [ Œ1; 1/:

The One-Electron Ion Near a point-like nucleus with Z protons, a stationary state of an electron is a normalized wave function 2 L2 .R3 ; C4 /, solution of the linear eigenvalue problem:  0 1  D D ˛  jj Here, ˛ is a dimensionless quantity called the finestructure constant. Its physical value is approximately 1=137. The ground state 1 corresponds to the choice:

where the Pauli matrices are defined as 1 D

Actually, in practical computations, it is quite difficult to deal properly with the Dirac sea. As a consequence, the notion of “ground state” (state of “lowest energy” which is supposed to be the most “stable” for the system under consideration) is problematic for many of the models found in the literature. Numerically, the unboundedness from below of the spectrum is also the source of important practical issues concerning the convergence of the considered algorithms, or the existence of spurious (unphysical) solutions. For a discussion and further references on the mathematical aspects of these questions, we refer to the review paper [9]. For references on the physics and chemistry side, we refer to the books [7, 12], the recent review papers [19, 21], and the contribution of T. Saue in the present Encyclopedia. Note that in the simulation of nuclei, relativistic models also play a role, and the indefiniteness of the Dirac operator causes similar difficulties (see the entry by B. Ducomet in this Encyclopedia).

(3)

To explain why there is no observable electron of negative energy, Dirac [4] made the assumption that the vacuum (called the Dirac sea) is filled with infinitely many virtual electrons occupying the negative energy states. With this interpretation, a real free electron cannot be in a negative state due to the Pauli principle which forbids it from being in the same state as a virtual electron of the Dirac sea.

However, the standard characterization of the ground state as minimizer of the Rayleigh quotient cannot be used, since D 0 is not bounded below. A consequence, in numerical computations, is the existence of “spurious states,” that is, eigenvalues of the discretized problem that do not approximate eigenvalues of the exact problem, even when the discretization is refined. In 1986, Talman [25] proposed a min-max principle which turns out to be very helpful, from a theoretical and from a practical viewpoint. If  is a positive measure of total mass Z with ˛Z < 1, then Talman’s principle is:

Relativistic Theories for Molecular Models

1 D

inf

'2Cc1 .R3 ;C 2 /nf0g

1249

. ; .D 0  ˛ 

sup D. ' /

1 jj /

/

. ; /

2Cc1 .R3 ;C2 /

Talman’s principle has been generalized to abstract operators [5, 14]. The study in [5] has led to the design of a new algorithm based on Talman’s principle, which is free of spurious states [6]. It has also led to general criteria for the choice of pollution-free Galerkin bases [20]. No-Photon Mean-Field QED From a physics point of view, the correct relativistic theory of electrons in atoms is quantum electrodynamics (QED), the prototype of field theories. Yet a direct calculation using only QED is impractical for atoms with more than one electron because of the complexity of the calculation, and approximations are necessary. As in nonrelativistic quantum mechanics, a natural idea is to try a Hartree–Fock approximation. In the relativistic case, Hartree–Fock states are special states in the electron-positron Fock space which are totally described by their one-body density matrix, an orthogonal projector P of infinite rank in L2 .R3 ; C4 /, or, more generally, a convex combination of such projectors. Denoting P 0 as the negative spectral projector of the free Dirac operator D 0 , it is natural to work with the difference  WD P  P 0 . Physically, P 0 represents the free Dirac sea, which is taken as a reference for normal ordering of the QED Hamiltonian. The Hartree–Fock approximation consists in restricting the QED Hamiltonian to the Hartree–Fock states. If we also neglect transverse photons, we obtain an energy functional depending only on  , called the Bogoliubov–Dirac–Fock energy (Chaix-Iracane [3]) “ EBDF . / D tr.D 0  /  ˛ C

˛ 2

“

 .x/  .y/ ˛ dx dy  jx  yj 2

“

.x/  .y/ dx dy jx  yj j .x; y/j2 dx dy jx  yj

Here:  .x/ D trC4 . .x; x//. The operator  satisfies the constraints  D   , P 0    1  P 0 , tr. / D N. The last constraint means that we consider a system of N “real” electrons together with the Dirac sea. To define properly this energy, one needs an ultraviolet cutoff  and a new definition of the trace [15].

Then, in order to interpret correctly the solutions, it is necessary to perform a charge renormalization [13]. It is possible to derive the BDF model as a thermodynamic limit by considering the Hartree–Fock approximation of no-photon QED in a box, with no a priori choice for normal ordering, and letting the size of the box go to infinity. But then the reference projector P 0 must be replaced by the solution P 0 of a nonlinear equation. The new reference minimizes the QED energy per unit volume, and the BDF energy EBDF . / is, in a suitable sense, the difference: < ˝ CP 0 ; H ˝ CP 0 >  < ˝P 0 ; H ˝P 0 > (Hainzl-Lewin-Solovej [17]). In the sequel, we denote by I .A/ the spectral projector of an operator A associated with the interval I. There exists a BDF ground state for neutral atoms and positively charged ions: Theorem 1 [16] Let N  Z,  > 0,  2 Cc1 . When ˛ is small enough, the functional EBDF possesses a global minimizer N in the charge sector N. The operator N is a solution of the self-consistent equation   8 0 < N D .1; / DN  P   N .x; y/ 1 : DN D D 0 C ˛ N    ˛ jj jx  yj

(4)

Moreover, one can split P D N C P 0 D  C ˘ where

 D Œ0;  .DN /

;

˘ D .1;0/ .DN / :

The “electronic projector”  has rank N and the “Dirac sea” projector ˘ satisfies trP 0 .˘  P 0 / D 0 (neutrality of the vacuum). The Dirac–Fock Model In practice, ˘  P 0 is small, so it is reasonable to replace ˘ by P 0 and N by  in (4) (see [1, 2] for a mathematical justification of this procedure). One gets the Dirac–Fock equation (Swirles [24]) which is widely used in relativistic quantum chemistry:

R

1250

Relativistic Theories for Molecular Models

8   <  D Œ0;  D   .x; y/ 1 ˛ : D D D 0 C ˛     jj jx  yj

in practical calculations. After a Dirac–Fock compu(5) tation, one can perform, for instance, a CI calculation or a multiconfiguration SCF calculation. In such a calculation, pair creation is neglected, and one works with 2 .0; 1/ such that D has exactly N eigenvalues with the so-called no-pair Hamiltonian, which reads: between 0 and . N X X The Dirac–Fock projector  is a critical point, under np  2 D h .r / C ˛ Uij ; (6) H D i the constraints  D  ,  D  , tr. / D N , of the i D1 i 0, and S D ˛ or ˇ, with ˛./ D ı;" and ˇ./ D ı;# . The N b X main advantage of Gaussian type orbitals is that all the ŒG.D/ WD Œ.j/  .j/ D ; integrals in (4)–(6) can be calculated analytically [5]. ;D1

S

1312

Self-Consistent Field (SCF) Algorithms

In order to simplify the notation, we assume in the sequel that the basis .1 ;    ; Nb / is orthonormal, or, equivalently, that S D INb . The molecular orbital and density matrix formulations of the discretized HartreeFock model then read

eigenvalues of F .D/:  D N C1  N > 0 [2, 9]. P  From a geometrical viewpoint, D D N i D1 ˚i ˚i is the matrix of the orthogonal projector on the vector space spanned by the lowest N eigenvalues of F .D/. Note that (13) can be reformulated without any reference to the molecular orbitals ˚i as follows:

˚ ˚ E0HF .N; V/ D min E HF .C C  /; C 2 C ; (10) (14) D 2 argmin Tr .F .D/D 0 /; D 0 2 P ; ˚ C D C 2 CNb N j C  C D IN ; and that, as  D N C1  N > 0, the right-hand side of ˚ E0HF .N; V/ D min E HF .D/; D 2 P ; (11) (14) is a singleton. This formulation is a consequence n of the following property: for any hermitian matrix b Nb P D D 2 CN j D 2 D D; b Nb h F 2 CN with eigenvalues 1      Nb and any h P o  orthogonal projector D of the form D D N i D1 ˚i ˚i Tr .D/ D N :  with ˚i ˚j D ıij and F ˚i D i ˚i , the following holds 8D 0 2 P,

Hartree-Fock-Roothaan-Hall Equations

Tr .FD 0 /  Tr .FD/ C

The matrix F .D/ WD h C G.D/

N C1  N kD  D 0 k2F : 2 (15)

(12)

Roothaan Fixed Point Algorithm

is called the Fock matrix associated with D. It is the gradient of the Hartree-Fock energy functional at D, It is very natural to try and solve (13) by means of the for the Frobenius scalar product. following fixed point algorithm, originally introduced It can be proved (see again [9] for details) that if D by Roothaan in [24]: is a local minimizer of (11), then 8 F .DkRth /˚i;kC1 D i;kC1 ˚i;kC1 8 ˆ N ˆ X ˆ  ˆ ˆ ˚i;kC1 ˚j;kC1 D ıij ˆ  ˆ ˆ ˆ D D ˚ ˚ ˆ i ˆ i ˆ < ˆ  1;kC1  2;kC1      N;kC1 are the lowest ˆ i D1 < (16) N eigenvalues F .DkRth / F .D/˚i D i ˚i ˆ (13) ˆ ˆ N  ˆ ˆ X ˆ ˚i ˚j D ıij ˆ ˆ Rth  ˆ ˆ ˆ D ˚i;kC1 ˚i;kC1 ; DkC1 ˆ : ˆ          are the lowest 1 2 N ˆ : i D1 N eigenvalues F .D/. The above system is a nonlinear eigenvalue problem: the vectors ˚i are orthonormal eigenvectors of the hermitian matrix F .D/ associated with the lowest N eigenvalues of F .D/, and the matrix F .D/ depends on the eigenvectors ˚i through the definition of D. The first three equations of (13) are called the Hartree-Fock-Roothaan-Hall equations. The property that 1 ;    ; N are the lowest N eigenvalues of the hermitian matrix F .D/ is referred to as the Aufbau principle. An interesting property of the Hartree-Fock model is that, for any local minimizer of (11), there is a positive gap between the N th and .N C 1/th

which also reads, in view of (15), ˚ Rth DkC1 2 argmin Tr .F .DkRth /D 0 /; D 0 2 P :

(17)

Solving the nonlinear eigenvalue problem (13) then boils down to solving a sequence of linear eigenvalue problems. It was, however, early realized that the above algorithm often fails to converge. More precisely, it can be proved that, under the assumption that inf .N C1;k  N;k / > 0;

k2N

(18)

Self-Consistent Field (SCF) Algorithms

1313

which seems to be always satisfied in practice, the E.D; D 0 / with the penalized functional E.D; D 0 / C sequence .DkRth /k2N generated by the Roothaan b2 kD  D 0 k2F (b > 0) suppresses the oscillations when algorithm either converges to a solution D of the b is large enough, but the resulting algorithm Hartree-Fock-Roothaan-Hall equations satisfying the ˚ b Aufbau principle DkC1 2 argmin Tr .F .Dkb  bDkb /D 0 /; D 0 2 P often converges toward a critical point of the Hartreewith D satisfying (13); (19) Fock-Roothaan-Hall equations, which does not satisfy k!1 the Aufbau principle, and is, therefore, not a local minimizer. This algorithm, introduced in [25], is called or asymptotically oscillates between two states Deven the level-shifting algorithm. It has been analyzed in and Dodd , none of them being solutions to the Hartree- [6, 17]. Fock-Roothaan-Hall equations kDkRth  Dk ! 0

Rth Rth kD2k Deven k ! 0 and kD2kC1 Dodd k ! 0: k!1

Direct Minimization Methods

k!1

(20) The molecular orbital formulation (10) and the density The behavior of the Roothaan algorithm can be matrix formulation (11) of the discretized Hartreeexplained mathematically, noticing that the sequence Fock model are constrained optimization problems. .DkRth /k2N is obtained by minimizing by relaxation the In both cases, the minimization set is a (non-convex) smooth compact manifold. The set C is a manifold functional of dimension N Nb  12 N.N C 1/, called the Stiefel manifold; the set P of the rank-N orthogonal projecE.D; D 0 / D Tr .hD/ C Tr .hD 0 / C Tr .G.D/D 0 /: tors in CNb is a manifold of dimension N.Nb  N /, called the Grassmann manifold. We refer to [12] for an Indeed, we deduce from ( 7), (12), and (17) that interesting review of optimization methods on Stiefel ˚ and Grassmann manifolds. D1Rth D argmin Tr .F .D0Rth /D 0 /; D 0 2 P From a historical viewpoint, the first minimization ˚ Rth 0 method for solving the Hartree-Fock-Roothaan-Hall D argmin Tr .hD0 / C Tr .hD / equations, the so-called steepest descent method, was CTr .G.D0Rth /D 0 /; D 0 2 P proposed in [20]. It basically consists in performing ˚ one unconstrained gradient step on the function D 7! D argmin E.D0Rth ; D 0 /; D 0 2 P ; e kC1 D Dk trE.Dk / D Dk tF .Dk /), ˚ E.D/ (i.e., D D2Rth D argmin Tr .F .D1Rth /D/; D 2 P e kC1 ! DkC1 2 followed by a “projection” step D ˚ D argmin Tr .hD/ C Tr .hD1Rth / P. The “projection” can be done using McWeeny’s purification, an iterative method consisting in replacing CTr .G.D1Rth /D/; D 0 2 P at each step D with 3D 2 2D 3 . It is easily checked that ˚ e kC1 is close enough to P, the purification method D argmin Tr .hD/ C Tr .hD1Rth / if D converges quadratically to the point of P closest to CTr .G.D/D1Rth /; D 0 2 P e kC1 for the Frobenius norm. The steepest descent D ˚ Rth D argmin E.D; D1 /; D 2 P ; method has the drawback of any basic gradient method: it converges very slowly, and is therefore never used in and so on, and so forth. Together with (15) and (18), practice. Rth  this leads to numerical convergence [6]: kDkC2 Newton-like algorithms for computing HartreeDkRth kF ! 0. The convergence/oscillation properties Fock ground states appeared in the early 1960s with (19)/(20) can then be obtained by resorting to the Bacskay quadratic convergent (QC) method [3]. Łojasiewicz inequality [17]. Bacskay’s approach was to lift the constraints and Oscillations of the Roothaan algorithm are called use a standard Newton algorithm for unconstrained charge sloshing in the physics literature. Replacing optimization. The local maps of the manifold P used

S

1314

Self-Consistent Field (SCF) Algorithms

in [3] are the following exponential maps: for any C 2 CNb Nb such that C  S C D INb ,

P D C e A D0 e A C  ; D0 D  AD





IN 0 ; 0 0 

0 Avo ; Avo 2 C.Nb N /N I Avo 0

the suffix vo denotes the virtual-occupied off-diagonal block of the matrix A. Starting from some reference matrix C, Bacskay QC algorithm performs one Newton step on the unconstrained optimization problem ˚ min E C .Avo / WD E HF .C e A D0 e A C  /; Avo 2 C.Nb N /N ; and updates the reference C by replacing C  matrix  e 0  A A vo ,e Avo denoting the , where e AD e with C ee Avo 0 result of the Newton step. Newton methods being very expensive in terms of computational costs, various attempts have been made to build quasi-Newton versions of Bacskay QC algorithm (see, for e.g., [10, 13]). A natural alternative to Bacskay QC is to use Newton-like algorithms for constrained optimization in order to directly tackle problems (10) or (11) (see, e.g., [26]). Trust-region methods for solving the constrained optimization problem (10) have also been developed by Helgaker and co-workers [27], and independently by Mart´ınez and co-workers [14]. Recently, gradient flow methods for solving (10) [1] and (11) [17] have been introduced and analyzed from a mathematical viewpoint. For molecular systems of moderate size, and when the Hartree-Fock model is discretized in atomic orbital basis sets, direct minimization methods are usually less efficient than the methods based on constraint relaxation or optimal mixing presented in the next two sections.

Lieb Variational Principle and Constraint Relaxation

where 0  D  1 means that 0  ˚  D˚  ˚  ˚ for all ˚ 2 CNb , or equivalently, that all the eigenvalues of D lay in the range Œ0; 1. It is easily seen that the set e is convex. It is in fact the convex hull of the set P. P A fundamental remark is that all the local minimizers of (21) are on P [9]. This is the discretized version of a result by Lieb [18]. It is, therefore, possible to solve the Hartree-Fock model by relaxing the nonconvex constraint D 2 D D into the convex constraint 0  D  1. The orthogonal projection of a given hermitian e for the Frobenius scalar product can matrix D on P be computed by diagonalizing D [8]. The cost of one iteration of the usual projected gradient algorithm [4] is therefore the same at the cost of one iteration of the Roothaan algorithm. A more efficient algorithm, the Optimal Damping Algorithm (ODA), is the following [7]

˚ e k /D 0 /; D 0 2 P DkC1 2 argmin ˚Tr .F .D e D e 2 SegŒD e k ; DkC1  ; e kC1 D argmin E HF .D/; D

˚ e k ; DkC1  D .1  /D e k C DkC1 ;  2 where SegŒD e k and DkC1 . Œ0; 1g denotes the line segment linking D HF As E is a second degree polynomial in the density matrix, the last step consists in minimizing a quadratic function of  on Œ0; 1, which can be done analytically. e 0 D D0 , D0 2 P The procedure is initialized with D being the initial guess. The ODA thus generates two sequences of matrices: • The main sequence of density matrices .Dk /k2N 2 P N which is proven to numerically converge to an Aufbau solution to the Hartree-Fock-Roothaan-Hall equations [9] e k /k1 of matrices belong• A secondary sequence .D e ing to P The Hartree-Fock energy is a Lyapunov functional of ODA: it decays at each iteration. This follows from the fact that for all D 0 2 P and all  2 Œ0; 1, e k C D 0 / D E HF .D e k / C Tr .F .D ek/ E HF ..1  /D 2   e k // C  Tr G.D 0  D e k /.D 0  D ek/ : .D 0  D 2

We now consider the variational problem ˚ e ; min E HF .D/; D 2 P

o n e D D 2 CNb Nb ; 0  D  1; Tr .D/ D N ; P h

(21)

(22)

Self-Consistent Field (SCF) Algorithms

1315

The “steepest descent” direction, that is, the density matrix D for which the slope se D k !D D e k // is minimum, is precisely DkC1 . e k /.D  D Tr .F .D In some sense, ODA is a combination of diagonalization and direct minimization. The practical implementation of ODA is detailed in [7], where numerical tests are also reported. The cost of one ODA iteration is approximatively the same as for the Roothaan algorithm. Numerical tests show that ODA is particularly efficient in the early steps of the iterative procedure.

In the EDIIS algorithm [16], the mixing coefficients are chosen to minimize the Hartree-Fock energy ek: of D 8
k k

 j D1 :j D1 ; F

The commutator ŒF .D/; D is in fact the gradient of the functional A 7! E HF .e A De A / defined on the vector space of the Nb  Nb antihermitian matrices (note that e A De A 2 P for all D 2 P and A antihermitian); it vanishes when D is a critical point of E HF on P.

SCF Algorithms for Kohn-Sham Models After discretization in a finite basis set, the Kohn-Sham energy functional reads 1 E KS .D/ D Tr .hD/ C Tr .J.D/D/ C Exc .D/; 2 P where ŒJ.D/ WD  .j/D is the Coulomb operator, and where Exc is the exchange-correlation energy functional [11]. In the standard Kohn-Sham model [15], E KS is minimized on P, while in the extended Kohn-Sham model [11], E KS is minimized e The algorithms presented in the on the convex set P. previous sections can be transposed mutatis mutandis to the Kohn-Sham setting, but as Exc .D/ is not a second order polynomial in D, the mathematical analysis is more complicated. In particular, no rigorous result on the Roothaan algorithm for Kohn-Sham has been published so far. P P Note that the equality F . i ci Di / D i ci F .Di / P whenever i ci D 1 is true for Hartree-Fock with F .D/ D rE HF .D/ D h C G.D/, but not for KohnSham with F .D/ D rE KS .D/ D h C J.D/ C rExc .D/. Consequently, in contrast with the situation encountered in the Hartree-Fock framework, mixing density matrices and mixing Kohn-Sham Hamiltonians are not equivalent procedures. This leads to a variety

S

1316

of acceleration schemes for Kohn-Sham that boil down to either DIIS or EDIIS in the Hartree-Fock setting. For the sake of brevity, and also because the situation is evolving fast (several new algorithms are proposed every year, and identifying the best algorithms is a matter of debate), we will not present these schemes here. Let us finally mention that, if iterative methods based on repeated diagonalization of the mean-field Hamiltonian, combined with mixing procedures, are more efficient than direct minimization methods for moderate size molecular systems, and when the KohnSham problem is discretized in atomic orbital basis sets, the situation may be different for very large systems, or when finite element or planewave discretization methods are used (see, e.g., [19,21] and references therein).

Cross-References  A Priori and a Posteriori Error Analysis in Chemistry  Density Functional Theory  Hartree–Fock Type Methods  Linear Scaling Methods  Numerical Analysis of Eigenproblems for Electronic

Structure Calculations

References 1. Alouges, F., Audouze, C.: Preconditioned gradient flows and applications to the Hartree-Fock functional. Numer. Methods PDE 25, 380–400 (2009) 2. Bach, V., Lieb, E.H., Loss, M., Solovej, J.P.: There are no unfilled shells in unrestricted Hartree-Fock theory. Phys. Rev. Lett. 72(19), 2981–2983 (1994) 3. Bacskay, G.B.: A quadratically convergent Hartree-Fock (QC-SCF) method. Application closed shell. Syst. Chem. Phys. 61, 385–404 (1961) 4. Bonnans, J.F., Gilbert, J.C., Lemar´echal, C., Sagastiz´abal, C.: Numerical Optimization. Theoretical and Practical Aspects. Springer, Berlin/New York (2006) 5. Boys, S.F.: Electronic wavefunctions. I. A general method of calculation for the stationary states of any molecular system. Proc. R. Soc. A 200, 542–554 (1950) 6. Canc`es, E., Le Bris, C.: On the convergence of SCF algorithms for the Hartree-Fock equations. M2AN Math. Model. Numer. Anal. 34, 749–774 (2000) 7. Canc`es, E., Le Bris, C.: Can we outperform the DIIS approach for electronic structure calculations? Int. J. Quantum Chem. 79, 82–90 (2000)

Self-Consistent Field (SCF) Algorithms 8. Canc`es, E., Pernal, K.: Projected gradient algorithms for Hartree-Fock and density-matrix functional theory. J. Chem. Phys. 128, 134108 (2008) 9. Canc`es, E., Defranceschi, M., Kutzelnigg, W., Le Bris, C., Maday, Y.: Computational quantum chemistry: A primer. In: Handbook of Numerical Analysis, vol. X, pp. 3–270. North-Holland, Amsterdam (2003) 10. Chaban, G., Schmidt, M.W., Gordon, M.S.: Approximate second order method for orbital optimization of SCF and MCSCF wavefunctions. Theor. Chem. Acc. 97, 88–95 (1997) 11. Dreizler, R.M., Gross, E.K.U.: Density Functional Theory. Springer, Berlin/New York (1990) 12. Edelman, A., Arias, T.A., Smith, S.T.: The geometry of algorithms with orthonormality constraints. SIAM J. Matrix Anal. Appl. 20, 303–353 (1998) 13. Fischer, T.H., Alml¨of, J.: General methods for geometry and wave function optimization. J. Phys. Chem. 96, 9768–9774 (1992) 14. Francisco, J., Mart´ınez, J.M., Mart´ınez, L.: Globally convergent trust-region methods for self-consistent field electronic structure calculations, J. Chem. Phys. 121, 10863–10878 (2004) 15. Kohn, K., Sham, L.J.: Self-consistent equations including exchange and correlation effects. Phys. Rev. 140, A1133 (1965) 16. Kudin, K., Scuseria, G.E., Canc`es, E.: A black-box selfconsistent field convergence algorithm: one step closer, J. Chem. Phys. 116, 8255–8261 (2002) 17. Levitt, A.: Convergence of gradient-based algorithms for the Hartree-Fock equations, preprint (2011) 18. Lieb, E.H.: Variational principle for many-fermion systems. Phys. Rev. Lett. 46, 457–459 (1981) 19. Marks, L.D., Luke, D.R.: Robust mixing for ab initio quantum mechanical calculations. Phys. Rev. B 78, 075114 (2008) 20. McWeeny, R.: The density matrix in self-consistent field theory. I. Iterative construction of the density matrix. Proc. R. Soc. Lond. A 235, 496–509 (1956) 21. Mostofi, A.A., Haynes, P.D., Skylaris, C.K., Payne, M.C.: Preconditioned iterative minimization for linear-scaling electronic structure calculations. J. Chem. Phys. 119, 8842–8848 (2003) 22. Pulay, P.: Improved SCF convergence acceleration. J. Comput. Chem. 3, 556–560 (1982) 23. Rohwedder, T., Schneider, R.: An analysis for the DIIS acceleration method used in quantum chemistry calculations. J. Math. Chem. 49, 1889–1914 (2011) 24. Roothaan, C.C.J.: New developments in molecular orbital theory. Rev. Mod. Phys. 23, 69–89 (1951) 25. Saunders, V.R., Hillier, I.H.: A “level-shifting” method for converging closed shell Hartree-Fock wavefunctions. Int. J. Quantum Chem. 7, 699–705 (1973) 26. Shepard, R.: Elimination of the diagonalization bottleneck in parallel direct-SCF methods. Theor. Chim. Acta 84, 343–351 (1993) 27. Thøgersen, L., Olsen, J., Yeager, D., Jørgensen, P., Sałek, P., Helgaker, T.: The trust-region self-consistent field method: towards a black-box optimization in HartreeFock and Kohn-Sham theories. J. Chem. Phys. 121, 16–27 (2004)

Semiconductor Device Problems

Semiconductor Device Problems Ansgar J¨ungel Institut f¨ur Analysis und Scientific Computing, Technische Universit¨at Wien, Wien, Austria

Mathematics Subject Classification 82D37; 35Q20; 35Q40; 35Q79; 76Y05

Definition Highly integrated electric circuits in computer processors mainly consist of semiconductor transistors which amplify and switch electronic signals. Roughly speaking, a semiconductor is a crystalline solid whose conductivity is intermediate between an insulator and a conductor. The modeling and simulation of semiconductor transistors and other devices is of paramount importance in the microelectronics industry to reduce the development cost and time. A semiconductor device problem is defined by the process of deriving physically accurate but computationally feasible model equations and of constructing efficient numerical algorithms for the solution of these equations. Depending on the device structure, size, and operating conditions, the main transport phenomena may be very different, caused by diffusion, drift, scattering, or quantum effects. This leads to a variety of model equations designed for a particular situation or a particular device. Furthermore, often not all available physical information is necessary, and simpler models are needed, helping to reduce the computational cost in the numerical simulation. One may distinguish four model classes: microscopic/mesoscopic and macroscopic semiclassical models and microscopic/mesoscopic and macroscopic quantum models (see Fig. 1).

Description In the following, we detail only some models from the four model classes since the field of semiconductor device problems became extremely large in recent years. For instance, we ignore compact models, hybrid model approaches, lattice heat equations, transport in

1317

subbands and magnetic fields, spintronics, and models for carbon nanotube, graphene, and polymer thin-film materials. For technological aspects, we refer to [9]. Microscopic Semiclassical Models We are interested in the evolution of charge carriers moving in an electric field. Their motion can be modeled by Newton’s law. However, in view of the huge number of electrons involved, the solution of the Newton equations is computationally too expensive. Moreover, we are not interested in the trajectory of each single particle. Hence, a statistical approach seems to be sufficient, introducing the distribution function (or “probability density”) f .x; v; t/ of an electron ensemble, depending on the position x 2 R3 , velocity v D xP D dx=dt 2 R3 , and time t > 0. By Liouville’s theorem, the trajectory of f .x.t/; v.t/; t/ does not change during time, in the absence of collisions, and hence, 0D

df D @t f Cxr P x f Cvr P vf dt

along trajectories; (1)

where @t f D @f =@t and rx f , rv f are gradients with respect to x, v, respectively. Since electrons are quantum particles (and position and velocity cannot be determined with arbitrary accuracy), we need to incorporate some quantum mechanics. As the solution of the many-particle Schr¨odinger equation in the whole space is out of reach, we need an approximate approach. First, by Bloch’s theorem, it is sufficient to solve the Schr¨odinger equation in a semiconductor lattice cell. Furthermore, the many-particle interactions are described by an effective Coulomb force. Finally, the properties of the semiconductor crystal are incorporated by the semiclassical Newton equations. More precisely, let p D „k denote the crystal momentum, where „ is the reduced Planck constant and k is the wave vector. For electrons with low energy, the velocity is proportional to the wave vector, xP D „k=m, where m is the electron mass at rest. In the general case, we have to take into account the energy band structure of the semiconductor crystal (see [4, 7, 8] for details). Newton’s third law is formulated as pP D qrx V , where q is the elementary charge and V .x; t/ is the electric potential. Then, using vP D p=m P and rk D .m=„/rv , (1) becomes the (mesoscopic) Boltzmann transport equation:

S

1318

Semiconductor Device Problems

Microscopic/mesoscopic semi-classical models

Microscopic/mesoscopic quantum models

Semi-classical Newton’s equations

Liouville-von Neumann equation

statistical approach

collisionless

Boltzmann transport equation

Hydrodynamic equations

Schrödinger equation

collisional Wigner-Boltzmann equation

moment method

moment method + Chapman-Enskog

moment method

Lindblad equation

Quantum hydrodynamic equations

Energy-transport equations

moment method + Chapman-Enskog Quantum energytransport equations

constant temperature

constant temperature Quantum driftdiffusion equations

Drift-diffusion equations Macroscopic quantum models

Macroscopic semi-classical models

Semiconductor Device Problems, Fig. 1 Hierarchy of some semiconductor models mentioned in the text

@t f C

„ q k  rx f C rx V  rk f m „

D Q.f /;

.x; k/ 2 R3  R3 ; t > 0;

(2)

alternative is the use of deterministic solvers, for example, expanding the distribution function with spherical harmonics [6].

Macroscopic Semiclassical Models When collisions become dominant in the semiconductor domain, that is, the mean free path (the length which a particle travels between two consecutive collision events) is much smaller than the device size, a Z Z fluid dynamical approach may be appropriate. Macro„ f d k; J D kf d k; nD scopic models are derived from (2) by multiplying m R3 R3 the equation by certain weight functions, that is 1, Z „2 2 ne D jkj2 f d k: (3) k, and jkj =2, and integrating over the wave-vector 2m R3 space. Setting all physical constants to one in the following, for notational simplicity, we obtain, using In the self-consistent setting, the electric potential V the definitions (3), the balance equations: is computed from the Poisson equation "s V D q.nC.x//, where "s is the semiconductor permittivity Z and C.x/ models charged background ions (doping Q.f /d k; x 2 R3 ; t > 0; (4) @t n C divx J D 3 R profile). Since n depends on the distribution function Z Z f , the Boltzmann–Poisson system is nonlinear. @t J C divx k ˝ kf d k  nrx V D kQ.f /d k; The Boltzmann transport equation is defined over R3 R3 (5) the six-dimensional phase space (plus time) whose Z high dimensionality makes its numerical solution a 1 @t .ne/ C divx kjkj2 f d k  rx V  very challenging task. One approach is to employ the 3 2 R Monte Carlo method which consists in simulating a Z 1 stochastic process. Drawbacks of the method are the J D jkj2 Q.f /d k: (6) 2 R3 stochastic nature and the huge computational cost. An where Q.f / models collisions of electrons with phonons, impurities, or other particles. The moments of f are interpreted as the particle density n.x; t/, current density J.x; t/, and energy density .ne/.x; t/:

Semiconductor Device Problems

The higher-order integrals cannot be expressed in terms of the moments (3), which is called the closure problem. It can be solved by approximating f by the equilibrium distribution f0 , which can be justified by a scaling argument and asymptotic analysis. The equilibrium f0 can be determined by maximizing the Boltzmann entropy under the constraints of given moments n, nu, and ne [4]. Inserting f0 in (4)–(6) gives explicit expressions for the higher-order moments, yielding the so-called hydrodynamic model. Formally, there is some similarity with the Euler equations of fluid dynamics, and there has been an extensive discussion in the literature whether electron shock waves in semiconductors are realistic or not [10]. Diffusion models, which do not exhibit shock solutions, can be derived by a Chapman–Enskog expansion around the equilibrium distribution f0 according to f D f0 C ˛f1 , where ˛ > 0 is the Knudsen number (the ratio of the mean free path and the device length) which is assumed to be small compared to one. The function f1 turns out to be the solution of a certain operator equation involving the collision operator Q.f /. Depending on the number of given moments, this leads to the drift-diffusion equations (particle density given):

1319

devices due to, for example, temperature effects, which can be modeled by the energy-transport equations. In the presence of high electric fields, the stationary equations corresponding to (7)–(9) are convection dominant. This can be handled by the Scharfetter– Gummel discretization technique. The key idea is to approximate the current density along each edge in a mesh by a constant, yielding an exponential approximation of the electric potential. This technique is related to mixed finite-element and finite-volume methods [2]. Another idea to eliminate the convective terms is to employ (dual) entropy variables. For instance, for the energy-transport equations, the dual entropy variables are w D .w1 ; w2 / D ..  V /=T; 1=T /, where  is the chemical potential, given by n D T 3=2 exp.=T /. Then (8) and (9) can be formulated as the system: @t b.w/  div.D.w; V /rw/ D 0; where b.w/ D .n; 32 nT /> and D.w; V / 2 R22 is a symmetric positive definite diffusion matrix [4] such that standard finite-element techniques are applicable.

Microscopic Quantum Models The semiclassical approach is reasonable if the carx 2 R3 ; t > 0; (7) riers can be treated as particles. The validity of this description is measured by the de Broglie wavelength or the energy-transport equations (particle and energy B corresponding to a thermal average carrier. When densities given) the electric potential varies rapidly on the scale of B or when the mean free path is much larger than n @t n C divx J D 0; J D rx n C rx V; B , quantum mechanical models are more appropriate. T A general description is possible by the Liouville–von 3 (8) Neumann equation: x 2 R ; t > 0; @t .ne/ C divx S C nu  rx V D 0; i "@tb  D ŒH;b  WD Hb  b H; t > 0; 3 (9) S D  .rx .nT /  nrx V /; 2 for the density matrix operator b , where i 2 D 1, 3 where ne D 2 nT , T being the electron temperature, " > 0 is the scaled Planck constant, and H is the and S is the heat flux. For the derivation of these quantum mechanical Hamiltonian. The operator b  is models; we have assumed that the equilibrium distri- assumed to possess a complete orthonormal set of bution is given by Maxwell–Boltzmann statistics and eigenfunctions . j / and eigenvalues .j /. The sethat the elastic scattering rate is proportional to the quence of Schr¨odinger equations i "@t j D H j wave vector. More general models can be derived too, (j 2 N), together with the numbers j  0, is called a mixed-state Schr¨ odinger system with the particle see [4, Chap. 6]. P 1 2 The drift-diffusion model gives a good description density n.x; t/ D j D1 j j j .x; t/j . In particular, of the transport in semiconductor devices close to j can be interpreted as the occupation probability of equilibrium but it is not accurate enough for submicron the state j . @t n C divx J D 0;

J D rx n C nrx V;

S

1320

Semiconductor Device Problems

The Schr¨odinger equation describes the evolution of a quantum state in an active region of a semiconductor device. It is used when inelastic scattering is sufficiently weak such that phase coherence can be assumed and effects such as resonant tunneling and quantum conductance can be observed. Typically, the device is connected to an exterior medium through access zones, which allows for the injection of charge carriers. Instead of solving the Schr¨odinger equation in the whole domain (self-consistently coupled to the Poisson equation), one wishes to solve the problem only in the active region and to prescribe transparent boundary conditions at the interfaces between the active and access zones. Such a situation is referred to as an open quantum system. The determination of transparent boundary conditions is a delicate issue since ad hoc approaches often lead to spurious oscillations which deteriorate the numerical solution [1]. Nonreversible interactions of the charge carriers with the environment can be modeled by the Lindblad equation:

Macroscopic Quantum Models Macroscopic models can be derived from the Wigner– Boltzmann equation (10) in a similar manner as from the Boltzmann equation (2). The main difference to the semiclassical approach is the definition of the equilibrium. Maximizing the von Neumann entropy under the constraints of given moments of a Wigner function w, the formal solution (if it exists) is given by the socalled quantum Maxwellian M Œw, which is a nonlocal version of the semiclassical equilibrium. It was first suggested by Degond and Ringhofer and is related to the (unconstrained) quantum equilibrium given by Wigner in 1932 [3, 5]. We wish to derive moment equations from the Wigner–Boltzmann equation (10) for the particle density n, current density J , and energy density ne, defined by: Z

Z M Œwdp;

nD

J D

R3

ne D

1 2

pM Œwdp; R3

Z

jpj2 M Œwdp: R3

 X

1 Lkb Lk  .Lk Lkb Cb Lk Lk / ; Such a program was carried out by Degond et al. [3], 2 using the simple relaxation-type operator Q.w/ D k M Œw  w. This leads to a hierarchy of quantum hydrowhere Lk are the so-called Lindblad operators and Lk dynamic and diffusion models which are, in contrast to is the adjoint of Lk . In the Fourier picture, this equation their semiclassical counterparts, nonlocal. When we employ only one moment (the particle can be formulated as a quantum kinetic equation, the density) and expand the resulting moment model in (mesoscopic) Wigner–Boltzmann equation: powers of " up to order O."4 / (to obtain local equations), we arrive at the quantum drift-diffusion (or @t w C p  rx w C ŒV w D Q.w/; density-gradient) equations:

DŒH;b Ci i "@tb

.x; p/ 2 R3  R3 ; t > 0;

(10) @t n C divx J D 0;

where p is the crystal momentum, ŒV w is the potential operator which is a nonlocal version of the drift term rx V rp w [4, Chap. 11], and Q.w/ is the collision operator. The Wigner function w D W Œb  , where W denotes the Wigner transform, is essentially the Fourier transform of the density matrix. A nice feature of the Wigner equation is that it is a phase-space description, similar to the semiclassical Boltzmann equation. Its drawbacks are that the Wigner function cannot be interpreted as a probability density, as the Boltzmann distribution function, and that the Wigner equation has to be solved in the high dimensional phase space. A remedy is to derive macroscopic models which are discussed in the following section.

J D rx n C nrx V C

 pn  x ; p n

x 2 R3 ;

"2 nrx 6 t>0:

This model is employed to simulate the carrier inversion layer near the oxide of a MOSFET (metaloxide-semiconductor field-effect transistor). The main difficulty of the numerical discretization is the treatment of the highly nonlinear fourth-order quantum correction. However, there exist efficient exponentially fitted finite-element approximations, see the references of Pinnau in [4, Chap. 12]. Formally, the moment equations for the charge carriers and energy density give the quantum

Shearlets

1321

energy-transport model. Since its mathematical structure is less clear, we do not discuss this model [4, Chap. 13.2]. Employing all three moments n, nu, ne, the moment equations, expanded up to terms of order O."4 /, become the quantum hydrodynamic equations:

4.



J ˝ J CP @t n C div J D 0; @t J C divx n Z C nrx V D  pQ.M Œw/dp;

7.

5. 6.

8.

R3

@t .ne/  divx ..P C neI/u  q/ C rx V  Z 1 jpj2 Q.M Œw/dp; x 2 R3 ; t > 0; J D 2 R3 where I is the identity matrix in R33 , the quantum stress tensor P and the energy density ne are given by: P D nT I 

"2 nr 2 log n; 12 x

ne D

3 nT 2

9. 10.

Quantum Transport: Modelling, Analysis and Asymptotics. Lecture Notes in Mathematics 1946, pp. 111–168. Springer, Berlin (2009) J¨ungel, A.: Transport Equations for Semiconductors. Springer, Berlin (2009) J¨ungel, A.: Dissipative quantum fluid models. Revista Mat. Univ. Parma, to appear (2011) Hong, S.-M., Pham, A.-T., Jungemann, C.: Deterministic Solvers for the Boltzmann Transport Equation. Springer, New York (2011) Lundstrom, M.: Fundamentals of Carrier Transport. Cambridge University Press, Cambridge (2000) Markowich, P., Ringhofer, C., Schmeiser, C.: Semiconductor Equations. Springer, Vienn (1990) Mishra, U., Singh, J.: Semiconductor Device Physics and Design. Springer, Dordrecht (2007) Rudan, M., Gnudi, A., Quade, W.: A generalized approach to the hydrodynamic model of semiconductor equations. In: Baccarani, G. (ed.) Process and Device Modeling for Microelectronics, pp. 109–154. Elsevier, Amsterdam (1993)

Shearlets

1 "2 C njuj2  nx log n; 2 24

Gitta Kutyniok Institut f¨ur Mathematik, Technische Universit¨at u D J =n is the mean velocity, and q D Berlin, Berlin, Germany ."2 =24/n.x u C 2rx divx u/ is the quantum heat flux. When applying a Chapman–Enskog expansion around the quantum equilibrium, viscous effects are Mathematics Subject Classification added, leading to quantum Navier–Stokes equations [5, Chap. 5]. These models are very interesting from 42C40; 42C15; 65T60 a theoretical viewpoint since they exhibit a surprising nonlinear structure. Simulations of resonant tunneling diodes using these models give qualitatively reasonable results. However, as expected, quantum phenomena are Synonyms easily destroyed by the occurring diffusive or viscous Shearlets; Shearlet system effects.

References 1. Antoine, X., Arnold, A., Besse, C., Ehrhardt, M., Sch¨adle, A.: A review of transparent and artificial boundary conditions techniques for linear and nonlinear Schr¨odinger equations. Commun. Comput. Phys. 4, 729–796 (2008) 2. Brezzi, F., Marini, L., Micheletti, S., Pietra, P., Sacco, R., Wang, S.: Discretization of semiconductor device problems. In: Schilders, W., ter Maten, W. (eds.) Handbook of Numerical Analysis. Numerical Methods in Electromagnetics, vol. 13, pp. 317–441. North-Holland, Amsterdam (2005) 3. Degond, P., Gallego, S., M´ehats, F., Ringhofer, C.: Quantum hydrodynamic and diffusion models derived from the entropy principle. In: Ben Abdallah, N., Frosali, G. (eds.)

Short Description Shearlets are multiscale systems in L2 .R2 / which efficiently encode anisotropic features. They extend the framework of wavelets and are constructed by parabolic scaling, shearing, and translation applied to one or very few generating functions. The main application area of shearlets is imaging science, for example, denoising, edge detection, or inpainting. Extensions of shearlet systems to L2 .Rn /, n  3 are also available.

S

1322

Shearlets

The continuous shearlet transform is an isometry, provided that satisfies some weak regularity conditions. One common class of generating functions are clasMultivariate Problems Multivariate problem classes are typically governed by sical shearlets, which are band-limited functions 2 anisotropic features such as singularities concentrated L2 .R2 / defined by on lower dimensional embedded manifolds. Examples

 are edges in images or shock fronts of transport domO . / D O . 1 ; 2 / D O 1 . 1 / O 2 2 ; 1 inated equations. Since due to their isotropic nature wavelets are deficient to efficiently encode such func2 1 2 L .R/ is a discrete wavelet, i.e., it tions, several directional representation systems were where P O j 2 proposed among which are ridgelets, contourlets, and satisfies j 2Z j 1 .2 /j D 1 for a.e. 2 R with O 1 2 C 1 .R/ and supp O 1  Œ 1 ;  1  [ Œ 1 ; 1 , curvelets. 2 16 16 2 2 Shearlets were introduced in 2006 [10] and are and P2 2 L .R/ is a “bump function” in the sense 1 2 to date the only directional representation system that kD1 j O 2 . C k/j D 1 for a.e. 2 Œ1; 1 1 which provides optimally sparse approximations with O 2 2 C .R/ and supp O 2  Œ1; 1. Figure 1 of anisotropic features while providing a unified illustrates classical shearlets and the tiling of Fourier treatment of the continuum and digital realm in domain they provide, which ensures their directional the sense of allowing faithful implementations. One sensitivity. From a mathematical standpoint, continuous shearimportant structural property is their membership in the class of affine systems, similar to wavelets. lets are being generated by a unitary representation of A comprehensive presentation of the theory and a particular semi-direct product, the shearlet group [2]. However, since those systems and their associated applications of shearlets can be found in [16]. transforms do not provide a uniform resolution of all directions but are biased towards one axis, for Continuous Shearlet Systems Continuous shearlet systems are generated by applica- applications cone-adapted continuous shearlet systems Q 2 L2 .R2 /, the conetion of parabolic scaling Aa , AQa , a > 0, shearing Ss , were introduced. For ; ; adapted continuous shearlet system SHcont . ; ; Q / is s 2 R and translation, where defined by   1=2   a 0 a 0 Aa D ; ; AQa D SHcont . ; ; Q / D ˚cont . / [ cont . / [ Q cont . Q /; 0 a1=2 0 a   1s where ; and Ss D 01

Description

to one or very few generating functions. For 2 L2 .R2 /, the associated continuous shearlet system is defined by

˚cont . / D f t D .  t/ W t 2 R2 g; cont . / D f

3

a;s;t

1 D a 4 .A1 a Ss .   t//

W a 2 .0; 1; jsj  1 C a1=2 ; t 2 R2 g; n

3

a;s;t

1 D a 4 .A1 a Ss .   t// W a > 0; s 2 R; o t 2 R2 ;

3 T Q cont . Q / D f Q a;s;t D a 4 Q .AQ1 a Ss .   t//

W a 2 .0; 1; jsj  1 C a1=2 ; t 2 R2 g:

The associated transform is defined in a similar manwith a determining the scale, s the direction, and t the ner as before. The induced uniform resolution of all position of a shearlet a;s;t . The associated continuous directions by a cone-like partition of Fourier domain is shearlet transform of some function f 2 L2 .R2 / is the illustrated in Fig. 2. mapping The high directional selectivity of cone-adapted continuous shearlet systems is reflected in the L2 .R2 / 3 f 7! hf; a;s;t i; a > 0; s 2 R; t 2 R2 : result that they precisely resolve wavefront sets of

Shearlets

1323

c a b

Shearlets, Fig. 1 Classical shearlets: (a) j O a;s;t for different values of a and s

a;s;t j

for exemplary values of a; s, and t . (b) Support of O . (c) Approximate support of

distributions f by the decay behavior of jhf; and jhf; Q a;s;t ij as a ! 0 [15].

a;s;t ij

Discrete Shearlet Systems Discretization of the parameters a; s, and t by a D S 1 m, 2j , j 2 Z, s D k2j=2 , k 2 Z, and t D A1 2j k 2 m 2 Z leads to the associated discrete systems. For 2 L2 .R2 /, the discrete shearlet system is defined by f

3

j;k;m

D 2 4 j .Sk A2j  m/ W j; k 2 Z; m 2 Z2 g;

with j determining the scale, k the direction, and m the position of a shearlet j;k;m . The associated discrete shearlet transform of some f 2 L2 .R2 / is the mapping L2 .R2 / 3 f 7! hf;

j;k;m i;

j; k 2 Z; m 2 Z2 :

Similarly, for ; ; Q 2 L2 .R2 /, the cone-adapted discrete shearlet system SHdisc . ; ; Q / is defined by SHdisc . ; ; Q / D ˚disc . / [ disc . / [ Q disc . Q /; where ˚disc . / D f m D .  m/ W m 2 Z2 g; disc . / D f

3

j;k;m

D 2 4 j .Sk A2j  m/

W j  0; jkj  d2j=2 e; m 2 Z2 g; Q disc . Q / D f Q j;k;m D 2

3 4j

Q .SkT AQ2j

To allow more flexibility in the denseness of the positioning of shearlets, sometimes the discretization of the translation parameter t is performed by t D S 1 diag.c1 ; c2 /m, m 2 Z2 ; c1 ; c2 > 0. A very A1 2j k general discretization approach is by coorbit theory which is however only applicable to the non-coneadapted setting [4]. For classical (band-limited) shearlets as generating functions, both the discrete shearlet system and the cone-adapted discrete shearlet system – the latter one with a minor adaption at the intersections of the cones and suitable – form tight frames for L2 .R2 /. A theory for compactly supported (cone-adapted) discrete shearlet systems is also available [14]. For a special class of separable generating functions, compactly supported cone-adapted discrete shearlet systems form a frame with the ratio of frame bounds being approximately 4. Discrete shearlet systems provide optimally sparse approximations of anisotropic features. A customarily employed model are cartoon-like functions, i.e., compactly supported functions in L2 .R2 / which are C 2 apart from a closed piecewise C 2 discontinuity curve. Up to a log-factor, discrete shearlet systems based on classical shearlets or compactly supported shearlets satisfying some weak regularity conditions provide the optimal decay rate of the best N -term approximation of cartoon-like functions f [7, 17], i.e., kf  fN k22  C N 2 .log N /3

as N ! 1;

 m/

W j  0; jkj  d2j=2 e; m 2 Z2 g:

where here fN denotes the N -term shearlet approximation using the N largest coefficients.

S

1324

Shearlets

b

a

c

1 0.8 0.6

1

0.4

0.5

0.2

0

0 1

–0.5 0.5

0

b

–0.5

–1 –1

Shearlets, Fig. 2 Cone-adapted shearlet system: (a) Partitioning into cones. (b) Approximate support of O a;s;t and Q a;s;t for

different values of a and s. (c) j O a;s;t j for some shearlet exemplary values of a; s, and t

Fast Algorithms The implementations of the shearlet transform can be grouped into two categories, namely, in Fourierbased implementations and in implementations in spatial domain. Fourier-based implementations aim to produce the same frequency tiling as in Fig. 2b typically by employing variants of the Pseudo-Polar transform [5, 21]. Spatial domain approaches utilize filters associated with the transform which are implemented by a convolution in the spatial domain. A fast implementation with separable shearlets was introduced in [22], subdivision schemes are the basis of the algorithmic approach in [19], and a general filter approach was studied in [11].

Several of the associated algorithms are provided at www.ShearLab.org.

and

Extensions to Higher Dimensions Continuous shearlet systems in higher dimensions have been introduced in [3]. In many situations, these systems inherit the property to resolve wavefront sets. The theory of discrete shearlet systems and their sparse approximation properties have been introduced and studied in [20] in dimension 3 with the possibility to extend the results to higher dimensions, and similar sparse approximation properties were derived.

Shift-Invariant Approximation

Applications Shearlets are nowadays used for a variety of applications which require the representation and processing of multivariate data such as imaging sciences. Prominent examples are deconvolution [23], denoising [6], edge detection [9], inpainting [13], segmentation [12], and separation [18]. Other application areas are sparse decompositions of operators such as the Radon operator [1] or Fourier integral operators [8].

References 1. Colonna, F., Easley, G., Guo, K., Labate, D.: Radon transform inversion using the shearlet representation. Appl. Comput. Harmon. Anal. 29, 232–250 (2010) 2. Dahlke, S., Kutyniok, G., Maass, P., Sagiv, C., Stark, H.-G., Teschke, G.: The uncertainty principle associated with the continuous shearlet transform. Int. J. Wavelets Multiresolut. Inf. Process. 6, 157–181 (2008) 3. Dahlke, S., Steidl, G., Teschke, G.: The continuous shearlet transform in arbitrary space dimensions. J. Fourier Anal. Appl. 16, 340–364 (2010) 4. Dahlke, S., Steidl, G., Teschke, G.: Shearlet coorbit spaces: compactly supported analyzing shearlets, traces and embeddings. J. Fourier Anal. Appl. 17, 1232–1255 (2011) 5. Easley, G., Labate, D., Lim, W-Q.: Sparse directional image representations using the discrete shearlet transform. Appl. Comput. Harmon. Anal. 25, 25–46 (2008) 6. Easley, G., Labate, D., Colonna, F.: Shearlet based total Variation for denoising. IEEE Trans. Image Process. 18, 260–268 (2009) 7. Guo, K., Labate, D.: Optimally sparse multidimensional representation using shearlets. SIAM J. Math. Anal. 39, 298–318 (2007) 8. Guo, K., Labate, D.: Representation of Fourier integral operators using shearlets. J. Fourier Anal. Appl. 14, 327–371 (2008) 9. Guo, K., Labate, D.: Characterization and analysis of edges using the continuous shearlet transform. SIAM J. Imaging Sci. 2, 959–986 (2009) 10. Guo, K., Kutyniok, G., Labate, D.: Sparse multidimensional representations using anisotropic dilation and shear operators. In: Wavelets and Splines (Athens, 2005), pp. 189–201. Nashboro Press, Nashville (2006) 11. Han, B., Kutyniok, G., Shen, Z.: Adaptive multiresolution analysis structures and shearlet systems. SIAM J. Numer. Anal. 49, 1921–1946 (2011) 12. H¨auser, S., Steidl, G.: Convex multiclass segmentation with shearlet regularization. Int. J. Comput. Math. 90, 62–81 (2013) 13. King, E.J., Kutyniok, G., Zhuang, X.: Analysis of Inpainting via Clustered Sparsity and Microlocal Analysis. J. Math. Imaging Vis. (to appear) 14. Kittipoom, P., Kutyniok, G., Lim, W-Q.: Construction of compactly supported shearlet frames. Constr. Approx. 35, 21–72 (2012)

1325 15. Kutyniok, G., Labate, D.: Resolution of the wavefront set using continuous shearlets. Trans. Am. Math. Soc. 361, 2719–2754 (2009) 16. Kutyniok, G., Labate, D. (eds.): Shearlets: Multiscale Analysis for Multivariate Data. Birkh¨auser, Boston (2012) 17. Kutyniok, G., Lim, W-Q.: Compactly supported shearlets are optimally sparse. J. Approx. Theory 163, 1564–1589 (2011) 18. Kutyniok, G., Lim, W-Q.: Image separation using wavelets and shearlets. In: Curves and Surfaces (Avignon, 2010). Lecture Notes in Computer Science, vol. 6920. Springer, Berlin, Heidelberg (2012) 19. Kutyniok, G., Sauer, T.: Adaptive directional subdivision schemes and shearlet multiresolution analysis. SIAM J. Math. Anal. 41, 1436–1471 (2009) 20. Kutyniok, G., Lemvig, J., Lim, W-Q.: Optimally sparse approximations of 3D functions by compactly supported shearlet frames. SIAM J. Math. Anal. 44, 2962–3017 (2012) 21. Kutyniok, G., Shahram, M., Zhuang, X.: ShearLab: a rational design of a digital parabolic scaling algorithm. SIAM J. Imaging Sci. 5, 1291–1332 (2012) 22. Lim, W-Q.: The discrete shearlet transform: a new directional transform and compactly supported shearlet frames. IEEE Trans. Image Process. 19, 1166–1180 (2010) 23. Patel, V.M., Easley, G., Healy, D.M.: Shearlet-based deconvolution. IEEE Trans. Image Process. 18, 2673–2685 (2009)

Shift-Invariant Approximation Robert Schaback Institut f¨ur Numerische und Angewandte Mathematik (NAM), Georg-August-Universit¨at G¨ottingen, G¨ottingen, Germany

Mathematics Subject Classification 41A25; 42C15; 42C25; 42C40

Synonyms Approximation by integer translates

Short Definition Shift-invariant approximation deals with functions f on the whole real line, e.g., time series and signals.

S

1326

Shift-Invariant Approximation

It approximates f by shifted copies of a single gen- the coefficients in (1) should be obtained instead as ck;h .f / D .f; '. h  k//2 for any f 2 L2 .IR/ to erator ', i.e., turn the approximation into an optimal L2 projection. 

x X  k ; x 2 IR: Surprisingly, these two approaches coincide for the f .x/ Sf;h;' .x/ WD ck;h .f /' h sinc case. k2Z Z (1) Analysis of shift-invariant approximation focuses   The functions ' h  k for k 2 ZZ span a space that on the error in (1) for various generators ' and for difis shift-invariant wrt. integer multiples of h. Exten- ferent ways of calculating useful coefficients ck;h .f /. sions [1, 2] allow multiple generators and multivariate Under special technical conditions, e.g., if the generfunctions. Shift-invariant approximation uses only a ator ' is compactly supported, the Strang–Fix condisingle scale h, while wavelets use multiple scales and tions [4] refinable generators.

'O .j /.2k/ D ı0k ; k 2 ZZ; 0  j < m

imply that the error of (1) is O.hm / for h ! 0 in Sobolev space W2m .IR/ if the coefficients are given via Nyquist–Shannon–Whittaker–Kotelnikov sampling L2 projection. This holds for B-spline generators of provides the formula order m. The basic tool for analysis of shift-invariant L2 

x X k f .x/ D f .kh/ sinc approximation is the bracket product h k2Z Z X Œ'; .!/ WD '.! O C 2k/ O .! C 2k/; ! 2 IR for band-limited functions with frequencies in k2Z Z Œ= h; C= h. It is basic in Electrical Engineering for AD/DA conversion of signals after low-pass filtering. which is a 2-periodic function. It should exist pointAnother simple example arises from the hat function wise, be in L2 Œ;  and satisfy a stability property or order 2 B-spline B2 .x/ WD 1  jxj for 1  x  1 and 0 elsewhere. Then the “connect-the-dots” formula 0 < A  Œ'; '.!/  B; ! 2 IR:

x  X k f .kh/B2 f .x/ Then the L2 projector for h D 1 has the convenient h k2Z Z Fourier transform

Description

is a piecewise linear approximation of f by connecting Œf; '.!/ '.!/; O ! 2 IR; SOf;1;' .!/ D the values f .kh/ by straight lines. These two examples Œ'; '.!/ arise from a generator ' satisfying the cardinal interpolation conditions '.k/ D ı0k ; k 2 ZZ, and then the and if Œ'; '.!/ D 1=2 for all !, the integer shifts right-hand side of the above formulas interpolates f at '.  k/ for k 2 ZZ are orthonormal in L2 .IR/. all integers. If the generator is a higher-order B-spline Fundamental results on shift-invariant approximaBm , the approximation tion are in [1, 2], and the survey [3] gives a comprehensive account of the theory and the historical

x  X background. k f .x/ f .kh/Bm h k2Z Z

goes back to I.J. Schoenberg and is not interpolatory in References general. So far, these examples of (1) have very special 1. de Boor, C., DeVore, R., Ron, A.: Approximation from shift– invariant subspaces of L2 .Rd /. Trans. Am. Math. Soc. 341, coefficients ck;h .f / D f .kh/ arising from sampling 787–806 (1994) the function f at data locations hZZ. This connects 2. de Boor, C., DeVore, R., Ron, A.: The structure of finitely shift-invariant approximation to sampling theory. If generated shift–invariant spaces in L2 .Rd /. J. Funct. Anal. 19, 37–78 (1994) the shifts of the generator are orthonormal in L2 .IR/,

Simulation of Stochastic Differential Equations

1327

stochastic processes, stochastic numerical methods are derived from these representations. We can distinguish several classes of stochastic numerical methods: Monte Carlo methods consist in simulating large numbers of independent paths of a given stochastic process; stochastic particle methods consist in simulating paths of interacting particles whose empirical distribution converges in law to a deterministic measure; and ergodic methods consist in simulating one single path of a given stochastic process up to a Simulation of Stochastic Differential large time horizon. Monte Carlo methods allow one Equations to approximate statistics of probability distributions of stochastic models or solutions to linear partial differenDenis Talay tial equations. Stochastic particle methods approximate INRIA Sophia Antipolis, Valbonne, France solutions to deterministic nonlinear McKean–Vlasov PDEs. Ergodic methods aim to compute statistics of Synonyms equilibrium measures of stochastic models or to solve elliptic PDEs. See, e.g., [2]. Numerical mathematics; Stochastic analysis; StochasIn all cases, one needs to develop numerical aptic numerics proximation methods for paths of stochastic processes. Most of the stochastic processes used as models or involved in stochastic representations of PDEs are obDefinition tained as solutions to stochastic differential equations Z t Z t The development and the mathematical analysis of b.Xs .x// ds C .Xs .x// dW s ; Xt .x/ D x C stochastic numerical methods to obtain approximate 0 0 (1) solutions of deterministic linear and nonlinear par3. Jetter, K., Plonka, G.: A survey on L2 -approximation orders from shift-invariant spaces. In: Multivariate approximation and applications, pp. 73–111. Cambridge University Press, Cambridge (2001) 4. Strang, G., Fix, G.: A Fourier analysis of the finite element variational method. In: Geymonat, G. (ed.) Constructive Aspects of Functional Analysis. C.I.M.E. II Ciclo 1971, pp 793–840 (1973)

tial differential equations and to simulate stochastic where .Wt / is a standard Brownian motion or, more models. generally, a L´evy process. Existence and uniqueness of solutions, in strong and weak senses, are exhaustively studied, e.g., in [10].

Overview Owing to powerful computers, one now desires to model and simulate more and more complex physical, chemical, biological, and economic phenomena at various scales. In this context, stochastic models are intensively used because calibration errors cannot be avoided, physical laws are imperfectly known (as in turbulent fluid mechanics), or no physical law exists (as in finance). One then needs to compute moments or more complex statistics of the probability distributions of the stochastic processes involved in the models. A stochastic process is a collection .Xt / of random variables indexed by the time variable t. This is not the only motivation to develop stochastic simulations. As solutions of a wide family of complex deterministic partial differential equations (PDEs) can be represented as expectations of functionals of

Monte Carlo Methods for Linear PDEs Set a.x/ WD .x/ .x/t , and consider the parabolic PDE d d X 1 X i @u .t; x/ D b i .x/ @i u.x/ C a .x/ @ij u.x/ @t 2 i;j D1 j i D1

(2) with initial condition u.0; x/ D f .x/. Under various hypotheses on the coefficients b and , it holds that u.t; x/ D Eu0 .t; Xt .x//, where Xt .x/ is the solution to (1). Let h > 0 be a time discretization step. Let .Gp / be independent centered Gaussian vectors with unit covariance matrix. Define the Euler scheme by XN 0h .x/ D x and the recursive relation

S

1328

Simulation of Stochastic Differential Equations h h h XN .pC1/h .x/ DXNph .x/ C b XN ph .x/ h p h .x// h GpC1 : C .XNph

The simulation of this random sequence only requires the simulation of independent Gaussian random variables. Given a time horizon M h, independent copies of the sequence .Gp ; 1  p  M / provide independent h;k .x/; 1  p  M /. paths .XNph The global error of the Monte Carlo method with N simulations which approximates u.ph; x/ is N 1 X N h;k  u0 XMh N kD1  h  D Eu0 .XMh /  Eu0 XN Mh „ ƒ‚ …

Eu0 .XMh / 

Z

DWd .h/

N X

 h  1  C Eu0 XNMh N kD1 ƒ‚ „ DWs .h;N /

European option prices, moments of solutions to mechanical systems with random excitations, etc. When the PDE (2) is posed in a domain D with Dirichlet boundary conditions u.t; x/ D g.x/ on @D, then u.t; x/ D Ef .Xt .x// It < C Eg.X .x// It  , where is the first hitting time of @D by .Xt .x//. An approximation method is obtained by substituting h .x/ to X .x/, where N h is the first hitting XN ph^N

h time of @D by the interpolated Euler scheme. For a convergence rate analysis, see, e.g., [7]. Let n.x/ denote the unit inward normal vector at point x on @D. When one adds Neumann boundary conditions ru.t; x/  n.x/ D 0 on @D to (2), then ] u.t; x/ D Ef .Xt .x//, where X ] := is the solution to an SDE with reflection

 h;k u0 XN Mh :

] Xt .x/

Dx C 0

Z

b.Xs] .x//

t

ds C 0

.Xs] .x// dW s

tn

C

…

Z

t

0

.Xs /dL]s .X /;

Nonasymptotic variants of the central limit theorem where .Lt .X // is the local time of X at the boundary. Then one constructs the reflected Euler scheme in such imply that the statistical error s .h/ satisfies a way that the simulation of the local time, which C.M / would be complex and numerically instable, is avoided. for all 8M  1; 9C.M / > 0; Ejs .h/j  p This construction and the corresponding error analysis N have been developed in [4]. 0 < h < 1: Local times also appear in SDEs related to PDEs with transmission conditions along the discontinuity Using estimates on the solution to the PDE (2) obmanifolds of the coefficient a.x/ as in the Poisson– tained by PDE analysis or stochastic analysis (stochasBoltzmann equation in molecular dynamics, Darcy tic flows theory, Malliavin calculus), one can prove law in fluid mechanics, etc. Specific numerical meththat the discretization error ed .h/ satisfies the so-called ods and error analyses were recently developed: see, Talay–Tubaro expansion e.g., [5]. Elliptic PDEs are interpreted by means of solutions ed .h/ D C.T; x/ h C Qh .f; T; x/ h2 ; to SDEs integrated from time 0 up to infinity or their equilibrium measures. Implicit Euler schemes often where jC.T; x/j C suph jQh .u0 ; T; x/j depend on b, are necessary to get stability: see [12]. An alternative , u0 , and T . Therefore, Romberg extrapolation tech- efficient methods are those with decreasing stepsizes niques can be used: introduced in [11]. ( ) N N 2 X N h=2;k  1 X N h;k   u0 XMh u0 XMh E D O.h2 /: N N Stochastic Particle Methods for Nonlinear kD1

kD1

PDEs For surveys of results in this direction and various extensions, see [8, 14, 17]. Consider the following stochastic particle system. The preceding statistical and discretization error The dynamics of the ith particle is as follows: .i / estimates have many applications: computations of given N independent Brownian motions .Wt /,

Simulation of Stochastic Differential Equations

1329

.i / multidimensional coefficients B and S , and McKean Note that the processes Xt are not independent. .i / interaction kernels b and , the positions Xt solve However, the propagation of chaos and nonlinear marthe stochastic differential system tingale problems theories developed in a seminal way by McKean and Sznitman allow one to prove that 1 0 N 

X the probability distribution of the particles empirical .i / .i / .i / .j / dX t DB @t; Xt ; N1 b Xt ; Xt A dt measure process converges weakly when N goes to j D1 infinity. The limit distribution is concentrated at the 1 0 probability law of the process .Xt / solution to the N 

X .i / .i / .j / .i / 1 A @  Xt ; Xt dW t : following stochastic differential equation which is nonC S t; Xt ; N linear in McKean’s sense (its coefficients depend on the j D1 (3) probability distribution of the solution):

(

R R dX t D B.t; Xt ; b.Xt ; y/t .dy//dt C S.t; Xt ; .Xt ; y/t .dy//dW t ; t .dy/ WD probability distribution of Xt :

In addition, the flow of the probability distributions t solves the nonlinear McKean–Vlasov–Fokker–Planck equation d t D Lt t ; (5) dt where, A denoting the matrix S  S t , L is the formal adjoint of the differential operator L WD

X

R Bk .t; x; b.x; y/.dy//@k

k

C

1 2

X

R Ajk .t; x; .x; y/.dy//@jk :

(4)

approach is numerically relevant in the cases of small viscosities. It is also intensively used, for example, in Lagrangian stochastic simulations of complex flows and in molecular dynamics: see, e.g., [9, 15]. When the functions B, S , b,  are smooth, optimal convergence rates have been obtained for finite time horizons, e.g., in [1, 3]. Other stochastic representations have been developed for backward SDEs related to quasi-linear PDEs and variational inequalities. See the survey [6].

(6)

j;k

From an analytical point of view, the SDEs (4) provide probabilistic interpretations for macroscopic equations which includes, e.g., smoothened versions of the Navier–Stokes and Boltzmann equations: see, e.g., the survey [16] and [13]. From a numerical point of view, whereas the time discretization of .Xt / does not lead to an algorithm since t is unknown, the Euler scheme for the par.i / ticle system f.Xt /; i D 1; : : : ; N g can be simulated: the solution t to (5) is approximated by the empirical distribution of the simulated particles at time t, the number N of the particles being chosen large enough. Compared to the numerical resolution of the McKean–Vlasov–Fokker–Planck equation by deterministic methods, this stochastic numerical

References 1. Antonelli, F., Kohatsu-Higa, A.: Rate of convergence of a particle method to the solution of the McKean-Vlasov equation. Ann. Appl. Probab. 12, 423–476 (2002) 2. Asmussen, S., Glynn, P.W.: Stochastic Simulation: Algorithms and Analysis. Stochastic Modelling and Applied Probability Series, vol. 57. Springer, New York (2007) 3. Bossy, M.: Optimal rate of convergence of a stochastic particle method to solutions of 1D viscous scalar conservation laws. Math. Comput. 73(246), 777–812 (2004) ´ Talay, D.: A symmetrized Euler 4. Bossy, M., Gobet, E., scheme for an efficient approximation of reflected diffusions. J. Appl. Probab. 41(3), 877–889 (2004) 5. Bossy, M., Champagnat, N., Maire, S., Talay D.: Probabilistic interpretation and random walk on spheres algorithms for the Poisson-Boltzmann equation in Molecular Dynamics. ESAIM:M2AN 44(5), 997–1048 (2010) 6. Bouchard, B., Elie, R., Touzi, N.: Discrete-time approximation of BSDEs and probabilistic schemes for fully

S

1330

7.

8.

9.

10.

11.

12.

13.

14. 15. 16.

17.

nonlinear PDEs. In: Advanced Financial Modelling. Radon Series on Computational and Applied Mathematics, vol. 8, pp. 91–124. Walter de Gruyter, Berlin (2008) Gobet, E., Menozzi, S.: Stopped diffusion processes: boundary corrections and overshoot. Stochastic Process. Appl. 120(2), 130–162 (2010) Graham, C., Talay, D.: Mathematical Foundations of Stochastic Simulations. Vol. I: Stochastic Simulations and Monte Carlo Methods. Stochastic Modelling and Applied Probability Series, vol. 68. Springer, Berlin/New York (2013, in press) Jourdain, B., Leli`evre, T., Roux, R.: Existence, uniqueness and convergence of a particle approximation for the Adaptive Biasing Force process. M2AN 44, 831–865 (2010) Karatzas, I., Shreve, S.E.: Brownian Motion and Stochastic Calculus. Graduate Texts in Mathematics, vol. 113. Springer, New York (1991) Lamberton, D., Pag`es, G.: Recursive computation of the invariant distribution of a diffusion. Bernoulli 8(23), 367–405 (2002) Mattingly, J., Stuart, A., Tretyakov, M.V.: Convergence of numerical time-averaging and stationary measures via Poisson equations. SIAM J. Numer. Anal. 48(2), 552–577 (2010) M´el´eard, S.: A trajectorial proof of the vortex method for the two-dimensional Navier-Stokes equation. Ann. Appl. Probab. 10(4), 1197–1211 (2000) Milstein, G.N., Tretyakov, M.V.: Stochastic Numerics for Mathematical Physics. Springer, Berlin/New York (2004) Pope, S.B.: Turbulent Flows. Cambridge University Press, Cambridge/New York (2003) ´ Sznitman, A.-S.: Topics in propagation of chaos. In: Ecole ´ e de Probabilit´es de Saint-Flour XIX-1989. Lecture d’Et´ Notes Mathematics, vol. 1464. Springer, Berlin/New York (1991) Talay, D.: Probabilistic numerical methods for partial differential equations: elements of analysis. In: Talay, D., Tubaro, L. (eds.) Probabilistic Models for Nonlinear Partial Differential Equations. Lecture Notes in Mathematics, vol. 1627, pp. 48–196, Springer, Berlin/New York (1996)

Singular Perturbation Problems

a singular perturbation problem (using the nomenclature of Friedrichs and Wasow [4], now universal). The prototype singular perturbation problem occurred as Prandtl’s boundary layer theory of 1904, concerning the flow of a fluid of small viscosity past an object (cf. [15, 20]). Applications have continued to motivate the subject, which holds independent mathematical interest involving differential equations. Prandtl’s G¨ottingen lectures from 1931 to 1932 considered the model y 00 C y 0 C y D 0 on 0  x  1 with prescribed endvalues y.0/ and y.1/ for a small positive parameter  (corresponding physically to a large Reynolds number). Linearly independent solutions of the differential equation are given by e  ./x and e ./x= where ./

p 1C 14 2

p 1 14 2 2

D 1 C O./ and ./

D 1   C O. / as  ! 0. Setting y.x; / D ˛e  ./x C ˇe ./x= ;

we will need y.0/ D ˛ C ˇ and y.1/ D ˛e  ./ C ˇe ./= . The large decay constant = implies that ˛  y.1/e  ./ , so y.x; /  e  ./.1x/ y.1/ C e ./x= .y.0/  e  ./ y.1// and y.x; / D e .1x/ y.1/Ce x= e x .y.0/ey.1//CO./:

Singular Perturbation Problems Robert O’Malley Department of Applied Mathematics, University of Washington, Seattle, WA, USA

The second term decays rapidly from y.0/  ey.1/ to zero in an O./-thick initial layer near x D 0, so the limiting solution e 1x y.1/ for x > 0 satisfies the reduced problem

Regular perturbation methods often succeed in proY00 C Y0 D 0 with Y0 .1/ D y.1/: viding approximate solutions to problems involving a small parameter  by simply seeking solutions as a formal power series (or even a polynomial) in . Convergence of y.x; / at x D 0 is nonuniform unless When the regular perturbation approach fails to pro- y.0/ D ey.1/. Indeed, to all orders  j , the asymptotic vide a uniformly valid approximation, one encounters solution for x > 0 is given by the outer expansion

Singular Perturbation Problems

Y .x; / D e  ./.1x/ y.1/ 

1331

X

Yj .x/ j

(

j 0

x.t; / D X0 .t/ C O./ and y.t; / D Y0 .t/ C 0 .t=/ C O./

(see Olver [12] for the definition of an asymptotic (at least for t finite) where  . / is the asymptotically 0 expansion). It is supplemented by an initial (boundary) stable solution of the stretched problem layer correction d0

x  D g.x.0/;0 C Y0 .0/; 0; 0/; 0 .0/ D y.0/  Y0 .0/: d

;  e ./x= .y.0/  e  ./ y.1//  X x  The theory supports practical numerical methods for j j   integrating stiff differential equations (cf. [5]). A more j 0 inclusive geometric theory, using normally hyperbolic invariant manifolds, has more recently been extenwhere terms j all decay exponentially to zero as the sively used (cf. [3, 9]). stretched inner variable x= tends to infinity. For many linear problems, classical analysis (cf. Traditionally, one learns to complement regular [2, 6, 23]) suffices. Multiscale methods (cf. [8, 10]), outer expansions by local inner expansions in regions however, apply more generally. Consider, for example, of nonuniform convergence. Asymptotic matching the two-point problem methods, generalizing Prandtl’s fluid dynamical insights involving inner and outer approximations, y 00 C a.x/y 0 C b.x; y/ D 0 then provide higher-order asymptotic solutions (cf. Van Dyke [20], Lagerstrom [11], and I’lin [7], noting b are smooth. that O’Malley [13] and Vasil’eva et al. [21] provide on 0  x  1 when a.x/ > 0 and a and C We will seek the solution as  ! 0 when y.0/ and more efficient direct techniques involving boundary layer corrections). The Soviet A.N. Tikhonov and y.1/ are given in the form American Norman Levinson independently provided X y.x; ; /  yj .x; / j methods, in about 1950, to solve initial value problems j 0 for the slow-fast vector system (

xP D f .x; y; t; /

using the fast variable

 yP D g.x; y; t; / on t  0 subject to initial values and y.0/. As we  x.0/  X0 .t/ might expect, the outer limit should satisfy Y0 .t/ the reduced (differential-algebraic) system (

XP0 D f .X0 ; Y0 ; t; 0/; X0 .0/ D x.0/ 0 D g.X0 ; Y0 ; t; 0/

1 D 

Z

x

a.s/ds 0

to provide boundary layer behavior near x D 0. Because a.x/ y y 0 D yx C  and 2 a0 .x/ a2 .x/ y C y 00 D yxx C a.x/yx C y ;   2

for an attracting root Y0 D .X0 ; t/

the given equation is converted to the partial differential equation  @2 y @y C a0 .x/ C  2a.x/ @x@ @  @y C b.x; y/ C  2 yxx D 0: C a.x/ @x 

of g D 0, along which gy remains a stable matrix. We must expect nonuniform convergence of the fast variable y near t D 0, unless y.0/ D Y0 .0/. Indeed, Tikhonov and Levinson showed that

a2 .x/

@y @2 y C @2 @



S

1332

Singular Perturbation Problems

We naturally ask the leading term y0 to satisfy @y0 @2 y0 D 0; C 2 @ @

Related two-time scale methods have long been used in celestial mechanics (cf. [17,22]) to solve initial value problems for nearly linear oscillators yR C y D f .y; y/ P

so y0 has the form y0 .x; / D A0 .x/ C B0 .x/e  : The boundary conditions moreover require that

on t  0. Regular perturbation methods suffice on bounded t intervals, but for t D O.1=/ one must seek solutions X yj .t; / j y.t; ; /  j 0

A0 .0/ C B0 .0/ D y.0/ and A0 .1/  y.1/ (since e  is negligible when x D 1). From the  coefficient, we find that y1 must satisfy 

@y1 @2 y1 C a .x/ 2 @ @ 2

 D 2a.x/

@2 y0 @y0  a0 .x/ @x@ @

@y0  b.x; y0 / @x D a.x/A00   C a.x/B00 C a0 .x/B0 e  a.x/

b.x; A0 C B0 e



/:

using the slow time

D t: We must expect a boundary layer (i.e., nonuniform convergence) at t D 1 to account for the cumulative effect of the small perturbation f . Instead of using two-timing or averaging (cf. [1] or [19]), let us directly seek an asymptotic solution of the initial value problem for the Rayleigh equation   1 2 yR C y D  yP 1  yP 3

Consider the right-hand side as a power series in e  . in the form Undetermined coefficient arguments show that its first two terms (multiplying 1 and e  ) will resonate with it 3i t 2 5i t the solutions of the homogeneous equation to produce y.t; ; / D A. ; /e C B. ; /e C  C. ; /e unbounded or secular solutions (multiples of  and C : : : C complex conjugate e  ) as  ! 1 unless we require that 1. A0 satisfies the reduced problem for undetermined slowly varying complex-valued coefficients A; B; C; : : : depending on  (cf. [16]). Differena.x/A00 C b.x; A0 / D 0; A0 .1/ D y.1/ tiating twice and separating the coefficients of the odd harmonics e i t ; e 3i t ; e 5i t ; : : : in the differential equation, (and continues to exist throughout 0  x  1) we obtain 2. B0 satisfies the linear problem  2    dA d A 2 2 dA    i A 1  jAj 2i  A C  0 0 a.x/B0 C by .x; A0 / C a .x/ B0 D 0; d

d 2 d

  B0 .0/ D y.0/  A0 .0/: dA   1  2jAj2  3i.A /2 B C : : : D 0; d

Thus, we have completely obtained the limiting soi 8B  A3 C : : : D 0; and lution Y0 .x; /. We note that the numerical solution 3 of restricted two-point problems is reported in Roos 24C  3i A2 B C    D 0: et al. [18]. Special complications, possibly shock layers, must be expected at turning points where a.x/ The resulting initial value problem vanishes (cf. [14, 24]).

Solid State Physics, Berry Phases and Related Issues

1333

  A dA i 2 4 D .1  jAj / C .jAj  2/ C : : : d

2 8 for the amplitude A. ; / can be readily solved for 10. finite by using polar coordinates and regular pertur11. bation methods on all equations. Thus, we obtain the asymptotic solution 12. i y.t; ; / D A. ; /e i t  A3 . ; /e 3i t 24   A3 . ; /.3jA. ; /j2  2/e 3i t C 64  A5 . ; / 5i t e C : : : C complex  3 conjugate there. We note that the oscillations for related coupled van der Pol equations are of special current interest in neuroscience. The reader who consults the literature cited will find that singular perturbations continue to provide asymptotic solutions to a broad variety of differential equations from applied mathematics. The underlying mathematics is also extensive and increasingly sophisticated.

13. 14. 15.

16.

17. 18.

19.

20. 21.

22. 23.

References 24. 1. Bogoliubov, N.N., Mitropolski, Y.A.: Asymptotic Methods in the Theory of Nonlinear Oscillations. Gordon and Breach, New York (1961) 2. Fedoryuk, M.V.: Asymptotic Analysis. Springer, New York (1994) 3. Fenichel, N.: Geometic singular perturbation theory for ordinary differential equations. J. Differ. Equ. 15, 77–105 (1979) 4. Friedrichs, K.-O., Wasow, W.: Singular perturbations of nonlinear oscillations. Duke Math. J. 13, 367–381 (1946) 5. Hairer, E., Wanner, G.: Solving Ordinary Differential Equations II. Stiff and Differential-Algebraic Problems, 2nd revised edn. Springer, Berlin (1996) 6. Hsieh, P.-F., Sibuya Y.: Basic Theory of Ordinary Differential Equations. Springer, New York (1999) 7. Il’in, A.M.: Matching of Asymptotic Expansions of Solutions of Boundary Value Problems. American Mathematical Society, Providence (1992) 8. Johnson, R.S.: Singular Perturbation Theory, Mathematical and Analytical Techniques with Applications to Engineering. Springer, New York (2005) 9. Kaper, T.J.: An introduction to geometric methods and dynamical systems theory for singular perturbation problems.

In: Cronin, J., O’Malley, R.E. (eds.) Analyzing Multiscale Phenomena Using Singular Perturbation Methods, pp. 85–131. American Mathematical Society, Providence (1999) Kevorkian, J., Cole J.D.: Multiple Scale and Singular Perturbation Methods. Springer, New York (1996) Lagerstrom, P.A.: Matched Asymptotic Expansions. Springer, New York (1988) Olver, F.W.J.: Asymptotics and Special Functions. Academic, New York (1974) O’Malley, R.E., Jr.: Singular Perturbation Methods for Ordinary Differential Equations. Springer, New York (1991) O’Malley, R.E., Jr.: Singularly perturbed linear two-point boundary value problems. SIAM Rev. 50, 459–482 (2008) O’Malley, R.E., Jr.: Singular perturbation theory: a viscous flow out of G¨ottingen. Annu. Rev. Fluid Mech. 42, 1–17 (2010) O’Malley, R.E., Jr., Kirkinis, E.: A combined renormalization group-multiple scale method for singularly perturbed problems. Stud. Appl. Math. 124, 383–410 (2010) Poincar´e, H.: Les Methodes Nouvelles de la Mecanique Celeste II. Gauthier-Villars, Paris (1893) Roos, H.-G., Stynes, M., Tobiska L.: Robust Numerical Methods for Singularly Perturbed Differential Equations, 2nd edn. Springer, Berlin (2008) Sanders, J.A., Verhulst, F., Murdock, J.: Averaging Methods in Nonlinear Dynamical Systems, 2nd edn. Springer, New York (2007) Van Dyke, M.: Perturbation Methods in Fluid Dynamics. Academic, New York (1964) Vasil’eva, A.B., Butuzov, V.F., Kalachev L.V.: The Boundary Function Method for Singular Perturbation Problems. SIAM, Philadelphia (1995) Verhulst, F.: Nonlinear Differential Equations and Dynamical Systems, 2nd edn. Springer, New York (2000) Wasow, W.: Asymptotic Expansions for Ordinary Differential Equations. Wiley, New York (1965) Wasow, W.: Linear Turning Point Theory. Springer, New York (1985)

S Solid State Physics, Berry Phases and Related Issues Gianluca Panati Dipartimento di Matematica, Universit di Roma “La Sapienza”, Rome, Italy

Synonyms Dynamics of Bloch electrons; Theory of Bloch bands; Semiclassical model of solid-state physics

1334

Solid State Physics, Berry Phases and Related Issues

Clearly, the case of a potential with Coulomb-like singularities is included.

Definition/Abstract Crystalline solids are solids in which the ionic cores of the atoms are arranged periodically. The dynamics of a test electron in a crystalline solid can be conveniently analyzed by using the Bloch-Floquet transform, while the localization properties of electrons are better described by using Wannier functions. The latter can also be obtained by minimizing a suitable localization functional, yielding a convenient numerical algorithm. Macroscopic transport properties of electrons in crystalline solids are derived, by using adiabatic theory, from the analysis of a perturbed Hamiltonian, which includes the effect of external macroscopic or slowly varying electromagnetic potentials. The geometric Berry phase and its curvature play a prominent role in the corresponding effective dynamics.

The Bloch–Floquet Transform (Bloch Representation) Since Hper commutes with the lattice translations, it can be decomposed as a direct integral of simpler operators by the (modified) Bloch–Floquet transform. Preliminarily, we define the dual lattice as   WD ˚ d k 2 R W k   2 2Z for all  2  : We denote by Y (resp. Y  ) the centered fundamental domain of  (resp.   ), namely,

Xd Y  D k 2 Rd W k D

  1 1 kj0 j for kj0 2  ; ; j D1 2 2

˚ where fj g is the dual basis to j , i.e., j  i D 2ıj;i . When the opposite faces of Y  are identified, The Periodic Hamiltonian one obtains the torus Td WD Rd =  . One defines, initially for 2 C0 .Rd /, the modified In a crystalline solid, the ionic cores are arranged Bloch–Floquet transform as periodically, according to a periodicity lattice o D n Pd X 1  2 Rd W  D j D1 nj j for some nj 2 Z ' .UQBF /.k; y/ WD eik.yC / .y C  /; 1  2 jY j  2 Zd ; where f1 ; : : : ; d g are fixed linearly independent

vectors in Rd . The dynamics of a test electron in the potential generated by the ionic cores of the solid and, in a meanfield approximation, by the remaining electrons is described by the Schr¨odinger equation i@t D Hper , where the Hamiltonian operator reads (in Rydberg units) Hper D  C V .x/

acting in L2 .Rd /:

(1)

y 2 Rd ; k 2 Rd :

  For any fixed k 2 Rd , UQBF .k; / is a  -periodic function and can thus be regarded as an element of Hf WD L2 .TY /, TY being the flat torus Rd = . The Q map defined by R ˚(3) extends to a unitary operator UBF W 2 d L .R / ! Y  Hf d k; with inverse given by  1  UQBF ' .x/ D

2

Here, D r is the Laplace operator and the function V W Rd ! R is periodic with respect to  , i.e., V .x C  / D V .x/ for all  2 ; x 2 Rd . A mathematical justification of such a model in the reduced Hartree-Fock approximation was obtained in Catto et al. [3] and Canc`es et al. [4], see  Mathematical Theory for Quantum Crystals and references therein. To assure that Hper is self-adjoint in L2 .Rd / on the Sobolev space W 2;2 .Rd /, we make an usual Kato-type assumption on the  -periodic potential:

p

Z

1 1

jY  j 2

Y

d k eikx '.k; Œx/;

where Œ   refers to the decomposition x D x C Œx, with x 2  and Œx 2 Y . The advantage of this construction is that the transformed Hamiltonian is a fibered operator over Y  . Indeed, one checks that 1 D UQBF Hper UQBF

Z

˚ Y

d k Hper .k/

with fiber operator

V 2 L2loc .Rd / for d  3; V 2 Lloc .Rd / with p > d=2 for d  4:

(3)

(2)

 2 Hper .k/ D  iry C k C V .y/ ;

k 2 Rd ; (4)

Solid State Physics, Berry Phases and Related Issues

1335

acting on the k-independent domain W 2;2 .TY /  L2 .TY /. The latter fact explains why it is mathematically convenient to use the modified BF transform. Each fiber operator Hper .k/ is self-adjoint, has compact resolvent, and thus pure point spectrum accumulating at infinity. We label the eigenvalue increasingly, i.e., E0 .k/  E1 .k/  E2 .k/  : : :. With this choice, they are   -periodic, i.e., En .k C / D En .k/ for all  2   . The function k 7! En .k/ is called the nth Bloch band. For fixed k 2 Y  , one considers the eigenvalue problem Hper .k/ un .k; y/ D En .k/ un .k; y/; kun .k; /kL2 .TY / D 1:

(5)

prominent role in the modern theories of macroscopic polarization [9, 18] and of orbital magnetization [21]. A convenient system of localized generalized eigenfunctions has been proposed by Wannier [22]. By definition, a Bloch function corresponding to the nth Bloch band is any u satisfying (5). Clearly, if u is a Bloch function then uQ , defined by uQ .k; y/ D ei#.k/ u.k; y/ for any   -periodic function #, is also a Bloch function. The latter invariance is often called Bloch gauge invariance. Definition 1 The Wannier function wn 2 L2 .Rd / corresponding to a Bloch function un for the Bloch band En is the preimage of un with respect to the Bloch-Floquet transform, namely  1  un .x/ D wn .x/ WD UQBF

1 1 jY  j 2

Z Y

d k eikx un .k; Œx/:

A solution to the previous eigenvalue equation (e.g., by numerical simulations) provides a complete solution to The translated Wannier functions are the dynamical equation induced by (1). Indeed, if the initial datum 0 satisfies wn; .x/ WD wn .x   / Z 1 .UQBF 0 /.k; y/ D '.k/ un .k; y/ for some ' 2 L2 .Y  /; d k eik eikx un .k; Œx/; D 1   2 jY j Y

 2 :

(one says in jargon that “ 0 is concentrated on the Thus, in view of the orthogonality of the trigonometric nth band”) then the solution .t/ to the Schr¨odinger polynomials and the fact that UQ is an isometry, BF equation with initial datum 0 is characterized by the functions fwn; g 2 are mutually orthogonal in L2 .Rd /. Moreover, the family fwn; g 2 is a complete   1 .UQBF .t//.k; y/ D eiEn .k/t '.k/ un .k; y/: Ran P , where P .k/ is the orthonormal basis of UQBF spectral projection of Hper .k/ corresponding to the R In particular, the solution is exactly concentrated on eigenvalue En .k/ and P D Y˚ P .k/ d k. the nth band at any time. By linearity, one recovers the In view of the properties of the Bloch–Floquet solution for any initial datum. Below, we will discuss transform, the existence of an exponentially localized to which extent this dynamical description survives Wannier function for the Bloch band En is equivalent when macroscopic perturbations of the operator (1) are to the existence of an analytic and   -pseudoperiodic considered. Bloch function (recall that (3) implies that the Bloch function must satisfy u.k C ; y/ D eiy u.k; y/ for all  2   ). A local argument assures that Wannier Functions and Charge Localization While the Bloch representation is a useful tool to there is always a choice of the Bloch gauge such that deal with dynamical and energetic problems, it is not the Bloch function is analytic around a given point. convenient to study the localization of electrons in However, as several authors noticed [5,13], there might solids. A related crucial problem is the construction be topological obstruction to obtain a global analytic of a basis of generalized eigenfunctions of the oper- Bloch function, in view of the competition between the ator Hper which are exponentially localized in space. analyticity and the pseudoperiodicity. Hereafter, we denote by  .k/ the set fEi .k/ W n  Indeed, such a basis allows to develop computational i  n C m  1g, corresponding to a physically relevant methods which scale linearly with the system size family of m Bloch bands, and we assume the following [6], makes possible the description of the dynamics gap condition: by tight-binding effective Hamiltonians, and plays a

S

1336

Solid State Physics, Berry Phases and Related Issues

(6) Although the terminology is not standard, we call Bloch frame a set fa gaD1;:::;m of quasi-Bloch functions such that f1 .k/; : : : ; m .k/g is an orthonormal basis If a Bloch band En satisfies (6) for m D 1 we say that of Ran P .k/ at (almost-)every k 2 Y  . As in the it is an single isolated Bloch band. For m > 1, we refer previous case, there is a gauge ambiguity: a Bloch to a composite family of Bloch bands. frame is fixed only up to a k-dependent unitary matrix U.k/ 2 U.m/, i.e., if fa gaD1;:::;m is a Bloch frame then P Single Isolated Bloch Band the functions e a .k/ D m bD1 b .k/Ub;a .k/ also define In the case of a single isolated Bloch band, the problem a Bloch frame. of proving the existence of exponentially localized Definition 2 The composite Wannier functions correWannier functions was raised in 1959 by W. Kohn [10], sponding to a Bloch frame fa gaD1;:::;m are the funcwho solved it in dimension d D 1. In higher dimentions sion, the problem has been solved, always in the case of a single isolated Bloch band, by J. des Cloizeaux [5]  1  (under the nongeneric hypothesis that V has a center wa .x/ WD UQBF a .x/; a 2 f1; : : : ; mg : of inversion) and finally by G. Nenciu under general hypothesis [12], see also [8] for an alternative proof. As in the case of a single Bloch band, the expoNotice, however, that in real solids, it might happen nential localization of the composite Wannier functions that the interesting Bloch band (e.g., the conduction is equivalent to the analyticity of the correspondband in graphene) is not isolated from the rest of the ing Bloch frame (which, in addition, must be   spectrum and that k 7! P .k/ is not smooth at the pseudoperiodic). As before, there might be topological degeneracy point. In such a case, the corresponding obstruction to the existence of such a Bloch frame. As Wannier function decreases only polynomially. far as the operator (1) is concerned, the existence of exponentially localized composite Wannier functions Composite Family of Bloch Bands has been proved in Nenciu [12] in dimension d D 1; It is well-known that, in dimension d > 1, the as for d > 1, the problem remained unsolved for Bloch bands of crystalline solids are not, in general, more than two decades, until recently [2, 16]. Notice isolated. Thus, the interesting problem, in view of real that for magnetic periodic Schr¨odinger operators the applications, concerns the case of composite families existence of exponentially localized Wannier functions of bands, i.e., m > 1 in (6), and in this context, the is generically false. more general notion of composite Wannier functions is relevant [1, 5]. Physically, condition (6) is always The Marzari–Vanderbilt Localization Functional satisfied in semiconductors and insulators by considTo circumvent the long-standing controversy about the ering the family of all the Bloch bands up to the Fermi existence of exponentially localized composite Wanenergy. nier functions, and in view of the application to numeriGiven a composite family of Bloch bands, we concal simulations, the solid-state physics community presider the orthogonal projector (in Dirac’s notation) ferred to introduce the alternative notion of maximally localized Wannier functions [11]. The latter are defined nCm1 X as the minimizers of a suitable localization functional, P .k/ WD jui .k/i hui .k/j ; known as the Marzari–Vanderbilt (MV) functional. i Dn For a single-band normalized Wannier function w 2 L2 .Rd /, the localization functional is which is independent from the Bloch gauge, and we R˚ Z pose P D Y  P .k/ d k. A function  is called a quasi-Bloch function if jxj2 jw.x/j2 dx FM V .w/ D inf dist . .k/; .H.k// n  .k// > 0:

k2T d

Rd

P .k/.k; / D .k; / and .k; / ¤ 0 8k 2 Y  : (7)



d Z X j D1

2 2

Rd

xj jw.x/j dx

;

(8)

Solid State Physics, Berry Phases and Related Issues

1337

R which is well defined at least whenever Rd jxj2 jw.x/j2 dx < C1. More generally, for a system of L2 -normalized composite Wannier functions w D fw1 ; : : : ; wm g  L2 .Rd /, the Marzari–Vanderbilt localization functional is FM V .w/ D

m X

FM V .wa / D

aD1



aD1

d Z m X X aD1 j D1

m Z X Rd

jxj2 jwa .x/j2 dx 2

Rd

xj jwa .x/j2 dx

effect of the periodic potential V is the modification of the relation between the momentum and the kinetic energy of the electron, from the free relation Efree .k/ D 12 k 2 to the function k 7! En .k/ given by the nth Bloch band. Therefore, the semiclassical equations of motion are (

rP D rEn ./ P D rV .r/ C rP  B.r/

:

(9)

We emphasize that the above definition includes the crucial constraint that the corresponding Bloch functions 'a .k; / D .UQBF wa /.k; /, for a 2 f1; : : : ; mg, are a Bloch frame. While such approach provided excellent results from the numerical viewpoint, the existence and exponential localization of the minimizers have been investigated only recently [17].

Dynamics in Macroscopic Electromagnetic Potentials

(11)

where r 2 R3 is the macroscopic position of the electron,  D k  A.r/ is the kinetic momentum with k 2 Td the Bloch momentum, rV the external electric field and B D r  A the external magnetic field. In fact, one can derive also the first-order correction to (11). At this higher accuracy, the electron acquires an effective k-dependent electric moment An .k/ and magnetic moment Mn .k/. If the nth Bloch band is non-degenerate (hence isolated), the former is given by the Berry connection An .k/ D i hun .k/ ; rk un .k/iHf Z D i un .k; y/ rk un .k; y/ dy;

Y To model the transport properties of electrons in solids, one modifies the operator (1) to include the effect of ˝ the external electromagnetic potentials. Since the latter and the latter reads Mn .k/ D 2i rk un .k/; .Hper .k/ vary at the laboratory scale, it is natural to assume that En .k//rk un .k/iHf ; i.e., explicitly the ratio " between the lattice constant a D jY j1=d and the length-scale of variation of the external potentials is ˝   i X ij l @kj un .k/; .Hper .k/ Mn .k/ i D small, i.e., " 1. The original problem is replaced by 2 1j; l3

˛ i"@ . ; x/ En .k//@kl un .k/ H f   1 2 .irx  A."x// C V .x/ C V ."x/ . ; x/ D where ij l is the totally antisymmetric symbol. The 2 refined semiclassical equations read

H" . ; x/ (10) ( rP D r .En ./  "B.r/  Mn .//  "P  ˝n ./ where D "t is the macroscopic time, and V 2 1 d 1 d Cb .R ; R/ and Aj 2 Cb .R ; R/, j 2 f1; : : : ; d g P D rr .V .r/  "B.r/  Mn .// C rP  B.r/ are respectively the external electrostatic and magnetic (12) potential. Hereafter, for the sake of a simpler notation, where ˝n .k/ D r  An .k/ corresponds to the curvawe consider only d D 3. ture of the Berry connection. The previous equations While the dynamical equation (10) is quantum have a hidden Hamiltonian structure [14]. Indeed, mechanical, physicists argued [1] that for suitable by introducing the semiclassical Hamiltonian function wavepackets, which are localized on the nth Bloch Hsc .r; / D En ./ C V .r/  "B.r/  Mn ./, (12) band and spread over many lattice spacings, the main become

S

1338

Solid State Physics, Berry Phases and Related Issues

B.r/

I

I

" An ./

!

rP P

! D

rr Hsc .r; /

!

r Hsc .r; /

and inserting the definition of the Weyl quantization for (13) a one arrives at the formula

where I is the identity matrix and B (resp. An ) is the 3  3 matrix corresponding to the vector field P B (resp. ˝n ), i.e., Bl;m .r/ D 1j 3 lmj Bj .r/ D .@l Am  @m Al /.r/. Since the matrix appearing on the l.h.s corresponds to a symplectic form B;" (i.e., a nondegenerate closed 2-form) on R6 , (13) has Hamiltonian form with respect to B;" . The mathematical derivation of the semiclassical model (12) from (10) as " ! 0 has been accomplished in Panati et al. [14]. The first-order correction to the semiclassical (11) was previously investigated in Sundaram and Niu [19], but the heuristic derivation in the latter paper does not yield the term of order " in the second equation. Without such a term, it is not clear if the equations have a Hamiltonian structure. As for mathematically related problems, both the semiclassical asymptotic of the spectrum of H" and the corresponding scattering problem have been studied in detail (see [7] and references therein). The effective quantum Hamiltonians corresponding to (10) for " ! 0 have also been deeply investigated [13]. The connection between (10) and (12) can be expressed either by an Egorov-type theorem involving quantum observables, or by using Wigner functions. Here we focus on the second approach. First we define the Wigner function. We consider the space C D Cb1 .R2d / equipped with the standard distance dC , and the subspace of   -periodic observables Cper D fa 2 C W a.r; k C / D a.r; k/ 8  2   g:

W" .q; p/ D

1 .2/d

Z d ei p



.q C " =2/

Rd

 .q  " =2/;

(14)

which yields W" 2 L2 .R2d /. Although W" is realvalued, it attains also negative values in general, so it does not define a probability distribution on phase space. After this preparation, we can vaguely state the link between (10) and (12), see [20] for the precise formulation. Let En be an isolated, nondegenerate Bloch

band. Denote by ˚ " .r; k/ the flow of the dynamical system (12) in canonical coordinates .r; k/ D .r;  C A.r// (recall that the Weyl quantization, and hence the definition of Wigner function, is not invariant under non-linear changes of canonical coordinates). Then for each finite time-interval I  R there is a constant C such that for 2 I , a 2 Cper and for 0 “wellconcentrated on the nth Bloch band” one has ˇZ ˇ ˇ ˇ

R2d

a.q; p/ W"

. /



.q; p/ W" ı

 "2 C dC .a; 0/ k

0

2 0k



˚ " .q; p/

ˇ ˇ dq dpˇˇ

;

where .t/ is the solution to the Schr¨odinger equation (10) with initial datum 0 .

Slowly Varying Deformations and Piezoelectricity

To investigate the contribution of the electrons to Recall that, according to the Calderon-Vaillancourt the macroscopic polarization and to the piezoelectric theorem, there is a constant C such that for a 2 C its effect, it is crucial to know how the electrons move in Weyl quantization b a 2 B.L2 .R3 // satisfies a crystal which is strained at the macroscopic scale. Assuming the usual fixed-lattice approximation, the jh ; b a iL2 .R3 / j  C dC .a; 0/ k k2 : problem can be reduced to study the solutions to Hence, the map C 3 a 7! h ; b a i 2 C is linear continuous and thus defines an element W" of the dual space C 0 , the Wigner function of . Writing h ;b a i DW hW" ; aiC 0 ;C Z DW a.q; p/ W" .q; p/ dq dp R2d

i @t

  1 .t; x/ D  C V .x; "t/ .t; x/ 2

(15)

for " 1, where V .; t/ is  -periodic for every t 2 R, i.e., the periodicity lattice does not depend on time. While a model with a fixed lattice might seem unrealistic at first glance, we refer to Resta [18] and

Solid State Physics, Berry Phases and Related Issues

1339

King-Smith and Vanderbilt [9] for its physical justification. The analysis of the Hamiltonian H.t/ D  12 C V .x; t/ yields a family of time-dependent Bloch functions fun .k; t/gn2N and Bloch bands fEn .k; t/gn2N . Assuming that the relevant Bloch band is isolated from the rest of the spectrum, so that (6) holds true at every time, and that the initial datum is wellconcentrated on the nth Bloch band, one obtains a semiclassical description of the dynamics analogous to (12). In this case, the semiclassical equations read (

rP D rk En .k; t/  " n .k; t/ P D 0

is the starting point of any state-of-the-art numerical simulation of macroscopic polarization in insulators.

Cross-References  Born–Oppenheimer

Approximation, Adiabatic Limit, and Related Math. Issues  Mathematical Theory for Quantum Crystals

References (16)

where n .k; t/ D @t An .k; t/  rk n .k; t/ with An .k; t/ D i hun .k; t/ ; rk un .k; t/iHf n .k; t/ D i hun .k; t/ ; @t un .k; t/iHf : The notation emphasizes the analogy with the electromagnetism: if An .k; t/ and n .k; t/ are interpreted as the geometric analogous of the vector potential and of the electrostatic scalar potential, then n .k; t/ and ˝n .k; t/ correspond, respectively, to the electric and to the magnetic field. One can rigorously connect (15) and the semiclassical model (16), in the spirit of the result stated at the end of the previous section, see [15]. From (16) one obtains the King-Smith and Vanderbilt formula [9], which approximately predicts the contribution P of the electrons to the macroscopic polarization of a crystalline insulator strained in the time interval Œ0; T , namely, X Z 1 .An .k; T /  An .k; 0// d k ; P D .2/d n2N Y occ (17) where the sum runs over all the occupied Bloch bands, i.e., Nocc D fn 2 N W En .k; t/  EF g with EF the Fermi energy. Notice that (17) requires the computation of the Bloch functions only at the initial and at the final time; in view of that, the previous formula

1. Blount, E.I.: Formalism of band theory. In: Seitz, F., Turnbull, D. (eds.) Solid State Physics, vol. 13, pp. 305–373. Academic, New York (1962) 2. Brouder, Ch., Panati, G., Calandra, M., Mourougane, Ch., Marzari, N.: Exponential localization of Wannier functions in insulators. Phys. Rev. Lett. 98, 046402 (2007) 3. Catto, I., Le Bris, C., Lions, P.-L.: On the thermodynamic limit for Hartree-Fock type problems. Ann. Henri Poincar´e 18, 687–760 (2001) 4. Canc`es, E., Deleurence, A., Lewin, M.: A new approach to the modeling of local defects in crystals: the reduced Hartree-Fock case. Commun. Math. Phys. 281, 129–177 (2008) 5. des Cloizeaux, J.: Analytical properties of n-dimensional energy bands and Wannier functions. Phys. Rev. 135, A698–A707 (1964) 6. Goedecker, S.: Linear scaling electronic structure methods. Rev. Mod. Phys. 71, 1085–1111 (1999) 7. G´erard, C., Martinez, A., Sj¨ostrand, J.: A mathematical approach to the effective Hamiltonian in perturbed periodic problems. Commun. Math. Phys. 142, 217–244 (1991) ´ 8. Helffer, B., Sj¨ostrand, J.: Equation de Schr¨odinger avec champ magn´etique et e´ quation de Harper. In: Holden, H., Jensen, A. (eds.) Schr¨odinger Operators. Lecture Notes in Physics, vol. 345, pp. 118–197. Springer, Berlin (1989) 9. King-Smith, R.D., Vanderbilt, D.: Theory of polarization of crystalline solids. Phys. Rev. B 47, 1651–1654 (1993) 10. Kohn, W.: Analytic properties of Bloch waves and Wannier functions. Phys. Rev. 115, 809–821 (1959) 11. Marzari, N., Vanderbilt, D.: Maximally localized generalized Wannier functions for composite energy bands. Phys. Rev. B 56, 12847–12865 (1997) 12. Nenciu, G.: Existence of the exponentially localised Wannier functions. Commun. Math. Phys. 91, 81–85 (1983) 13. Nenciu, G.: Dynamics of band electrons in electric and magnetic fields: rigorous justification of the effective Hamiltonians. Rev. Mod. Phys. 63, 91–127 (1991) 14. Panati, G., Spohn, H., Teufel, S.: Effective dynamics for Bloch electrons: Peierls substitution and beyond. Commun. Math. Phys. 242, 547–578 (2003) 15. Panati, G., Sparber, Ch., Teufel, S.: Geometric currents in piezoelectricity. Arch. Ration. Mech. Anal. 191, 387–422 (2009) 16. Panati, G.: Triviality of Bloch and Bloch-Dirac bundles. Ann. Henri Poincar´e 8, 995–1011 (2007)

S

1340

Source Location

17. Panati, G., Pisante, A.: Bloch bundles, Marzari-Vanderbilt functional and maximally localized Wannier functions. preprint arXiv.org (2011) 18. Resta, R.: Theory of the electric polarization in crystals. Ferroelectrics 136, 51–75 (1992) 19. Sundaram, G., Niu, Q.: Wave-packet dynamics in slowly perturbed crystals: gradient corrections and Berry-phase effects. Phys. Rev. B 59, 14915–14925 (1999) 20. Teufel, S., Panati, G.: Propagation of Wigner functions for the Schr¨odinger equation with a perturbed periodic potential. In: Blanchard, Ph., Dell’Antonio, G. (eds.) Multiscale Methods in Quantum Mechanics. Birkh¨auser, Boston (2004) 21. Thonhauser, T., Ceresoli, D., Vanderbilt, D., Resta, R.: Orbital magnetization in periodic insulators. Phys. Rev. Lett. 95, 137205 (2005) 22. Wannier, G.H.: The structure of electronic excitation levels in insulating crystals. Phys. Rev. 52, 191–197 (1937)

f0 D Au0 . Due to linearity, A.u C u0 / D f C f0 . Obviously, u and u C u0 have the same Cauchy data on 0 , so f and f C f0 produce the same data (2), but they are different in ˝. It is clear that there is a very large (infinite dimensional) manifold of solutions to the inverse source problem (1) and (2). To regain uniqueness, one has to restrict unknown distributions to a smaller but physically meaningful uniqueness class.

Inverse Problems of Potential Theory We start with an inverse source problem which has a long and rich history. Let ˚ be a fundamental solution of a linear second-order elliptic partial differential operator A in Rn . The potential of a (Radon) measure  supported in ˝ is

Source Location

Z u.xI / D

Victor Isakov Department of Mathematics and Statistics, Wichita State University, Wichita, KS, USA

General Problem Let A be a partial differential operator of second order Au D f in ˝:

˚.x; y/d.y/:

The general inverse problem of potential theory is to find ; supp  ˝; from the boundary data (2). Since Au.I / D  (in generalized sense), the inverse problem of potential theory is a particular case of the inverse source problem. In the inverse problem of gravimetry, one considers A D ,

(1) ˚.x; y/ D

In the inverse source problem, one is looking for the source term f from the boundary data u D g0 ; @ u D g1 on 0  @˝;

(3)

˝

(2)

where g0 ; g1 are given functions. In this short expository note, we will try to avoid technicalities, so we assume that (in general nonlinear) A is defined by a known C 2 -function and f is a function of x 2 ˝ where ˝ is a given bounded domain in Rn with C 2 boundary.  denotes the exterior unit normal to the boundary of a domain. H k .˝/ is the Sobolev space with the norm kk.k/ .˝/. A first crucial question is whether there is enough data to (uniquely) find f . If A is a linear operator, then solution f of this problem is not unique. Indeed, let u0 be a function in the Sobolev space H 2 .˝/ with zero Cauchy data u0 D @ u0 D 0 on 0 , and let

1 ; 4jx  yj

and the gravity field is generated by volume mass distribution f 2 L1 .˝/. We will identify f with a measure . Since f with the data (2) is not unique, one can look for f with the smallest (L2 .˝/-) norm. The subspace of harmonic functions fh is L2 -closed, so for any f , there is a unique fh such that f D fh C f0 where f0 is (L2 )-orthogonal to fh . Since the fundamental solution is a harmonic function of y when x is outside ˝, the term f0 produces zero potential outside ˝. Hence, the harmonic orthogonal component of f has the same exterior data and minimal L2 norm. Applying the Laplacian to the both sides of the equation u.I fh / D fh , we arrive at the biharmonic equation 2 u.I fh / D 0 in ˝. When 0 D @˝, we have a well-posed first boundary value problem for the biharmonic equation for u.I fh /. Solving this problem, we find fh from the previous Poisson equation.

Source Location

1341

However, it is hard to interpret fh (geo)physically, knowing fh does not help much with finding f . A (geo)physical intuition suggests looking for a perturbing inclusion D of constant density, i.e., for f D D (characteristic function of an open set D). Since (in distributional sense) u.I / D  in ˝, by using the Green’s formula (or the definition of a weak solution), we yield Z

u d D

 ˝

Z

..@ u/u  .@ u /u/ @˝

(4)

equation, u1 D u2 near @˝. Then from (4) (with d D .D1  D2 /d m, d m is the Lebesgue measure) Z

u D D1

Z

u D2

for any function u which is harmonic in ˝. Novikov’s method of orthogonality is to assume that D1 and D2 are different and then to select u in such way that the left integral is less than the right one. To achieve this goal, u is replaced by its derivative, and one integrates by parts to move integrals to boundaries and makes use of the maximum principles to bound interior integrals. The inverse problem of potential theory is a severely (exponentially) ill-conditioned problem of mathematical physics. The character of stability, conditional stability estimates, and regularization methods of numerical solutions of such problems are studied starting from pioneering work of Fritz John and Tikhonov in 1950–1960s. To understand the degree of ill conditioning, one can consider harmonic continuation from the circle 0 D fx W jxj D Rg onto the inner circle  D fx W jxj D g. By using polar coordinates .r; /, any harmonic function decaying at infinity can be (in a stable way) PM m i m e . approximated by u.r; I M / D mD1 um r Let us define the linear operator of the continuation as A.@r .; R// D @r u.; /. Using the formula for u.I M /, it is easy to see that the condition number of the corresponding matrix is . R /M which is growing exponentially with respect to M . If R D 10, then the use of computers is only possible when M < 16, and typical practical measurements errors of 0:01 allow meaningful computational results when M < 3. The following logarithmic stability estimate holds and can be shown to be best possible. We denote by jj2 .S 2 / the standard norm in the space C 2 .S 2 /.

for any function u 2 H 1 .˝/ which is harmonic in ˝. If 0 D @˝, then the right side in (4) is known; we are given all harmonic moments of . In particular, letting u D 1, we obtain the total mass of , and by letting u to be coordinate (linear) functions, we obtain moments of  of first order and hence the center of gravity of . Even when one assumes that f D D , there is a nonuniqueness due to possible disconnectedness of the complement of D. Indeed, it is well known that if D is the ball B.a; R/ with center a of radius R, then 1 , where its Newtonian potential u.x; D/ D M 4jxaj M is the total mass of D. So the exterior potentials of all annuli B.a; R2 / n B.a; R1 / are the same when R13 R23 D C where C is a positive constant. Moreover, by using this simple example and some reflections in Rn , one can find two different domains with connected boundaries and equal exterior Newtonian potentials. Augmenting this construction by the condensation of singularities argument from the theory of functions of complex variables, one can construct a continuum of different domains with connected boundaries and the same exterior potential. So there is a need to have geometrical conditions on D. A domain D is called star shaped with respect to a point a if any ray originated at a intersects D over an interval. An open set D is x1 convex if any straight line Theorem 2 Let D1 ; D2 be two domains given in polar parallel to the x1 -axis intersects D over an interval. coordinates .r; / by the equations @Dj D fr D In what follows 0 is a non-void open subset of @˝. dj ./g where jdj j2 .S 2 /  M2 ; M12 < dj ; j D 1; 2. Theorem 1 Let D1 ; D2 be two domains which are star Let " D jju1  u2 jj.1/ .0 / C jj@ .u1  u2 /jj.0/ .0 /. shaped with respect to their centers of gravity or two Then there is a constant C depending only on 1 x1 convex domains in Rn . Let u1 ; u2 be potentials of M2 ; 0 such that jd1  d2 j  C.log"/ C . D D D1 ; D2 . A proof in [4] is using some ideas from the proof If u1 D u2 ; @ u1 D @ u2 on 0 , then D1 D D2 . of Theorem 1 and stability estimates for harmonic Returning to the uniqueness proof, we assume that continuation. there are two x1 -convex D1 ; D2 with the same data. Moreover, while it is not possible to obtain (even loBy uniqueness in the Cauchy problem for the Laplace cal) existence results, a special local existence theorem

S

1342

is available [4], chapter 5. In more detail, if one assumes that u0 is a potential of some C 3 -domain D0 , that the Cauchy data for a function u are close to the Cauchy data of u0 , and that, moreover, u admits harmonic continuation across @D0 , as well as suitable behavior at infinity, then u is a potential of a domain D which is close to D0 . The exterior gravity field of a polygon (polyhedron) D develops singularities at the corner points of D. Indeed, @j @k u.xI D / where D is a polyhedron with corner at x0 behaves as C logjx  x0 j, [4], section 4.1. Since these singularities are uniquely identified by the Cauchy data, one has obvious uniqueness results under mild geometrical assumptions on D. Moreover, the use of singularities provides us with constructive identification tools, based on range type algorithms in the harmonic continuation, using, for example, the operator of the single layer potential. For proofs and further results on inverse problems of potential theory, we refer to the work of V. Ivanov, Isakov, and Prilepko [4, 7]. An inverse source problem for nonlinear elliptic equations arises when detecting doping profile (source term in equations modeling semiconductors). In the inverse problem of magnetoencephalography, A is defined to be Maxwell’s system, and f is a first-order distribution supported in ˝ (e.g., head of a patient). As above, there are difficulties due to nonuniqueness and severe PM instability. One of the simple cases is when f D mD1 am @d.m/ ı.x.m//, where ı.x.m// is the Dirac delta function with the pole x.m/ and d.m/ is a direction. Then uniqueness of f is obvious, and for not large M , the problem of determining am ; x.m/ is well conditioned. However, such simplification is not satisfactory for medical diagnostics. For simplicity of exposition, we let now A D . One of the more realistic assumptions is that f is a double layer distributed with density g over a so-called cortical surface  , i.e., f D g@ d .  can be found by using different methods, so one can assume that it is known. So one looks for a function g 2 L1 . / on  from the Cauchy data (2) for the double layer potential

Source Location

For biomedical inverse (source) problems, we refer to [1].

Finding Sources of Stationary Waves Stationary waves of frequency k in a simple case are solutions to the Helmholtz equation, i.e., A D k 2 . The radiating fundamental solution of this equation is ˚.x; y/ D

e i kjxyj : 4jx  yj

The inverse source problem at a fixed k has many similarities with the case k D 0, except that maximum principles are not valid anymore. In particular, Theorem 1 is not true: potential of a ball u.I B / can be zero outside ˝ containing B for certain choices of k and the radius of B. Looking for f supported in DN (D is a subdomain of ˝) can be viewed as finding acoustical sources N Besides, this inverse source probdistributed over D. lem has immediate applications to so-called acoustical holography. This is a method to detect (mechanical) vibrations of  D @D from measurements of acoustical pressure u on 0  @˝. In simple accepted models, the normal speed of  is @ u on  . By solving the exterior Dirichlet problem for the Helmholtz equation outside ˝, one can uniquely and in a stable way determine @ u on 0 . One can show that if k is not a Dirichlet eigenvalue, then any H 1 .˝/ solution u to the Helmholtz equation can be uniquely represented by u.I gd /, so we can reduce the continuation problem to the inverse source problem for  D gd (single layer distribution over  ). The continuation of solutions of the Helmholtz equation is a severely ill-posed problem, but its ill conditioning is decreasing when k grows, and if one is looking for the “low frequency” part of g, then stability is Lipschitz. This “low frequency” part is increasing with growing k. As above, the inverse source problem at fixed k has the similar uniqueness features. However, if f D f0 C kf1 , where f0 ; f1 , depend only on x, one regains Z uniqueness. This statement is easier to understand cong.y/@.y/ ˚.x; y/d .y/: u.xI f / D sidering u as the time Fourier transform of a solution of  a wave equation with f0 ; f1 as the initial data. In tranUniqueness of g (up to a constant) is obvious, and sient (nonstationary) problems, one collects additional stability is similar to the inverse problem of gravimetry. boundary data over a period of time.

Source Location

1343

Hyperbolic Equations The inverse source problem in a wave motion is to find .u; f / 2 H .2/ .˝/  L2 .˝/ from the following partial differential equation @2t u  u D f; @2t f D 0 in ˝ D G  .0; T /; with natural lateral boundary and initial conditions @ u D 0 on @˝  .0; T /; u D @t u D 0 on ˝  f0g; (5) and the additional data u D g on 0 D S0  .0; T /; where S0 is a part of @G. Assuming @2t u 2 H 2 .˝/ and letting v D @2t u

(6)

and differentiating twice with respect to t transform this problem into finding the initial data in the following hyperbolic mixed boundary value problem @2t v  v D 0 in ˝;

(7)

with the lateral boundary condition @ v D 0 on @G  .0; T /;

(8)

from the additional data v D @2t g on 0 :

(9)

Indeed, one can find u from (6) and the initial conditions (5). Theorem 3 Let 2d i st.x; S0 I G/ < T; x 2 @G: Then the data (9) on 0 for a solution v of (7) and (8) uniquely determine v on ˝. If, in addition, S0 D @G, then kv.; 0/k.1/ .G/ C k@t v.; 0/k.0/ .G/  C k@2t gk.1/ .0 /: (10) Here, d.x; S0 I G/ is the (minimal) distance from x to S0 inside G.

The statement about uniqueness for arbitrary S0 follows from the sharp uniqueness of the continuation results for second-order hyperbolic equations and some geometric ideas [5], section 3.4. For hyperbolic equations with analytic coefficients, these sharp results are due to Fritz John and are based on the Holmgren Theorem. For C 1 -space coefficients, the Holmgren Theorem was extended by Tataru. Stability of continuation (and hence in the inverse source problem) is (as for the harmonic continuation) at best of the logarithmic type (i.e., we have severely ill-conditioned inverse problem). When S0 D @G, one has a very strong (best possible) Lipschitz stability estimate (10). This estimate was obtained by Lop-Fat Ho (1986) by using the technique of multipliers; for more general hyperbolic equations by Klibanov, Lasiecka, Tataru, and Triggiani (1990s) by using Carleman-type estimates; and by Bardos, Lebeau, and Rauch (1992) by propagation of singularities arguments. Similar results are available for general linear hyperbolic equations of second order with time-independent coefficients. However, for Lipschitz stability, one has to assume the existence of a suitable pseudo-convex function or absence of trapped bicharacteristics. Looking for the source of the wave motion (in the more complicated elasticity system), in particular, can be interpreted as finding location and intensity of earthquakes. The recent medical diagnostic technique called thermoacoustical tomography can be reduced to looking for the initial displacement u0 . One of the versions of this problem is a classical one of looking for a function from its spherical means. In a limiting case when radii of spheres are getting large, one arrives at one of the most useful problems of tomography whose mathematical theory was initiated by Radon (1917) and Fritz John (1940s). For a recent advance in tomography in case of general attenuation, we refer to [2]. Detailed references are in [4, 5]. In addition to direct applications, the inverse source problems represent linearizations of (nonlinear) problems of finding coefficients of partial differential equations and can be used in the study of uniqueness and stability of identification of coefficients. For example, subtracting two equations @2t u2 a2 u2 D 0 and @2t u1  a1 u1 D 0 yields @2t u  a2 u D ˛f with ˛ D u1 (as a known weight function) and unknown f D a2  a1 . A general technique to show uniqueness and stability of such inverse source problems by utilizing Carleman estimates was introduced in [3].

S

1344

Sparse Approximation

sparse representation in an appropriate basis/frame. Concrete applications include compression, denoising, signal separation, and signal reconstruction (compressed sensing). On the one hand, the theory of sparse approximation References is concerned with identifying the type of vectors, functions, etc. which can be well approximated by a sparse 1. Ammari, H., Kang, H.: An Introduction to Mathematics of expansion in a given basis or frame and with quantifyEmerging Biomedical Imaging. Springer, Berlin (2008) 2. Arbuzov, E.V., Bukhgeim, A.L., Kazantsev, S.G.: Two- ing the approximation error. For instance, when given dimensional tomography problems and the theory of A- a wavelet basis, these questions relate to the area of analytic functions. Siber. Adv. Math. 8, 1–20 (1998) function spaces, in particular, Besov spaces. On the 3. Bukhgeim, A.L., Klibanov, M.V.: Global uniqueness of class of multidimensional inverse problems. Sov. Math. Dokl. 24, other hand, algorithms are required to actually find a 244–247 (1981) sparse approximation to a given vector or function. 4. Isakov, V.: Inverse Source Problems. American Mathematical In particular, if the frame at hand is redundant or if Society, Providence (1990) only incomplete information is available – as it is the 5. Isakov, V.: Inverse Problems for PDE. Springer, New York case in compressed sensing – this is a nontrivial task. (2006) 6. Novikov, P.: Sur le probleme inverse du potentiel. Dokl. Several approaches are available, including convex Akad. Nauk SSSR 18, 165–168 (1938) relaxation (`1 -minimization), greedy algorithms, and 7. Prilepko, A.I., Orlovskii, D.G., Vasin, I.A.: Methods for certain iterative procedures. Solving Inverse Problems in Mathematical Physics. Marcel Acknowledgements This research was in part supported by the NSF grant DMS 10-07734 and by Emylou Keith and Betty Dutcher Distinguished Professorship at WSU.

Dekker, New York (2000)

Sparse Approximation

Sparsity

Holger Rauhut Lehrstuhl C f¨ur Mathematik (Analysis), RWTH Aachen University, Aachen, Germany

Let x be a vector in RN or CN or `2 . / for some possibly infinite set  . We say that x is s-sparse if

Definition

For a general vector x, the error of best s-term approximation quantifies the distance to sparse vectors,

The aim of sparse approximation is to represent an object – usually a vector, matrix, function, image, or operator – by a linear combination of only few elements from a basis, or more generally, from a redundant system such as a frame. The tasks at hand are to design efficient computational methods for finding sparse representations and to estimate the approximation error that can be achieved for certain classes of objects.

kxk0 WD #f` W x` ¤ 0g  s:

s .x/p WD

inf

zWkzk0 s

kx  zkp :

P Here, kxkp D . j jxj jp /1=p is the usual `p -norm for 0 < p < 1 and kxk1 D supj jxj j. Note that the vector z minimizing s .c/p equals x on the indices corresponding to the s largest absolute coefficients of x and is zero on the remaining indices. We say that x is compressible if s .x/p decays quickly in s, that is, for suitable s we can approximate x well by Overview an s-sparse vector. This occurs for instance in the particular case when x is taken from the `q -unit ball Sparse approximations are motivated by several types Bq D fx W kxkq  1g for small q. Indeed, an inequality of applications. An important source is the various due to Stechkin (see, e.g., [21, Lemma 3.1]) states that, tasks in signal and image processing tasks, where it for 0 < q < p, is an empirical finding that many types of signals and images can indeed be well approximated by a s .x/p  s 1=p1=q kxkq : (1)

Sparse Approximation

1345

This inequality enlightens the importance of `q -spaces with q < 1 in this context. The situation above describes sparsity with respect to the canonical basis. For a more general setup, consider a (finite- or infinite-dimensional) Hilbert space H (often a space of functions) endowed with an orthonormal basis f j ; j 2 J g. Given an element f 2 H, our aim is to approximate it by a finite linear combination of the j , that is, by X

orthonormal basis also does not make sense anymore, so that f j ; j 2 J g is then just some system of elements spanning the space – possibly a basis. Important types of systems f j ; j 2 J g considered in this context include the trigonometric system fe 2 i k ; k 2 Zg  L2 Œ0; 1, wavelet systems [9, 36], or Gabor frames [23].

Quality of a Sparse Approximation xj

j;

j 2S

One important task in the field of sparse approximation is to quantify how well an element f 2 H or a where S  J is of cardinality at most s, say. In contrast whole class B  H of elements can be approximated to linear approximation, the index set S is not fixed a by sparse expansions. An abstract way [12, 20] of priori but is allowed to depend on f . Analogously as describing good approximation classes is to introduce above, the error of best s-term approximation is then X defined as xj j ; kxkp < 1g Bp WD ff 2 H W f D X j 2J s .f /H WD inf kf  xj j kH xWkxk0 s

j 2J

P with norm kf kBp D inffkxkp W f D j xj j g. If and the element j xj j with kxk0  s realizing f j W j 2 J g is an orthonormal basis, then it follows the infimum is called a best s-term approximation to directly from (1) that, for 0 < p < 2, f . Due to the fact that the support set of x (i.e., the index set of nonzero entries of x) is not fixed a (2) s .f /H  s 1=21=p kf kBp : priori, the set of such elements does not form a linear space, so that one sometimes simply refers to nonlinear In concrete situations, the task is then to characterize approximation [12, 30]. the spaces Bp . In the case, that f j W j 2 J g One may generalize this setup further. For instance, is a wavelet system, then one obtains Besov spaces instead of requiring that f j W j 2 J g forms an [32, 36], and in the case of the trigonometric system, orthonormal basis, one may assume that it is a frame this results in the classical Fourier algebra when p D 1. [7,19,23], that is, there are constants 0 < A  B < 1 If f j W j 2 J g is a frame, then (2) remains valid such that up to a multiplicative constant. In the special case of X Gabor frames, the space Bp coincides with a class of jh j ; f ij2  Bkf k2H : Akf k2H  modulation spaces [23]. P

j 2J

This definition includes orthonormal bases but allows also redundancy,P that is, the coefficient vector x in the expansion f D j 2J xj j is no longer unique. Redundancy has several advantages. For instance, since there are more possibilities for a sparse approximation of f , the error of s-sparse approximation may potentially be smaller. On the other hand, it may get harder to actually find a sparse approximation (see also below). In another direction, one may relax the assumption that H is a Hilbert space and only require it to be a Banach space. Clearly, then the notion of an

Algorithms for Sparse Approximation For practical purposes, it is important to have algorithms for computing optimal or at least near-optimal sparse approximations. When H D CN is finite dimensional and f j ; j D 1; : : : ; N g  CN is an orthonormal basis, then this is easy. P In fact, the coefficients in the expansion f D N j D1 xj j are given by xj D hf; j i, so that a best s-term approximation to f in H is given by

S

1346

Sparse Approximation

X

hf;

ji

j

j 2S

where S is an index set of s largest absolute entries of the vector .hf; j i/N j D1 . When f j ; j D 1; : : : ; M g  CN is redundant, that is, M > N , then it becomes a nontrivial problem to find the sparsest approximation to a given f 2 CN . Denoting by  the N  M matrix whose columns are the vectors j , this problem can be expressed as finding the minimizer of min kxk0

subject to

k x  f k2  ";

(3)

on this greedy algorithm which avoids this drawback consists in the orthogonal matching pursuit algorithm [31, 33] outlined next. Again, starting with r0 D f , S0 D ; and k D 0, the following steps are conducted: n o jhr ; ij 1. jk WD argmax kk j kj W j 2 J . 2. SkC1 WD Sk [ fjk g. P 3. x .kC1/ WD argminzWsupp.z/SkC1 kf  j 2SkC1 zj j k2 . P .kC1/ 4. rkC1 WD f  j 2SkC1 xj j. 5. Repeat from step (1) with k 7! k C 1 until a stopping criterion is met. .k/ eDf ek D P 6. Output f j. j 2Sk xj The essential difference to matching pursuit is the orthogonal projection step in (3). Orthogonal matching pursuit may require a smaller number of iterations than matching pursuit. However, the orthogonal projection makes an iteration computationally more demanding than an iteration of matching pursuit.

for a given threshold " > 0. In fact, this problem is known to be NP hard in general [11, 25]. Several tractable alternatives have been proposed. We discuss the greedy methods matching pursuit and orthogonal matching pursuit as well as the convex relaxation method basis pursuit (`1 -minimization) next. Other sparse approximation algorithms include itera- Convex Relaxation tive schemes, such as iterative hard thresholding [1] A second tractable approach to sparse approximation is to relax the `0 -minimization problem to the convex and iteratively reweighted least squares [10]. optimization problem of finding the minimizer of Matching Pursuits Given a possibly redundant system f j ; j 2 J g  H – often called a dictionary – the greedy algorithm matching pursuit [24,27,31,33] iteratively builds up the support set and the sparse approximation. Starting with r0 D f , S0 D ; and k D 0 it performs the following steps: n o jhr ; ij 1. jk WD argmax kk j kj W j 2 J . 2. SkC1 WD Sk [ fjk g. hr ; i 3. rkC1 D rk  kk j jkk2 jk . k 4. k 7! k C 1. 5. Repeat from step (1) with k 7! k C 1 until a stopping criterion is met. eDf ek D Pk hr` ; j` i j . 6. Output f ` `D1 k j` k2 Clearly, if s steps of matching pursuit are performed, e has an s-sparse representation with then the output f e. It is known that the sequence f ek conrespect to f verges to f when k tends to infinity [24]. A possible stopping criterion for step (5) is a maximal number of iterations, or that the residual norm krk k   for some prescribed tolerance  > 0. Matching pursuit has the slight disadvantage that an index k may be selected more than once. A variation

min kxk1

subject to

k x  f k2  ":

(4)

This program is also known as basis pursuit [6] and can be solved using various methods from convex optimization [2]. At least in the real-valued case, the minimizer x of the above problem will always have at most N nonzero entries, and the support of x defines a linear independent set f j W xj ¤ 0g, which is a basis of CN if x has exactly N nonzero entries – thus, the name basis pursuit. Finding the Sparsest Representation When the dictionary f j g is redundant, it is of great interest to provide conditions which ensure that a specific algorithm is able to identify the sparsest possible representation. For this purpose, it is helpful to define the coherence  of the system f j g, or equivalently of the matrix  having the vectors j as its columns. Assuming the normalization k j k2 D 1, it is defined as the maximal inner product between different dictionary elements,  D max jh j ; k ij: j ¤k

Sparse Approximation

1347

Suppose that f has a representation with s terms, that P is, f D j xj j with kxk0  s. Then s iterations of orthogonal matching pursuit [3, 33] as well as basis pursuit (4) with " D 0 [3, 14, 34] find the sparsest representation of f with respect to f j g provided that .2s  1/ < 1:

(5)

Moreover, for a general f , both orthogonal matching pursuit and basis pursuit generate an s-sparse approximation whose approximation error is bounded by the error of best s-term approximation up to constants; see [3, 18, 31] for details. N For typical “good” dictionaries f j gM j D1  C , the p coherence scales as   N [29], so that the bound (5) implies that s-sparse representations in such dictiop naries with small enough sparsity, that is, s  c N , can be found efficiently via the described algorithms.

representing the noise. The additional knowledge that the signal at hand can be approximated well by a sparse representation can be exploited to clean the signal by essentially removing the noise. The P essential idea is to find a sparse approximation j xj j of e with respect to a suitable dictionary f j g and to f use it as an approximation to the original f . One algorithmic approach is to solve the `1 -minimization problem (4), where  is now a suitable estimate of the `2 -norm of the noise . If  is a wavelet basis, this principle is often called wavelet thresholding or wavelet shrinkage [16] due to connection of the softthresholding function [13].

Signal Separation Suppose one observes the superposition f D f1 C f2 of two signals f1 and f2 of different nature, for instance, the “harmonic” and the “spiky” component of an acoustic signal, or stars and filaments in an astronomical image. The task is to separate the two components f1 and f2 from the knowledge of f . Applications of Sparse Approximation Knowing that both f1 and f2 have sparse represenj 2 Sparse approximation find a variety of applications. tations in dictionaries f j g and f j g of “different Below we shortly describe compression, denoising, nature,” one can indeed recover both f1 and f2 by and signal separation. Sparse representations play also similar algorithms as outlined above, for instance, by a major role in adaptive numerical methods for solving solving the `1 -minimization problem operator equations such as PDEs. When the solution X X kz1 kCkz2 k1 subject to f D z1j j1 C z2j j2 : has a sparse representation with a suitable basis, say min 1 2 z ;z j j finite elements or wavelets, then a significant acceleration with respect to standard linear methods can .x1 ; x2 / defines the reconstructions be achieved. The algorithms used in this context are The solution P 1 1 e e2 D P x 2 2 . If, for instance, of different nature than the ones described above. We f 1 D j xj j and f j j j 2 N refer to [8] for details. f j1 gN and f g are mutually incoherent bases j D1 j j D1 in CN , that is, k j1 k2 D k j2 k2 D 1 for all j and the Compression maximal inner product  D jh j1 ; j2 ij is small, then An obvious application of sparse approximation is the above optimization problem recovers both f1 and image and signal compression. Once a sparse approx- f2 provided they have representations with altogether imation is found, one only needs to store the nonzero s terms where s < 1=.2/ [15]. An example of two coefficients of the representation. If the representation mutually incoherent bases are the Fourier basis and p is sparse enough, then this requires significantly less the canonical basis, where  D 1= N [17]. Under memory than storing the original signal or image. a probabilistic model, better estimates are possible This principle is exploited, for instance, in the JPEG, [5, 35]. MPEG, and MP3 data compression standards.

Compressed Sensing Denoising Often acquired signals and images are corrupted by e D The theory of compressed sensing [4, 21, 22, 26] builds noise, that is, the observed signal can be written as f f C , where f is the original signal and  is a vector on sparse representations. Assuming that a vector

S

1348

Sparse Approximation

x 2 CN is s-sparse (or approximated well by a sparse vector), one would like to reconstruct it from only limited information, that is, from y D Ax;

with A 2 CmN

where m is much smaller than N . Again the algorithms outlined above apply, for instance, basis pursuit (4) with A replacing  . In this context, one would like to design matrices A with the minimal number m of rows (i.e., the minimal number of linear measurements), which are required to reconstruct x from y. The recovery criterion based on coherence  of A described above applies but is highly suboptimal. In fact, it can be shown that for certain random matrices m  cs log.eN=s/ measurements suffice to (stably) reconstruct an s-sparse vector using `1 -minimization with high probability, where c is a (small) universal constant. This bound is sufficiently better than the ones that can be deduced from coherence based bounds as described above. A particular case of interest arise when A consists of randomly selected rows of the discrete Fourier transform matrix. This setup corresponds to randomly sampling entries of the Fourier transform of a sparse vector. When m  cs log4 N , then `1 -minimization succeeds to (stably) recover ssparse vectors from m samples [26, 28]. This setup generalizes to the situation that one takes limited measurements of a vector f 2 CN , which is sparse with respect to a basis or frame f j gM j D1 . In fact, then f D  x for a sparse x 2 CM and with a measurement matrix A 2 CmN , we have y D Af D A x; so that we reduce to the initial situation with A0 D A replacing A. Once x is recovered, one forms f D  x. Applications of compressed sensing can be found in various signal processing tasks, for instance, in medical imaging, analog-to-digital conversion, and radar.

References 1. Blumensath, T., Davies, M.: Iterative thresholding for sparse approximations. J. Fourier Anal. Appl. 14, 629–654 (2008) 2. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)

3. Bruckstein, A., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51(1), 34–81 (2009) 4. Cand`es, E.J.: Compressive sampling. In: Proceedings of the International Congress of Mathematicians, Madrid (2006) 5. Cand`es, E.J., Romberg, J.K.: Quantitative robust uncertainty principles and optimally sparse decompositions. Found. Comput. Math. 6(2), 227–254 (2006) 6. Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998) 7. Christensen, O.: An Introduction to Frames and Riesz Bases. Applied and Numerical Harmonic Analysis. Birkh¨auser, Boston (2003) 8. Cohen, A.: Numerical Analysis of Wavelet Methods. Studies in Mathematics and its Applications, vol. 32, xviii, 336p. EUR 95.00. North-Holland, Amsterdam (2003) 9. Daubechies, I.: Ten Lectures on Wavelets. CBMS-NSF Regional Conference Series in Applied Mathematics, vol 61. SIAM, Philadelphia (1992) 10. Daubechies, I., DeVore, R.A., Fornasier, M., G¨unt¨urk, C.: Iteratively re-weighted least squares minimization for sparse recovery. Commun. Pure Appl. Math. 63(1), 1–38 (2010) 11. Davis, G., Mallat, S., Avellaneda, M.: Adaptive greedy approximations. Constr. Approx. 13(1), 57–98 (1997) 12. DeVore, R.A.: Nonlinear approximation. Acta Numer. 7, 51–150 (1998) 13. Donoho, D.: De-noising by soft-thresholding. IEEE Trans. Inf. Theory 41(3), 613–627 (1995) 14. Donoho, D.L., Elad, M.: Optimally sparse representation in general (nonorthogonal) dictionaries via `1 minimization. Proc. Natl. Acad. Sci. U.S.A. 100(5), 2197–2202 (2003) 15. Donoho, D.L., Huo, X.: Uncertainty principles and ideal atomic decompositions. IEEE Trans. Inf. Theory 47(7), 2845–2862 (2001) 16. Donoho, D.L., Johnstone, I.M.: Minimax estimation via wavelet shrinkage. Ann. Stat. 26(3), 879–921 (1998) 17. Donoho, D.L., Stark, P.B.: Uncertainty principles and signal recovery. SIAM J. Appl. Math. 48(3), 906–931 (1989) 18. Donoho, D.L., Elad, M., Temlyakov, V.N.: Stable recovery of sparse overcomplete representations in the presence of noise. IEEE Trans. Inf. Theory 52(1), 6–18 (2006) 19. Duffin, R.J., Schaeffer, A.C.: A class of nonharmonic Fourier series. Trans. Am. Math. Soc. 72, 341–366 (1952) 20. Fornasier, M., Gr¨ochenig, K.: Intrinsic localization of frames. Constr. Approx. 22(3), 395–415 (2005) 21. Fornasier, M., Rauhut, H.: Compressive sensing. In: Scherzer, O. (ed.) Handbook of Mathematical Methods in Imaging, pp. 187–228. Springer, New York (2011) 22. Foucart, S., Rauhut, H.: A mathematical introduction to compressive sensing. Applied and Numerical Harmonic Analysis. Birkh¨auser, Basel 23. Gr¨ochenig, K.: Foundations of Time-Frequency Analysis. Applied and Numerical Harmonic Analysis. Birkh¨auser, Boston (2001) 24. Mallat, S.G., Zhang, Z.: Matching pursuits with timefrequency dictionaries. IEEE Trans. Signal Process. 41(12), 3397–3415 (1993) 25. Natarajan, B.K.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24, 227–234 (1995) 26. Rauhut, H.: Compressive sensing and structured random matrices. In: Fornasier, M. (ed.) Theoretical Foundations

Special Functions: Computation

27. 28.

29.

30. 31.

32. 33.

34.

35. 36.

and Numerical Methods for Sparse Recovery. Radon Series on Computational and Applied Mathematics, vol. 9, pp. 1–92. De Gruyter, Berlin/New York (2010) Roberts, D.H.: Time series analysis with clean I. Derivation of a spectrum. Astronom. J. 93, 968–989 (1987) Rudelson, M., Vershynin, R.: On sparse reconstruction from Fourier and Gaussian measurements. Commun. Pure Appl. Math. 61, 1025–1045 (2008) Strohmer, T., Heath, R.W.: Grassmannian frames with applications to coding and communication. Appl. Comput. Harmon Anal. 14(3), 257–275 (2003) Temlyakov, V.N.: Nonlinear methods of approximation. Found Comput. Math. 3(1), 33–107 (2003) Temlyakov, V.: Greedy Approximation. Cambridge Monographs on Applied and Computational Mathematics, No. 20. Cambridge University Press, Cambridge (2011) Triebel, H.: Theory of Function Spaces. Monographs in Mathematics, vol. 78. Birkh¨auser, Boston (1983) Tropp, J.A.: Greed is good: Algorithmic results for sparse approximation. IEEE Trans. Inf. Theory 50(10), 2231–2242 (2004) Tropp, J.A.: Just relax: Convex programming methods for identifying sparse signals in noise. IEEE Trans. Inf. Theory 51(3), 1030–1051 (2006) Tropp, J.A.: On the conditioning of random subdictionaries. Appl. Comput. Harmon Anal. 25(1), 1–24 (2008) Wojtaszczyk, P.: A Mathematical Introduction to Wavelets. Cambridge University Press, Cambridge (1997)

1349

Bessel, Legendre, or parabolic cylinder functions are well known for everyone involved in physics. This is not surprising because Bessel functions appear in the solution of partial differential equations in cylindrically symmetric domains (such as optical fibers) or in the Fourier transform of radially symmetric functions, to mention just a couple of applications. On the other hand, Legendre functions appear in the solution of electromagnetic problems involving spherical or spheroidal geometries. Finally, parabolic cylinder functions are involved, for example, in the analysis of the wave scattering by a parabolic cylinder, in the study of gravitational fields or quantum mechanical problems such as quantum tunneling or particle production. But there are many more functions under the term “special functions” which, differently from the examples mentioned above, are not of hypergeometric type, such as some cumulative distribution functions [3, Chap. 10]. These functions also need to be evaluated in many problems in statistics, probability theory, communication theory, or econometrics.

Basic Methods

Special Functions: Computation

The methods used for the computation of special functions are varied, depending on the function under Amparo Gil1 , Javier Segura2 , and Nico M. Temme3 consideration as well as on the efficiency and the 1 Departamento de Matem´atica Aplicada y Ciencias de accuracy demanded. Usual tools for evaluating special la Computaci´on, Universidad de Cantabria, E.T.S. functions are the evaluation of convergent and diverCaminos, Canales y Puertos, Santander, Spain gent series, the computation of continued fractions, the 2 Departamento de Matem´aticas, Estad´ıstica y use of Chebyshev approximations, the computation of Computaci´on, Universidad de Cantabria, Santander, the function using integral representations (numerical Spain quadrature), and the numerical integration of ODEs. 3 Centrum voor Wiskunde and Informatica (CWI), Usually, several of these methods are needed in order Amsterdam, The Netherlands to build an algorithm able to compute a given function for a large range of values of parameters and argument. Also, an important bonus in this kind of algorithms Mathematics Subject Classification will be the possibility of evaluating scaled functions: if, for example, a function f .z/ increases exponentially 65D20; 41A60; 33C05; 33C10; 33C15 for large jzj, the factorization of the exponential term and the computation of a scaled function (without the exponential term) can be used to avoid degradations in Overview the accuracy of the functions and overflow problems as z increases. Therefore, the appropriate scaling of a The special functions of mathematical physics [4] special function could be useful for increasing both the are those functions that play a key role in many range of computation and the accuracy of the computed problems in science and engineering. For example, expression.

S

1350

Special Functions: Computation

for N D 0; 1; 2; : : :, and the order estimate holds for fixed N . This is the Poincar´e-type expansion and for special functions like the gamma and Bessel functions they are crucial for evaluating these functions. Other variants of the expansion are also important, in particular expansions that hold for a certain range of additional parameters (this leads to the uniform asymptotic Convergent and Divergent Series Convergent series for special functions usually arise in expansions in terms of other special functions like Airy functions, which are useful in turning point problems). the form of hypergeometric series: Next, we briefly describe three important techniques for computing special functions which appear ubiquitously in algorithms for special function evaluation: convergent and divergent series, recurrence relations, and numerical quadrature.

0

1 1 a1 ;    ; ap X .a1 /n    .ap /n zn @ A ; Iz D p Fq .b1 /n    .bq /n nŠ nD0 b1 ;    ; bq

(1)

where p  q C 1 and .a/n is the Pochhammer symbol, also called the shifted factorial, defined by .a/0 D 1; .a/n D

.a/n D a.a C 1/    .a C n  1/ .n  1/;

.a C n/ : .a/

(2)

The series is easy to evaluate because of the recursion .a/nC1 D .a C n/.a/n , n  0, of the Pochhammer symbols. For example, for the modified Bessel function 1

 X 1 z 2

. 14 z2 /n . C n C 1/ nŠ nD0 !

  1 1 2 I 4z ; D 2 z 0 F1 C1

I .z/ D

(3)

Recurrence Relations In many important cases, there exist recurrence relations relating different values of the function for different values of its variables; in particular, one can usually find three-term recurrence relations [3, Chap. 4]. In these cases, the efficient computation of special functions uses at some stage the recurrence relations satisfied by such families of functions. In fact, it is difficult to find a computational task which does not rely on recursive techniques: the great advantage of having recursive relations is that they can be implemented with ease. However, the application of recurrence relations can be risky: each step of a recursive process generates not only its own rounding errors but also accumulates the errors of the previous steps. An important aspect is then the study of the numerical condition of the recurrence relations, depending on the initial values for starting recursion. If we write the three-term recurrence satisfied by the function yn as

ynC1 C bn yn C an yn1 D 0; (6) this is a stable representation when z > 0 and   0 and it is an efficient representation when z is not large compared with . .m/ With divergent expansion we mean asymptotic ex- then, if a solution yn of (6) exists that satisfies .m/ y .D/ pansions of the form lim n D 0 for all solutions yn that are linearly n!C1 yn.D/ .m/ .m/ 1 independent of yn , we will call yn the minimal X cn .D/ F .z/  ; z ! 1: (4) solution. The solution yn is said to be a dominant zn nD0 solution of the three-term recurrence relation. From a computational point of view, the crucial point is the The series usually diverges, but it has the property identification of the character of the function to be evaluated (either minimal or dominant) because the N 1 stable direction of application of the recurrence relation X cn   F .z/ D C RN .z/; RN .z/ D O zN ; is different for evaluating the minimal or a dominant n z nD0 solution of (6): forward for dominant solutions and z ! 1; (5) backward for minimal solutions.

Special Functions: Computation

1351

For analyzing whether a special function is minimal gnC1 z 2n fnC1 ; : (11)   or not, analytical information is needed regarding its fn 2n gn z behavior as n ! C1. Assume that for large values of n the coefficients As the known asymptotic behavior of the Bessel funcan ; bn behave as follows. tions reads ˛

an  an ;

ˇ

bn  bn ;

ab ¤ 0

(7)

1 z n Jn .z/  ; nŠ 2

.n  1/Š Yn .z/   

 n 2 ; z

n ! 1; with ˛ and ˇ real; assume that t1 ; t2 are the zeros of the characteristic polynomial ˚.t/ D t 2 C bt C a with jt1 j  jt2 j. Then it follows from Perron’s theorem [3, p. 93] that we have the following results: 1. If ˇ > 12 ˛, then the difference equation (6) has two linearly independent solutions fn and gn , with the property

it is easy to identify Jn .z/ and Yn .z/ as the minimal (fn ) and a dominant (gn ) solutions, respectively, of the three-term recurrence relation (10). Similar results hold for the modified Bessel functions, with recurrence relation ynC1 C

a fn   n˛ˇ ; fn1 b

gn  bnˇ ; gn1

n ! 1: (8)

(12)

2n yn  yn1 D 0; z

z ¤ 0;

(13)

with solutions In .z/ (minimal) and .1/n Kn .z/ (dominant).

In this case, the solution fn is minimal. 2. If ˇ D 12 ˛ and jt1 j > jt2 j, then the difference Numerical Quadrature equation (6) has two linear independent solutions fn Another example where the study of numerical and gn , with the property stability is of concern is the computation of special functions via integral representations. It is tempting, but usually wrong, to believe that once an integral g fn n  t1 nˇ ;  t2 nˇ ; n ! 1; (9) representation is given, the computational problem fn1 gn1 is solved. One has to choose a stable quadrature rule and this choice depends on the integral In this case, the solution fn is minimal. under consideration. Particularly problematic is the 3. If ˇ D 12 ˛ and jt1 j D jt2 j, or if ˇ < 12 ˛, then integration of strongly oscillating integrals (Bessel some information is still available, but the theorem and Airy functions, for instance); in these cases is inconclusive with respect to the existence of an alternative approach consists in finding nonminimal and dominant solutions. oscillatory representations by properly deforming the Let’s consider three-term recurrence relations sat- integration path in the complex plane. Particularly isfy by Bessel functions as examples. Ordinary Bessel useful is the saddle point method for obtaining functions satisfy the recurrence relation integral representations which are suitable for applying the trapezoidal rule, which is optimal for computing certain integrals in R. Let’s explain the 2n yn C yn1 D 0; z ¤ 0; (10) saddle point method taking the Airy function Ai.z/ ynC1  z as example. Quadrature methods for evaluating complex Airy functions can be found in, for with solutions Jn .z/ (the Bessel function of the first example, [1, 2]. kind) and Yn .z/ (the Bessel function of the second We start from the following integral representation kind). This three-term recurrence relation corresponds in the complex plane: to (8), with the values a D 1, ˛ D 0, b D  2z , ˇ D 1. Z Then, there exist two independent solutions fn and gn 1 1 3 e 3 w zw d w; (14) Ai.z/ D satisfying 2 i C

S

1352

Special Functions: Computation

5.0

−5.0

5.0

−5.0

p the circle with radius r and are indicated by small dots. The saddle point on the positive real axis corresponds with the case  D 0 and the two saddles on the imaginary axis with the case  D . It is interesting to see that the contour may split up and run through both saddle points ˙w0 . When  D 23  both saddle points are on one path, and the half-line in the zplane corresponding with this  is called a Stokes line. Integrating with respect to D v  v0 (and writing  D u  u0 ), we obtain e  Ai.z/ D 2 i

Special Functions: Computation, Fig. 1 Saddle point contours for  D 0; 13 ; 23 ;  and r D 5

Z



1

e 1

r .; /

 d C i d ; d

(18)

3

where  D 23 z 2 and

where z 2 C and C is a contour starting at 1e i =3

. C 3v0 / and terminating at 1e Ci =3 (in the valleys of  D  ; 1 < < 1; q the integrand). In the example we take ph z 2 3 u0 C 13 . 2 C 4v0 C 3r/ Œ0; 23 . (19) Let .w/ D 13 w3  zw. The saddle points are p w0 D z and w0 and follow from solving 0 .w/ D w2  z D 0. The saddle point contour (the path of 2 2 r .; / D 3 when dealing with second-order Œa Œb eh.F CF / Œg.x/, where F Œa and F Œb are the Lie ODEs of the form y 00 D g.y/ when they are rewritten derivatives corresponding to f Œa and f Œb , respec- as (1) [1]. tively, acting as For time-symmetric methods, the order conditions at even orders are automatically satisfied, which leads D to n1 C n3 C    C n2k1 order conditions to achieve X @g Œa fj .x/ .x/; F Œa Œg.x/ D order r D 2k. For instance, n1 C n3 D 4 conditions @xj j D1 need to be fulfilled for a symmetric method (5–6) to be of order 4. D X Œb @g F Œb Œg.x/ D fj .x/ .x/: (7) @xj j D1 Splitting and Composition Methods When the original system (1) is split in m > 2 parts, Similarly, the approximation h .x/ 'h .x/ given higher-order schemes can be obtained by considering by the splitting method (5) satisfies the identity a composition of the basic first-order splitting method Œ1 Œm1 Œm g. h .x// D  .h/Œg.x/, where ı 'h . More (3) and its adjoint h D 'h ı    ı 'h specifically, compositions of the general form a1 hF Œa b1 hF Œb as hF Œa bs hF Œb asC1 hF Œa e  e e e :  .h/ D e   (10) h D ˛2s h ı ˛2s1 h ı    ı ˛2 h ı ˛1 h ; (8)

Hence, the coefficients ai ; bi must be chosen in such a way that the operator  .h/ is a good approximation of can be considered with2sappropriately chosen coeffiŒa Œb cients .˛1 ; : : : ; ˛2s / 2 R so as to achieve a prescribed eh.F CF / , or equivalently, h1 log. / F Œa CF Œb . order of approximation. Applying repeatedly the Baker–Campbell– In the particular case when system (1) is split in Hausdorff (BCH) formula [2], one arrives at Œb Œa m D 2 parts so that h D 'h ı 'h , method (10) reduces to (5) with a1 D ˛1 and 1 log. .h// D .va F Œa C vb F Œb / C hvab F Œab h bj D ˛2j 1 C ˛2j ; aj C1 D ˛2j C ˛2j C1 ; Ch2 .vabb F Œabb C vaba F Œaba / for j D 1; : : : ; s; (11) Ch3 .vabbb F Œabbb C vabba F Œabba Cvabaa F Œabaa / C O.h4 /; where F Œab D ŒF Œa ; F Œb ;

(9) where ˛2sC1 D 0. In that case, the coefficients ai and bi are such that sC1 X

F Œabb D ŒF Œab ; F Œb ;

i D1

ai D

s X i D1

bi :

(12)

Splitting Methods

1355

Conversely, any splitting method (5) satisfying (12) can Œb Œa be written in the form (10) with h D 'h ı 'h . Moreover, compositions of the form (10) make sense for an arbitrary basic first-order integrator h (and its adjoint h ) of the original system (1). Obviously, if the coefficients ˛j of a composition method (10) are such that h is of order r for arbitrary basic integrators h of (1), then the splitting method (5) with (11) is also of order r. Actually, as shown in [6], the integrator (5) is of order r for ODEs of the form (1) with f D f Œa C f Œb if and only if the integrator (10) (with coefficients ˛j obtained from (11)) is of order r for arbitrary first-order integrators h . This close relationship allows one to establish in an elementary way a defining feature of splitting methods (5) of order r  3: at least one ai and one bi are necessarily negative [1]. In other words, splitting schemes of order r  3 always involve backward fractional time steps. Preserving Properties Œi  Assume that the individual flows 'h share with the exact flow 'h some defining property which is preserved by composition. Then it is clear that any composition of the form (5) and (10) with h given by (3) also possesses this property. Examples of such features are symplecticity, unitarity, volume preservation, conservation of first integrals, etc. [7]. In this sense, splitting methods form an important class of geometric numerical integrators [2]. Repeated application of the BCH formula can be used (see (9)) to show that there exists a modified (formal) differential equation Q f .x/ Q C hf2 .x/ Q C h2 f3 .x/ Q C ; xQ 0 D fh .x/ x.0/ Q D x0 ;

(13)

associated to any splitting method h such that the numerical solution xn D h .xn1 / (n D 1; 2; : : :) Q for the exact solution x.t/ Q satisfies xn D x.nh/ of (13). An important observation is that the vector fields fk in (13) belong to the Lie algebra generated by f Œ1 ; : : : ; f Œm . In the particular case of autonomous Hamiltonian systems, if f Œi  are Hamiltonian, then each fk is also Hamiltonian. Then one may study the long-time behavior of the numerical integrator by analyzing the solutions of (13) viewed as a small perturbation of the original system (1) and obtain

rigorous statements with techniques of backward error analysis [2]. Further Extensions Several extensions can be considered to reduce the number of stages necessary to achieve a given order and get more efficient methods. One of them is the use of a processor or corrector. The idea consists in enhancing an integrator h (the kernel) with a map h (the processor) as O h D h ı h ı h1 . Then, after n steps, one has O hn D h ı hn ı h1 , and so only the cost of h is relevant. The simplest example of a processed integrator is provided by the St¨ormer–Verlet Œb Œa method (4). In that case, h D h D 'h ı 'h and Œa h D 'h=2 . The use of processing allows one to get methods with fewer stages in the kernel and smaller error terms than standard compositions [1]. The second extension uses the flows corresponding to other vector fields in addition to F Œa and F Œb . For instance, one could consider methods (5) such Œa Œb Œabb that, in addition to 'h and 'h , use the h-flow 'h of the vector field F Œabb when its computation is straightforward. This happens, for instance, for secondorder ODEs y 00 D g.y/ [1, 7]. Splitting is particularly appropriate when kf Œa k kf Œb k in (1). Introducing a small parameter ", we can write x 0 D "f Œa .x/ C f Œb .x/, so that the error of scheme (5) is O."/. Moreover, since in many practical applications " < h, one is mainly interested in eliminating error terms with small powers of " instead of satisfying all the order conditions. In this way, it is possible to get more efficient schemes. In addition, the use of a processor allows one to eliminate the errors of order "hk for all 1 < k < n and all n [7]. Although only autonomous differential equations have been considered here, several strategies exist for adapting splitting methods also to nonautonomous systems without deteriorating their overall efficiency [1]. Some Good Fourth-Order Splitting Methods In the following table, we collect the coefficients of a few selected fourth-order symmetric methods of the form (5–6). Higher-order and more elaborated schemes can be found in [1, 2, 7] and references therein. They are denoted as Xs 4, where s indicates the number of stages. S6 4 is a general splitting method, whereas SN6 4 refers to a method tailored for second-order ODEs of

S

1356

Splitting Methods

the form y 00 D g.y/ when they are rewritten as a firstorder system (1), and the coefficients ai are associated to g.y/. Finally, SNI5 4 is a method especially designed for problems of the form x 0 D "f Œa .x/ C f Œb .x/. With s D 3 stages, there is only one solution, S3 4, given by a1 D b1 =2; b1 D 2=.2  21=3 /. In all cases, the remaining coefficients are fixed by symmetry and P P consistency ( i ai D i bi D 1).

and the two extra stages to achieve an even higher efficiency. It requires solving the Kepler problem separately from the perturbation. This requires a more elaborated algorithm with a slightly increase in the computational cost (not reflected in the figure). Results provided by the leapfrog method S2 and the standard fourth-order Runge–Kutta integrator RK4 are also included for reference.

Splitting Methods for PDEs S6 4 a1 D 0:07920369643119565 a2 D 0:353172906049774

b1 D 0:209515106613362 b2 D 0:143851773179818

a3 D 0:04206508035771952

SN6 4 a1 D 0:08298440641740515 a2 D 0:396309801498368

b1 D 0:245298957184271 b2 D 0:604872665711080

a3 D 0:3905630492234859

SNI5 4a1 D 0:81186273854451628884

b1 D 0:0075869131187744738

a2 D 0:67748039953216912289 b2 D 0:31721827797316981388

In the numerical treatment of evolutionary PDEs of parabolic or mixed hyperbolic-parabolic type, splitting time-integration methods are also widely used. In this setting, the overall evolution operator is formally written as a sum of evolution operators, typically representing different aspects of a given model. Consider an evolutionary PDE formulated as an abstract Cauchy problem in a certain function space U  fu W RD  R ! Rg,

(15) ut D L.u/; u.t0 / D u0 ; Numerical Example: A Perturbed Kepler where L is a spatial partial differential operator. For Problem instance, To illustrate the performance of the previous splitting methods, we apply them to the time integration of the ! d d X X @ @ @ perturbed Kepler problem described by the Hamiltou.x; t/ D ci .x/ u.x; t/ @t @xj i D1 @xi nian j D1  "  1 1 H D .p12 C p22 /   5 q22  2q12 ; (14) Cf .x; u.x; t//; u.x; t0 / D u0 .x/ 2 r 2r q where r D q12 C q22 . We take " D 0:001 and integrate the equations of motion qi0 D pi , pi0 D @H=@qi , i D 1; 2, with initial p conditions 3=2. Splitting q1 D 4=5, q2 D p1 D 0, p2 D methods are used with the partition into kinetic and potential energy. We measure the two-norm error in the position at tf D 2; 000, .q1 ; q2 / D .0:318965403761932; 1:15731646810481/, for different time steps and plot the corresponding error as a function of the number of evaluations for each method in Fig. 1. Notice that although the generic method S6 4 has three more stages than the minimum given by S3 4, this extra cost is greatly compensated by a higher accuracy. On the other hand, since this system corresponds to the second-order ODE q 00 D g.q/, method SN6 4 leads to a higher accuracy with the same computational cost. Finally, SNI5 4 takes profit of the near-integrable character of the Hamiltonian (14)

or in short, L.x; u/ D r  .cru/ C f .u/ corresponds to a diffusion-reaction problem. In that case, it makes sense to split the problem into two subequations, corresponding to the different physical contributions, ut D La .u/ r  .cru/;

ut D Lb .u/ f .u/; (16)

solve numerically each equation in (16), thus giving Œa Œb uŒa .h/ D 'h .u0 /, uŒb .h/ D 'h .u0 /, respectively, for a time step h, and then compose the operators Œa Œb 'h , 'h to construct an approximation to the soluŒb Œa tion of (15). Thus, u.h/ 'h .'h .u0 // provides a first-order approximation, whereas the Strang splitŒa Œb Œa ting u.h/ 'h=2 .'h .'h=2 .u0 /// is formally secondorder accurate for sufficiently smooth solutions. In this way, especially adapted numerical methods can be used to integrate each subproblem, even in parallel [3, 4].

Splitting Methods

0

S2 RK4 S34 S64 SN64 SNI54

−1 −2 LOG10(ERROR)

Splitting Methods, Fig. 1 Error in the solution .q1 .tf /; q2 .tf // vs. the number of evaluations for different fourth-order splitting methods (the extra cost in the method SNI5 4, designed for perturbed problems, is not taken into account)

1357

−3 −4 −5 −6 −7 −8

4

4.2

4.4

4.6

4.8

5

5.2

5.4

5.6

5.8

6

LOG10(N. EVALUATIONS)

Systems of hyperbolic conservation laws, such as ut C f .u/x C g.u/x D 0;

u.x; t0 / D u0 .x/;

can also be treated with splitting methods, in this case, by fixing a step size h and applying a especially tailored numerical scheme to each scalar conservation law ut C f .u/x D 0 and ut C g.u/x D 0. This is a particular example of dimensional splitting where the original problem is approximated by solving one space direction at a time. Early examples of dimensional splitting are the so-called locally onedimensional (LOD) methods (such as LOD-backward Euler and LOD Crank–Nicolson schemes) and alternating direction implicit (ADI) methods (e.g., the Peaceman–Rachford algorithm) [4]. Although the formal analysis of splitting methods in this setting can also be carried out by power series expansions, several fundamental difficulties arise, however. First, nonlinear PDEs in general possess solutions that exhibit complex behavior in small regions of space and time, such as sharp transitions and discontinuities. Second, even if the exact solution of the original problem is smooth, it might happen that the composition defining the splitting method provides nonsmooth approximations. Therefore, it is necessary to develop sophisticated tools to analyze whether the numerical solution constructed with a splitting method

leads to the correct solution of the original problem or not [3]. On the other hand, even if the solution is sufficiently smooth, applying splitting methods of order higher than two is not possible for certain problems. This happens, in particular, when there is a diffusion term in the equation; since then the presence of negative coefficients in the method leads to an ill-posed problem. When c D 1 in (16), this order barrier has been circumvented, however, with the use of complexvalued coefficients with positive real parts: the operator Œa 'zh corresponding to the Laplacian La is still well defined in a reasonable distribution set for z 2 C, provided that 1 integer, be the grid size, and define the grid domain, ˝h D f .lh; mh/ j 0 < l; m < n g, as the set of grid points in ˝. The nearest neighbors of a grid point p D .p1 ; p2 / are the four grid points pW D .p1  h; p2 /, pE D .p1 C h; p2 /, pS D .p1 ; p2  h/, and pN D .p1 ; p2 C h/. The grid points which do not themselves belong to ˝, but which have a nearest neighbor in ˝ constitute the grid boundary, @˝h , and we set ˝N h D ˝h [ @˝h . Now let v W ˝N h ! R be a grid function. Its five-point Laplacian h v is defined by

Overview v.pE /Cv.pW /Cv.pS /Cv.pN /4v.p/ ;

h v.p/ D h2 A problem in differential equations can rarely be solved analytically, and so often is discretized, p 2 ˝h : resulting in a discrete problem which can be solved in a finite sequence of algebraic operations, efficiently The finite difference discretization then seeks uh W implementable on a computer. The error in a ˝N h ! R satisfying

Stability, Consistency, and Convergence of Numerical Discretizations

1359

Stability, Consistency, and Convergence of Numerical Discretizations, Fig. 1 The grid domain ˝N h consists of the points in ˝h , marked with solid dots, and in @˝h , marked with hollow dots. On the right is the stencil of the five-point Laplacian, which consists of a grid point p and its four nearest neighbors

h uh .p/ D f .p/; p 2 ˝h ;

uh .p/ D 0; p 2 @˝h :

If we regard as unknowns, the N D .n  1/2 values uh .p/ for p 2 ˝h , this gives us a systems of N linear equations in N unknowns which may be solved very efficiently. A Finite Element Method A second example of a discretization is provided by a finite element solution of the same problem. In this case we assume that ˝ is a polygon furnished with a triangulation Th , such as pictured in Fig. 2. The finite element method seeks a function uh W ˝ ! R which is continuous and piecewise linear with respect to the mesh and vanishing on @˝, and which satisfies

the discrete operator as a linear map Lh from a vector space Vh , called the discrete solution space, to a second vector space Wh , called the discrete data space. In the case of the finite difference operator, the discrete solution space is the space of mesh functions on ˝N h which vanish on @˝h , the discrete data space is the space of mesh functions on ˝h , and the discrete operator Lh D h , the five-point Laplacian. In the case of the finite element method, Vh is the space of continuous piecewise linear functions with respect to the given triangulation that vanish on @˝, and Wh D Vh , the dual space of Vh . The operator Lh is given by Z .Lh w/.v/ D 

rw  rv dx;

w; v 2 Vh :

˝

For the finite difference method, we define the discrete data fh 2 Wh by fh D f j˝h , while for the finite ruh  rv dx D f v dx;  Relement method fh 2 Wh is given by fh .v/ D ˝ ˝ f v dx. In both cases, the discrete solution uh 2 Vh for all test functions v which are themselves continuous is found by solving the discrete equation and piecewise linear with respect to the mesh and (1) Lh uh D fh : vanish on @˝. If we choose a basis for this set of space of test functions, then the computation of uh may be reduced to an efficiently solvable system of Of course, a minimal requirement on the discretization N linear equations in N unknowns, where, in this is that the finite dimensional linear system (1) has case, N is the number of interior vertices in the a unique solution, i.e., that the associated matrix is triangulation. invertible (so Vh and Wh must have the same dimension). Then, the discrete solution uh is well-defined. Discretization The primary goal of numerical analysis is to ensure We may treat both these examples, and many other that the discrete solution is a good approximation of discretizations, in a common framework. We regard the true solution u in an appropriate sense. Z

Z

S

1360

Stability, Consistency, and Convergence of Numerical Discretizations

Stability, Consistency, and Convergence of Numerical Discretizations, Fig. 2 A finite element mesh of the domain ˝. The solution is sought as a piecewise linear function with respect to the mesh

Representative and Error Since we are interested in the difference between u and uh , we must bring these into a common vector space, where the difference makes sense. To this end, we suppose that a representative Uh 2 Vh of u is given. The representative is taken to be an element of Vh which, though not practically computable, is a good approximation of u. For the finite difference method, a natural choice of representative is the grid function Uh D uj˝h . If we show that the difference Uh  uh is small, we know that the grid values uh .p/ which determine the discrete solution are close to the exact values u.p/. For the finite element method, a good possibility for Uh is the piecewise linear interpolant of u, that is, Uh is the piecewise linear function that coincides with u at each vertex of the triangulation. Another popular possibility is to take Uh to be the best approximation of u in Vh in an appropriate norm. In any case, the quantity Uh  uh , which is the difference between the representative of the true solution and the discrete solution, defines the error of the discretization. At this point we have made our goal more concrete: we wish to ensure that the error, Uh  uh 2 Vh , is small. To render this quantitative, we need to select a norm on the finite dimensional vector space Vh with which to measure the error. The choice of norm is an important aspect of the problem presentation, and an appropriate choice must reflect the goal of the computation. For example, in some applications, a large error at a single point of the domain could be catastrophic, while in others only the average error over the domain is significant. In yet other cases, derivatives of u are the true quantities of interest. These cases would lead to different choices of norms. We shall denote the chosen norm of v 2 Vh by kvkh . Thus, we

now have a quantitative goal for our computation that the error kUh  uh kh be sufficiently small. Consistency and Stability Consistency Error Having used the representative Uh of the solution to define the error, we also use it to define a second sort of error, the consistency error, also sometimes called the truncation error. The consistency error is defined to be Lh Uh  fh , which is an element of Wh . Now Uh represents the true solution u, so the consistency error should be understood as a quantity measuring the extent to which the true solution satisfies the discrete equation (1). Since Lu D f , the consistency error should be small if Lh is a good representative of L and fh a good representative of f . In order to relate the norm of the error to the consistency error, we need a norm on the discrete data space Wh as well. We denote this norm by kwk0h for w 2 Wh and so our measure of the consistency error is kLh Uh  fh k0h .

Stability If a problem in differential equations is well-posed, then, by definition, the solution u depends continuously on the data f . On the discrete level, this continuous dependence is called stability. Thus, stability refers to the continuity of the mapping L1 h W Wh ! Vh , which takes the discrete data fh to the discrete solution uh . Stability is a matter of degree, and an unstable discretization is one for which the modulus of continuity of L1 h is very large. To illustrate the notion of instability, and to motivate the quantitative measure of stability we shall introduce below, we consider a simpler numerical problem than

Stability, Consistency, and Convergence of Numerical Discretizations

1361

the discretization of a differential equation. Suppose Relating Consistency, Stability, and Error we wish to compute the definite integral The Fundamental Error Bound Z 1 Let us summarize the ingredients we have introduced x n e x1 dx; (2) in our framework to assess a discretization: nC1 D 0 • The discrete solution space, Vh , a finite dimensional vector space, normed by k  kh for n D 15. Using integration by parts, we obtain a simple recipe to compute the integral in short sequence • The discrete data space, Wh , a finite dimensional vector space, normed by k  k0h of arithmetic operations: • The discrete operator, Lh W Vh ! Wh , an invertible linear operator nC1 D 1  nn ; n D 1; : : : ; 15; • The discrete data fh 2 Wh 1 D 1  e 1 D 0:632121 : : : : (3) • The discrete solution uh determined by the equation Lh uh D fh Now suppose we carry out this computation, beginning • The solution representative Uh 2 Vh with 1 D 0:632121 (so truncated after six decimal • The error Uh  uh 2 Vh places). We then find that 16 D 576; 909, which is • The consistency error Lh Uh  fh 2 Wh truly a massive error, since the correct value is 16 D • The stability constant Chstab D kL1 h kL.Wh ;Vh / 0:0590175 : : :. If we think of (3) as a discrete solution With this framework in place, we may prove a rigorous operator (analogous to L1 h above) taking the data 1 error bound, stating that the error is bounded by the to the solution 16 , then it is a highly unstable scheme: product of the stability constant and the consistency a perturbation of the data of less than 106 leads to error: a change in the solution of nearly 6  105 . In fact, it is easy to see that for (3), a perturbation  in the data leads kUh  uh kh  Chstab kLh Uh  fh k0h : (4) to an error of 15Š   in solution – a huge instability. It is important to note that the numerical computation The proof is straightforward. Since Lh is invertible, of the integral (2) is not a difficult numerical problem. It could be easily computed with Simpson’s rule, for Uh  uh D L1 ŒLh .Uh  uh / D L1 .Lh Uh  Lh uh / h h example. The crime here is solving the problem with 1 D Lh .Lh Uh  fh /: the unstable algorithm (3). Returning to the case of the discretization (1), imagine that we perturb the discrete data fh to some fQh D Taking norms, gives fh C h , resulting in a perturbation of the discrete 0 Q kUh  uh kh  kL1 solution to uQ h D L1 h kL.Wh ;Vh / kLh Uh  fh kh ; h fh . Using the norms in Wh and Vh to measure the perturbations and then computing as claimed. the ratio, we obtain kL1 solution perturbation kQuh  uh kh h h kh D D : 0 data perturbation kh k0h kfQh  fh kh We define the stability constant Chstab , which is our quantitative measure of stability, as the maximum value this ratio achieves for any perturbation h of the data. In other words, the stability constant is the norm of the operator L1 h : Chstab D

sup 0¤h 2Wh

kL1 h h kh D kL1 h kL.Wh ;Vh / : kh k0h

The Fundamental Theorem A discretization of a differential equation always entails a certain amount of error. If the error is not small enough for the needs of the application, one generally refines the discretization, for example, using a finer grid size in a finite difference method or a triangulation with smaller elements in a finite element method. Thus, we may consider a whole sequence or family of discretizations, corresponding to finer and finer grids or triangulations or whatever. It is conventional to parametrize these by a positive real number h called the discretization parameter. For example, in the finite

S

1362

Stability, Consistency, and Convergence of Numerical Discretizations

difference method, we may use the same h as before, the grid size, and in the finite element method, we can take h to be the maximal triangle diameter or something related to it. We shall call such a family of discretizations a discretization scheme. The scheme is called convergent if the error norm kUh  uh kh tends to 0 as h tends to 0. Clearly convergence is a highly desirable property: it means that we can achieve whatever level of accuracy we need, as long as we do a fine enough computation. Two more definitions apply to a discretization scheme. The scheme is consistent if the consistency error norm kLh Uh  fh k0h tends to 0 with h. The scheme is stable if the stability constant Chstab is bounded uniformly in h: Chstab  C stab for some number C stab and all h. From the fundamental error bound, we immediately obtain what may be called the fundamental theorem of numerical analysis: a discretization scheme which is consistent and stable is convergent.

Historical Perspective Consistency essentially requires that the discrete equations defining the approximate solution are at least approximately satisfied by the true solution. This is an evident requirement and has implicitly guided the construction of virtually all discretization methods, from the earliest examples. Bounds on the consistency error are often not difficult to obtain. For finite difference methods, for example, they may be derived from Taylor’s theorem, and, for finite element methods, from simple approximation theory. Stability is another matter. Its central role was not understood until the midtwentieth century, and there are still many differential equations for which it is difficult to devise or to assess stable methods. That consistency alone is insufficient for the convergence of a finite difference method was pointed out in a seminal paper of Courant, Friedrichs, and Lewy [2] in 1928. They considered the one-dimensional wave equation and used a finite difference method, analogous to the five-point Laplacian, with a space-time grid of points .j h; lk/ with 0  j  n, 0  l  m integers and h; k > 0 giving the spatial and temporal grid size, respectively. It is easy to bound the consistency error by O.h2 Ck 2 /, so setting k D h for some constant  > 0 and letting h tend to 0, one obtains a consistent scheme. However, by comparing the domains of dependence of the true solution and of the discrete solution on

the initial data, one sees that this method, though consistent, cannot be convergent if  > 1. Twenty years later, the property of stability of discretizations began to emerge in the work of von Neumann and his collaborators. First, in von Neumann’s work with Goldstine on solving systems of linear equations [5], they studied the magnification of roundoff error by the repeated algebraic operations involved, somewhat like the simple example (3) of an unstable recursion considered above. A few years later, in a 1950 article with Charney and Fjørtoft [1] on numerical solution of a convection diffusion equation arising in atmospheric modeling, the authors clearly highlighted the importance of what they called computational stability of the finite difference equations, and they used Fourier analysis techniques to assess the stability of their method. This approach developed into von Neumann stability analysis, still one of the most widely used techniques for determining stability of finite difference methods for evolution equations. During the 1950s, there was a great deal of study of the nature of stability of finite difference equations for initial value problems, achieving its capstone in the 1956 survey paper [3] of Lax and Richtmeyer. In that context, they formulated the definition of stability given above and proved that, for a consistent difference approximation, stability ensured convergence.

Techniques for Ensuring Stability Finite Difference Methods We first consider an initial value problem, for example, the heat equation or wave equation, discretized by a finite difference method using grid size h and time step k. The finite difference method advances the solution from some initial time t0 to a terminal time T by a sequence of steps, with the lth step advancing the discrete solution from time .l  1/k to time lk. At each time level lk, the discrete solution is a spatial grid function ulh , and so the finite difference method to ulh , called defines an operator G.h; k/ mapping ul1 h the amplification matrix. Since the amplification matrix is applied many times in the course of the calculation (m D .T  t0 /=k times to be precise, a number which tends to infinity as k tends to 0), the solution at the final step um h involves a high power of the amplification matrix, namely G.h; k/m , applied to the

Stability, Consistency, and Convergence of Numerical Discretizations

1363

Stability, Consistency, and Convergence of Numerical Discretizations, Fig. 3 Finite difference solution of the heat equation using (5). Left: initial data. Middle: discrete solution at

t D 0:03 computed with h D 1=20, k D 1=2;000 (stable). Right: same computation with k D 1=1;000 (unstable)

data u0h . Therefore, the stability constant will depend on a bound for kG.h; k/m k. Usually this can only be obtained by showing that kG.h; k/k  1 or, at most, kG.h; k/k  1 C O.k/. As a simple example, we may consider an initial value problem for the heat equation with homogeneous boundary conditions on the unit square:

Several methods are used to bound the norm of the amplification matrix. If an L1 norm is chosen, one can often use a discrete maximum principle based on the structure of the matrix. If an L2 norm is chosen, then Fourier analysis may be used if the problem has constant coefficients and simple enough boundary conditions. In other circumstances, more sophisticated matrix or eigenvalue analysis is used. For time-independent PDEs, such as the Poisson equation, the requirement is to show that the inverse of the discretization operator is bounded uniformly in the grid size h. Similar techniques as for the timedependent problems are applied.

@u D u; x 2 ˝; 0 < t  T; @t u.x; t/ D 0; x 2 @˝; 0 < t  T; u.x; 0/ D u0 .x/;

x 2 ˝;

Galerkin Methods which we discretize with the five-point Laplacian and Galerkin methods, of which finite element methods are forward differences in time: an important case, treat a problem which can be put into the form: find u 2 V such that B.u; v/ D F .v/ for ul .p/  ul1 .p/ l1 all v 2 V . Here, V is a Hilbert space, B W V  V ! R D h u .p/; p 2 ˝h ; k is a bounded bilinear form, and F 2 V  , the dual 0 < l  m; (5) space of V . (Many generalizations are possible, e.g., to the case where B acts on two different Hilbert spaces or the case of Banach spaces.) This problem l u .p/ D 0; p 2 @˝h ; 0 < l  m; is equivalent to a problem in operator form, find u u0 .p/ D u0 .p/; p 2 ˝h : (6) such Lu D F , where the operator L W V ! V  is defined by Lu.v/ D B.u; v/. An example is the considered In this case the norm condition on the amplification Dirichlet problem for the Poisson equation R 1 V 2 matrix kG.h; k/k  1 holds if 4k  h , but not earlier. Then, VR D H .˝/, B.u; v/ D ˝ ru  rv dx, otherwise, and, indeed, it can be shown that this dis- and F .v/ D ˝ f v dx. The operator is L D  W cretization scheme is stable, if and only if that con- HV 1 .˝/ ! HV 1 .˝/ . dition is satisfied. Figure 3 illustrates the tremendous A Galerkin method is a discretization which seeks difference between a stable and unstable choice of time uh in a subspace Vh of V satisfying B.uh ; v/ D F .v/ for all v 2 Vh . The finite element method discussed step.

S

1364

Stability, Consistency, and Convergence of Numerical Discretizations

Stability, Consistency, and Convergence of Numerical Discretizations, Fig. 4 Approximation of the problem (7), with u D cos x shown on left and  D u0 on the right. The exact solution is shown in blue, and the stable finite element method, using piecewise linears for  and piecewise constants for u, is

shown in green (in the right plot, the blue curve essentially coincides with the green curve, and so is not visible). An unstable finite element method, using piecewise quadratics for  , is shown in red

above took Vh to be the subspace of continuous piece- If we partition I into subintervals and choose Sh wise linears. If the bilinear form B is coercive in the and Wh both to be the space of continuous piecewise linears, then the resulting matrix problem is singular, sense that there exists a constant  > 0 for which so the method is unusable. If we choose Sh to continuous piecewise linears, and Wh to be piecewise B.v; v/   kvk2V ; v 2 V; constants, we obtain a stable method. But if we choose then stability of the Galerkin method with respect to Sh to contain all continuous piecewise quadratic the V norm is automatic. No matter how the subspace functions and retain the space of piecewise constants Vh is chosen, the stability constant is bounded by 1= . for Wh , we obtain an unstable scheme. The stable If the bilinear form is not coercive (or if we consider and unstable methods can be compared in Fig. 4. a norm other than the norm in which the bilinear For the same problem of the Poisson equation in form is coercive), then finding stable subspaces for first-order form, but in more than one dimension, Galerkin’s method may be quite difficult. As a very the first stable elements were discovered in 1975 simple example, consider a problem on the unit interval [4]. I D .0; 1/, to find .; u/ 2 H 1 .I /  L2 .I / such that Z

Z

1

1

 dx C 0

0

Z

1

u dx C 0

. ; v/ 2 H 1 .I /  L2 .I /:

0

Z

 v dx D 0

References

1

f v dx; 0

(7)

This is a weak formulation of system  D u0 ,  0 D f , with Dirichlet boundary conditions (which arise from this weak formulation as natural boundary conditions), so this is another form of the Dirichlet problem for Poisson’s equation u00 D f on I , u.0/ D u.1/ D 0. In higher dimensions, there are circumstances where such a first-order formulation is preferable to a standard second-order form. This problem can be discretized by a Galerkin method, based on subspaces Sh  H 1 .I / and Wh  L2 .I /. However, the choice of subspaces is delicate, even in this one-dimensional context.

1. Charney, J.G., Fjørtoft, R., von Neumann, J.: Numerical integration of the barotropic vorticity equation. Tellus 2, 237–254 (1950) ¨ 2. Courant, R., Friedrichs, K., Lewy, H.: Uber die partiellen Differenzengleichungen der mathematischen Physik. Math. Ann. 100(1), 32–74 (1928) 3. Lax, P.D., Richtmyer, R.D.: Survey of the stability of linear finite difference equations. Commun. Pure Appl. Math. 9(2), 267–293 (1956) 4. Raviart, P.A., Thomas, J.M.: A mixed finite element method for 2nd order elliptic problems. In: Mathematical Aspects of Finite Element Methods. Proceedings of the Conference, Consiglio Naz. delle Ricerche (C.N.R.), Rome, 1975. Volume 606 of Lecture Notes in Mathematics, pp. 292–315. Springer, Berlin (1977) 5. von Neumann, J., Goldstine, H.H.: Numerical inverting of matrices of high order. Bull. Am. Math. Soc. 53, 1021–1099 (1947)

Statistical Methods for Uncertainty Quantification for Linear Inverse Problems

Statistical Methods for Uncertainty Quantification for Linear Inverse Problems Luis Tenorio Mathematical and Computer Sciences, Colorado School of Mines, Golden, CO, USA

To solve an inverse problem means to recover an unknown object from indirect noisy observations. As an illustration, consider an idealized example of the blurring of a one-dimensional signal, f .x/, by a measuring instrument. Assume the function is parametrized so that x 2 Œ0; 1 and that the actual data can be modeled as noisy observations of a blurred version of f . We may model the blurring as a convolution with a kernel, K.x/, determined by the instrument. The forward operator Rmaps f to the blurred function  given by 1 .x/ D 0 K.x  t/f .t/ dt (i.e., a Fredholm integral equation of the first kind.) The statistical model for the data is then y.x/ D .x/ C ".x/; where ".x/ is measurement noise. The inverse problem consists of recovering f from finitely many measurements y.x1 /,. . . ,y.xn /. However, inverse problems are usually ill-posed (e.g., the estimates may be very sensitive to small perturbations of the data) and deblurring is one such example. A regularization method is required to solve the problem. An introduction to regularization of inverse problems can be found in [11] and more general references are [10, 27]. Since the observations y.xi / are subject to systematic errors (e.g., discretization) as well as measurement errors that will be modeled as random variables, the solution of an inverse problem should include a summary of the statistical characteristics of the inversion estimate such as means, standard deviations, bias, mean squared errors, and confidence sets. However, the selection of proper statistical methods to assess estimators depends, of course, on the class of estimators, which in turn is determined by the type of inverse problem and chosen regularization method. Here we use a general framework that encompasses several different and widely used approaches. We consider the problem of assessing the statistics of solutions of linear inverse problems whose data are modeled as yi D Ki Œf  C "i .i D 1; : : : ; n/; where the functions f belong to a linear space H , each Ki

1365

is a continuous linear operator defined on H , and the errors "i are random variables. Since the errors are random, an estimate fO of f is a random variable taking values in H . Given the finite amount of data, we can only hope to recover components of f that admit a finite-dimensional parametrization. Such parametrizations also help us avoid defining probability measures in function spaces. For example, we can discretize the operators and the function so that the estimate fO is a vector in Rm . Alternatively, one may be able to use Pma finite-dimensional parametrization such as f D kD1 ak k , where k are fixed functions defined on H . This time the random variable is the estimate aO of the vector of coefficients a D .ak /. In either case the problem of finding an estimate of a function reduces to a finite-dimensional linear algebra problem. Example 1 Consider the inverse problem for a Fredholm integral equation: Z

1

yi D y.xi / D

K.xi t/f .t/ dtC"i ;

.i D 1; : : : ; n/:

0

To discretize the integral, we can use m equally spaced points ti in Œ0; 1 and define tj0 D .tj C tj 1 /=2. Then, Z 1 m1 1 X .xi / D K.xi y/f .y/ dy K.xi tj0 / f .tj0 /: m j D1 0 Hence we have the approximation  D ..x1 /; : : : ; .xn //t K f with Kij D K.xi tj0 / and fi D f .ti0 /. Writing the discretization error as ı D   K f , we arrive at the following model for the data vector y: y D K f C ı C ":

(1)

As m ! 1 the approximation of the integral improves but the matrix K becomes more ill-conditioned. To regularize the problem, we define an estimate fO of f using penalized least squares: fO D arg minm ky  K gk2 C 2 kDgk2 g2R

D .K K C 2 D t D/1 K t y Ly; t

where  > 0 is a fixed regularization parameter and D is a chosen matrix (e.g., a matrix that computes discrete derivatives). This regularization addresses the ill-conditioning of the matrix K ; it is a way of adding

S

1366

Statistical Methods for Uncertainty Quantification for Linear Inverse Problems

the prior information that we expect kDf k to be small. The case D D I is known as (discrete) TikhonovPhillips regularization. Note that we may write fO.xi / as a linear function of y: fO.xi / D e ti Ly where fe i g is the standard orthonormal basis in Rm . 

at a point (i.e., f ! f .x/) is a continuous linear functional.

Example 3 Let I D Œ0; 1. Let Wm .I / be the linear space of real-valued functions on I such that f has m1 continuous derivatives on I , f .m1/ is absolutely In the next two examples, we assume that H is a continuous on I (so f .m/ exists almost everywhere on Hilbert space, and each Ki W H ! R is a bounded lin- I ), and f .m/ 2 L2 .I /. The space Wm .I / is a Hilbert ear operator (and thus continuous). We write KŒf  D space with inner product .K1 Œf ; : : : ; Kn Œf /t . Z m1 Example 2 Since K.H /  Rn is finite dimensional, hf; gi D X f .k/ .0/ g .k/ .0/ C f .m/ .x/ g .m/ .x/ dx it follows that K is compact as is its adjoint K W I kD0 Rn ! H , and K K is a self-adjoint compact operator on H . In addition, there is a collection of orthonormal and has the following properties [2, 29]: (a) For every functions f k g in H , orthonormal vectors f vk g in Rn , x 2 I , there is a function x 2 Wm .I / such that and a positive, nonincreasing sequence .k / such that the linear functional f ! f .x/ is continuous on [22]: (a) f k g is an orthonormal basis for Null.K/? ; Wm .I / and given by f ! hx ; f i. The function (b) f vk g is an orthonormal basis for the closure of R W I  I ! R, R.x; y/ D hx ; y i is called Range.K/ in Rn ; and (c) KŒ k  D k vk and K Œvk  D a reproducing kernel of the Hilbert space, and (b) k k . Write f D f0 C f1 , with f0 2 Null.K/ and Wm .I / D Nm1 ˚ Hm , where Nm1 is the space ? f1 2 Null.K/ Pn . Then, there are constants ak such of polynomials of degree at most m  1 and Hm D that f1 D kD1 ak k . The data do not provide any f f 2 Wm .I / W f .k/ .0/ D 0 for k D 0; : : : ; m  1 g: information about f0 so without any other information Since the space Wm .I / satisfies (a), it is called a we have no way of estimating such component of reproducing kernel Hilbert space (RKHS). To control f . This introduces a systematic bias. The problem of the smoothness of the Tikhonov estimate, we put a estimating f is thus reduced to estimating f1 , that is, penalty on the derivative of f1 , which is the projection the coefficients ak . In fact, we may transform the data f1 D PH f onto Hm . To write the penalized sum to hy; vk i D k ak C h"; vk i and use them to estimate of squares, we use the fact that each functional Ki W the vector of coefficients a D .ak /; the transformed Wm .I / ! R is continuous and thus Ki f D hi ; f i data based on this sequence define a sequence space for some function i 2 Wm .I /. We can then write model [7, 18]. We may also rewrite the data as y D Z V a C "; where V D . 1 v1    n vn /. An estimate of .m/ 2 2 k yKf k C  .f1 .x//2 dx f is obtained using a penalized least-squares estimate I of a: n X . yi  hi ; f i /2 C 2 k PH f k2 : (3) D aO D arg minn ky  V bk2 C 2 kbk2 : j D1 b2R

This leads again to an estimate that is linear in y; write Define k .x/ D x k1 for k D 1; : : : ; m and k D it as aO D Ly for some matrix L. The estimate of f .x/ PH km for k D m C 1; : : : ; m C n. Then f D P is then similar to that in Example 1: k ak k C ı, where ı belongs to the orthogonal complement of the span of f k g. It can be shown that (2) the minimizer fO of (3) is again of the form (2) [29]. To fO.x/ D .x/t aO D .x/t Ly; estimate a we rewrite (3) as a function of a and use the t following estimate: with .x/ D . 1 .x/; : : : ; n .x// .  If the goal is to estimate pointwise values of f 2 aO D arg min k y  X b k2 C 2 btH PbH ; H , then the Hilbert space H needs to be defined b appropriately. For example, if H D L2 .Œ0; 1/, then pointwise values of f are not well defined. The fol- where X is the matrix of inner products hi ; j i, Pij D lowing example introduces spaces where evaluation hPH i ; PH j i and aH D .amC1 ; : : : ; amCn /t . 

Statistical Methods for Uncertainty Quantification for Linear Inverse Problems

These examples describe three different frameworks where the functional estimation is reduced to a finitedimensional penalized least-squares problem. They serve to motivate the framework we will use in the statistical analysis. We will focus on frequentist statistical methods. Bayesian methods for inverse problems are discussed in [17]; [1,23] provide a tutorial comparison of frequentist and Bayesian procedures for inverse problems. Consider first the simpler case of general linear regression: the n  1 data vector is modeled as y D K a C ", where K is an n  m matrix, n > m, K t K is non-singular and " is a random vector with mean zero and covariance matrix  2 I. The least-squares estimate of a is aO D arg min ky  K bk2 D . K t K /1 K t y; (4) b

ky  yk O 2 ky  yk O 2 D : O D n  dof.y/ O tr.I  H /

aO D arg min ky  K bk2 C 2 bt S b b

D . K K C 2 S /1 K t y; t

(6)

where S is a symmetric non-negative matrix and  > 0 is a fixed regularization parameter. The estimate of f O is defined as fO.x/ D .x/t a: Bias, Variance, and MSE For a fixed regularization parameter , the estimator fO.x/ is linear in y, and therefore its mean, bias, variance, and mean squared error can be determined using only knowledge of the first two moments of the distribution of the noise vector ". Using (6) we find the mean, bias, and variance of fO.x/: E. fO.x/ / D .x/t G  1 K t KŒf ; Bias. fO.x/ / D .x/t G  1 K t KŒf   f .x/

and it has the following properties: Its expected value is E.a/ O D a regardless of the true value a; that is, aO is an unbiased estimator of a. The covariance matrix of aO is Var.a/ O  2 .K t K /1 . An unbiased estimator of  2 2 is O D ky K ak O 2 =.nm/: Note that the denominator is the difference between the number of observations; it is a kind of “effective number of observations.” It can be shown that m D tr.H /, where H D K .K t K /1 K t is the hat matrix; it is the matrix defined by H y

yO D K a. O The degrees of freedom (dof) of yO is defined as the sum of the covariances of .K a/ O i with yi divided O D by  2 [24]. For linear regression we have dof.K a/ m D tr.H /. Hence we may write O 2 as the residual sum of squares normalized by the effective number of observations: 2

1367

(5)

Var. fO.x/ / D  2 kK G  .x/k2 ; where G  D . K t K C 2 S /1 . Hence, unlike the least-squares estimate of a (4), the penalized leastsquares estimate (6) is biased even when ı D 0. This bias introduces a bias in the estimates of f . In terms of a and ı, this bias is Bias. fO.x/ / D E. fO.x/ /  f .x/ D .x/t B  a C .x/t G  K t ı;

(7)

where B  D 2 G  S . Prior information about a should be used to choose the matrix S so that kB  ak is small. Note that similar formulas can be derived for correlated noise provided the covariance matrix is known. Also, analogous closed formulas can be derived for estimates of linear functionals of f . The mean squared error (MSE) can be used to include the bias and variance in the uncertainty evaluation of fO.x/; it is defined as the expected value of .fO.x/f .x//2 , which is equivalent to the squared bias plus the variance:

We now return to ill-posed inverse problems and define a general framework motivated by Examples 1– 2 that is similar to general linear regression. We assume that the data vector has a representation of the form y D KŒf  C " D K a C ı C "; where K is a linear MSE. fO.x/ / D Bias. fO.x/ /2 C Var. fO.x/ /: operator H ! Rn , K is an n  m matrix, ı is a fixed unknown vector (e.g., discretization error), and " is O a random vector of mean zero and covariance matrix The integrated mean squared error of f is Z  2 I. We also assume that there is an n  1 vector a t O and a vector function  such that f .x/ D .x/ a for IMSE.f / D E j fO.x/  f .x/ j2 dx all x. The vector a is estimated using penalized least D Bias.a/ O t F Bias.a/Ctr. O F G  K t K G  /; squares:

S

1368

Statistical Methods for Uncertainty Quantification for Linear Inverse Problems

R where F D .x/.x/t dx. The bias component of the MSE is the most difficult to assess as it depends on the unknown f (or a and ı), but, depending on the available prior information, some inequalities can be derived [20, 26].

information leads to the modeling of a and ı as random variables with means and covariance matrices a D Ea, ˙ a D Var.a/, ı D Eı and ˙ ı D Var.ı/, then the average bias is

EŒ Bias. fO.x/ /  D .x/t B  a C .x/t G  K t ı : Example 4 If H is a Hilbert space and the functionals Ki are bounded, then there are function i 2 H such Similarly, we can easily derive a bound for the mean that Ki Œf  D hi ; f i, and we may write squared bias that can be used to put a bound on the X average MSE. O ai .x/i ; f i D hAx ; f i; EŒ f .x/  D h Since the bias may play a significant factor in the i inference (in some geophysical applications the bias is P where the function Ax .y/ D i ai .x/i .y/ is called the dominant component of the MSE), it is important to the Backus-Gilbert averaging kernel for fO at x [3]. study the residuals of the fit to determine if a significant In particular, since we would like to have f .x/ D bias is present. The mean and covariance matrix of the hAx ; f i, we would like Ax to be as concentrated as residual vector r D y  K aO are possible around x; a plot of the function Ax may provide useful information about the mean of the estimate Er D K Bias.a/ O C ı D K B  a C .I  H  /ı (8) fO.x/. One may also summarize characteristics of jAx j such as its center and spread about the center (e.g., Var.r/ D  2 .I  H  /2 ; (9) [20]). Heuristically, jAx j should be like a ı-function centered at x. This can be formalized in an RKHS H . where H  D K G  K t is the hat matrix. Equation (8) In this case, there is a function x 2 H such that shows that if there is a significant bias, then we may f .x/ D hx ; f i and the bias of fO.x/ can be written see a trend in the residuals. From (8) we see that as Bias. fO.x/ / D h Ax  x ; f i and therefore the residuals are correlated and heteroscedastic (i.e., Var.ri / depends on i ) even if the bias is zero, which complicates the interpretation of the plots. To stabilize the variance, it is better to plot residuals that have been We can guarantee a small bias when Ax is close to x corrected for heteroscedasticity, for example, r 0 D i in H . In actual computations, averaging kernels can ri =.1  .H /ii /. be approximated using splines. A discussion of this topic as well as information about available software Confidence Intervals for splines and reproducing kernels can be found in In addition to the mean and variance of fO.x/, we [14, 20].  may construct confidence intervals that are expected to Another bound for the bias follows from (7) via the contain E.fO.x// with some prescribed probability. We j Bias. fO.x/ / j  k Ax  x k kf k:

Cauchy-Schwarz and triangle inequalities: j Bias. fO.x/ / j  kG  .x/k. 2 kS ak C kK t ık /: Plots of kG  .x/k (or k Ax  x k) as a function of x may provide geometric information (usually conservative) about the bias. Other measures such as the worst or an average bias can be obtained depending on the available prior information we have on f or its parametric representation. For example, if a and ı are known to lie in convex sets S1 and S2 , respectively, then we may determine the maximum of j Bias. fO / j subject to a 2 S1 and ı 2 S2 . Or, if the prior

now assume that the noise is Gaussian, "  N.0;  2 I/. We use ˚ to denote the cumulative distribution function of the standard Gaussian N.0; 1/ and write z˛ D ˚ 1 .1  ˛/. Since fO.x/ is a biased estimate of f .x/, we can only construct confidence intervals for EfO.x/. We should therefore interpret the intervals with caution as they may be incorrectly centered if the bias is significant. Under the Gaussianity assumption, a confidence interval I˛ .x; / for EfO.x/ of coverage 1  ˛ is I˛ .x; / D fO.x/ ˙ z˛=2 kK G  .x/k:

Statistical Methods for Uncertainty Quantification for Linear Inverse Problems

That is, for each x the probability that I˛ .x; / contains EfO.x/ is 1  ˛. If we have prior information to find an upper bound for the bias, jBias.fO.x//j  B.x/, then a confidence interval for f .x/ with coverage at least 1  ˛ is fO.x/ ˙ .z˛=2 kK G  .x/k C B.x//: If the goal is to detect structure by studying the confidence intervals I˛ .x; / for a range of values of x, then it is advisable to correct for the total number of intervals considered so as to control the rate of incorrect detections. One way to do this is by constructing 1  ˛ confidence intervals of the form I˛ .x; ˇ/ with simultaneous coverage for all x in some closed set S . This requires finding a constant ˇ > 0 such that PŒ EfO.x/ 2 I˛ .x; ˇ/; 8x 2 S   1  ˛; which is equivalent to  P sup jZ t V .x/j  ˇ  ˛; x2S

where V .x/ D K G  .x/=kK G  .x/k and Z1 ; : : : ; Zn are independent N.0; 1/. We can use results regarding the tail behavior of maxima of Gaussian processes to find an appropriate value of ˇ. For example, for the case when x 2 Œa; b [19] (see also [25]) shows that for ˇ large  v 2 P sup jZ t V .x/j  ˇ e ˇ =2 C 2.1  ˚.ˇ//;  x2S Rb where v D a kV 0 .x/k dx. It is not difficult to find a root of this nonlinear equation; the only potential problem may be computing v, but even an upper bound for it leads to intervals with simultaneous coverage at least 1  ˛. Similar results can be derived for the case when S is a subset of R2 or R3 [25]. An alternative approach is to use methods based on controlling the false discovery rate to correct for the interval coverage after the pointwise confidence intervals have been selected [4]. Estimating  and  The formulas for the bias, variance, and confidence intervals described so far require knowledge of  and a selection of  that is independent of the data y. If y is also used to estimate  or choose , then fO.x/ is no longer linear in y and closed formulas for the moments, bias, or confidence intervals are

1369

not available. Still, the formulas derived above with “reasonable” estimates O and O in place of  and  are approximately valid. This depends of course on the class of possible functions f , the noise distribution, the signal-to-noise ratio, and the ill-posedness of the problem. We recommend conducting realistic simulation studies to understand the actual performance of the estimates for a particular problem. Generalized cross-validation (GCV) methods to select  have proved useful in applications and theoretical studies. A discussion of these methods can be found in [13, 14, 29, 30]. We now summarize a few methods for obtaining an estimate of . The estimate of  2 given by (5) could be readily used for, once again, dof.K a/ O D tr.H  /, with the corresponding hat matrix H  (provided ı D 0). However, because of the bias of K aO and the fixed error ı, it is sometimes better to estimate  by considering the data as noisy observations of  D Ey D KŒf  – in which case we may assume ı D 0; that is, yi D i C"i . This approach is natural as  2 is the variance of the errors, "i , in the observations of i . To estimate the variance of "i , we need to remove the trend i . This trend may be seen as the values of a function .x/: i D .xi /. A variety of nonparametric regression methods can be used to estimate the function  so it can be removed, and an estimate of the noise variance can be obtained (e.g., [15, 16]). If the function can be assumed to be reasonably smooth, then we can use the framework described in 3 with Ki Œ D .xi / and a penalty in the second derivative P of . In this case .x/ O D ai i .x/ D at .x/ is called a spline smoothing estimate because it is a finite linear combination of spline functions i [15, 29]. The estimate of  2 defined by (5) with the corresponding hat matrix H  was proposed by [28]. Using (8) and (9) we find that the expected value of the residual sum of squares is

Eky  yk O 2 D  2 trΠ.I  H  /2  C at B  K t K B  a; (10) and thus O 2 is not an unbiased estimator of  2 even if the bias is zero (i.e., B  a D 0, which happens when  is linear), but it has been shown to have good asymptotic properties when  is selected using generalized cross-validation [13,28]. From (10) we see that a slight modification of O 2 leads to an estimate that is unbiased when B  a D 0 [5]

S

1370

Statistical Methods for Uncertainty Quantification for Linear Inverse Problems

O B2 D

ky  yk O 2 : trΠ.I  H  /2 

Simulation studies seem to indicate that this estimate has a smaller bias for a wider set of values of  [6]. This property is desirable as  is usually chosen adaptively. In some cases the effect of the trend can also be reduced using first- or second-order finite differences without having to choose a regularization parameter . For example, a first-order finite-difference estimate proposed by [21] is X 1 .yi C1  yi /2 : 2.n  1/ i D1

an estimate fOj of f following the same procedure used to obtain fO. The statistics of the sample of fOj are used as estimates of those of fO. One problem with this approach is that the bias of fO may lead to a poor estimate of KŒf  and thus to unrealistic simulated data. An introduction to bootstrap methods can be found in [9, 12]. For an example of bootstrap methods to construct confidence intervals for estimates of a function based on smoothing splines, see [31].

n1

O R2 D

The bias of O R2 is small if the local changes of .x/ are small. In particular, the bias is zero for a linear trend. Other estimators of  as well performance comparisons can be found in [5, 6]. Resampling Methods We have assumed a Gaussian noise distribution for the construction of confidence intervals. In addition, we have only considered linear operators and linear estimators. Nonlinear estimators arise even when the operator K is linear. For example, if  and  are estimated using the same data or if the penalized leastsquares estimate of a includes interval constraints (e.g., positivity), then the estimate aO is no longer linear in y. In some cases the use of bootstrap (resampling) methods allows us to assess statistical properties while relaxing the distributional and linearity assumptions. The idea is to simulate data y  as follows: the function estimate fO is used as a proxy for the unknown function f . Noise " is simulated using a parametric or nonparametric method. In the parametric bootstrap, " is sampled from the assumed distribution whose parameters are estimated from the data. For example, if the "i are independent N.0;  2 /, then "i is sampled from N.0; O 2 /. In the nonparametric bootstrap, "i is sampled with replacement from the vector of residuals of the fit. However, as Eqs. (8) and (9) show, even in the linear case, the residuals have to be corrected to behave approximately like the true errors. Of course, due to the bias and correlation of the residuals, these corrections are often difficult to derive and implement. Using "i and fO, one generates simulated data vectors y j D KŒfO C "j . For each such y j one computes

References 1. Aguilar, O., Allmaras, M., Bangerth, W., Tenorio, L.: Statistics of parameter estimates: a concrete example. SIAM Rev. 57, 131 (2015) 2. Aronszajn, N.: Theory of reproducing kernels. Trans. Am. Math. Soc. 89, 337 (1950) 3. Backus, G., Gilbert, F.: Uniqueness in the inversion of inaccurate gross earth data. Philos. Trans. R. Soc. Lond. A 266, 123 (1970) 4. Benjamini, Y., Yekutieli, D.: False discovery rate–adjusted multiple confidence intervals for selected parameters. J. Am. Stat. Assoc. 100, 71 (2005) 5. Buckley, M.J., Eagleson, G.K., Silverman, B.W.: The estimation of residual variance in nonparametric regression. Biometrika 75, 189 (1988) 6. Carter, C.K., Eagleson, G.K.: A comparison of variance estimators in nonparametric regression. J. R. Stat. Soc. B 54, 773 (1992) 7. Cavalier, L.: Nonparametric statistical inverse problems. Inverse Probl. 24, 034004 (2008) 8. Craven, P., Wahba, G.: Smoothing noisy data with splines. Numer. Math. 31, 377 (1979) 9. Davison, A.C., Hinkley, D.V.: Bootstrap Methods and Their Application. Cambridge University Press, Cambridge (1997) 10. Engl, H., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Kluwer, Dordrecht (1996) 11. Engl H his chapter 12. Efron, B., Tibshirani, R.J.: An Introduction to the Bootstrap. Chapman & Hall, New York (1993) 13. Gu, C.: Smoothing Spline ANOVA Models. Springer, Berlin/Heidelberg/New York (2002) 14. Gu, C.: Smoothing noisy data via regularization: statistical perspectives. Inverse Probl. 24, 034002 (2008) 15. Green, P.J., Silverman, B.W.: Nonparametric Regression and Generalized Linear Models: A Roughness Penalty Approach. Chapman & Hall, London (1993) 16. H¨ardle, W.K., M¨uller, M., Sperlich, S., Werwatz, A.: Nonparametric and Semiparametric Models. Springer, Berlin/Heidelberg/New York (2004) 17. Kaipio, J., Somersalo, E.: Statistical and Computational Inverse Problems. Springer, Berlin/Heidelberg/New York (2004)

Step Size Control 18. Mair, B., Ruymgaart, F.H.: Statistical estimation in Hilbert scale. SIAM J. Appl. Math. 56, 1424 (1996) 19. Naiman, D.Q.: Conservative confidence bands in curvilinear regression. Ann. Stat. 14, 896 (1986) 20. O’Sullivan, F.: A statistical perspective on ill-posed inverse problems. Stat. Sci. 1, 502 (1986) 21. Rice, J.: Bandwidth choice for nonparametric regression. Ann. Stat. 12, 1215 (1984) 22. Rudin, W.: Functional Analysis. McGraw-Hill, New York (1973) 23. Stark, P.B., Tenorio, L.: A primer of frequentist and Bayesian inference in inverse problems. In: Biegler, L., et al. (eds.) Computational Methods for Large-Scale Inverse Problems and Quantification of Uncertainty, pp. 9–32. Wiley, Chichester (2011) 24. Stein, C.: Estimation of the mean of a multivariate normal distribution. Ann. Stat. 9, 1135 (1981) 25. Sun, J., Loader, C.R.: Simultaneous confidence bands for liner regression and smoothing. Ann. Stat. 22, 1328 (1994) 26. Tenorio, L., Andersson, F., de Hoop, M., Ma, P.: Data analysis tools for uncertainty quantification of inverse problems. Inverse Probl. 29, 045001 (2011) 27. Vogel, C.R.: Computational Methods for Inverse Problems. SIAM, Philadelphia (2002) 28. Wahba, G.: Bayesian “confidence intervals” for the crossvalidated smoothing spline. J. R. Stat. Soc. B 45, 133 (1983) 29. Wahba, G.: Spline Models for Observational Data. CBMSNSF Regional Conference Series in Applied Mathematics, vol. 50. SIAM, Philadelphia (1990) 30. Wahba G her chapter 31. Wang, Y., Wahba, G.: Bootstrap confidence intervals for smoothing splines and their comparison to Bayesian confidence intervals. J. Stat. Comput. Simul. 51, 263 (1995)

Step Size Control Gustaf S¨oderlind Centre for Mathematical Sciences, Numerical Analysis, Lund University, Lund, Sweden

Introduction Step size control is used to make a numerical method that proceeds in a step-by-step fashion adaptive. This includes time stepping methods for solving initial value problems, nonlinear optimization methods, and continuation methods for solving nonlinear equations. The objective is to increase efficiency, but also includes managing the stability of the computation. This entry focuses exclusively on time stepping adaptivity in initial value problems. Special control

1371

algorithms continually adjust the step size in accordance with the local variation of the solution, attempting to compute a numerical solution to within a given error tolerance at minimal cost. As a typical integration may run over thousands of steps, the task is ideally suited to proven methods from automatic control. Assume that the problem to be solved is a dynamical system, dy D f .y/I y.0/ D y0 ; (1) dt with y.t/ 2 Rm . Without loss of generality, we may assume that the problem is solved numerically using a one-step integration procedure, explicit or implicit, written formally as ynC1 D ˆh .yn /I

y0 D y.0/;

(2)

where the map ˆh advances the solution one step of size h, from time tn to tnC1 D tn C h. Here the sequence yn is the numerical approximations to the exact solution, y.tn /. The difference en D yn  y.tn / is the global error of the numerical solution. If the method is of convergence order p, and the vector field f in (1) is sufficiently differentiable, then ken k D O.hp / as h ! 0. The accuracy of the numerical solution can also be evaluated locally. The local error ln is defined by y.tnC1 / C ln D ˆh .y.tn // :

(3)

Thus, if the method would take a step of size h, starting on the exact solution y.tn /, it will deviate from the exact solution at tnC1 by a small amount, ln . If the method is of order p, the local error will satisfy kln k D 'n hpC1 C O.hpC2 / I

h ! 0:

(4)

Here the principal error function 'n varies along the solution, and depends on the problem (in terms of derivatives of f ) as well as on the method. Using differential inequalities, it can be shown that the global and local errors are related by the a priori error bound ken k / max mn

klm k eM Œf tn  1  ; h M Œf 

(5)

S

1372

Step Size Control

where M Œf  is the logarithmic Lipschitz constant of f . Thus, the global error is bounded in terms of the local error per unit step, ln = h. For this reason, one can manage the global error by choosing the step size h so as to keep kln k= h D TOL during the integration, where TOL is a user-prescribed local error tolerance. The global error is then proportional to TOL, by a factor that reflects the intrinsic growth or decay of solutions to (1). Good initial value problem solvers usually produce numerical results that reflect this tolerance proportionality. By reducing TOL, one reduces the local error as well as the global error, while computational cost increases, as h  TOL1=p . Although it is possible to compute a posteriori global error estimates, such estimates are often costly. All widely used solvers therefore control the local error, relying on the relation (5), and the possibility of comparing several different numerical solutions computed for different values of TOL. There is no claim that the step size sequences are “optimal,” but in all problems where the principal error function varies by several orders of magnitude, as is the case in stiff differential equations, local error control is an inexpensive tool that offers vastly increased performance. It is a necessity for efficient computations. A time stepping method is made adaptive by providing a separate procedure for updating the step size as a function of the numerical solution. Thus, an adaptive method can be written formally as the interactive recursion ynC1 D ˆhn .yn /

(6)

hnC1 D ‰ynC1 .hn /;

(7)

where the first equation represents the numerical method and the second the step size control. If ‰y I (the identity map), the scheme reduces to a constant step size method. Otherwise, the interaction between the two dynamical systems implies that step size control interferes with the stability of the numerical method. For this reason, it is important that step size control algorithms are designed to increase efficiency without compromising stability.

Basic Multiplicative Control Modern time stepping methods provide a local error estimate. By using two methods of different orders,

computing two results, ynC1 and yOnC1 from yn , the solver estimates the local error by rn D kynC1  yOnC1 k. To control the error, the relation between step size and error is modeled by rn D 'On hkn :

(8)

Here k is the order of the local error estimator. Depending on the estimator’s construction, k may or may not equal p C 1, where p is the order of the method used to advance the solution. For control purposes, however, it is sufficient that k is known, and that the method operates in the asymptotic regime, meaning that (8) is an accurate model of the error for the step sizes in use. The common approach to varying the step size is multiplicative control hnC1 D n  hn ;

(9)

where the factor n needs to be determined so that the error estimate rn is kept near the target value TOL for all n. A simple control heuristic is derived by requiring that the next step size hnC1 solves the equation TOL D 'On hknC1 ; this assumes that 'n varies slowly. Thus, dividing this equation by (8), one obtains  hnC1 D

TOL

1=k

rn

hn :

(10)

This multiplicative control is found in many solvers. It is usually complemented by a range of safety measures, such as limiting the maximum step size increase, preventing “too small” step size changes, and special schemes for recomputing a step, should the estimated error be much larger than TOL. Although it often works well, the control law (10) and its safety measures have several disadvantages that call for more advanced feedback control schemes. Control theory and digital signal processing, both based on linear difference equations, offer a wide range of proven tools that are suitable for controlling the step size. Taking logarithms, (10) can be written as the linear difference equation log hnC1 D log hn 

1 log rOn ; k

(11)

Step Size Control

1373

where rOn D rn =TOL. This recursion continually changes the step size, unless log rOn is zero (i.e., rn D TOL). If rOn > 1 the step size decreases, and if rOn < 1 it increases. Thus, the error rn is kept near the set point TOL. As (11) is a summation process, the controller is referred to as an integrating controller, or I control. This integral action is necessary in order to eliminate a persistent error, and to find the step size that makes rn D TOL. The difference equation (11) may be viewed as using the explicit Euler method for integrating a differential equation that represents a continuous control. Just as there are many different methods for solving differential equations, however, there are many different discrete-time controllers that can potentially be optimized for different numerical methods or problem types, and offering different stability properties.

General Multiplicative Control In place of (11), a general controller takes the error sequence log rO D flog rOn g as input, and produces a step size sequence log h D flog hn g via a linear difference equation, .E  1/Q.E/ log h D P .E/ log rO :

(12)

Here E is the forward shift operator, and P and Q are two polynomials of equal degree, making the recursion explicit. The special case (11) has Q.E/ 1 and P .E/ 1=k, and is a one-step controller, while (12) in general is a multistep controller. Finally, the factor E  1 in (12) is akin to the consistency condition in linear multistep methods. Thus, if log rO 0, a solution to (12) is log h const: If, for example, P .z/ D ˇ1 z C ˇ0 and Q.z/ D z C ˛0 , then the recursion (12) is equivalent to the two-step multiplicative control,  hnC1 D

TOL

rn

ˇ1 

TOL

rn1

ˇ0 

hn hn1

˛0 hn : (13)

advanced controllers in existing codes, keeping in mind that a multistep controller is started either by using (10), or by merely putting all factors representing nonexistent starting data equal to one. Examples of how to choose the parameters in (13) are found in Table 1. A causal digital filter is a linear difference equation of the form (12), converting the input signal log rO to an output log h. This implies that digital control and filtering are intimately related. There are several important filter structures that fit the purpose of step size control, all covered by the general controller (13). Among them are finite impulse response (FIR) filters; proportional–integral (PI) controllers; autoregressive (AR) filters; and moving average (MA) filters. These filter classes are not mutually exclusive but can be combined. The elementary controller (10) is a FIR filter, also known as a deadbeat controller. Such controllers have the quickest dynamic response to variations in log ', O but also tend to produce nonsmooth step size sequences, and sometimes display ringing or stability problems. These problems can be eliminated by using PI controllers and MA filters that improve stability and suppress step size oscillations. Filter design is a matter of determining the filter coefficients with respect to order conditions and stability criteria, and is reminiscent of the construction of linear multistep methods [11].

Stability and Frequency Response Controllers are analyzed and designed by investigating the closed loop transfer function. In terms of the z transform of (12), the control action is log h D C.z/ log rO

(14)

where the control transfer function is given by

P .z/ : C.z/ D By taking logarithms, it is easily seen to correspond .z  1/Q.z/ to (12). One could include more factors following the same pattern, but in general, it rarely pays off to use a longer step size – error history than two Similarly, the error model (8) can be written to three steps. Because of the simple structure of log rO D k  log h C log 'O  log TOL: (13), it is relatively straightforward to include more

(15)

(16)

S

1374

Step Size Control

log ϕ ˆ log TOL

Controller

log h

Process

log

( ) −1

Step Size Control, Fig. 1 Time step adaptivity viewed as a feedback control system. The computational process takes a stepsize log h as input and produces an error estimate log r D k log h C log '. O Representing the ODE, the principal error function log 'O

enters as an additive disturbance, to be compensated by the controller. The error estimate log r is fed back and compared to log TOL. The controller constructs the next stepsize through log h D C.z/  .log TOL  log r/ (From [11])

These relations and their interaction are usually illustrated in a block diagram, see Fig. 1. Overall stability depends on the interaction between the controller C.z/ and the computational process. Inserting (16) into (14) and solving for log h yields

Step Size Control, Table 1 Some recommended two-step controllers. The H 211 controllers produce smooth step size sequences, using a moving average low-pass filter. In H 211b, the filter can be adjusted. Starting at b D 2 it is a deadbeat (FIR) filter; as the parameter b increases, dynamic response slows and high frequency suppression (smoothing) increases. Note that the ˇj coefficients are given in terms of the product kˇj for use with error estimators of different orders k. The ˛ coefficient is however independent of k (From [11])

log h D

C.z/ C.z/ log 'O C log TOL: 1 C kC.z/ 1 C kC.z/ (17) kˇ1

Here the closed loop transfer function H.z/ W log 'O 7! log h is defined by H.z/ D

C.z/ P .z/ D : (18) 1 C kC.z/ .z  1/Q.z/ C kP .z/

It determines the performance of the combined system of step size controller and computational process, and, in particular, how successful the controller will be in adjusting log h to log 'O so that log r log TOL. For a controller or a filter to be useful, the closed loop must be stable. This is determined by the poles of H.z/, which are the roots of the characteristic equation .z  1/Q.z/ C kP .z/ D 0. These must be located well inside the unit circle, and preferably have positive real parts, so that homogeneous solutions decay quickly without oscillations. To asses frequency response, one takes log 'O D fei! n g with ! 2 Œ0;  to investigate the output log h D H.ei! /fei! n g. The amplitude jH.ei! /j measures the attenuation of the frequency !. By choosing P such  that P .ei! / D 0 for some !  , it follows that H.ei! / D 0. Thus, zeros of P .z/ block signal transmission. The natural choice is !  D  so that P .1/ D 0, as this will annihilate .1/n oscillations, and produce a smooth step size sequence. This is achieved by the two H 211 controllers in Table 1. A smooth step size

kˇ0

˛0

Type

Name

Usage

3=5 1=b

1=5 1=b

– 1=b

PI MA

PI.4.2 H 211b

1=6

1=6

–

MA+PI

H 211 PI

Nonstiff solvers Stiff solvers; b 2 Œ2; 6 Stiff problems, smooth solutions

sequence is of importance, for example, to avoid higher order BDF methods to suffer stability problems.

Implementation and Modes of Operation Carefully implemented adaptivity algorithms are central for the code to operate efficiently and reliably for broad classes of problems. Apart from the accuracy requirements, which may be formulated in many different ways, there are several other factors of importance in connection with step size control. EPS Versus EPUS For a code that emphasizes asymptotically correct error estimates, controlling the local error per unit step kln k= hn is necessary in order to accumulate a targeted global error, over a fixed integration range, regardless of the number of steps needed to complete the integration. Abbreviated EPUS, this approach is viable for nonstiff problems, but tends to be costly for

Step Size Control

stiff problems, where strong dissipation usually means that the global error is dominated by the most recent local errors. There, controlling the local error per step kln k, referred to as EPS, is often a far more efficient option, if less well aligned with theory. In modern codes, the trend is generally to put less emphasis on asymptotically correct estimates, and control kln k directly. This has few practical drawbacks, but it makes it less straightforward to compare the performance of two different codes.

1375

di TOL

D

lOi : TOL  i C TOL  jyi j

(21)

Most codes employ two different tolerance parameters, ATOL and RTOL , defined by ATOLi D TOL  i and RTOL D TOL , respectively, replacing the denominator in (21) by ATOLi C RTOL  jyi j. Thus, the user controls the accuracy by the vector ATOL and the scalar RTOL. For scaling purposes, it is also important to note that TOL and RTOL are dimensionless, whereas ATOL is not. The actual computational setup will make the step Computational Stability size control operate differently as the user-selected Just as a well-conditioned problem depends continutolerance parameters affect both the set point and the ously on the data, the computational procedure should control objective. depend continuously on the various parameters that control the computation. In particular, for tolerance proportionality, there should be constants c and C such Interfering with the Controller that the global error e can be bounded above and below, In most codes, a step is rejected if it exceeds TOL by a small amount, say 20 %, calling for the step to   c  TOL  kek  C  TOL ; (19) be recomputed. As a correctly implemented controller is expectation-value correct, a too large error is alwhere the method is tolerance proportional if  D 1. most invariably compensated by other errors being too The smaller the ratio C =c, the better is the compu- small. It is, therefore, in general harmless to accept tational stability, but C =c can only be made small steps that exceed TOL by as much as a factor of 2, and with carefully implemented tools for adaptivity. Thus, indeed often preferable to minimize interference with with the elementary controller (10), prevented from the controller’s dynamics. Other types of interference may come from conmaking small step size changes, C =c is typically large, ditions that prevent “small” step size changes, as this whereas if the controller is based on a digital filter (here H 211b with b D 4, cf. Table 1) allowing a continual might call for a refactorization of the Jacobian. Howchange of the step size, the global error becomes a ever, such concerns are not warranted with smooth smooth function of TOL, see Fig. 2. This also shows controllers, which usually make small enough changes that the behavior of an implementation is significantly not to disturb the Newton process beyond what can be affected by the control algorithms, and how they are managed. On the contrary, a smoothly changing step size is beneficial for avoiding instability in multistep implemented. methods such as the BDF methods. It is however necessary to interfere with the conAbsolute and Relative Errors All modern codes provide options for controlling both troller’s action when there is a change of method order, absolute and relative errors. If at any given time, the or when a too large step size change is suggested. This estimated error is lO and the computed solution is y, then is equivalent to encountering an error that is much larger or smaller than TOL. In the first case, the step a weighted error vector d with components needs to be rejected, and in the second, the step size increase must be held back by a limiter. lOi (20) di D i C jyi j is constructed, where i is a scaling factor, determining a gradual switchover from relative to absolute error as yi ! 0. The error (and the step) is accepted if kd k  TOL , and the expression TOL=kd k corresponds to the factors TOL=r in (10) and (13). By (20),

Special Problems Conventional multiplicative control is not useful in connection with geometric integration, where it fails to preserve structure. The interaction (6, 7) shows that

S

Step Size Control 10−4

10−4

10−5

10−5

10−6

10−6 achieved accuracy

achieved accuracy

1376

10−7

10−8

10−9

10−10 10−10

10−7

10−8

10−9

10−8

10−6

10−4

10−10 10−10

10−8

TOL

Step Size Control, Fig. 2 Global error vs. TOL for a linear multistep code applied to a stiff nonlinear test problem. Left panel shows results when the controller is based on (10). In the right panel, it has been replaced by the digital filter H 211b.

adaptive step size selection adds dynamics, interfering with structure preserving integrators. A one-step method ˆh W yn 7! ynC1 is called symmetric if ˆ1 h D ˆh . This is a minimal requirement for the numerical integration of, for example, reversible Hamiltonian systems, in order to nearly preserve action variables in integrable problems. To make such a method adaptive, symmetric step size control is also needed. An invertible step size map ‰y W R ! R is called symmetric if ‰y is an involution, see Fig. 3. A symmetric ‰y then maps hn1 to hn and hn to hn1 , and only depends on yn ; with these conditions satisfied, the adaptive integration can be run in reverse time and retrace the numerical trajectory that was generated in forward time, [8]. However, this cannot be achieved by multiplicative controllers (9), and a special, nonlinear controller is therefore necessary. An explicit control recursion satisfying the requirements is either additive or inverse-additive, with the latter being preferable. Thus, a controller of the form 1 1  D G.yn / (22) hn hn1

10−6

10−4

TOL

Although computational effort remains unchanged, stability is much enhanced. The graphs also reveal that the code is not tolerance proportional

h

−1

−1

−h

Ψ Φh

Φh

Φ− h

Φ−h Ψ

−1

h

+1

−h

Step Size Control, Fig. 3 Symmetric adaptive integration in forward time (upper part), and reverse time (lower part) illustrate the interaction (6, 7). The symmetric step size map ‰y governs both h and h (From [8])

can be used, where the function G needs to be chosen with respect to the symmetry and geometric properties of the differential equation to be solved. This approach corresponds to constructing a Hamiltonian continuous control system, which is converted to the discrete controller (22) by geometric integration of the control system. This leaves the long-term behavior of the geometric integrator intact, even in the presence

Stochastic and Statistical Methods in Climate, Atmosphere, and Ocean Science

of step size variation. It is also worth noting that (22) generates a smooth step size sequence, as hn  hn1 D O.hn hn1 /. This type of control does not work with an error estimate, but rather tracks a prescribed target function; it corresponds to keeping hQ.y/ D const:, where Q is a given functional reflecting the geometric structure of (1). One can then take G.y/ D grad Q.y/f .y/=Q.y/. For example, in celestial mechanics, Q.y/ could be selected as total centripetal acceleration; then the step size is small when centripetal acceleration is large and vice versa, concentrating the computational effort to those intervals where the solution of the problem changes rapidly and is more sensitive to perturbations.

Literature Step size control has a long history, starting with the first initial value problem solvers around 1960, often using a simple step doubling/halving strategy. The controller (10) was soon introduced, and further developments quickly followed. Although the schemes were largely heuristic, performance tests and practical experience developed working standards. Monographs such as [1, 2, 6, 7, 10] all offer detailed descriptions. The first full control theoretic analysis is found in [3, 4], explaining and overcoming some previously noted difficulties, developing proportional-integral (PI) and autoregressive (AR) controllers. Synchronization with Newton iteration is discussed in [5]. A complete framework for using digital filters and signal processing is developed in [11], focusing on moving average (MA) controllers. Further developments on how to obtain improved computational stability are discussed in [12]. The special needs of geometric integration are discussed in [8], although the symmetric controllers are not based on error control. Error control in implicit, symmetric methods is analyzed in [13].

1377

3. Gustafsson, K.: Control theoretic techniques for stepsize selection in explicit Runge–Kutta methods. ACM TOMS 17, 533–554 (1991) 4. Gustafsson, K.: Control theoretic techniques for stepsize selection in implicit Runge–Kutta methods. ACM TOMS 20, 496–517 (1994) 5. Gustafsson, K., S¨oderlind, G.: Control strategies for the iterative solution of nonlinear equations in ODE solvers. SIAM J. Sci. Comp. 18, 23–40 (1997) 6. Hairer, E., N rsett, S.P., Wanner, G.: Solving Ordinary Differential Equations I: Nonstiff Problems, 2nd edn. Springer, Berlin (1993) 7. Hairer, E., Wanner, G.: Solving Ordinary Differential Equations II: Stiff and Differential-Algebraic Problems, 2nd edn. Springer, Berlin (1996) 8. Hairer, E., S¨oderlind, G.: Explicit, time reversible, adaptive step size control. SIAM J. Sci. Comp. 26, 1838–1851 (2005) 9. Hairer, E., Lubich, C., Wanner, G.: Geometric Numerical Integration. Structure-Preserving Algorithms for Ordinary Differential Equations, 2nd edn. Springer, Berlin (2006) 10. Shampine, L., Gordon, M.: Computer Solution of Ordinary Differential Equations: The Initial Value Problem. Freeman, San Francisco (1975) 11. S¨oderlind, G.: Digital filters in adaptive time-stepping. ACM Trans. Math. Softw. 29, 1–26 (2003) 12. S¨oderlind, G., Wang, L.: Adaptive time-stepping and computational stability. J. Comp. Methods Sci. Eng. 185, 225– 243 (2006) 13. Stoffer, D.: Variable steps for reversible integration methods. Computing 55, 1–22 (1995)

Stochastic and Statistical Methods in Climate, Atmosphere, and Ocean Science Daan Crommelin1;2 and Boualem Khouider3 1 Scientific Computing Group, Centrum Wiskunde and Informatica (CWI), Amsterdam, The Netherlands 2 Korteweg-de Vries Institute for Mathematics, University of Amsterdam, Amsterdam, The Netherlands 3 Department of Mathematics and Statistics, University of Victoria, Victoria, BC, Canada

Introduction References 1. Butcher, J.C.: Numerical Methods for Ordinary Differential Equations. Wiley, Chichester (2008) 2. Gear, C.W.: Numerical Initial Value Problems in Ordinary Differential Equations. Prentice Hall, Englewood Cliffs (1971)

The behavior of the atmosphere, oceans, and climate is intrinsically uncertain. The basic physical principles that govern atmospheric and oceanic flows are well known, for example, the Navier-Stokes equations for fluid flow, thermodynamic properties of moist air, and the effects of density stratification and Coriolis force.

S

1378

Stochastic and Statistical Methods in Climate, Atmosphere, and Ocean Science

Notwithstanding, there are major sources of randomness and uncertainty that prevent perfect prediction and complete understanding of these flows. The climate system involves a wide spectrum of space and time scales due to processes occurring on the order of microns and milliseconds such as the formation of cloud and rain droplets to global phenomena involving annual and decadal oscillations such as the EL Nio-Southern Oscillation (ENSO) and the Pacific Decadal Oscillation (PDO) [5]. Moreover, climate records display a spectral variability ranging from 1 cycle per month to 1 cycle per 100;000 years [23]. The complexity of the climate system stems in large part from the inherent nonlinearities of fluid mechanics and the phase changes of water substances. The atmosphere and oceans are turbulent, nonlinear systems that display chaotic behavior (e.g., [39]). The time evolutions of the same chaotic system starting from two slightly different initial states diverge exponentially fast, so that chaotic systems are marked by limited predictability. Beyond the so-called predictability horizon (on the order of 10 days for the atmosphere), initial state uncertainties (e.g., due to imperfect observations) have grown to the point that straightforward forecasts are no longer useful. Another major source of uncertainty stems from the fact that numerical models for atmospheric and oceanic flows cannot describe all relevant physical processes at once. These models are in essence discretized partial differential equations (PDEs), and the derivation of suitable PDEs (e.g., the so-called primitive equations) from more general ones that are less convenient for computation (e.g., the full Navier-Stokes equations) involves approximations and simplifications that introduce errors in the equations. Furthermore, as a result of spatial discretization of the PDEs, numerical models have finite resolution so that small-scale processes with length scales below the model grid scale are not resolved. These limitations are unavoidable, leading to model error and uncertainty. The uncertainties due to chaotic behavior and unresolved processes motivate the use of stochastic and statistical methods for modeling and understanding climate, atmosphere, and oceans. Models can be augmented with random elements in order to represent time-evolving uncertainties, leading to stochastic models. Weather forecasts and climate predictions are increasingly expressed in probabilistic terms,

making explicit the margins of uncertainty inherent to any prediction.

Statistical Methods For assessment and validation of models, a comparison of individual model trajectories is typically not suitable, because of the uncertainties described earlier. Rather, the statistical properties of models are used to summarize model behavior and to compare against other models and against observations. Examples are the mean and variance of spatial patterns of rainfall or sea surface temperature, the time evolution of global mean temperature, and the statistics of extreme events (e.g., hurricanes or heat waves). Part of the statistical methods used in this context is fairly general, not specifically tied to climate-atmosphere-ocean science (CAOS). However, other methods are rather specific for CAOS applications, and we will highlight some of these here. General references on statistical methods in CAOS are [61, 62]. EOFs A technique that is used widely in CAOS is Principal Component Analysis (PCA), also known as Empirical Orthogonal Function (EOF) analysis in CAOS. Consider a multivariate dataset ˆ 2 RM N . In CAOS this will typically be a time series .t1 /; .t2 /; : : : ; .tN / where each .tn / 2 RM is a spatial field (of, e.g., temperature or pressure). For simplicity we assume that the time mean has been subtracted from the dataset, so PN nD1 ˆmn D 0 8m. Let C be the M  M (sample) covariance matrix for this dataset: C D

1 ˆ ˆT : N 1

We denote by .m ; v m /, m D 1; : : : ; M the ordered eigenpairs of C : C v m D m v m ; m  mC1 8m: The ordering of the (positive) eigenvalues implies that the projection of the dataset onto the leading eigenvector v 1 gives the maximum variance among all projections. The next eigenvector v 2 gives the maximum variance among all projections orthogonal to v 1 , v 3 gives maximum variance among all projections

Stochastic and Statistical Methods in Climate, Atmosphere, and Ocean Science

P orthogonal to v 1 and v 2 , etc. The fraction m = l l equals the fraction of the total variance of the data captured by projection onto the m-th eigenvector v m . The eigenvectors v m are called the Empirical Orthogonal Functions (EOFs) or Principal Components (PCs). Projecting the original dataset ˆ onto the leading EOFs, i.e., the projection/reduction 0

r

.tn / D

M X

˛m .tn /v m ;

M 0 M;

be the result of, e.g., projecting the dataset ˆ onto the EOFs (in which case the data are time series of ˛.t/). These models are often cast as SDEs whose parameters must be estimated from the available time series. If the SDEs are restricted to have linear drift and additive noise (i.e., restricted to be those of a multivariate Ornstein-Uhlenbeck (OU) process), the estimation can be carried out for high-dimensional SDEs rather easily. That is, assume the SDEs have the form

mD1

can result in a substantial data reduction while retaining most of the variance of the original data. PCA is discussed in great detail in [27] and [59]. Over the years, various generalizations and alternatives for PCA have been formulated, for example, Principal Interaction and Oscillation Patterns [24], Nonlinear Principal Component Analysis (NLPCA) [49], and Nonlinear Laplacian Spectral Analysis (NLSA) [22]. These more advanced methods are designed to overcome limitations of PCA relating to the nonlinear or dynamical structure of datasets. In CAOS, the EOFs v m often correspond to spatial patterns. The shape of the patterns of leading EOFs can give insight in the physical-dynamical processes underlying the dataset ˆ. However, this must be done with caution, as the EOFs are statistical constructions and cannot always be interpreted as having physical or dynamical meaning in themselves (see [50] for a discussion). The temporal properties of the (time-dependent) coefficients ˛m .t/ can be analyzed by calculating, e.g., autocorrelation functions. Also, models for these coefficients can be formulated (in terms of ordinary differential equations (ODEs), stochastic differential equations (SDEs), etc.) that aim to capture the main dynamical properties of the original dataset or model variables .t/. For such reduced models, the emphasis is usually on the dynamics on large spatial scales and long time scales. These are embodied by the leading EOFs v m ; m D 1; : : : ; M 0 , and their corresponding coefficients ˛m .t/, so that a reduced model .M 0 M / can be well capable of capturing the main large-scale dynamical properties of the original dataset.

1379

d˛.t/ D B ˛.t/ dt C  dW.t/ ;

(1)

in which B and  are both a constant real M 0  M 0 matrix and W .t/ is an M 0 -dimensional vector of independent Wiener processes (for simplicity we assume that ˛ has zero mean). The parameters of this model are the matrix elements of B an . They can be estimated from two (lagged) covariance matrices of the time series. If we define Rij0 D E˛i .t/˛j .t/ ;

Rij D E˛i .t/˛j .t C / ;

with E denoting expectation, then for the OU process (1), we have the relations R D exp.B / R0 and BR0 C R0 B T C  T D 0

The latter of these is the fluctuation-dissipation relation for the OU process. By estimating R0 and R (with some > 0) from time series of ˛, estimates for B and A WD  T can be easily computed using these relations. This procedure is sometimes referred to as linear inverse modeling (LIM) in CAOS [55]. The matrix  cannot be uniquely determined from A; however, any  for which A D  T (e.g., obtained by Cholesky decomposition of A) will result in an OU process with the desired covariances R0 and R . As mentioned, LIM can be carried out rather easily for multivariate processes. This is a major advantage of LIM. A drawback is that the OU process (1) cannot capture non-Gaussian properties, so that LIM can only be used for data with Gaussian distributions. Also, the Inverse Modeling One way of arriving at reduced models is inverse mod- estimated B and A are sensitive to the choice of , eling, i.e., the dynamical model is obtained through unless the available time series is a an exact sampling statistical inference from time series data. The data can of (1).

S

1380

Stochastic and Statistical Methods in Climate, Atmosphere, and Ocean Science

Estimating diffusion processes with non-Gaussian properties is much more complicated. There are various estimation procedures available for SDEs with nonlinear drift and/or multiplicative noise; see, e.g., [30, 58] for an overview. However, the practical use of these procedures is often limited to SDEs with very low dimensions, due to curse of dimension or to computational feasibility. For an example application in CAOS, see, e.g., [4]. The dynamics of given time series can also be captured by reduced models that have discrete state spaces, rather than continuous ones as in the case of SDEs. There are a number of studies in CAOS that employ finite-state Markov chains for this purpose (e.g., [8,48,53]). It usually requires discretization of the state space; this can be achieved with, e.g., clustering methods. A more advanced methodology, building on the concept of Markov chains yet resulting in continuous state spaces, is that of hidden Markov models. These have been used, e.g., to model rainfall data (e.g., [3, 63]) and to study regime behavior in large-scale atmospheric dynamics [41]. Yet a more sophisticated methodology that combines the clustering and Markov chain concepts, specifically designed for nonstationary processes, can be found in [25]. Extreme Events The occurrence of extreme meteorological events, such as hurricanes, extreme rainfall, and heat waves, is of great importance because of their societal impact. Statistical methods to study extreme events are therefore used extensively in CAOS. The key question for studying extremes with statistical methods is to be able to assess the probability of certain events, having only a dataset available that is too short to contain more than a few of these events (and occasionally, too short to contain even a single event of interest). For example, how can one assess the probability of sea water level at some coastal location being more than 5 m above average if only 100 years of observational data for that location is available, with a maximum of 4 m above average? Such questions can be made accessible using extreme value theory. General introductions to extreme value theory are, e.g., [7] and [11]. For recent research on extremes in the context of climate science, see, e.g., [29] and the collection [1]. The classical theory deals with sequences or observations of N independent and identically distributed (iid) random variables, denoted here by r1 ; : : : ; rN .

Let MN be the maximum of this sequence, MN D maxfr1 ; : : : ; rN g. If the probability distribution for MN can be rescaled so that it converges in the limit of increasingly long sequences (i.e., N ! 1), it converges to a generalized extreme value (GEV) distribution. More precisely, if there are sequences aN .> 0/ and bN such that Prob..MN  bN /=aN  z/ ! G.z/ as N ! 1, then  

z   1=  : G.z/ D exp  1 C  G.z/ is a GEV distribution, with parameters  (location),  > 0 (scale), and (shape). It combines the Fr´echet ( > 0), Weibull ( < 0), and Gumbel ( ! 0) families of extreme value distributions. Note that this result is independent of the precise distribution of the random variables rn . The parameters ; ; can be inferred by dividing the observations r1 ; r2 ; : : : in blocks of equal length and considering the maxima on these blocks (the so-called block maxima approach). An alternative method for characterizing extremes, making more efficient use of available data than the block maxima approach, is known as the peaks-overthreshold (POT) approach. The idea is to set a threshold, say r  , and study the distribution of all observations rn that exceed this threshold. Thus, the object of interest is the conditional probability distribution Prob.rn  r  > z j rn > r  /, with z > 0. Under fairly general conditions, this distribution converges to 1  H.z/ for high thresholds r  , where H.z/ is the generalized Pareto distribution (GPD):   z 1= : H.z/ D 1  1 C Q The parameters of the GPD family of distributions are directly related to those of the GEV distribution: the shape parameter is the same in both, whereas the threshold-dependent scale parameter is Q D  C .r   / with  and  as in the GEV distribution. By inferring the parameters of the GPD or GEV distributions from a given dataset, one can calculate probabilities of extremes that are not present themselves in that dataset (but have the same underlying distribution as the available data). In principle, this makes it possible to assess risks of events that have not been observed, provided the conditions on convergence to GPD or GEV distributions are met.

Stochastic and Statistical Methods in Climate, Atmosphere, and Ocean Science

As mentioned, classical results on extreme value theory apply to iid random variables. These results have been generalized to time-correlated random variables, both stationary and nonstationary [7]. This is important for weather and climate applications, where datasets considered in the context of extremes are often time series. Another relevant topic is the development of multivariate extreme value theory [11].

Stochastic Methods Given the sheer complexity of climate-atmosphereocean (CAO) dynamics, when studying the global climate system or some parts of global oscillation patterns such ENSO or PDO, it is natural to try to separate the global dynamics occurring on longer time scales from local processes which occur on much shorter scales. Moreover, as mentioned before, climate and weather prediction models are based on a numerical discretization of the equations of motion, and due to limitations in computing resources, it is simply impossible to represent the wide range of space and time scales involved in CAO. Instead, general circulation models (GCMs) rely on parameterization schemes to represent the effect of the small/unresolved scales on the large/resolved scales. Below, we briefly illustrate how stochastic models are used in CAO both to build theoretical models that separate small-scale (noise) and large-scale dynamics and to “parameterize” the effect of small scales on large scales. A good snapshot on the state of the art, during the last two decades or so, in stochastic climate modeling research can be found in [26, 52].

1381

where t is time and u.x; v/ and v.x; y/ contain the external forcing and internal dynamics that couple the slow and fast variables. Hasselmann assumes a large scale-separation D between the slow and fast time scales: ! y 1 ! 1   dyj dxi x D O xi , for O yj dt dt all components i and j . The time scale separation was used earlier to justify statistical dynamical models (SDM) used then to track the dynamics of the climate system alone under the influence of external forcing. Without the variability due to the internal interactions of CAO, the SDMs failed badly to explain the observed “red” spectrum which characterizes low-frequency variability of CAO. Hasselmann made the analogy with the Brownian motion (BM), modeling the erratic movements of a few large particles immersed in a fluid that are subject to bombardments by the rapidly moving fluid molecules as a “natural” extension of the SDM models. Moreover, Hasselmann [23] assumes that the variability of x can be divided into a mean tendency hdx=dti D hu.x; y/i (Here h:i denotes average with respect to the joint distribution of the fast variables.) and a fluctuation tendency dx0 =dt D u.x; y/  hu.x; y/i D u0 .x; y/ which, according to the Brownian motion problem, is assumed to be a pure diffusion process or white noise. However, unlike BM, Hasselmann argued that for the weather and climate system, the statistics of y are not in equilibrium but depend on the slowly evolving large-scale dynamics and thus can only be obtained empirically. To avoid linear growth of the covariance matrix hx 0 ˝ x 0 i, Hasselmann assumes a damping term proportional to the divergence of the background 0 .!/ D frequency F .0/ of hx 0 ˝ x 0 i, where ı.! R 1 !0 /Fiji 1 0 hVi .!/Vj .! /i with V .!/ D 2 1 u .t/e !t dt. This leads to the Fokker-Plank equation: [23]

Model Reduction for Noise-Driven Large-Scale Dynamics In an attempt to explain the observed low-frequency variability of CAO, Hasselmann [23] splits the system @p.x; t/ Crx .Ou.x/p.x; t// D rx .Drx p.x; t// (3) into slow climate components (e.g., oceans, biosphere, @t cryosphere), denoted by the vector x, and fast components representing the weather, i.e., atmospheric for the distribution p.x; t/ of x.t/ as a stochastic variability, denoted by a vector y. The full climate process given that x.0/ D x0 , where D is the normalized covariance matrix D D hx 0 ˝ x 0 i=2t and system takes the form uO D huirx F .0/. Given the knowledge of the mean dx statistical forcing hui, the evolution equation for p can D u.x; y/ (2) dt be determined from the time series of x obtained either from a climate model simulation of from observations. dy D v.x; y/; Notice also that for a large number of slow variables xi , dt

S

1382

Stochastic and Statistical Methods in Climate, Atmosphere, and Ocean Science

the PDE in (3) is impractical; instead, one can always discrete form, is possible as illustrated by the MTV resort to Monte Carlo simulations using the associated theory presented next. Langevin equation: The Systematic Mode Reduction MTV Methodology A systematic mathematical methodology to derive (4) dx D uO .x/dt C †.x/dW t Langevin-type equations (4) a` la Hasselmann, for the slow climate dynamics from the coupled atmospherewhere †.x/†.x/T D D.x/. However, the functional ocean-land equations of motion, which yields the dependence of uO and D remains ambiguous, and rely- propagation (or drift) and diffusion terms uO .x/ and ing on rather empirical methods to define such terms D.x/ in closed form, is presented in [44,45] by Majda, is unsatisfactory. Nonetheless, Hasselmann introduced Timofeyev, and Vanden-Eijnden (MTV). Starting from the generalized form of the discretized a “linear feedback” version of his model where the drift or propagation term is a negative definite linear equations of motion operator: uO .x/ D Ux and D is constant, independent dz of x as an approximation for short time excursions of D Lz C B.z; z/ C f .t/ dt the climate variables. In this case, p.x; t/ is simply a Gaussian distribution whose time-dependent mean and variance are determined by the matrices D and U as where L and B are a linear and a bilinear operators while f .t/ represent external forcing, MTV operate noted in the inverse modeling section above. Due to its simplicity, the linear feedback model is the same dichotomy as Hasselmann did of splitting widely used to study the low-frequency variability of the vector z into slow and fast variables x and y, revarious climate processes. It is, for instance, used in spectively. However, they introduced a nondimensional [17] to reproduce the observed red spectrum of the parameter  D y = x which measures the degree of sea surface temperature in midlatitudes using simula- time scale separation between the two sets of variables. tion data from a simplified coupled ocean-atmosphere This leads to the slow-fast coupled system model. However, this linear model has severe limi1 1 tations of, for example, not being able to represent dx D .L11 x C L12 y/ dt C B11 .x; x/dt  1  1 deviations from Gaussian distribution of some climate C  1 B12 .x; y/ C B22 .y; y/ dt phenomena [13, 14, 17, 36, 51, 54]. It is thus natural C Dxdt C F1 .t/dt C  1 f1 . 1 t/ (5) to try to reincorporate a nonlinearity of some kind   into the model. The most popular idea consisted in dy D 1 L x C L y C B 2 .x; y/ C B 2 .y; y/ dt 21 22 12 22 making the matrix D or equivalently † dependent on 2 1 1 1   ydt C  dW t C  f2 . t/ x (quadratically for D or linearly for † as a next order Taylor correction) to which is tied the notion of multiplicative versus additive (when D is constant) under a few key assumptions, including (1) the nonnoise [37, 60]. Beside the crude approximation, the linear self interaction term of the fast variables is apparent advantage of this approach is the maintaining “parameterized” by an Ornstein-Uhlenbeck process: p 1 2 .y; y/dt WD  1 ydt C  dW t and (2) a small of the stabilizing linear operator U in place although it B22 dissipation term Dxdt is added to the slow dynamics is not universally justified. A mathematical justification for Hasselmann’s while (3) the slow variable forcing term assumes slow framework is provided by Arnold and his collaborators and fast contributions f1 .t/ D F1 .t/ C f1 .t/. More(see [2] and references therein). It is based on the over, the system in (5) is written in terms of the slow well-known technique of averaging (the law of large time t ! t. MTV used the theory of asymptotic expansion apnumbers) and the central limit theorem. However, as in Hasselmann’s original work, it assumes the existence plied to the backward Fokker-Plank equation associand knowledge of the invariant measure of the ated with the stochastic differential system in (5) to fast variables. Nonetheless, a rigorous mathematical obtain an effective reduced Langevin equation (4) for derivation of such Langevin-type models for the slow the slow variables x in the limit of large separation climate dynamics, using the equations of motion in of time scales  ! 0 [44, 45]. The main advantage

Stochastic and Statistical Methods in Climate, Atmosphere, and Ocean Science

of the MTV theory is that unlike Hasselmann’s ad hoc formulation, the functional form of the drift and diffusion coefficients, in terms of the slow variables, are obtained and new physical phenomena can emerge from the large-scale feedback besides the assumed stabilization effect. It turns out that the drift term is not always stabilizing, but there are dynamical regimes where growing modes can be excited and, depending on the dynamical configuration, the Langevin equation (4) can support either additive or multiplicative noise. Even though MTV assumes strict separation of scales,  1, it is successfully used for a wide range of examples including cases where  D O.1/ [46]. Also in [47], MTV is successfully extended to fully deterministic systems where the requirement that the fast-fast interaction term B22 .y; y/ in (5) is parameterized by an Ornstein-Uhlenbeck process is relaxed. Furthermore, MTV is applied to a wide range of climate problems. It is used, for instance, in [19] for a realistic barotropic model and extended in [18] to a three-layer quasi-geostrophic model. The example of midlatitude teleconnection patterns where multiplicative noise plays a crucial role is studied in [42]. MTV is also applied to the triad and dyad normal mode (EOF) interactions for arbitrary time series [40]. Stochastic Parametrization In a typical GCM, the parametrization of unresolved processes is based on theoretical and/or empirical deterministic equations. Perhaps the area where deterministic parameterizations have failed the most is moist convection. GCMs fail very badly in simulating the planetary and intra-seasonal variability of winds and rainfall in the tropics due to the inadequate representation of the unresolved variability of convection and the associated crossscale interactions behind the multiscale organization of tropical convection [35]. To overcome this problem, some climate scientists introduced random variables to mimic the variability of such unresolved processes. Unfortunately, as illustrated below, many of the existing stochastic parametrizations were based on the assumptions of statistical equilibrium and/or of a stationary distribution for the unresolved variability, which are only valid to some extent when there is scale separation. The first use of random variables in CGMs appeared in Buizza et al. [6] as means for improving the skill of the ECMWF ensemble prediction system (EPS).

1383

Buizza et al. [6] used uniformly distributed random scalars to rescale the parameterized tendencies in the governing equations. Similarly, Lin and Neelin [38] introduced a random perturbation in the tendency of convective available potential energy (CAPE). In [38], the random noise is assumed to be a Markov process of the form t C t D t t C zt where zt is a white noise with a fixed standard deviation and t is a parameter. Plant and Craig [57] used extensive cloud-permitting numerical simulations to empirically derive the parameters for the PDF of the cloud base mass flux itself whose Poisson shape is determined according to arguments drawn from equilibrium statistical mechanics. Careful simulations conducted by Davoudi et al. [10] revealed that while the Poisson PDF is more or less accurate for isolated deep convective clouds, it fails to extend to cloud clusters where a variety of cloud types interact with each other: a crucial feature of organized tropical convection. Majda and Khouider [43] borrowed an idea from material science [28] of using the Ising model of ferromagnetization to represent convective inhibition (CIN). An order parameter , defined on a rectangular lattice, embedded within each horizontal grid box of the climate model, takes values 1 or 0 at a given site, according to whether there is CIN or there is potential for deep convection (PAC). The lattice model makes transitions at a given site according to intuitive probability rules depending both on the large-scale climate model variables and on local interactions between lattice sites based on a Hamiltonian energy principle. The Hamiltonian is given by

H.; U / D 

X 1X J.jxyj/.x/.y/Ch.U / x 2 x;y x

where J.r/ is the local interaction potential and h.U / is the external potential which depends on the climate variables U and where the summations are taken over all lattice sites x; y. A transition (spin-flip by analogy to the Ising model of magnetization) occurs at a site y if for a small time , we have t C .y/ D 1  t .y/ and t C .x/ D t .x/ if x ¤ y. Transitions occur at a rate C.y; ; U / set by Arrhenius dynamics: C.x; ; U / D 1I exp. x H.; U // if x D 0 and C.x; ; U / D 1I if x D 1 so that the resulting Markov process satisfies detailed balance with respect to the Gibbs distribution .; U / / exp.H.; U /. Here

S

1384

Stochastic and Statistical Methods in Climate, Atmosphere, and Ocean Science

x H.; U // D H. CŒ1.x/ex /; U /H.; U / D P  z J.jx  zj/.z/ C h.U / with ex .y/ D 1 if y D x and 0 otherwise. For computational efficiency, a coarse graining of the stochastic CIN model is used in [34] to derive a stochastic birth-death process for the mesoscopic area P coverage X D x2X .x/ where X represents a generic site of a mesoscopic lattice, which in practice can be considered to be the GCM grid. The stochastic CIN model is coupled to a toy GCM where it is successfully demonstrated how the addition of such a stochastic model could improve the climatology and waves dynamics in a deficient GCM [34, 42]. This Ising-type modeling framework is extended in [33] to represent the variability of organized tropical convection (OTC). A multi-type order parameter is introduced to mimic the multimodal nature of OTC. Based on observations, tropical convective systems (TCS) are characterized by three cloud types, cumulus congestus whose height does not exceed the freezing level develop when the atmosphere is dry, and there is convective instability, positive CAPE. In return congestus clouds moisten the environment for deep convective towers. Stratiform clouds that develop in the upper troposphere lag deep convection as a natural freezing phase in the upper troposphere. Accordingly, the new order parameter  takes the multiple values 0,1,2,3, on a given lattice site, according to whether the given site is, respectively, clear sky or occupied by a congestus, deep, or stratiform cloud. Similar Arrhenius-type dynamics are used to build transition rates resulting in an ergodic Markov process with a well-defined equilibrium measure. Unphysical transitions of congestus to stratiform, stratiform to deep, stratiform to congestus, clear to stratiform, and deep to congestus were eliminated by setting the associated rates to zero. When local interactions are ignored, the equilibrium measure and the transition rates depend only on the large-scale climate variables U where CAPE and midlevel moisture are used as triggers and the coarse-graining process is carried with exact statistics. It leads to a multidimensional birthdeath process with immigration for the area fractions of the associated three cloud types. The stochastic multicloud model (SMCM) is used very successfully in [20, 21] to capture the unresolved variability of organized convection in a toy GCM. The simulation of convectively coupled gravity waves and mean

climatology were improved drastically when compared to their deterministic counterparts. The realistic statistical behavior of the SMCM is successfully assessed against observations in [56]. Local interaction effects are reintroduced in [32] where a coarse-graining approximation based on conditional expectation is used to recover the multidimensional birth-death process dynamics with local interactions. A Bayesian methodology for inferring key parameters for the SMCM is developed and validated in [12]. A review of the basic methodology of the CIN and SMCM models, which is suitable for undergraduates, is found in [31]. A systematic data-based methodology for inferring a suitable stochastic process for unresolved processes conditional on resolved model variables was proposed in [9]. The local feedback from unresolved processes on resolved ones is represented by a small Markov chain whose transition probability matrix is made dependent on the resolved-scale state. The matrix is estimated from time series data that is obtained from highly resolved numerical simulations or observations. This approach was developed and successfully tested on the Lorenz ’96 system [39] in [9]. [16] applied it to parameterize shallow cumulus convection, using data from large eddy simulation (LES) of moist atmospheric convection. A two-dimensional lattice, with at each lattice node a Markov chain, was used to mimic (or emulate) the convection as simulated by the highresolution LES model, at a fraction of the computational cost. Subsequently, [15] combined the conditional Markov chain methodology with elements from the SMCM [33]. They applied it to deep convection but without making use of the Arrhenius functional forms of the transition rates in terms of the large-scale variables (as was done in [33]). Similar to [16], LES data was used for estimation of the Markov chain transition probabilities. The inferred stochastic model in [15] was well capable of generating cloud fractions very similar to those observed in the LES data. While the main cloud types of the original SMCM were preserved, an important improvement in [15] resides in the addition of a fifth state for shallow cumulus clouds. As an experiment, direct spatial coupling of the Markov chains on the lattice was also considered in [15]. Such coupling amounts to the structure of a stochastic cellular automaton (SCA). Without this direct coupling, the Markov chains are still coupled,

Stochastic and Statistical Methods in Climate, Atmosphere, and Ocean Science

1385

but indirectly, through their interaction with the large- 18. Franzke, C., Majda, A.: Low order stochastic mode reduction for a prototype atmospheric GCM. J. Atmos. Sci. 63, scale variables (see, e.g., [9]).

References 1. AghaKouchak, A., Easterling, D., Hsu, K., Schubert, S., Sorooshian, S. (eds.): Extremes in a Changing Climate, p. 423. Springer, Dordrecht (2013) 2. Arnold, L.: Hasselmann’s program revisited: the analysis of stochasticity in deterministic climate models. In: Imkeller, P., von Storch, J.-S. (eds) Stochastic Climate Models. Birkh¨auser, Basel (2001) 3. Bellone, E., Hughes, J.P., Guttorp, P.: A Hidden Markov model for downscaling synoptic atmospheric patterns to precipitation amounts. Clim. Res. 15, 1–12 (2000) 4. Berner, J.: Linking nonlinearity and non-Gaussianity of planetary wave behavior by the Fokker-Planck equation. J. Atmos. Sci. 62, 2098–2117 (2005) 5. Bond, N.A., Harrison, D.E.: The pacific decadal oscillation, air-sea interaction and central north pacific winter atmospheric regimes. Geophys. Res. Lett. 27, 731–734 (2000) 6. Buizza, R., Milleer, M., Palmer, T.N.: Stochastic representation of model uncertainties in the ECMWF ensemble prediction system. Quart. J. R. Meteorol. Soc. 125, 2887–2908 (1999) 7. Coles, S.: An Introduction to Statistical Modeling of Extreme Values, p. 208. Springer, London (2001); Statistical Methods in the Atmospheric Sciences, 3rd edn., p. 704. Academic, Oxford 8. Crommelin, D.T.: Observed non-diffusive dynamics in large-scale atmospheric flow. J. Atmos. Sci. 61, 2384–239 (2004) 9. Crommelin, D.T., Vanden-Eijnden, E.: Subgrid-scale parameterization with conditional Markov chains. J. Atmos. Sci. 65, 2661–2675 (2008) 10. Davoudi, J., McFarlane, N.A., Birner, T.: Fluctuation of mass flux in a cloud-resolving simulation with interactive radiation. J. Atmos. Sci. 67, 400–418 (2010) 11. de Haan, L., Ferreira, A.: Extreme Value Theory: An Introduction, p. 417. Springer, New York (2006) 12. de la Chevroti`ere, M., Khouider, B., Majda, A.: Calibration of the stochastic multicloud model using Bayesian inference. SIAM J. Sci. Comput. 36(3), B538–B560 (2014) 13. DelSole, T.: Stochastic models of quasi-geostrophic turbulence. Surv. Geophys. 25, 107–149 (2004) 14. DelSole, T., Farrel, B.F.: Quasi-linear equilibration of a thermally maintained, stochastically excited jet in a quasigeostrophic model. J. Atmos. Sci. 53, 1781–1797 (1996) 15. Dorrestijn, J., Crommelin, D.T., Biello, J.A., B¨oing, S.J.: A data-driven multicloud model for stochastic parameterization of deep convection. Phil. Trans. R. Soc. A 371, 20120374 (2013) 16. Dorrestijn, J., Crommelin, D.T., Siebesma, A.P., Jonker, H.J.J.: Stochastic parameterization of shallow cumulus convection estimated from high-resolution model data. Theor. Comput. Fluid Dyn. 27, 133–148 (2013) 17. Frankignoul, C., Hasselmann, K.: Stochastic climate models, Part II application to sea-surface temperature anomalies and thermocline variability. Tellus 29, 289–305 (1977)

457–479 (2006) 19. Franzke, C., Majda, A., Vanden-Eijnden, E.: Low-order stochastic mode reduction for a realistic barotropic model climate. J. Atmos. Sci. 62, 1722–1745 (2005) 20. Frenkel, Y., Majda, J., Khouider, B.: Using the stochastic multicloud model to improve tropical convective parameterization: a paradigm example. J. Atmos. Sci. 69, 1080–1105 (2012) 21. Frenkel, Y., Majda, J., Khouider, B.: Stochastic and deterministic multicloud parameterizations for tropical convection. Clim. Dyn. 41, 1527–1551 (2013) 22. Giannakis, D., Majda, A.J.: Nonlinear Laplacian spectral analysis for time series with intermittency and lowfrequency variability. Proc. Natl. Acad. Sci. USA 109, 2222–2227 (2012) 23. Hasselmann, K.: Stochastic climate models. Part I, theory. Tellus 28, 473–485 (1976) 24. Hasselmann, K.: PIPs and POPs: the reduction of complex dynamical systems using principal interaction and oscillation patterns. J. Geophys. Res. 93(D9), 11015–11021 (1988) 25. Horenko, I.: Nonstationarity in multifactor models of discrete jump processes, memory, and application to cloud modeling. J. Atmos. Sci. 68, 1493–1506 (2011) 26. Imkeller P., von Storch J.-S. (eds.): Stochastic Climate Models. Birkh¨auser, Basel (2001) 27. Jolliffe, I.T.: Principal Component Analysis, 2nd edn. Springer, New York (2002) 28. Katsoulakis, M., Majda, A.J., Vlachos, D.: Coarse-Grained stochastic processes and Monte-Carlo simulations in lattice systems. J. Comp. Phys. 186, 250–278 (2003) 29. Katz, R.W., Naveau P.: Editorial: special issue on statistics of extremes in weather and climate. Extremes 13, 107–108 (2010) 30. Kessler, M., Lindner, A., Sørensen, M. (eds.): Statistical Methods for Stochastic Differential Equations. CRC, Boca Raton (2012) 31. Khouider, B.: Markov-jump stochastic models for organized tropical convection. In: Yang, X.-S. (ed.) Mathematical Modeling With Multidisciplinary Applications. Wiley, (2013). ISBN: 978-1-1182-9441-3 32. Khouider, B.: A stochastic coarse grained multi-type particle interacting model for tropical convection. Commun. Math. Sci. 12, 1379–1407 (2014) 33. Khouider, B., Biello, J., Majda, A.J.: A stochastic multicloud model for tropical convection. Commun. Math. Sci. 8, 187–216 (2010) 34. Khouider, B., Majda, A.J., Katsoulakis, M.: Coarse grained stochastic models for tropical convection. Proc. Natl. Acad. Sci. USA 100, 11941–11946 (2003) 35. Khouider, B., Majda, A.J., Stechmann, S.: Climate science in the tropics: waves, vortices, and PDEs. Nonlinearity 26, R1-R68 (2013) 36. Kleeman, R., Moore, A.: A theory for the limitation of ENSO predictability due to stochastic atmospheric transients. J. Atmos. Sci. 54, 753–767 (1997) 37. Kondrasov, D., Krastov, S., Gill, M.: Empirical mode reduction in a model of extra-tropical low-frequency variability. J. Atmos. Sci. 63, 1859–1877 (2006)

S

1386 38. Lin, J.B.W., Neelin, D.: Toward stochastic deep convective parameterization in general circulation models. Geophy. Res. Lett. 27, 3691–3694 (2000) 39. Lorenz, E.N.: Predictability: a problem partly solved. In: Proceedings, Seminar on Predictability ECMWF, Reading, vol. 1, pp. 1–18 (1996) 40. Majda, A., Franzke, C., Crommelin, D.: Normal forms for reduced stochastic climate models. Proc. Natl. Acad. Sci. USA 16, 3649–3653 (2009) 41. Majda, A.J., Franzke, C.L., Fischer, A., Crommelin, D.T.: Distinct metastable atmospheric regimes despite nearly Gaussian statistics: a paradigm model. Proc. Natl. Acad. Sci. USA 103, 8309–8314 (2006) 42. Majda, A.J., Franzke, C.L., Khouider, B.: An applied mathematics perspective on stochastic modelling for climate. Phil. Trans. R. Soc. 366, 2427–2453 (2008) 43. Majda, A.J., Khouider, B.: Stochastic and mesoscopic models for tropical convection. Proc. Natl. Acad. Sci. 99, 1123–1128 (2002) 44. Majda, A.J., Timofeyev, I., Vanden-Eijnden, E.: Models for stochastic climate prediction. Proc. Natl. Acad. Sci. 96, 14687–14691 (1999) 45. Majda, A.J., Timofeyev, I., Vanden-Eijnden, E.: A mathematical framework for stochastic climate models. Commun. Pure Appl. Math. LIV, 891–974 (2001) 46. Majda, A.J., Timofeyev, I., Vanden-Eijnden, E.: A priori tests of a stochastic mode reduction strategy. Physica D 170, 206–252 (2002) 47. Majda A., Timofeyev, I., Vanden-Eijnden, E.: Stochastic models for selected slow variables in large deterministic systems. Nonlinearity 19, 769–794 (2006) 48. Mo, K.C., Ghil, M.: Statistics and dynamics of persistent anomalies. J. Atmos. Sci. 44, 877–901 (1987) 49. Monahan, A.H.: Nonlinear principal component analysis by neural networks: theory and application to the Lorenz system. J. Clim. 13, 821–835 (2000) 50. Monahan, A.H., Fyfe, J.C., Ambaum, M.H.P., Stephenson, D.B., North, G.R.: Empirical orthogonal functions: the medium is the message. J. Clim. 22, 6501–6514 (2009) 51. Newman, M., Sardeshmukh, P., Penland, C.: Stochastic forcing of the wintertime extratropical flow. J. Atmos. Sci. 54, 435–455 (1997) 52. Palmer, T., Williams, P. (eds.): Stochastic Physics and Climate Modelling. Cambridge University Press, Cambridge (2009) 53. Pasmanter, R.A., Timmermann, A.: Cyclic Markov chains with an application to an intermediate ENSO model. Nonlinear Process. Geophys. 10, 197–210 (2003) 54. Penland, C., Ghil, M.: Forecasting northern hemisphere 700-mb geopotential height anomalies using empirical normal modes. Mon. Weather Rev. 121, 2355–2372 (1993) 55. Penland, C., Sardeshmukh, P.D.: The optimal growth of tropical sea surface temperature anomalies. J. Clim. 8, 1999–2024 (1995) 56. Peters, K., Jakob, C., Davies, L., Khouider, B., Majda, A.: Stochastic behaviour of tropical convection in observations and a multicloud model. J. Atmos. Sci. 70, 3556–3575 (2013) 57. Plant, R.S., Craig, G.C.: A stochastic parameterization for deep convection based on equilibrium statistics. J. Atmos. Sci. 65, 87–105 (2008)

Stochastic Eulerian-Lagrangian Methods 58. Prakasa Rao, B.L.S.: Statistical Inference for Diffusion Type Processes. Arnold Publishers, London (1999) 59. Preisendorfer, R.W.: Principal Component Analysis in Meteorology and Oceanography. Elsevier, Amsterdam (1988) 60. Sura, P., Newman, M., Penland, C., Sardeshmukh, P.: Multiplicative noise and non-Gaussianity: a paradigm for atmospheric regimes. J. Atmos. Sci. 62, 1391–1409 (2005) 61. Von Storch, H., Zwiers, F.W.: Statistical Analysis in Climate Research. Cambridge University Press, Cambridge (1999) 62. Wilks, D.S.: Statistical Methods in the Atmospheric Sciences, 3rd edn., p. 704. Academic, Oxford (2011) 63. Zucchini, W., Guttorp, P.: A Hidden Markov model for space-time precipitation. Water Resour. Res. 27, 1917–1923 (1991)

Stochastic Eulerian-Lagrangian Methods Paul J. Atzberger Department of Mathematics, University of California Santa Barbara (UCSB), Santa Barbara, CA, USA

Synonyms Fluid-structure interaction; Fluid dynamics; Fluctuating hydrodynamics; Immersed Boundary Method; SELM; Statistical mechanics; Stochastic Eulerian Lagrangian method; Thermal fluctuations

Abstract We present approaches for the study of fluid-structure interactions subject to thermal fluctuations. A mechanical description is utilized combining Eulerian and Lagrangian reference frames. We establish general conditions for the derivation of operators coupling these descriptions and for the derivation of stochastic driving fields consistent with statistical mechanics. We present stochastic numerical methods for the fluidstructure dynamics and methods to generate efficiently the required stochastic driving fields. To help establish the validity of the proposed approach, we perform analysis of the invariant probability distribution of the stochastic dynamics and relate our results to statistical mechanics. Overall, the presented approaches are expected to be applicable to a wide variety of systems involving fluid-structure interactions subject to thermal fluctuations.

Stochastic Eulerian-Lagrangian Methods

Introduction Recent scientific and technological advances motivate the study of fluid-structure interactions in physical regimes often involving very small length and time scales [26, 30, 35, 36]. This includes the study of microstructure in soft materials and complex fluids, the study of biological systems such as cell motility and microorganism swimming, and the study of processes within microfluidic and nanofluidic devices. At such scales thermal fluctuations play an important role and pose significant challenges in the study of such fluid-structure systems. Significant past work has been done on the formulation of descriptions for fluidstructure interactions subject to thermal fluctuations. To obtain descriptions tractable for analysis and numerical simulation, these approaches typically place an emphasis on approximations which retain only the structure degrees of freedom (eliminating the fluid dynamics). This often results in simplifications in the descriptions having substantial analytic and computational advantages. In particular, this eliminates the many degrees of freedom associated with the fluid and avoids having to resolve the potentially intricate and stiff stochastic dynamics of the fluid. These approaches have worked especially well for the study of bulk phenomena in free solution and the study of many types of complex fluids and soft materials [3, 3, 9, 13, 17, 23]. Recent applications arising in the sciences and in technological fields present situations in which resolving the dynamics of the fluid may be important and even advantageous both for modeling and computation. This includes modeling the spectroscopic responses of biological materials [19, 25, 37], studying transport in microfluidic and nanofluidic devices [16, 30], and investigating dynamics in biological systems [2, 11]. There are also other motivations for representing the fluid explicitly and resolving its stochastic dynamics. This includes the development of hybrid fluid-particle models in which thermal fluctuations mediate important effects when coupling continuum and particle descriptions [12, 14], the study of hydrodynamic coupling and diffusion in the vicinity of surfaces having complicated geometries [30], and the study of systems in which there are many interacting mechanical structures [8, 27, 28]. To facilitate the development of methods for studying such phenomena in fluid-structure systems, we present a rather general

1387

formalism which captures essential features of the coupled stochastic dynamics of the fluid and structures. To model the fluid-structure system, a mechanical description is utilized involving both Eulerian and Lagrangian reference frames. Such mixed descriptions arise rather naturally, since it is often convenient to describe the structure configurations in a Lagrangian reference frame while it is convenient to describe the fluid in an Eulerian reference frame. In practice, this presents a number of challenges for analysis and numerical studies. A central issue concerns how to couple the descriptions to represent accurately the fluid-structure interactions, while obtaining a coupled description which can be treated efficiently by numerical methods. Another important issue concerns how to account properly for thermal fluctuations in such approximate descriptions. This must be done carefully to be consistent with statistical mechanics. A third issue concerns the development of efficient computational methods. This requires discretizations of the stochastic differential equations and the development of efficient methods for numerical integration and stochastic field generation. We present a set of approaches to address these issues. The formalism and general conditions for the operators which couple the Eulerian and Lagrangian descriptions are presented in section “Stochastic Eulerian Lagrangian Method.” We discuss a convenient description of the fluid-structure system useful for working with the formalism in practice in section “Derivations for the Stochastic Eulerian Lagrangian Method.” A derivation of the stochastic driving fields used to represent the thermal fluctuations is also presented in section “Derivations for the Stochastic Eulerian Lagrangian Method.” Stochastic numerical methods are discussed for the approximation of the stochastic dynamics and generation of stochastic fields in sections “Computational Methodology.” To validate the methodology, we perform analysis of the invariant probability distribution of the stochastic dynamics of the fluid-structure formalism. We compare this analysis with results from statistical mechanics in section “Equilibrium Statistical Mechanics of SELM Dynamics.” A more detailed and comprehensive discussion of the approaches presented here can be found in our paper [6].

S

1388

Stochastic Eulerian-Lagrangian Methods

Stochastic Eulerian Lagrangian Method To study the dynamics of fluid-structure interactions subject to thermal fluctuations, we utilize a mechanical description involving Eulerian and Lagrangian reference frames. Such mixed descriptions arise rather naturally, since it is often convenient to describe the structure configurations in a Lagrangian reference frame while it is convenient to describe the fluid in an Eulerian reference frame. In principle more general descriptions using other reference frames could also be considered. Descriptions for fluid-structure systems having these features can be described rather generally by the following dynamic equations du D Lu C Œ$ .v   u/ C  C fthm dt dv D $ .v   u/  rX ˚ŒX C  C Fthm m dt dX D v: dt 

(1) (2) (3)

The u denotes the velocity of the fluid, and  the uniform fluid density. The X denotes the configuration of the structure, and v the velocity of the structure. The mass of the structure is denoted by m. To simplify the presentation, we treat here only the case when

Stochastic Eulerian-Lagrangian Methods, Fig. 1 The description of the fluid-structure system utilizes both Eulerian and Lagrangian reference frames. The structure mechanics are often most naturally described using a Lagrangian reference frame. The fluid mechanics are often most naturally described using an Eulerian reference frame. The mapping X(q) relates the Lagrangian reference frame to the Eulerian reference frame. The

 and m are constant, but with some modifications these could also be treated as variable. The ;  are Lagrange multipliers for imposed constraints, such as incompressibility of the fluid or a rigid body constraint of a structure. The operator L is used to account for dissipation in the fluid, such as associated with Newtonian fluid stresses [1]. To account for how the fluid and structures are coupled, a few general operators are introduced, ; $; . The linear operators ; ; $ are used to model the fluid-structure coupling. The  operator describes how a structure depends on the fluid flow, while $ is a negative definite dissipative operator describing the viscous interactions coupling the structure to the fluid. We assume throughout that this dissipative operator is symmetric, $ D $ T . The linear operator  is used to attribute a spatial location for the viscous interactions between the structure and fluid. The linear operators are assumed to have dependence only on the configuration degrees of freedom  D  ŒX,  D ŒX. We assume further that $ does not have any dependence on X. For an illustration of the role these coupling operators play, see Fig. 1. To account for the mechanics of structures, ˚ŒX denotes the potential energy of the configuration X. The total energy associated with this fluid-structure system is given by

operator  prescribes how structures are to be coupled to the fluid. The operator  prescribes how the fluid is to be coupled to the structures. A variety of fluid-structure interactions can be represented in this way. This includes rigid and deformable bodies, membrane structures, polymeric structures, or point particles

Stochastic Eulerian-Lagrangian Methods

Z EŒu; v; X D ˝

1389

1 ju.y/j2 d y 2

1 C mv2 C ˚ŒX: 2

(4)

The first two terms give the kinetic energy of the fluid and structures. The last term gives the potential energy of the structures. As we shall discuss, it is natural to consider coupling operators  and  which are adjoint in the sense Z

Z . u/.q/  v.q/d q D S

u.x/  .v/.x/d x (5) ˝

for any u and v. The S and ˝ denote the spaces used to parameterize respectively the structures and the fluid. We denote such an adjoint by  D  % or  D % . This adjoint condition can be shown to have the important consequence that the fluid-structure coupling conserves energy when $ ! 1 in the inviscid and zero temperature limit. To account for thermal fluctuations, a random force density fthm is introduced in the fluid equations and Fthm in the structure equations. These account for spontaneous changes in the system momentum which occurs as a result of the influence of unresolved microscopic degrees of freedom and unresolved events occurring in the fluid and in the fluid-structure interactions. The thermal fluctuations consistent with the form of the total energy and relaxation dynamics of the system are taken into account by the introduction of stochastic driving fields in the momentum equations of the fluid and structures. The stochastic driving fields are taken to be Gaussian processes with mean zero and with ıcorrelation in time [29]. By the fluctuation-dissipation principle [29], these have covariances given by

the thermal fluctuations is given in section “Derivations for the Stochastic Eulerian Lagrangian Method.” It is important to mention that some care must be taken when using the above formalism in practice and when choosing operators. An important issue concerns the treatment of the material derivative of the fluid, d u=dt D @u=@t C u  ru. For stochastic systems the field u is often highly irregular and not defined in a point-wise sense, but rather only in the sense of a generalized function (distribution) [10, 24]. To avoid these issues, we shall treat d u=dt D @u=@t in this initial presentation of the approach [6]. The SELM provides a rather general framework for the study of fluidstructure interactions subject to thermal fluctuations. To use the approach for specific applications requires the formulation of appropriate coupling operators  and  to model the fluid-structure interaction. We provide some concrete examples of such operators in the paper [6]. Formulation in Terms of Total Momentum Field When working with the formalism in practice, it turns out to be convenient to reformulate the description in terms of a field describing the total momentum of the fluid-structure system at a given spatial location. As we shall discuss, this description results in simplifications in the stochastic driving fields. For this purpose, we define p.x; t/ D u.x; t/ C Œmv.t/.x/:

The operator  is used to give the distribution in space of the momentum associated with the structures for given configuration X.t/. Using this approach, the fluid-structure dynamics are described by dp DLu C ŒrX ˚.X/ dt

hfthm .s/fTthm .t/i D  .2kB T / .L  $  / ı.t  s/ (6) hFthm .s/FTthm .t/i D .2kB T / $ ı.t  s/ T

hfthm .s/Fthm .t/i D  .2kB T / $ ı.t  s/:

(7) (8)

We have used that  D % and $ D $ T . We remark that the notation ghT which is used for the covariance operators should be interpreted as the tensor product. This notation is meant to suggest the analogue to the outer-product operation which holds in the discrete setting [5]. A more detailed discussion and derivation of

(9)

C .rX Œmv/  v C  C gthm

(10)

dv D  $ .v   u/  rX ˚.X/ C  C Fthm (11) dt dX Dv (12) dt

m

where u D 1 .p  Œmv/ and gthm D fthm C ŒFthm . The third term in the first equation arises from the dependence of  on the configuration of the structures, Œmv D .ŒX /Œmv. The Lagrange

S

1390

Stochastic Eulerian-Lagrangian Methods

multipliers for imposed constraints are denoted by ; . For the constraints, we use rather liberally the notation with the Lagrange multipliers denoted here not necessarily assumed to be equal to the previous definition. The stochastic driving fields are again Gaussian with mean zero and ı-correlation in time [29]. The stochastic driving fields have the covariance structure given by hgthm.s/gTthm .t/i D  .2kB T / L ı.t  s/ hFthm .s/FTthm .t/i D .2kB T / $ ı.t  s/ hgthm .s/FTthm .t/i D 0:

“Stochastic Eulerian Lagrangian Method.” For these equations, we focus primarily on the motivation for the stochastic driving fields used for the fluid-structure system. For the thermal fluctuations of the system, we assume Gaussian random fields with mean zero and ıcorrelated in time. For such stochastic fields, the central challenge is to determine an appropriate covariance structure. For this purpose, we use the fluctuation(13) dissipation principle of statistical mechanics [22, 29]. For linear stochastic differential equations of the (14) form (15) (19) d Zt D LZt dt C Qd Bt

This formulation has the convenient feature that the stochastic driving fields become independent. This is a consequence of using the field for the total momentum for which the dissipative exchange of momentum between the fluid and structure no longer arises. In the equations for the total momentum, the only source of dissipation remaining occurs from the stresses of the fluid. This approach simplifies the effort required to generate numerically the stochastic driving fields and will be used throughout.

the fluctuation-dissipation principle can be expressed as G D QQT D .LC /  .LC /T :

(20)

This relates the equilibrium covariance structure C of the system to the covariance structure G of the stochastic driving field. The operator L accounts for the dissipative dynamics of the system. For the Eqs. 16–18, the dissipative operators only appear in the momentum equations. This can be shown to have the consequence that there is no thermal forcing in the equation for X.t/; Derivations for the Stochastic Eulerian this will also be confirmed in section “Formulation in Lagrangian Method Terms of Total Momentum Field.” To simplify the presentation, we do not represent explicitly the stochastic We now discuss formal derivations to motivate the dynamics of the structure configuration X. stochastic differential equations used in each of For the fluid-structure system, it is convenient to the physical regimes. For this purpose, we do not work with the stochastic driving fields by defining present the most general derivation of the equations. For brevity, we make simplifying assumptions when (21) q D Œ1 fthm ; m1 Fthm T : convenient. In the initial formulation of SELM, the fluid- The field q formally is given by q D Qd B =dt and det structure system is described by termined by the covariance structure G D QQT . This du DLu C Œ$ .v   u/ C  C fthm  dt dv D  $ .v   u/  rX ˚.X/ C  m dt C Fthm dX Dv: dt

covariance structure is determined by the fluctuation(16) dissipation principle expressed in Eq. 20 with 

1 .L  $  / 1 $ m1 $ m1 $   1  kB T I 0 C D : 0 m1 kB T I LD

(17) (18)

(22) (23)

The I denotes the identity operator. The covariance The notation and operators appearing in these C was obtained by considering the fluctuations at equations have been discussed in detail in section equilibrium. The covariance C is easily found since

Stochastic Eulerian-Lagrangian Methods

1391

the Gibbs-Boltzmann distribution is a Gaussian with formal density  .u; v/ D Z10 exp ŒE=kB T . The Z0 is the normalization constant for  . The energy is given by Eq. 4. For this purpose, we need only consider the energy E in the case when ˚ D 0. This gives the covariance structure

hgthm gTthm i Dhfthm fTthm i C hfthm FTthm T i C hFthm fTthm i C hFthm FTthm T i  D.2kB T /  L C $    $T  $T C $T

D  .2kB T / L: (29) 2 .L  $  / m1 1 $ : G D .2kB T / m1 1 $  m2 $ This makes use of the adjoint property of the coupling (24) operators % D  . One particularly convenient feature of this reformu% % lation is that the stochastic driving fields Fthm and gthm To obtain this result, we use that  D  and $ D $ . become independent. This can be seen as follows: From the definition of q, it is found that the covariance of the stochastic driving fields of SELM is given by (30) hgthm FTthm i D hfthm FTthm i C hFthm FTthm i Eqs. 6–8. This provides a description of the thermal fluctuations in the fluid-structure system. D .2k T /.$ C $ / D 0: 

B

Formulation in Terms of Total Momentum Field It is convenient to reformulate the description of the fluid-structure system in terms of a field for the total momentum of the system associated with spatial location x. For this purpose we define p.x; t/ D u.x; t/ C Œmv.t/.x/:

This decoupling of the stochastic driving fields greatly reduces the computational effort to generate the fields with the required covariance structure. This shows that the covariance structure of the stochastic driving fields of SELM is given by Eqs. 13–15.

(25)

Computational Methodology The operator  is used to give the distribution in space of the momentum associated with the structures. We now discuss briefly numerical methods for the Using this approach, the fluid-structure dynamics are SELM formalism. For concreteness we consider the specific case in which the fluid is Newtonian and described by incompressible. For now, the other operators of the SELM formalism will be treated rather generally. This dp DLu C ŒrX ˚.X/ case corresponds to the dissipative operator for the dt fluid (26) C .rX Œmv/  v C  C gthm m

Lu D  u:

dv D  $ .v   u/  rX ˚.X/ C  dt C Fthm

dX Dv dt

(27) (28)

The denotes the Laplacian u D @xx uC@yy uC@zz u. The incompressibility of the fluid corresponds to the constraint r  u D 0:

where u D 1 .p  Œmv/ and gthm D fthm C ŒFthm . The third term in the first equation arises from the dependence of  on the configuration of the structures, Œmv.t/ D .ŒX /Œmv.t/. The thermal fluctuations are taken into account by two stochastic fields gthm and Fthm . The covariance of gthm is obtained from

(31)

(32)

This is imposed by the Lagrange multiplier . By the Hodge decomposition,  is given by the gradient of a function p with  D rp. The p can be interpreted as the local pressure of the fluid. A variety of methods could be used in practice to discretize the SELM formalism, such as finite

S

1392

Stochastic Eulerian-Lagrangian Methods

difference methods, spectral methods, and finite In this expression, we let fthm D gthm  ŒFthm  for the element methods [20, 32, 33]. We present here particular realized values of the fields gthm and Fthm . discretizations based on finite difference methods. We remark that in fact the semi-discretized equations of the SELM formalism in this regime can also be given in terms of u directly, which may provide Numerical Semi-discretizations for a simpler approach in practice. The identity fthm D Incompressible Newtonian Fluid The Laplacian will be approximated by central differ- gthm  ŒFthm  could be used to efficiently generate the required stochastic driving fields in the equations for ences on a uniform periodic lattice by u. We present the reformulation here, since it more 3 X umCej  2um C umej directly suggests the semi-discretized equations to be : (33) ŒLum D 2 used for the reduced stochastic equations.

x j D1 For this semi-discretization, we consider a total energy for the system given by The m D .m1 ; m2 ; m3 / denotes the index of the lattice site. The ej denotes the standard basis vector in three m X dimensions. The incompressibility of the fluid will be ju.xm /j2 x3m C jvj2 C ˚ŒX: EŒu; v; X D 2 2 approximated by imposing the constraint m (40) j 3 uj X mCej  umej : (34) This is useful in formulating an adjoint condition 5 for ŒD  um D 2 x j D1 the semi-discretized system. This can be derived by The superscripts denote the vector component. In prac- considering the requirements on the coupling operators tice, this will be imposed by computing the projection  and  which ensure the energy is conserved when of a vector u to the subspace fu 2 R3N j D  u D 0g, $ ! 1 in the inviscid and zero temperature limit. To obtain appropriate behaviors for the thermal where N is the total number of lattice sites. We denote fluctuations, it is important to develop stochastic drivthis projection operation by ing fields which are tailored to the specific semiu D }u : (35) discretizations used in the numerical methods. Once the stochastic driving fields are determined, which is The semi-discretized equations for SELM to be used in the subject of the next section, the equations can be practice are integrated in time using traditional methods for SDEs, such as the Euler-Maruyama method or a stochastic dp D Lu C ŒrX ˚ C .rX Œmv/  v C  C gthm Runge-Kutta method [21]. More sophisticated intedt grators in time can also be developed to cope with (36) sources of stiffness but are beyond the scope of this entry [7]. For each of the reduced equations, similar semi-discretizations can be developed as the one predv D $ Œv   u C Fthm (37) sented above. dt dX Stochastic Driving Fields for D v: (38) Semi-discretizations dt To obtain behaviors consistent with statistical mechanThe component um D 1 .pm  Œmvm /. Each ics, it is important stochastic driving fields be used of the operators now appearing is understood to be which are tailored to the specific numerical discretizadiscretized. We discuss specific discretizations for  tion employed [5–7, 15]. To ensure consistency with and  in paper [6]. To obtain the Lagrange multiplier  statistical mechanics, we will again use the fluctuationwhich imposes incompressibility, we use the projection dissipation principle but now apply it to the semioperator and discretized equations. For each regime, we then discuss the important issues arising in practice concerning the  D .I  }/ .Lu C $ Œv   u C fthm / (39) efficient generation of these stochastic driving fields.

Stochastic Eulerian-Lagrangian Methods

1393

1 Formulation in Terms of Total Momentum Field exp ŒE.z/=kB T  : (42) GB .z/ D To obtain the covariance structure for this regime, we Z apply the fluctuation-dissipation principle as expressed in Eq. 20 to the semi-discretized Eqs. 36–38. This gives The z is the state of the system, E is the energy, kB is Boltzmann’s constant, T is the system temthe covariance perature, and Z is a normalization constant for the 2 2 3 3 distribution [29]. We show that this Gibbs-Boltzmann  x L 0 0 distribution is the equilibrium distribution of both the m2 $ 05 : G D 2LC D .2kB T / 40 full stochastic dynamics and the reduced stochastic 0 0 0 (41) dynamics in each physical regime. We present here both a verification of the invariance 3 of the Gibbs-Boltzmann distribution for the general The factor of x arises from the form of the energy for the discretized system which gives covariance for formalism and for numerical discretizations of the the equilibrium fluctuations of the total momentum formalism. The verification is rather formal for the 1 x 3 kB T ; see Eq. 40. In practice, achieving the undiscretized formalism given technical issues which covariance associated with the dissipative operator of would need to be addressed for such an infinite dimenthe fluid L is typically the most challenging to generate sional dynamical system. However, the verification is efficiently. This arises from the large number N of rigorous for the semi-discretization of the formalism, which yields a finite dimensional dynamical system. lattice sites in the discretization. One approach is to determine a factor Q such that The latter is likely the most relevant case in practice. the block Gp;p D QQT ; subscripts indicate block Given the nearly identical calculations involved in the entry of the matrix. The required random field with verification for the general formalism and its semicovariance Gp;p is then given by g D Q, where discretizations, we use a notation in which the key  is the uncorrelated Gaussian field with the covari- differences between the two cases primarily arise in ance structure I. For the discretization used on the the definition of the energy. In particular, the energy is uniform periodic mesh, the matrices L and C are understood to be given by Eq. 4 when considering the cyclic [31]. This has the important consequence that general SELM formalism and Eq. 40 when considering they are both diagonalizable in the discrete Fourier semi-discretizations. basis of the lattice. As a result, the field fthm can be generated using the fast Fourier transform (FFT) with at most O.N log.N // computational steps. In fact, in this special case of the discretization, “random fluxes” at the cell faces can be used to generate the field in O.N / computational steps [5]. Other approaches can be used to generate the random fields on nonperiodic meshes and on multilevel meshes; see [4, 5].

Formulation in Terms of Total Momentum Field The stochastic dynamics given by Eqs. 10–12 is a change of variable of the full stochastic dynamics of the SELM formalism given by Eqs. 1–3. Thus verifying the invariance using the reformulated description is also applicable to Eqs. 1–3 and vice versa. To verify the invariance in the other regimes, it is convenient to work with the reformulated description. The energy associated with the reformulated description is given by Z Equilibrium Statistical Mechanics of SELM 1 jp.y/  Œmv.y/j2 d y EŒp; v; X D Dynamics 2 ˝ m C jvj2 C ˚ŒX: (43) We now discuss how the SELM formalism and the 2 presented numerical methods capture the equilibrium The energy associated with the semi-discretization is statistical mechanics of the fluid-structure system. This is done through an analysis of the invariant probability 1 X EŒp; v; X D jp.xm /  Œmvm j2 x3m (44) distribution of the stochastic dynamics. For the fluid2 m structure systems considered, the appropriate probam bility distribution is given by the Gibbs-Boltzmann C jvj2 C ˚ŒX: 2 distribution

S

1394

Stochastic Eulerian-Lagrangian Methods

The probability density  .p; v; X; t/ for the current In this regime, G is given by Eq. 13 or 41. In the state of the system under the SELM dynamics is notation Œr  G.z/i D @zj Gij .z/ with the summation convention for repeated indices. To simplify the governed by the Fokker-Planck equation notation, we have suppressed denoting the specific @ D r  J (45) functions on which each of the operators acts; see @t Eqs. 10–12 for these details. The requirement that the Gibbs-Boltzmann distriwith probability flux bution GB given by Eq. 42 be invariant under the 3 2 stochastic dynamics is equivalent to the distribution L C  C rX   v C  yielding r J D 0. We find it convenient to group terms 5 J D 4 $  rX ˚ C  and express this condition as v 1 1  .r  G/  Gr: 2 2

(46)

r  J D A1 C A2 C r  A3 C r  A4 D 0

(47)

The covariance operator G is associated with the Gaussian field g D Œgthm ; Fthm ; 0T byhg.s/gT .t/i D Gı.t  s/. where

  A1 D . C rX   v C 1 /  rp E C .rX ˚ C 1 /  rv E C .v/  rX E .kB T /1 GB   A2 D rp  . C rX   v C 1 / C rv  .rX ˚ C 2 / C rX  .v/ GB 1 A3 D  .r  G/GB 2   3 2 Lu C 2 C Gpp rp E C Gpv rv E C GpX rX E  .2kB T /1 A4 D 4 $ .2kB T /1 5 GB :  C 2 C Gvp rp E C Gvv rv E C  GvX rX E 1 GXp rp E C GXv rv E C GXX rX E .2kB T /

(48)

where u D 1 .pŒmv/. Similar expressions for the energy of the undiscretized formalism can be obtained by using the calculus of variations [18]. We now consider r J and each term A1 ; A2 ; A3 ; A4 . The term A1 can be shown to be the time derivative of the energy A1 D dE=dt when considering only a subset of the contributions to the dynamics. Thus, conservation of the energy under this restricted dynamics would result in A1 being zero. For the SELM formalism, we find by direct substitution of the gradients of 3 (49) rpn E D u.xn / xn E given by Eqs. 49–51 into Eq. 48 that A1 D 0. When X   there are constraints, it is important to consider only 3 u.xm /  rvq Œmvm xm C mvq rvq E D admissible states .p; v; X/. This shows in the inviscid m (50) and zero temperature limit of SELM, the resulting dynamics are nondissipative. This property imposes conX   3 u.xm /  rXq Œmvm xm C rXq ˚: straints on the coupling operators and can be viewed as rXq E D a further motivation for the adjoint conditions imposed m (51) in Eq. 5.

We assume here that the Lagrange multipliers can be split  D 1 C 2 and  D 1 C 2 to impose the constraints by considering in isolation different terms contributing to the dynamics; see Eq. 48. This is always possible for linear constraints. The block entries of the covariance operator G are denoted by Gi;j with i; j 2 fp; v; Xg. For the energy of the discretized system given by Eq. 4, we have

Stochastic Eulerian-Lagrangian Methods

The term A2 gives the compressibility of the phasespace flow generated by the nondissipative dynamics of the SELM formalism. The flow is generated by the vector field . C rX   v C 1 ; rX ˚ C 1 ; v/ on the phase-space .p; v; X/. When this term is nonzero, there are important implications for the Liouville theorem and statistical mechanics of the system [34]. For the current regime, we have A2 D 0 since in the divergence each component of the vector field is seen to be independent of the variable on which the derivative is computed. This shows in the inviscid and zero temperature limit of SELM, the phase-space flow is incompressible. For the reduced SELM descriptions, we shall see this is not always the case. The term A3 corresponds to fluxes arising from multiplicative features of the stochastic driving fields. When the covariance G has a dependence on the current state of the system, this can result in possible changes in the amplitude and correlations in the fluctuations. These changes can yield asymmetries in the stochastic dynamics which manifest as a net probability flux. In the SELM formalism, it is found that in the divergence of G, each contributing entry is independent of the variable on which the derivative is being computed. This shows for the SELM dynamics there is no such probability fluxes, A3 D 0. The last term A4 accounts for the fluxes arising from the primarily dissipative dynamics and the stochastic driving fields. This term is calculated by substituting the gradients of the energy given by Eqs. 49–51 and using the choice of covariance structure given by Eq. 13 or 41. By direct substitution this term is found to be zero, A4 D 0. This shows the invariance of the Gibbs-Boltzmann distribution under the SELM dynamics. This provides a rather strong validation of the stochastic driving fields introduced for the SELM formalism. This shows the SELM stochastic dynamics are consistent with equilibrium statistical mechanics [29].

Conclusions An approach for fluid-structure interactions subject to thermal fluctuations was presented based on a mechanical description utilizing both Eulerian and Lagrangian reference frames. General conditions were established for operators coupling these descriptions. A reformulated description was presented for the

1395

stochastic dynamics of the fluid-structure system having convenient features for analysis and for computational methods. Analysis was presented establishing for the SELM stochastic dynamics that the Gibbs-Boltzmann distribution is invariant. The SELM formalism provides a general framework for the development of computational methods for applications requiring a consistent treatment of structure elastic mechanics, hydrodynamic coupling, and thermal fluctuations. A more detailed and comprehensive discussion of SELM can be found in our paper [6]. Acknowledgements The author P. J. A. acknowledges support from research grant NSF CAREER DMS - 0956210.

References 1. Acheson, D.J.: Elementary Fluid Dynamics. Oxford Applied Mathematics and Computing Science Series. Oxford University Press, Oxford/Clarendon Press/New York (1990) 2. Alberts, B., Johnson, A., Lewis, J., Raff, M., Roberts, K., Walker, P.: Molecular Biology of the Cell. Garland Publishing, New York (2002) 3. Armstrong, R.C., Byron Bird, R., Hassager, O.:Dynamic Polymeric Liquids, Vols. I, II. Wiley, New York (1987) 4. Atzberger, P.: Spatially adaptive stochastic multigrid methods for fluid-structure systems with thermal fluctuations, technical report (2010). arXiv:1003.2680 5. Atzberger, P.J.: Spatially adaptive stochastic numerical methods for intrinsic fluctuations in reaction-diffusion systems. J. Comput. Phys. 229, 3474–3501 (2010) 6. Atzberger, P.J.: Stochastic eulerian lagrangian methods for fluid-structure interactions with thermal fluctuations. J. Comput. Phys. 230, 2821–2837 (2011) 7. Atzberger, P.J., Kramer, P.R., Peskin, C.S.: A stochastic immersed boundary method for fluid-structure dynamics at microscopic length scales. J. Comput. Phys. 224, 1255–1292 (2007) 8. Banchio, A.J., Brady, J.F.: Accelerated stokesian dynamics: Brownian motion. J. Chem. Phys. 118, 10323–10332 (2003) 9. Brady, J.F. Bossis, G.: Stokesian dynamics. Annu. Rev. Fluid Mech. 20, 111–157 (1988) 10. Da Prato, G., Zabczyk, J.: Stochastic Equations in Infinite Dimensions. Cambridge University Press, Cambridge/New York (1992) 11. Danuser, G., Waterman-Storer, C.M: Quantitative fluorescent speckle microscopy of cytoskeleton dynamics. Annu. Rev. Biophys. Biomol. Struct. 35, 361–387 (2006) 12. De Fabritiis, G., Serrano, M., Delgado-Buscalioni, R., Coveney, P.V.: Fluctuating hydrodynamic modeling of fluids at the nanoscale. Phys. Rev. E 75, 026307 (2007) 13. Doi, M., Edwards, S.F.: The Theory of Polymer Dynamics. Oxford University Press, New York (1986) 14. Donev, A., Bell, J.B., Garcia, A.L., Alder, B.J.: A hybrid particle-continuum method for hydrodynamics of complex fluids. SIAM J. Multiscale Model. Simul. 8 871–911 (2010)

S

1396 15. Donev, A., Vanden-Eijnden, E., Garcia, A.L., Bell, J.B.: On the accuracy of finite-volume schemes for fluctuating hydrodynamics, Commun. Appl. Math. Comput. Sci. 5(2), 149–197 (2010) 16. Eijkel, J.C.T., Napoli, M., Pennathur, S.: Nanofluidic technology for biomolecule applications: a critical review. Lab on a Chip 10, 957–985 (2010) 17. Ermak, D.L. McCammon, J.A.: Brownian dynamics with hydrodynamic interactions. J. Chem. Phys. 69, 1352–1360 (1978) 18. Gelfand, I.M., Fomin, S.V.: Calculus of Variations. Dover, Mineola (2000) 19. Gotter, R., Kroy, K., Frey, E., Barmann, M., Sackmann, E.: Dynamic light scattering from semidilute actin solutions: a study of hydrodynamic screening, filament bending stiffness, and the effect of tropomyosin/troponin-binding. Macromolecules 29 30–36 (1996) 20. Gottlieb, D., Orszag, S.A.: Numerical Analysis of Spectral Methods Theory and Applications. SIAM, Philadelphia (1993) 21. Kloeden, P.E., Platen, E.: Numerical Solution of Stochastic Differential Equations. Springer, Berlin/New York (1992) 22. Landau, L.D., Lifshitz, E.M.: Course of Theoretical Physics. Statistical Physics, vol. 9. Pergamon Press, Oxford (1980) 23. Larson, R.G.: The Structure and Rheology of Complex Fluids. Oxford University Press, New York (1999) 24. Lieb, E.H., Loss, M.: Analysis. American Mathematical Society, Providence (2001) 25. Mezei, F., Pappas, C., Gutberlet, T.: Neutron Spin Echo Spectroscopy: Basics, Trends, and Applications. Spinger, Berlin/New York (2003) 26. Moffitt, J.R., Chemla, Y.R., Smith, S.B., Bustamante, C.: Recent advances in optical tweezers. Annu. Rev. Biochem. 77, 205–228 (2008) 27. Peskin, C.S.: Numerical analysis of blood flow in the heart. J. Comput. Phys. 25, 220–252 (1977) 28. Peskin, C.S.: The immersed boundary method. Acta Numerica 11, 479–517 (2002) 29. Reichl, L.E.: A Modern Course in Statistical Physics. Wiley, New York (1998) 30. Squires, T.M., Quake, S.R.: Microfluidics: fluid physics at the nanoliter scale. Rev. Mod. Phys. 77, 977–1026 (2005) 31. Strang, G.: Linear Algebra and Its Applications. Harcourt Brace Jovanovich College Publishers, San Diego (1988) 32. Strang, G., Fix, G.: An Analysis of the Finite Element Method. Wellesley-Cambridge Press, Wellesley (2008) 33. Strikwerda, J.C.: Finite Difference Schemes and Partial Differential Equations. SIAM, Philadelphia (2004) 34. Tuckerman, M.E., Mundy, C.J., Martyna, G.J.: On the classical statistical mechanics of non-hamiltonian systems. EPL (Europhys. Lett.) 45, 149–155 (1999) 35. Valentine, M.T., Weeks, E.R., Gisler, T., Kaplan, P.D., Yodh, A.G., Crocker, J.C., Weitz, D.A.: Two-point microrheology of inhomogeneous soft materials. Phys. Rev. Lett. 85, 888–91 (2000) 36. Watari, N., Doi, M., Larson, R.G.: Fluidic trapping of deformable polymers in microflows. Phys. Rev. E 78, 011801 (2008) 37. Watson, M.C., Brown, F.L.H.: Interpreting membrane scattering experiments at the mesoscale: the contribution of dissipation within the bilayer. Biophys. J. 98, L9–L11 (2010)

Stochastic Filtering

Stochastic Filtering M.V. Tretyakov School of Mathematical Sciences, University of Nottingham, Nottingham, UK

Mathematics Subject Classification 60G35; 93E11; 93E10; 62M20; 60H15; 65C30; 65C35; 60H35

Synonyms Filtering problem for hidden Markov models; Nonlinear filtering

Short Definition Let T1 and T2 be either a time interval Œ0; T  or a discrete set. The stochastic filtering problem consists in estimating an unobservable signal Xt ; t 2 T1 ; based on an observation fys ; s  t; s 2 T2 g; where the process yt is related to Xt via a stochastic model.

Description We restrict ourselves to the case when an unobservable signal and observation are continuous time processes with T1 D T2 D Œ0; T  (see discrete filtering in, e.g., [1, 3, 7]). Let .&; F ; P/ be a complete probability space, Ft ; 0  t  T; be a filtration satisfying the usual hypotheses, and .wt ; Ft / and .vt ; Ft / be d1 dimensional and r-dimensional independent standard Wiener processes, respectively. We consider the classical filtering scheme in which the unobservable signal process (“hidden” state) Xt 2 Rd and the observation process yt 2 Rr satisfy the system of ˆIto stochastic differential equations (SDE): dX D ˛.X /ds C .X /d ws C .X /dvs ; X0 D x;

(1)

dy D ˇ.X /ds C dvs ; y0 D 0;

(2)

Stochastic Filtering

1397

where ˛.x/ and ˇ.x/ are d -dimensional and rdimensional vector functions, respectively, and .x/ and .x/ are d d1 -dimensional and d r-dimensional matrix functions, respectively. The vector X0 D x in the initial condition for (1) is usually random (i.e., uncertain), it is assumed to be independent of both w and v, and its density './ is assumed to be known. Let f .x/ be a function on Rd . We assume that the coefficients in (1), (2) and the function f are bounded and have bounded derivatives up to some order. The stochastic filtering problem consists in constructing the estimate fO.Xt / based on the observation ys ; 0  s  t; which is the best in the mean-square sense, i.e., the problem amounts to computing the conditional expectation:

filtering (4) is usually called the Kushner-Stratonovich equation or the Fujisaki-Kallianpur-Kunita equation. If the conditional measure E.I.X.t/ 2 dx/ j ys ; 0  s  t/ has a smooth density .t; x/ with respect to the Lebesgue measure, then it solves the following nonlinear stochastic equation:

Optimal Filter Equations In this section we give a number of expressions for the optimal filter which involve solving some stochastic evolution equations. Proofs of the results presented in this section and their detailed exposition and extensions are available, e.g., in [1, 4, 6–11]. The solution t Œf  to the filtering problem (3), (1)– (2) satisfies the nonlinear stochastic evolution equation:

According to our assumptions, we have E1 D 1; t 0  t  T: We introduce theR new probability Q measure PQ on .&; F / W P./ D  1 T d P.!/: The Q measures P and P are mutually absolutely continuous. Due to the Girsanov theorem, ys is a Wiener process on Q the processes Xs and ys are independent .&; F ; Ft ; P/; Q and the process Xs satisfies the ˆIto on .&; F ; Fs ; P/; SDE

d .t; x/ D L .t; x/dt C .M  t Œˇ/> .t; x/ .dy  t Œˇdt/ ; .0; x/ D '.x/;

(5)

to L W L f WD where L is an adjoint  operator Pd 1 Pd @2 @  i;j D1 @xi @xj aij f  i D1 @xi .˛i f / and M is 2

an adjoint operator to M W M D .M1 ; : : : ; Mr />   P with Mj f D  diD1 @x@ i ij f C ˇj f: We note that R t Œf  D Rd f .x/.t; x/dx: We also remark that the Rt process vN t WD yt  0 s Œˇds is called the innovation t Œf  D fO.Xt / D E .f .Xt / j ys ; 0  s  t/ process. D W Ey f .Xt /: (3) Now we will give another expression for the optimal filter. Let Applications of nonlinear filtering include track Z t  Z 1 t 2 ing, navigation systems, cryptography, image process> t W D exp ˇ .Xs /dvs C ˇ .Xs /ds 2 0 ing, weather forecasting, financial engineering, speech 0

Z t  Z recognition, and many others (see, e.g., [2, 11] and ref1 t 2 > D exp ˇ .X /dy  ˇ .X /ds : erences therein). For a historical account, see, e.g., [1]. s s s 2 0 0



 d t Œf  D t ŒLf dt C t ŒM> f   t Œf t Œˇ >  .dy  t Œˇdt/ ;

C.X /dys ; X0 D x:

(6)

(4)

where L is the generator for the diffusion process Xt : d d X 1 X @2 f @f aij C ˛i 2 i;j D1 @xi @xj @x i i D1

S dX D .˛.X /  .X /ˇ.X // ds C .X /d ws

Due to the Kallianpur-Striebel formula (a particular case of the general Bayes formula [7, 9]) for the conditional mean (3), we have

Q y .f .Xt /t / EQ .f .Xt /t j ys ; 0  s  t/ E ; D Q .t j ys ; 0  s  t/ E EQ y t (7) with a D faij g being a d  d -dimensional matrix de- where Xt is from (6), EQ means expectation according Q and EQ y ./ WD EQ . j ys ; 0  s  t/ : fined by a.x/ D .x/ > .x/ C .x/ > .x/ and M D to the measure P; > .M ; : : : ; Mr / is a vector of the operators Mj f WD Let Pd 1 @f t Œg WD EQ y .g.Xt /t / ; i D1 ij @xi C ˇj f: The equation of optimal nonlinear Lf WD

t Œf  D

1398

Stochastic Filtering

where g is a scalar function on Rd : The process t is therein and for a number of recent developments often called the unnormalized optimal filter. It satisfies see [2]. the linear evolution equation Linear Filtering and Kalman-Bucy Filter dt Œg D t ŒLgdt C t ŒM> gdyt ; 0 Œg D 0 Œg; There are a very few cases when explicit formulas for (8) optimal filters are available [1, 7]. The most notable which is known as the Zakai equation. Assuming case is when the filtering problem is linear. Consider that there is the corresponding smooth unnormalized the system of linear SDE: filtering density .t; x/, it solves the linear stochastic partial differential equation (SPDE) of parabolic type: dX D .as C As X / ds C Qs d ws C Gs dvs ;   > X0 D x; (13) d.t; x/ D L .t; x/dt C M .t; x/ dyt ; .0; x/ D '.x/:

(9) dy D .bs C Bs X / ds C dvs ; y0 D 0;

R

We note that t Œg D Rd g.x/.t; x/dx: The unnormalized optimal filter can also be found as a solution of a backward SPDE. To this end, let us fix a time moment t and introduce the function

 ; (10) ug .s; xI t/ D EQ y g.Xts;x /s;x;1 t s;x;1 where x 2 Rd is deterministic and Xss;x ; s 0  s; 0 ; s 0 ˆ is the solution of the Ito SDE

dX D .˛.X /  .X /ˇ.X // ds 0 C .X /d ws 0 C.X /dys 0 ; Xs D x; d D ˇ > .X /dys 0 ; s D 1: The function ug .s; xI t/; s  t; is the solution of the Cauchy problem for the backward linear SPDE:  d u D Luds C M> u dys ; u.t; x/ D g.x/: (11) The notation “ dy” means backward ˆIto integral [5,9]. If X0 D x D is a random variable with the density './; we can write uf;' .0; t/ ; u1;' .0; t/

(14)

where As ; Bs ; Qs ; and Gs are deterministic matrix functions of time s having the appropriate dimensions; as and bs are deterministic vector functions of time s having the appropriate dimensions; the initial condition X0 D x is a Gaussian random vector with mean M0 2 Rd and covariance matrix C0 2 Rd  Rd and it is independent of both w and v; the other notation is as in (1) and (2). We note that the solution Xt ; yt of the SDE (13) and (14) is a Gaussian process. The conditional distribution of Xt given fys ; 0  s  tg is Gaussian with mean XOt and covariance Pt ; which satisfy the following system of differential equations [1, 7, 10, 11]:

   d XO D as C As XO ds C Gs C PBs> O .dys  .bs C Bs X/ds/; XO0 D M0 ;

(15)

 >  d > P D PA> Gs C PBs> s C As P  Gs C PBs dt CQs Qs> C Gs Gs> ; P0 D C0 :

(16)

(12) The solution XO t ; Pt is called the Kalman-Bucy filter (or linear quadratic estimation). We remark that (15) R for the conditional mean XOt D E.Xt j ys ; 0  s  t/ where D Rd ug .0; xI t/'.x/dx

ug;' .0; t/  WD is a linear SDE, while the solution Pt of the matrix 0; 0; ;1 y Q E g.Xt /t D t Œg: Riccati equation (16) is deterministic and it can be preGenerally, numerical methods are required to solve computed off-line. Online updating of XO t with arrival optimal filtering equations. For an overview of various of new data yt from observations is computationally numerical approximations for the nonlinear filtering very cheap, and the Kalman-Bucy filter and its various problem, see [1, Chap. 8] together with references modifications are widely used in practical applications. t Œf  D

Stochastic ODEs

References 1. Bain, A., Crisan, D.: Fundamentals of Stochastic Filtering. Springer, New York/London (2008) 2. Crisan, D., Rozovskii, B. (eds.): Handbook of Nonlinear Filtering. Oxford University Press, Oxford (2011) 3. Fristedt, B., Jain, N., Krylov, N.: Filtering and Prediction: A Primer. AMS, Providence (2007) 4. Kallianpur, G.: Stochastic Filtering Theory. Springer, New York (1980) 5. Kunita, H.: Stochastic Flows and Stochastic Differential Equations. Cambridge University Press, Cambridge/New York (1990) 6. Kushner, H.J.: Probability Methods for Approximations in Stochastic Control and for Elliptic Equations. Academic, New York (1977) 7. Liptser, R.S., Shiryaev, A.N.: Statistics of Random Processes. Springer, New York (1977) ´ 8. Pardoux, E.: Equations du filtrage non lin´eaire de la pr´ediction et du lissage. Stochastics 6, 193–231 (1982) 9. Rozovskii, B.L.: Stochastic Evolution Systems, Linear Theory and Application to Nonlinear Filtering. Kluwer Academic, Dordrecht/Boston/London (1991) 10. Stratonovich, R.L.: Conditional Markov Processes and Their Applications to Optimal Control Theory. Elsevier, New York (1968) 11. Xiong, J.: An Introduction to Stochastic Filtering Theory. Oxford University Press, Oxford (2008)

1399

where the increments Wt2  Wt1 and Wt4  Wt3 on nonoverlapping intervals (i.e., with 0  t1 < t2  t3 < t4 ) are independent random variables. The sample paths of a Wiener process are continuous, but they are nowhere differentiable. Consequently, an SODE is not a differential equation at all, but just a symbolic representation for the stochastic integral equation Z

Z

t

X t D X t0 C

t

f .s; Xs / ds C t0

g.s; Xs / d Ws ; t0

where the first integral is a deterministic Riemann integral for each sample path. The second integral cannot be defined pathwise as a Riemann-Stieltjes integral because the sample paths of the Wiener process do not have even bounded variation on any bounded time interval. Thus, a new type of stochastic integral RT is required. An Itˆo stochastic integral t0 f .t/d Wt is defined as the mean-square limit of sums of products of an integrand f evaluated at the left end point of each partition subinterval times Œtn ; tnC1 , the increment of the Wiener process, i.e., Z

T

f .t/d Wt WD m.s.  lim

N !1

t0

Stochastic ODEs Peter Kloeden FB Mathematik, J.W. Goethe-Universit¨at, Frankfurt am Main, Germany



NX

1

f .tn /

j D0

 WtnC1  Wtn ;

where tnC1 tn D =N for n D 0, 1, : : :, N 1. The integrand function f may be random or even depend on the path of the Wiener process, but f .t/ should be independent of future increments of the Wiener A scalar stochastic ordinary differential equation process, i.e., Wt Ch  Wt for h > 0. (SODE) The Itˆo stochastic integral has the important properties (the second is called the Itˆo isometry) that dXt D f .t; Xt / dt C g.t; Xt / d Wt (1) "Z 2 # Z T T f .t/d Wt D 0; E f .t/d Wt E t0 t0 involves a Wiener process Wt , t  0, which is one of the most fundamental stochastic processes and is Z T often called a Brownian motion. A Wiener process is a   D E f .t/2 dt: Gaussian process with W0 D 0 with probability 1 and t0 normally distributed increments Wt  Ws for 0  s < t with However, the solutions of Itˆo SODE satisfy a different chain rule to that in deterministic calculus, called the E .Wt  Ws / D 0; E .Wt  Ws /2 D t  s; Itˆo formula, i.e.,

S

1400

Stochastic ODEs

Z

t

L0 U.s; Xs / ds

U.t; Xt / D U.t0 ; Xt0 / C Z

t0 t

L1 .s; Xs / d Ws ;

C t0

The Itˆo and Stratonovich stochastic calculi are both mathematically correct. Which one should be used is really a modeling issue, but once one has been chosen, the advantages of the other can be used through a modification of the drift term to obtain the corresponding SODE of the other type that has the same solutions.

where @U 1 @2 U @U Cf C g2 2 ; @t @x 2 @x

@U : Numerical Solution of SODEs @x The simplest numerical method for the above An immediate consequence is that the integration rules SODE (1) is the Euler-Maruyama scheme given by and tricks from deterministic calculus do not hold and different expressions result, e.g., YnC1 D Yn C f .tn ; Yn / n C g.tn ; Yn / Wn ; L0 U D

Z

T

L1 U D g

1 2 1 W  T: 2 T 2

There is another stochastic integral called the Stratonovich integral, for which the integrand function is evaluated at the midpoint of each partition subinterval rather than at the left end point. It is written with ıd Wt to distinguish it from the Itˆo integral. A Stratonovich SODE is thus written

where n D tnC1  tn and Wn D WtnC1  Wtn . This is intuitively consistent with the definition of the Itˆo integral. Here Yn is a random variable, which is supposed to be an approximation on Xtn . The stochastic increments Wn , which are N .0; n / distributed, can be generated using, for example, the Box-Muller method. In practice, however, only individual realizations can be computed. Depending on whether the realizations of the solutions or only their probability distributions are required to be close, one distinguishes between strong and weak convergence of numerical schemes, respectively, on a given interval Œt0 ; T . Let D maxn n be the maximum step size. Then a numerical scheme is said to converge with strong order  if, for sufficiently small ,

dXt D f .t; Xt / dt C g.t; Xt / ı d Wt :

ˇ

ˇ ˇ . / ˇ E ˇXT  YNT ˇ  KT 

Ws d Ws D 0

The situation for vector-valued SODE and vectorvalued Wiener processes is similar. Details can be found in Refs. [3, 4, 6].

Stratonovich SODEs

Stratonovich stochastic calculus has the same chain rule as deterministic calculus, which means that Stratonovich SODE can be solved with the same integration tricks as for ordinary differential equations. However, Stratonovich stochastic integrals do not satisfy the nice properties above for Itˆo stochastic integrals, which are very convenient for estimates in proofs. Nor does the Stratonovich SODE have same direct connection with diffusion process theory as the Itˆo SODE, e.g., the coefficient of the Fokker-Planck equation correspond to those of the Itˆo SODE (1), i.e., @ 1 @2 p @p Cf C g 2 2 D 0: @t @x 2 @x

and with weak order ˇ if ˇ

ˇ ˇ . / ˇ ˇE .p.XT //  E p.YNT / ˇ  Kp;T ˇ for each polynomial p. These are global discretization errors, and the largest possible values of  and ˇ give the corresponding strong and weak orders, respectively, of the scheme for a whole class of stochastic differential equations, e.g., with sufficiently often continuously differentiable coefficient functions. For example, the Euler-Maruyama scheme has strong order  D 12 and weak order ˇ D 1, while the Milstein scheme

Stochastic ODEs

YnC1 D Yn C f .tn ; Yn / n C g.tn ; Yn / Wn ˚ 1 @g C g.tn ; Yn / .tn ; Yn / . Wn /2  n 2 @x has strong order  D 1 and weak order ˇ D 1; see [2, 3, 5]. Note that these convergence orders may be better for specific SODE within the given class, e.g., the EulerMaruyama scheme has strong order  D 1 for SODE with additive noise, i.e., for which g does not depend on x, since it then coincides with the Milstein scheme. The Milstein scheme is derived by expanding the integrand of the stochastic integral with the Itˆo formula, the stochastic chain rule. The additional involves R t Rterm s the double stochastic integral tnnC1 tn d Wu d Ws , which provides more information about the nonsmooth Wiener process inside the ˚ discretization subinterval and is equal to 12 . Wn /2  n . Numerical schemes of even higher order can be obtained in a similar way. In general, different schemes are used for strong and weak convergence. The strong stochastic Taylor schemes have strong order  D 12 , 1, 32 , 2, : : :, whereas weak stochastic Taylor schemes have weak order ˇ D 1,2, 3, : : :. See [3] for more details. In particular, one should not use heuristic adaptations of numerical schemes for ordinary differential equations such as Runge-Kutta schemes, since these may not converge to the right solution or even converge at all. The proofs of convergence rates in the literature assume that the coefficient functions in the above stochastic Taylor schemes are uniformly bounded, i.e., the partial derivatives of appropriately high order of the SODE coefficient functions f and g exist and are uniformly bounded. This assumption, however, is not satisfied in many basic and important applications, for example, with polynomial coefficients such as dXt D .1 C Xt /.1  Xt2 / dt C .1  Xt2 / d Wt or with square-root coefficients such as in the CoxIngersoll-Ross volatility model

1401

methods have been suggested to prevent this. The paper [1] provides a systematic method to handle both of these problems by using pathwise convergence, i.e., sup nD0;:::;NT

ˇ ˇ ˇXt .!/  Y . / .!/ˇ ! 0 as ! 0; n n ! 2 &:

It is quite natural to consider pathwise convergence since numerical calculations are actually carried out path by path. Moreover, the solutions of some SODE do not have bounded moments, so pathwise convergence may be the only option.

Iterated Stochastic Integrals Vector-valued SODE with vector-valued Wiener processes can be handled similarly. The main new difficulty is how to simulate the multiple stochastic integrals since these cannot be written as simple formulas of the basic increments as in the double integral above when they involve different Wiener processes. In general, such multiple stochastic integrals cannot be avoided, so they must be approximated somehow. One possibility is to use random Fourier series for Brownian bridge processes based on the given Wiener processes; see [3, 5]. Another way is to simulate the integrals themselves by a simpler numerical scheme. For example, double integral Z

tnC1

Z

t

I.2;1/;n D tn

tn

d Ws2 d Wt1

S

for two independent Wiener processes Wt1 and Wt2 can be approximated by applying the (vector-valued) Euler-Maruyama scheme to the 2-dimensional Itˆo SODE (with superscripts labeling components) dXt1 D Xt2 d Wt1 ;

dXt2 D d Wt2 ;

(2)

over the discretization subinterval Œtn ; tnC1  with a suitable step size ı D .tnC1  tn /=K. The solution of the SODE (2) with the initial condition Xt1n D 0, 2 2 which requires Vt  0. The second is more difficult Xtn D Wtn at time t D tnC1 is given by because there is a small probability that numerical Xt1nC1 D I.2;1/;n ; Xt2nC1 D Wn2 : iterations may become negative, and various ad hoc p dVt D  .#  Vt / dt C  Vt d Wt ;

1402

Stochastic Simulation

j j j The important thing is to decide first what kind of Writing k D tn C kı and ıWn;k D W kC1  W k , the approximation one wants, strong or weak, as this will stochastic Euler scheme for the SDE (2) reads determine the type of scheme that should be used, and 1 1 2 2 then to exploit the structural properties of the SODE D Yk1 C Yk2 ıWn;k ; YkC1 D Yk2 C ıWn;k ; YkC1 under consideration to simplify the scheme that has for 0  k  K  1; (3) been chosen to be implemented.

with the initial value Y01 D 0, Y02 D Wtn2 . The strong order of convergence of  D 12 of the Euler-Maruyama References scheme ensures that p ˇ ˇ E ˇYK1  I.2;1/;n ˇ  C ı; so I.2;1/;n can be approximated in the Milstein scheme by YK1 with ı 2n , i.e., K 1 n , without affecting the overall order of convergence.

Commutative Noise Identities such as Z tnC1 Z t Z j2 j1 d Ws d Wt C tn

tn

D

Wnj1

tnC1 tn

Z

t tn

d Wsj2 d Wt

j1

Wnj2

1. Jentzen, A., Kloeden, P.E., Neuenkirch, A.: Convergence of numerical approximations of stochastic differential equations on domains: higher order convergence rates without global Lipschitz coefficients. Numer. Math. 112, 41–64 (2009) 2. Kloeden, P.E.: The systematic deviation of higher order numerical methods for stochastic differential equations. Milan J. Math. 70, 187–207 (2002) 3. Kloeden, P.E., Platen, E.: The Numerical Solution of Stochastic Differential Equations, 3rd rev. printing. Springer, Berlin, (1999) 4. Mao, X.: Stochastic Differential Equations and Applications, 2nd edn. Horwood, Chichester (2008) 5. Milstein, G.N.: Numerical Integration of Stochastic Differential Equations. Kluwer, Dordrecht (1995) 6. Øksendal, B.: Stochastic Differential Equations. An Introduction with Applications, 6th edn. 2003, Corr. 4th printing. Springer, Berlin (2007)

allow one to avoid calculating the multiple integrals if the corresponding coefficients in the numerical scheme Stochastic Simulation are identical, in this case if L1 g2 .t; x/ L2 g1 .t; x/ (where L2 is defined analogously to L1 ) for an SODE Dieter W. Heermann of the form Institute for Theoretical Physics, Heidelberg University, Heidelberg, Germany 1 2 dXt D f .t; Xt / dt C g1 .t; Xt / d Wt C g2 .t; Xt / d Wt : (4) Then the SODE (4) is said to have commutative noise.

Synonyms

Concluding Remarks

Brownian Dynamics Simulation; Langevin Simulation; Monte Carlo Simulations

The need to approximate multiple stochastic integrals places a practical restriction on the order of strong schemes that can be implemented for a general SODE. Wherever possible special structural properties like commutative noise of the SODE under investigation should be exploited to simplify strong schemes as much as possible. For weak schemes the situation is easier as the multiple integrals do not need to be approximated so accurately. Moreover, extrapolation of weak schemes is possible.

Modelling a system or data one is often faced with the following: • Exact data is unavailable or expensive to obtain • Data is uncertain and/or specified by a probability distribution or decisions have to be made with respect to the degrees of freedom that are taken explicitly into account. This can be seen by looking at a system with two components. One of the components could be water

Stochastic Simulation

1403

molecules and the other component large molecules. The decision is to take the water molecules explicitly into account or to treat them implicitly. Since the water molecules move much faster than the large molecules, we can eliminate the water by subsuming their action on the larger molecules by a stochastic force, i.e., as a random variable with a specific distribution. Thus we have eliminated some details in favor of a probabilistic description where perhaps some elements of the model description are given by deterministic rules and other contributes stochastically. Overall a model derived along the outlined path can be viewed as if an individual state has a probability that may depend on model parameters. In most of the interesting cases, the number of available states the model has will be so large that they simply cannot be enumerated. A sampling of the states is necessary such that the most relevant states will be sampled with the correct probability. Assume that the model has some deterministic part. In the above example, the motion of larger molecules is governed by Newton’s equation of motion. These are augmented by stochastic forces mimicking the water molecules. Depending on how exactly this is implemented results in Langevin equations

AN D

X

A.x/P .x; ˛/;

(5)

x

where P .x; ˛/ is the probability of the state and ˛ a set of parameters (e.g., the temperature T , mass m, etc.). A point of view that can be taken is that what the Eqs. (1) and (4) accomplish is the generation of a stochastic trajectory through the available states. This can equally be well established by other means. As long as we satisfy the condition that the right probability distribution is generated, we could generate the trajectory by a Monte Carlo method [1]. In a Monte Carlo formulation, a transition probability from a state x to another state x 0 is specified W .x 0 jx/:

(6)

Together with the proposition probability for the state x 0 , a decision is made to accept or reject the state (Metropolis-Hastings Monte Carlo Method). An advantage of this formulation is that it allows freedom in the choice of proposition of states and the efficient sampling of the states (importance sampling). In more general terms, what one does is to set up a biased random walk that explores the target distribution (Markov p (1) Chain Monte Carlo). mxR D rU.x/   mxP C .t/ 2kB T m; A special case of the sampling that yields a Markov where x denotes the state (here the position), U the chain is the Gibbs sampler. Assume x D .x 1 ; x 2 / with potential, m the mass of the large molecule, kB the target P .x; ˛/ Boltzmann constant, T the temperature, and the stochastic force with the properties: Algorithm 1 Gibbs Sampler Algorithm: h .t/i D 0 ˛ ˝ .t/ .t 0 / D ı.t  t 0 /:

(2) (3)

1: initialize x0 D .x01 ; x02 / 2: while i  max number of samples do 2 ; ˛/ 3: sample xi1 P .x 1 jxi1 4: sample xi2 P .x 2 jxi1 ; ˛/ 5: end while

Neglecting the acceleration in the Langevin equation yields the Brownian dynamics equation of motion p

x.t/ P D rU.x/= C .t/ 2D

(4)

with  D  m and D D kB T =. Hence the sampling is obtained using the equations of motion to transition from one state to the next. If enough of the available states (here x) are sampled, quantities of interest that depend on the states can be calculated as averages over the generated states:

then fx 1 ; x 2 g is a Markov chain. Thus we obtain a sequence of states such that we can again apply (5) to compute the quantities of interest. Common to all of the above stochastic simulation methods is the use of random numbers. They are either used for the implementation of the random contribution to the force or the decision whether a state is accepted or rejected. Thus the quality of the result depends on the quality of the random number generator.

S

1404

Reference 1. Binder, K., Heermann, D.W.: Monte Carlo Simulation in Statistical Physics: An Introduction. Graduate Texts in Physics, 5th edn. Springer, Heidelberg (2010)

Stochastic Systems Guang Lin1;2;3 and George Em Karniadakis4 1 Fundamental and Computational Sciences Directorate, Pacific Northwest National Laboratory, Richland, WA, USA 2 School of Mechanical Engineering, Purdue University, West Lafayette, IN, USA 3 Department of Mathematics, Purdue University, West Lafayette, IN, USA 4 Division of Applied Mathematics, Brown University, Providence, RI, USA

Mathematics Subject Classification 93E03

Synonyms Noisy Systems; Random Systems; Stochastic Systems

Short Definition A stochastic system may contain one or more elements of random, i.e., nondeterministic behavior. Compared to a deterministic system, a stochastic system does not always generate the same output for a given input. The elements of systems that can be stochastic in nature may include noisy initial conditions, random boundary conditions, random forcing, etc.

Description Stochastic systems (SS) are encountered in many application domains in science, engineering, and business. They include logistics, transportation, communication networks, financial markets, supply chains, social systems, robust engineering design, statistical physics,

Stochastic Systems

systems biology, etc. Stochasticity or randomness is perhaps associated with a bad outcome, but harnessing stochasticity has been pursued in arts to create beauty, e.g., in the paintings of Jackson Pollock or in the music of Iannis Xenakis. Similarly, it can be exploited in science and engineering to design new devices (e.g., stochastic resonances in the Bang and Olufsen speakers) or to design robust and cost-effective products under the framework of uncertainty-based design and real options [17]. Stochasticity is often associated with uncertainty, either intrinsic or extrinsic, and specifically with the lack of knowledge of the properties of the system; hence quantifying uncertain outcomes and system responses is of great interest in applied probability and scientific computing. This uncertainty can be further classified as aleatory, i.e., statistical, and epistemic which can be reduced by further measurements or computations of higher resolution. Mathematically, stochasticity can be described by either deterministic or stochastic differential equations. For example, the molecular dynamics of a simple fluid is described by the classical deterministic Newton’s law of motion or by the deterministic Navier-Stokes equations whose outputs, in both cases, however may be stochastic. On the other hand, stochastic elliptic equations can be used to predict the random diffusion of water in porous media, and similarly a stochastic differential equation may be used to model neuronal activity in the brain [9, 10]. Here we will consider systems that are governed by stochastic ordinary and partial differential equations (SODEs and SPDEs), and we will present some effective methods for obtaining stochastic solutions in the next section. In the classical stochastic analysis, these terms refer to differential equations subject to white noise either additive or multiplicative, but in more recent years, the same terminology has been adopted for differential equations with color noise, i.e., processes that are correlated in space or time. The color of noise, which can also be pink or violet, may dictate the numerical method used to predict efficiently the response of a stochastic system, and hence it is important to consider this carefully at the modeling stage. Specifically, the correlation length or time scale is the most important parameter of a stochastic process as it determines the effective dimension of the process; the smaller the correlation scale, the larger the dimensionality of the stochastic system.

Stochastic Systems

1405

Example: To be more specific in the following, we present a tumor cell growth model that involves stochastic inputs that need to be represented according to the correlation structure of available empirical data [27]. The evolution equation is x.tI P !/ DG.x/ C g.x/f1 .tI !/ C f2 .tI !/; x.0I !/ Dx0 .!/;

(1)

minimizes the mean square error [3, 18, 20]. The K-L expansion has been employed to represent random input processes in many stochastic simulations (see, e.g., [7, 20]).

Stochastic Modeling and Computational Methods

We present two examples of two fundamentally differwhere x.tI !/ denotes the concentration of tumor cell ent descriptions of SS in order to show some of the at time t, complexity but also rich stochastic response that can be obtained: the first is based on a discrete particle x x ; g.x/ D  ; model and is governed by SODEs, and the second is G.x/ D x.1  x/  ˇ xC1 xC1 a continuum system and is governed by SPDEs. ˇ is the immune rate, and  is related to the rate of growth of cytotoxic cells. The random process f1 .tI !/ represents the strength of the treatment (i.e., the dosage of the medicine in chemotherapy or the intensity of the ray in radiotherapy), while the process f2 .tI !/ is related to other factors, such as drugs and radiotherapy, that restrain the number of tumor cells. The parameters ˇ,  and the covariance structure of random processes f1 and f2 are usually estimated based on empirical data. If the processes f1 .t; !/ and f2 .t; !/ are independent, they can be represented using the KarhunenLoeve (K-L) expansion by a zero mean, second-order random process f .t; !/ defined on a probability space .&; F; P/ and indexed over t 2 Œa; b. Let us denote the continuous covariance function of f .tI !/ as C.s; t/. Then the process f .tI !/ can be represented as

A Stochastic Particle System: We first describe a stochastic model for a mesoscopic system governed by a modified version of Newton’s law of motion, the so-called dissipative particle dynamics (DPD) equations [19]. It consists of particles which correspond to coarse-grained entities, thus representing molecular clusters rather than individual atoms. The particles move off-lattice interacting with each other through a set of prescribed (conservative and stochastic) and velocity-dependent forces. Specifically, there are three types of forces acting on each dissipative particle: (a) a purely repulsive conservative force, (b) a dissipative force that reduces velocity differences between the particles, and (c) a stochastic force directed along the line connecting the center of the particles. The last two forces effectively implement a thermostat so that thermal equilibrium is achieved. Correspondingly, the amplitude of these forces is dictated by the fluctuationN d p X dissipation theorem that ensures that in thermodynamic k ek .t/ k .!/; f .tI !/ D equilibrium the system will have a canonical distrikD1 bution. All three forces are modulated by a weight where k .!/ are uncorrelated random variables with function which specifies the range of interaction or zero mean and unitary variance, while k and ek .t/ are, cutoff radius rc between the particles and renders the respectively, eigenvalues and (normalized) eigenfunc- interaction local. tions of the integral operator with kernel C.s; t/, i.e., The DPD equations for a system consisting of N particles have equal mass (for simplicity in the preZ b sentation) m, position ri , and velocities ui , which are C.s; t/ek .s/ds D k ek .t/: stochastic in nature. The aforementioned three types of a forces exerted on a particle i by particle j are given by The dimension Nd depends strongly on the correlation scale of the kernel C.s; t/. If we rearrange the eigen- Fcij D F .c/ .rij /eij ; Fdij D ! d .rij /.vij  eij /eij ; values k in a descending order, then any truncation r r of the expansion f .tI !/ is optimal in the sense that it Fij D ! .rij / ij eij ;

S

1406

Stochastic Systems

5

VLJ averaged

4.5 4

V(r)/kB T

3.5 3 2.5 2 1.5 1 0.5 0 0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

2.2

distance (r)

Stochastic Systems, Fig. 1 Left: Lennard-Jones potential and its averaged soft repulsive-only potential. Right: Polymer chains flowing in a sea of solvent in DPD. For more details see [19]

where rij D ri  rj , vij D vi  vj , rij D jrij j r and the unit vector eij D rijij . The variables  and  determine the strength of the dissipative and random forces, respectively, ij are symmetric Gaussian random variables with zero mean and unit variance, and ! d and ! r are weight functions. The time evolution of DPD particles is described by Newton’s law

d ri D vi ıtI

Fc ıt C Fdi ıt C Fri d vi D i mi

p ıt

;

P where Fci D i ¤j Fcij is the total conservative force acting on particle i ; Fdi and Fri are defined similarly. The velocity p increment due to the random force has a factor ıt since it represents Brownian motion, which is described by a standard Wiener process with jt1 t2 j a covariance kernel given by CFF .ti ; tj / D e  A , where A is the correlation time for this stochastic process. The conservative force Fc is typically given in terms of a soft potential in contrast to the LennardJones potential used in molecular dynamics studies (see Fig. 1(left)). The dissipative and random forces are characterized by strengths ! d .rij / and ! r .rij / coupled due to the fluctuation-dissipation theorem. Several complex fluid systems in industrial and biological applications (DNA chains, polymer gels,

lubrication) involve multiscale processes and can be modeled using modifications of the above stochastic DPD equations [19]. Dilute polymer solutions are a typical example, since individual polymer chains form a group of large molecules by atomic standards but still governed by forces similar to intermolecular ones. Therefore, they form large repeated units exhibiting slow dynamics with possible nonlinear interactions (see Fig. 1(right)). A Stochastic Continuum System: Here we present an example from classical aerodynamics on shock dynamics by reformulating the one-dimensional piston problem within the stochastic framework, i.e., we allow for random piston motions which may be changing in time [11]. In particular, we superimpose small random velocity fluctuations to the piston velocity and aim to obtain both analytical and numerical solutions of the stochastic flow response. We consider a piston having a constant velocity, Up , moving into a straight tube filled with a homogeneous gas at rest. A shock wave will be generated ahead of the piston. A sketch of the piston-driven shock tube with random piston motion superimposed is shown in Fig. 2(left). As shown in Fig. 2 (left), Up and S are the deterministic speed of the piston and deterministic speed of the shock, respectively, and 0 , P0 , C0 , 1 , P1 , and C1 are the deterministic density, pressure, and local sound

Stochastic Systems

1407

Perturbation Analysis Later time: ∼τ

U = Up + vp ( t ) P = P1 ρ = ρ1 C = C1

S + vs ( t )

U=0 P = P0 ρ = ρ0 C = C0

< ξ2(t)>/(UpqS’A / α)2

20

Early time: ∼τ2 15

10

Up + vp ( t )

5

0 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 τ

Stochastic Systems, Fig. 2 Left: Sketch of piston-driven shock tube with random piston motion. Right: Normalized variance of perturbed shock paths. Solid line: perturbation analysis results.

Dashed line: early-time asymptotic results, h 2 . /i 2 . Dashdotted line: late-time asymptotic results, h 2 . /i

speed ahead and after of the shock, respectively. We now define the stochastic motion of the piston by superimposing a small stochastic component to the steady speed of the piston, i.e., up .t/ D Up Œ1 C V .t; !/, where  is the amplitude of the random perturbation. Here V .t; !/ is modeled as a random process with zero jt1 t2 j mean and covariance hV .t1 ; !/; V .t2 ; !/i D e  A , where A is the correlation time; it can be represented by a truncated K-L expansion as explained earlier. Our objective is to quantify the deviation of the perturbed shock paths due to the random piston motion from the unperturbed ones, which are given by X.t/ D S  t. If the amplitude  is small, 0 <  1, the analytical solutions for the perturbed shock paths can be expressed as follows:

where S 0 D C1 CUp S , C1 CS Up

dS , m d Up 2 and 1Ck

C1 CUp S , ˇ D C1 S CS 0 U 1k . Here k D C 1CS Upp 1Ck

< n, ˛ D

qD rD and  D cp =cv is the ratio of specific heats. In Fig. 2 (right), the variance of the perturbed shock path, h 2 . /i=.Up qS 0 A=˛/2 , is plotted as a function of with Up D 1:25, i.e., corresponding to Mach number of the shock M D 2. The asymptotic formulas for small and large are also included in the plot. In Fig. 2 (right), we observe that the variance of the shock location grows quadratically with time for early times and switches to linear growth for longer times. The stochastic solutions for shock paths, for either small or large piston motions, can also be obtained numerically by solving the full nonlinear Euler equations with an unsteady stochastic boundary, namely, the piston position to model the stochastic piston prob" 1 n1 XX lem. Classic Monte Carlo simulations [4] or quasi.r/nCm In;m . /C Monte Carlo simulations [2] can be performed for h 2 . /i D.Up qS 0 A=˛/2 2 nD1 mD0 these stochastic simulations. However, due to the slow # 1 X convergence rate of Monte Carlo methods, it may take r 2n In;n . / (2) thousands of equivalent deterministic simulations to nD0 achieve acceptable accuracy. Recently, methods based on generalized polynomial chaos (gPC) expansions where D ˛t=A, and have become popular for such SPDEs due to their fast convergence rate for SS with color noise. The term polynomial chaos was first coined by Norbert Wiener 2

1 h m n in 1938 in his pioneering work on representing GausIn;m . / D m C nCm e ˇ C e ˇ

ˇ ˇ sian stochastic processes [22] as generalized Fourier i .ˇ m ˇ n /

series. In Wiener’s work, Hermite polynomials serve 1  e ;

S

1408

Stochastic Systems

as an orthogonal basis. The gPC method for solving SPDEs is an extension of the polynomial chaos method developed in [7], inspired by the theory of WienerHermite polynomial chaos. The use of Hermite polynomials may not be optimum in applications involving non-Gaussian processes, and hence gPC was proposed in [25] to alleviate the difficulty. In gPC, different kinds of orthogonal polynomials are chosen as a basis depending on the probability distribution of the random inputs. The P th-order, gPC approximations of the solution u.x; / can be obtained by projecting u onto the space WNP , i.e., PPN u D uPN .x; / D

M X

uO m .x/ m . /;

(3)

mD1

where PPN u denotes the orthogonal projection operator /Š O m are from L2 . / onto WNP , M C 1 D .NNCP ŠP Š , and u the coefficients, and  the probability measure. Although gPC was shown to exhibit exponential convergence in approximating stochastic solutions at finite times, gPC may converge slowly or fail to converge even in short-time integration due to a discontinuity of the approximated solution in random space. To this end, the Wiener-Haar method [14, 15] based on wavelets, random domain decomposition [12], multielement-gPC (ME-gPC) [21], and multielement probabilistic collocation method (MEPCM) [5] were developed to address problems related to the aforementioned discontinuities in random space. Additionally, a more realistic representation of stochastic inputs associated with various sources of uncertainty in the stochastic systems may lead to highdimensional representations, and hence exponential computational complexity, running into the socalled curse of dimensionality. Sparse grid stochastic collocation method [24] and various versions ANOVA (ANalysis Of VAriance) method [1, 6, 8, 13, 23, 26] have been employed as effective dimension-reduction techniques for quantifying the uncertainty in stochastic systems with dimensions up to 100.

Conclusion Aristotle’s logic has ruled our scientific thinking in the past two millennia. Most scientific models and theories have been constructed from exact models and logic

reasoning. It is argued in [16] that SS models and statistical reasoning are more relevant “i) to the world, ii) to science and many parts of mathematics and iii) particularly to understanding the computations in our own minds, than exact models and logical reasoning.” Indeed, many real-world problems can be viewed or modeled as SS with great potential benefits across disciplines from physical sciences and engineering to social sciences. Stochastic modeling can bring in more realism and flexibility and account for uncertain inputs and parametric uncertainty albeit at the expense of mathematical and computational complexity. However, the rapid mathematical and algorithmic advances already realized at the beginning of the twentyfirst century along with the simultaneous advances in computer speeds and capacity will help alleviate such difficulties and will make stochastic modeling the standard norm rather than the exception in the years ahead. The three examples we presented in this chapter illustrate the diverse applications of stochastic modeling in biomedicine, materials processing, and fluid mechanics. The same methods or proper extensions can also be applied to quantifying uncertainties in climate modeling; in decision making under uncertainty, e.g., in robust engineering design and in financial markets; but also for modeling the plethora of emerging social networks. Further work on the mathematical and algorithmic formulations of such more complex and high-dimensional systems is required as current approaches cannot yet deal satisfactorily with white noise, system discontinuities, high dimensions, and long-time integration. Acknowledgements The authors would like to acknowledge support by DOE/PNNL Collaboratory on Mathematics for Mesoscopic Modeling of Materials (CM4).

References 1. Bieri, M., Schwab, C.: Sparse high order fem for elliptic sPDEs. Comput. Methods Appl. Mech. Eng. 198, 1149–1170 (2009) 2. Caflisch, R.: Monte Carlo and quais-Monte Carlo methods. Acta Numer. 7, 1–49 (1998) 3. Chien, Y., Fu, K.S.: On the generalized Karhunen-Lo`eve expansion. IEEE Trans. Inf. Theory 13, 518–520 (1967) 4. Fishman, G.: Monte Carlo: Concepts, Algorithms, and Applications. Springer Series in Operations Research and Financial Engineering. Springer, New York (2003) 5. Foo, J., Wan, X., Karniadakis, G.E.: A multi-element probabilistic collocation method for PDEs with parametric un-

Stokes or Navier-Stokes Flows

6.

7. 8.

9. 10. 11.

12.

13.

14.

15.

16.

17. 18. 19.

20.

21.

22. 23.

24.

25.

certainty: error analysis and applications. J. Comput. Phys. 227, 9572–9595 (2008) Foo, J.Y., Karniadakis, G.E.: Multi-element probabilistic collocation in high dimensions. J. Comput. Phys. 229, 1536–1557 (2009) Ghanem, R.G., Spanos, P.: Stochastic Finite Eelements: A Spectral Approach. Springer, New York (1991) Griebel, M.: Sparse grids and related approximation schemes for higher-dimensional problems. In: Proceedings of the Conference on Foundations of Computational Mathematics, Santander (2005) van Kampen, N.: Stochastic Processes in Physics and Chemistry, 3rd edn. Elsevier, Amsterdam (2008) Laing, C., Lord, G.J.: Stochastic Methods in Neuroscience. Oxford University Press, Oxford (2009) Lin, G., Su, C.H., Karniadakis, G.E.: The stochastic piston problem. Proc. Natl. Acad. Sci. U. S. A. 101(45), 15,840–15,845 (2004) Lin, G., Tartakovsky, A.M., Tartakovsky, D.M.: Uncertainty quantification via random domain decomposition and probabilistic collocation on sparse grids. J. Comput. Phys. 229, 6995–7012 (2010) Ma, X., Zabaras, N.: An adaptive hierarchical sparse grid collocation method for the solution of stochastic differential equations. J. Comput. Phys. 228, 3084–3113 (2009) Maitre, O.P.L., Njam, H.N., Ghanem, R.G., Knio, O.M.: Multi-resolution analysis of Wiener-type uncertainty propagation schemes. J. Comput. Phys. 197, 502–531 (2004) Maitre, O.P.L., Njam, H.N., Ghanem, R.G., Knio, O.M.: Uncertainty propagation using Wiener-Haar expansions. J. Comput. Phys. 197, 28–57 (2004) Mumford, D.: The dawning of the age of stochasticity. In: Mathematics Towards the Third Millennium, Rome (1999) de Neufville, R.: Uncertainty Management for Engineering Systems Planning and Design. MIT, Cambridge (2004) Papoulis, A.: Probability, Random Variables and Stochastic Processes, 3rd edn. McGraw-Hill, Europe (1991) Symeonidis, V., Karniadakis, G., Caswell, McGraw-Hill Europe B.: A seamless approach to multiscale complex fluid simulation. Comput. Sci. Eng. 7, 39–46 (2005) Venturi, D.: On proper orthogonal decomposition of randomly perturbed fields with applications to flow past a cylinder and natural convection over a horizontal plate. J. Fluid Mech. 559, 215–254 (2006) Wan, X., Karniadakis, G.E.: An adaptive multi-element generalized polynomial chaos method for stochastic differential equations. J. Comput. Phys. 209, 617–642 (2005) Wiener, N.: The homogeneous chaos. Am. J. Math. 60, 897–936 (1938) Winter, C., Guadagnini, A., Nychka, D., Tartakovsky, D.: Multivariate sensitivity analysis of saturated flow through simulated highly heterogeneous groundwater aquifers. J. Comput. Phys. 217, 166–175 (2009) Xiu, D., Hesthaven, J.: High order collocation methods for differential equations with random inputs. SIAM J. Sci. Comput. 27(3), 1118–1139 (2005) Xiu, D., Karniadakis, G.E.: The Wiener-Askey polynomial chaos for stochastic differential equations. SIAM J. Sci. Comput. 24(2), 619–644 (2002)

1409 26. Yang, X., Choi, M., Lin, G., Karniadakis, G.E.: Adaptive anova decomposition of stochastic incompressible and compressible flows. J. Comput. Phys. 231, 1587–1614 (2012) 27. Zeng, C., Wang, H.: Colored noise enhanced stability in a tumor cell growth system under immune response. J. Stat. Phys. 141, 889–908 (2010)

Stokes or Navier-Stokes Flows Vivette Girault and Fr´ed´eric Hecht Laboratoire Jacques-Louis Lions, UPMC University of Paris 06 and CNRS, Paris, France

Mathematics Subject Classification 76D05; 76D07; 35Q30; 65N30; 65F10

Definition Terms/Glossary Boundary layer It refers to the layer of fluid in the immediate vicinity of a bounding surface where the effects of viscosity are significant. GMRES Abbreviation for the generalized minimal residual algorithm. It refers to an iterative method for the numerical solution of a nonsymmetric system of linear equations. Precondition It consists in multiplying both sides of a system of linear equations by a suitable matrix, called the preconditioner, so as to reduce the condition number of the system.

The Incompressible Navier-Stokes Model The incompressible Navier-Stokes system of equations is a widely accepted model for viscous Newtonian incompressible flows. It is extensively used in meteorology, oceanography, canal flows, pipeline flows, automotive industry, high-speed trains, wind turbines, etc. Computing accurately its solutions is a difficult and important challenge. A Newtonian fluid is a model whose Cauchy stress tensor depends linearly on the strain tensor, in contrast to non-Newtonian fluids for which this relation is

S

1410

Stokes or Navier-Stokes Flows

nonlinear and possibly implicit. For a Navier-Stokes are complemented with boundary conditions, such as fluid model, the constitutive equation defining the the no-slip condition Cauchy stress tensor T is: u D 0; on @˝;

T D I C 2D.u/;

(6)

(1) and an initial condition

where  > 0 is the constant viscosity coefficient, representing friction between molecules,  isthe pressure,  1 u is the velocity, D.u/ D 2 r u C r uT is the strain @ui tensor, and .r u/ij D @x is the gradient tensor. When j substituted into the balance of linear momentum

u.; 0/ D u0 ./ in ˝; satisfying divu0 D 0; and u0 D 0; on @˝:

(7)

In practical situations, other boundary conditions may be prescribed. One of the most important occurs in (2) flows past a moving obstacle, in which case (6) is replaced by where  > 0 is the fluid’s density, f is an external Z body force (e.g., gravity), and ddtu is the material time g  n D 0; (8) u D g; on @˝ where derivative du  D div T C f ; dt

@˝

d u @u D Cur u; where ur u D Œr uu D dt @t

X

@u ; where g is the velocity of the moving body and n @x i i denotes the unit exterior normal vector to @˝. To (3) simplify, we shall only discuss (6), but we shall present (1) gives, after division by , numerical experiments where (8) is prescribed. If the boundary conditions do not involve the bound@u 1  ary traction vector T n, (4) can be substantially simpliC u  r u D  r  C 2 div D.u/ C f : @t   fied by using the fluid’s incompressibility. Indeed, (5) implies 2div D.u/ D  u, and (4) becomes But the density  is constant, since the fluid is incompressible. Therefore renaming the quantities p D  @u C u  r u   u C r p D f : (9) and the kinematic viscosity  D  , the momentum @t equation reads: ui

@u C u  r u D r p C 2 div D.u/ C f : @t

When written in dimensionless variables (denoted by (4) the same symbols), (9) reads

As the fluid is incompressible, the conservation of mass @ C div.u/ D 0; @t

@u 1 Curu uC r p D f ; @t Re

(10)

where Re is the Reynolds number, ReD LU  , L is a characteristic length, and U a characteristic velocity. When 1  Re  105 , the flow is said to be laminar. reduces to the incompressibility condition Finally, when the Reynolds number is small and the div u D 0: (5) force f does not depend on time, the material time derivative can be neglected. Then reverting to the original variables, this yields the Stokes system: From now on, we assume that ˝ is a bounded, connected, open set in R3 , with a suitably piecewise   u C r p D f ; (11) smooth boundary @˝ (essentially, without cusps or multiple points). The relations (4) and (5) are the incompressible Navier-Stokes equations in ˝. They complemented with (5) and (6).

Stokes or Navier-Stokes Flows

1411

where ˇ > 0, only depends on ˝. This inequality is equivalent to the following “inf-sup condition”:

Some Theoretical Results Let us start with Stokes problems (11), (5), and (6). In view of both theory and numerics, it is useful to write it in variational form. Let H 1 .˝/ D fv 2 L2 .˝/ I r v 2 L2 .˝/3 g ; H01 .˝/

1

D fv 2 H .˝/ I v D 0 on @˝g;

L2ı .˝/ D fv 2 L2 .˝/ I .v; 1/ D 0g; where .; / denotes the scalar product of L2 .˝/: Z .f; g/ D

f g:

inf

sup

q2L2ı .˝/ v2H01 .˝/3

.div v; q/  ˇ: kr vkL2 .˝/ kqkL2 .˝/

(14)

Interestingly, (14) is not true when @˝ has an outward cusp, a situation that occurs, for instance, in a flow exterior to two colliding spheres. There is no simple proof of (14). Its difficulty lies in the no-slip boundary condition prescribed on v: The proof is much simpler when it is replaced by the weaker condition v  n D 0. The above isomorphism easily leads to the following result:

Theorem 1 For any f in H 1 .˝/3 and any  > 0, Problem (12) has exactly one solution and this solution Let H 1 .˝/ denote the dual space of H01 .˝/ and depends continuously on the data: h; i the duality pairing between them. The space H01 takes into account the no-slip boundary condition on 1 1 the velocity, and the space L2ı is introduced to lift kr ukL2 .˝/  kf kH 1 .˝/ ; kpkL2 .˝/  kf kH 1 .˝/ :  ˇ the undetermined constant in the pressure; note that (15) it is only defined by its gradient and hence up to one Next we consider the steady Navier-Stokes system. additive constant in a connected region. Assume that f belongs to H 1 .˝/3 . For our purpose, a suitable The natural extension of (12) is: Find a pair .u; p/ 2 variational form is: Find a pair .u; p/ 2 H01 .˝/3  H01 .˝/3  L2ı .˝/ solution of L2ı .˝/ solution of 8.v; q/ 2 H01 .˝/3  L2ı .˝/ ; 1 3 2 8.v; q/ 2H0 .˝/  Lı .˝/ ; .r u; r v/ C .u  r u; v/ .r u; r v/  .p; div v/.q; div u/ D hf ; vi:  .p; div v/  .q; div u/ D hf ; vi: (16) (12) ˝

Its analysis is fairly simple because on one hand the Albeit linear, this problem is difficult both from theononlinear convection term u  r u has the following retical and numerical standpoints. The pressure can be antisymmetry: eliminated from (12) by working with the space V of divergence-free velocities: 8u 2 V; 8v 2 H 1 .˝/3 ; .u  r u; v/ D .u  r v; u/; (17) V D fv 2 H 1 .˝/3 I div v D 0g; 0

and on the other hand, it belongs to L3=2 .˝/3 , which, roughly speaking, is significantly smoother than the data f in H 1 .˝/3 . This enables to prove existence of solutions, but uniqueness is only guaranteed for small force or large viscosity. More precisely, let

but the difficulty lies in recovering the pressure. Existence and continuity of the pressure stem from the following deep result: The divergence operator is an isomorphism from V ? onto L2ı .˝/, where V ? is the orthogonal of V in H01 .˝/3 . In other words, for every .w  r u; v/ q 2 L2ı .˝/, there exists one and only one v 2 V ? so: N D sup lution of div v D q. Moreover v depends continuously w;u;v2V;w;u;v¤0 kr wkL2 .˝/ kr ukL2 .˝/ kr vkL2 .˝/ on q: 1 kr vkL2 .˝/  kqkL2 .˝/ ; (13) Then we have the next result. ˇ

S

1412

Stokes or Navier-Stokes Flows

Theorem 2 For any f in H 1 .˝/3 and any  > 0, Discretization Problem (16) has at least one solution. A sufficient Solving numerically a steady Stokes system is costly condition for uniqueness is because the theoretical difficulty brought by the pressure is inherited both by its discretization, whatever N kf kH 1 .˝/ < 1: (18) the scheme, and by the computer implementation of 2 its scheme. This computational difficulty is aggraNow we turn to the time-dependent Navier-Stokes vated by the need of fine meshes for capturing comsystem. Its analysis is much more complex because plex flows produced by the Navier-Stokes system. In in R3 the dependence of the pressure on time holds comparison, when the flow is laminar, at reasonable in a weaker space. To simplify, we do not treat the Reynolds numbers, the additional cost of the nonlinear most general situation. For a given time interval Œ0; T , convection term is minor. There are some satisfacBanach space X , and number r  1, the relevant tory schemes and algorithms but so far no “miracle” spaces are of the form Lr .0; T I X /, which is the space method. of functions defined and measurable in 0; T Œ, such that Three important methods are used for discretizing flow problems: Finite-element, finite-difference, or Z T finite-volume methods. For the sake of simplicity, we kvkrX dt < 1; shall mainly consider discretization by finite-element 0 methods. Usually, they consist in using polynomial functions on cells: triangles or quadrilaterals in R2 or and tetrahedra or hexahedra in R3 . Most finite-difference schemes can be derived from finite-element methods dv W 1;r .0; T I X / D fv 2 Lr .0; T I X / I 2 Lr .0; T I X /g; on rectangles in R2 or rectangular boxes in R3 , coupled dt with quadrature formulas, in which case the mesh W01;r .0; T I X / D fv 2 W 1;r .0; T I X / Iv.0/ D v.T / D 0g; may not fit the boundary and a particular treatment may be required near the boundary. Finite volumes 0 with dual space W 1;r .0; T I X /, 1r C r10 D 1. There are closely related to finite differences but are more are several weak formulations expressing (4)–(7). For complex because they can be defined on very general numerical purposes, we shall use the following one: cells and do not involve functions. All three methods Find u 2 L2 .0; T I V / \ L1 .0; T I L2 .˝/3 /, with require meshing of the domain, and the success of du 3=2 .0; T I V 0 /, and p 2 W 1;1 .0; T I L2ı .˝// these methods depends not only on their accuracy but dt in L satisfying a.e. in 0; T Œ also on how well the mesh is adapted to the problem under consideration. For example, boundary layers may appear at large Reynolds numbers and require 8.v; q/ 2 H01 .˝/3  L2ı .˝/; locally refined meshes. Constructing a “good” mesh d .u.t/; v/C.r u.t/; r v/C.u.t/  r u.t/; v/ can be difficult and costly, but these important meshing dt issues are outside the scope of this work. Last, but not  .p.t/; div v/  .q; div u.t// D hf .t/; vi; least, in many practical applications where the Stokes (19) system is coupled with other equations, it is important that the scheme be locally mass conservative, i.e., the with the initial condition (7). This problem always has integral mean value of the velocity’s divergence be zero in each cell. at least one solution. Theorem 3 For any f in L2 .0; T I H 1 .˝/3 /, any  > 0, and any initial data u0 2 V , Problem (19), (7) Discretization of the Stokes Problem has at least one solution. Let Th be a triangulation of ˝ made of tetrahedra (also Unfortunately, unconditional uniqueness (which is called elements) in R3 , the discretization parameter h true in R2 ) is to this date an open problem in R3 . In being the maximum diameter of the elements. For apfact, it is one of the Millennium Prize Problems. proximation purposes, Th is not completely arbitrary:

Stokes or Navier-Stokes Flows

1413

it is assumed to be shape regular in the sense that its dihedral angles are uniformly bounded away from 0 and , and it has no hanging node in the sense that the intersection of two cells is either empty, or a vertex, or a complete edge, or a complete face. For a given integer k  0, let Pk denote the space of polynomials in three variables of total degree k and Qk that of degree k in each variable. The accuracy of a finite-element space depends on the degree of the polynomials used in each cell; however, we shall concentrate on low-degree elements, as these are most frequently used. Let us start with locally mass conservative methods and consider first conforming finite-element methods, i.e., where the finite-element space of discrete velocities, say Xh , is contained in H01 .˝/3 . Strictly speaking, the space of discrete pressures should be contained in L2ı .˝/. However, the zero mean-value constraint destroys the band structure of the matrix, and therefore, this constraint is prescribed weakly by means of a small, consistent perturbation. Thus the space of discrete pressures, say Qh , is simply a discrete subspace of L2 .˝/, and problem (12) is discretized by: Find .uh ; ph / 2 Xh  Qh solution of

in order to guarantee continuity at the interfaces of elements. We begin with constant pressures with one degree of freedom at the center of each tetrahedron. It can be checked that, except on some very particular meshes, a conforming P1 velocity space is not sufficiently rich to satisfy (21). This can be remedied by adding one degree of freedom (a vector in the normal direction) at the center of each face, and it is achieved by enriching the velocity space with one polynomial of P3 per face. This element, introduced by Bernardi and Raugel, is inf-sup stable and is of order one. It is frugal in number of degrees of freedom and is locally mass conservative but complex in its implementation because the velocity components are not independent. Of course, three polynomials of P3 (one per component) can be used on each face, and thus each component of the discrete velocity is the sum of a polynomial of P1 , which guarantees accuracy, and a polynomial of P3 , which guarantees inf-sup stability, but the element is more expensive. The idea of degrees of freedom on faces motivates a nonconforming method where Xh is contained in L2 .˝/3 and problem (12) is discretized by the following: Find .uh ; ph / 2 Xh  Qh solution of

8.vh ; qh / 2 Xh  Qh ; 8.vh ; qh / 2 Xh  Qh ; .r uh ; r vh /.ph ; div vh /  .qh ; div uh /  ".ph ; qh / D hf ; vh i;

 (20)

X T 2Th

.r uh ; r vh /T 

X

.ph ; div vh /T

T 2Th

X  .qh ; div uh /T  ".ph ; qh / D hf ; vh i: (22) where " > 0 is a small parameter. Let Mh D Qh \ 2 T 2Th Lı .˝/. Regardless of their individual accuracy, Xh and Mh cannot be chosen independently of each other because they must satisfy a uniform discrete analogue The inf-sup condition (21) is replaced by: of (14), namely, P T 2Th .div vh ; qh /T inf sup  ˇ  where .div vh ; qh / qh 2Mh vh 2Xh kr vh kh kqh kL2 .˝/  inf sup ˇ ; (21) qh 2Mh vh 2Xh kr vh kL2 .˝/ kqh kL2 .˝/ 1=2

X k  k2L2 .˝/ : (23) k  kh D for some real number ˇ  > 0, independent of h. T 2Th Elements that satisfy (21) are called inf-sup stable. For such elements, the accuracy of (20) depends directly on The simplest example, introduced by Crouzeix and the individual approximation properties of Xh and Qh . Raviart, is that of a constant pressure and each velocRoughly speaking, (21) holds when a discrete ve- ity’s component P1 per tetrahedron and each velocity’s locity space is sufficiently rich compared to a given component having one degree of freedom at the center discrete pressure space. Observe also that the discrete of each face. The functions of Xh must be continuous velocity’s degree in each cell must be at least one at the center of each interior face and vanish at the

S

1414

Stokes or Navier-Stokes Flows

center of each boundary face. Then it is not hard to prove that (23) is satisfied. Thus this element has order one, it is fairly economical and mass conservative, and its implementation is fairly straightforward. The above methods easily extend to hexahedral triangulations with Cartesian structure (i.e., eight hexahedra meeting at any interior vertex) provided the polynomial space Pk is replaced by the inverse image of Qk on the reference cube. Furthermore, such hexahedral triangulations offer more possibilities. For instance, a conforming, inf-sup stable, locally mass conservative scheme of order two can be obtained by taking, in each cell, a P1 pressure and each component of the velocity in Q2 . Now we turn to conforming methods that use continuous discrete pressures; thus the pressure must be at least P1 in each cell and continuous at the interfaces. Therefore the resulting schemes are not locally mass conservative. It can be checked that velocities with P1 components are not sufficiently rich. The simplest alternative, called “mini-element” or “P1 –bubble,” enriches each velocity component in each cell with a polynomial of P4 that vanishes on the cell’s boundary, whence the name bubble. This element is inf-sup stable and has order one. Its extension to order two, introduced in R2 by Hood and Taylor, associates with the same pressure, velocities with components in P2 . It is inf-sup stable and has order two. Discretization of the Navier-Stokes System Here we present straightforward discretizations of (19). The simplest one consists in using a linearized backward Euler finite-difference scheme in time. Let N > 1 be an integer, ı t D T =N the corresponding time step, and tn D nı t the discrete times. Starting from a finiteelement approximation or interpolation, say u0h of u0 satisfying the discrete divergence constraint of (20), we construct a sequence .unh ; phn / 2 Xh  Qh such that for 1  n  N: 8.vh ; qh / 2 Xh  Qh ; 1 n .u  uhn1 ; vh /C.r unh ; r vh /Cc.uhn1 I unh ; vh / ıt h  .phn ; div vh /  .qh ; div unh /  ".phn ; qh / D hf n ; vh i; (24)

convection term .w  r u; v/. As (17) does not necessarily extend to the discrete spaces, the preferred choice, from the standpoint of theory, is c.wh I uh ; vh / D

i 1h .wh  r uh ; vh /  .wh  r vh ; uh / ; 2 (25)

because it is both consistent and antisymmetric, which makes the analysis easier. But from the standpoint of numerics, the choice c.wh I uh ; vh / D .wh  r uh ; vh /

(26)

is simpler and seems to maintain the same accuracy. Observe that at each step n, (24) is a discrete Stokes system with two or three additional linear terms, according to the choice of form c. In both cases, the matrix of the system is not symmetric, which is a strong disadvantage. This can be remedied by completely time lagging the form c, i.e., replacing it by .uhn1  r uhn1 ; vh /. There are cases when none of the above linearizations are satisfactory, and the convection term is approximated by c.unh I unh ; vh / with c defined by (25) or (26). The resulting scheme is nonlinear and must be linearized, for instance, by an inner loop of Newton’s iterations. Recall Newton’s method for solving the equation f .x/ D 0 in R: Starting from an initial guess x0 , compute the sequence .xk / for k  0 by xkC1 D xk 

f .xk / : f 0 .xk /

Its generalization is straightforward and at step n, starting from uh;0 D uhn1 , the inner loop reads: Find .uh;kC1 ; ph;kC1 / 2 Xh  Qh solution of 8.vh ; qh /2Xh Qh ;

1 .uh;kC1 ; vh /C.r uh;kC1 ;r vh / ıt

C c.uh;kC1 I uh;k ; vh / C c.uh;k I uh;kC1 ; vh /  .ph;kC1 ; div vh /  .qh ; div uh;kC1 /  ".ph;kC1 ; qh / D c.uh;k I uh;k ; vh / C hf n ; vh i C

1 n1 .u ; vh /: ıt h (27)

Experience shows that only a few iterations are suffiwhere f is an approximation of f .tn ; / and cient to match the discretization error. Once this inner n n c.wh I uh ; vh / a suitable approximation of the loop converges, we set uh WD uh;kC1 , ph WD ph;kC1 . n

Stokes or Navier-Stokes Flows

1415

An interesting alternative to the above schemes is A U  B T P D F ; B U  C P D 0; (29) the characteristics method that uses a discretization of whose matrix is symmetric but indeed indefinite. Since the material time derivative (see (3)): A is nonsingular, a partial solution of (29) is  1 n d n1 n1   u.tn ; x/ ' u .x/  uh . .x// ; dt ıt h BA1 B T C C P D BA 1 F ; U D A 1 .F C B T P/: (30) where n1 .x/ gives the position at time tn1 of a particle located at x at time tn . Its first-order approximation As B has maximal rank, the Schur complement is BA 1 B T C C is symmetric positive definite, and n1 n1  .x/ D x  .ı t/uh .x/: an iterative gradient algorithm is a good candidate for Thus (24) is replaced by solving (30). Indeed, (30) is equivalent to minimizing with respect to Q the quadratic functional 8.vh ; qh / 2 Xh  Qh ;   1 1  1 n K.Q/ D AvQ ; vQ C C Q; Q ; with uh  uhn1 ı n1 ; vh C .r unh ; r vh / 2 2 ıt AvQ D F C B T Q: (31)  .phn ; div vh /  .qh ; div unh /  ".phn ; qh / D hf n ; vh i; (28) A variety of gradient algorithms for approximating the minimum are obtained by choosing a sequence whose matrix is symmetric and constant in time and of direction vectors W k and an initial vector P 0 and requires no linearization. On the other hand, computing computing a sequence of vectors P k defined for each the right-hand side is more complex. k  1 by:

Algorithms

P k D P k1  k1 W k1 ;

where

K.P k1  k1 W k1 / D inf K.P k1  W k1 /: 2R

In this section we assume that the discrete spaces are inf-sup stable, and to simplify, we restrict the discussion to conforming discretizations. Any discretization of the time-dependent Navier-Stokes system requires the solution of at least one Stokes problem per time step, whence the importance of an efficient Stokes solver. But since the matrix of the discrete Stokes system is large and indefinite, in R3 the system is rarely solved simultaneously for uh and ph . Instead the computation of ph is decoupled from that of uh .

(32) Usually the direction vectors W k are related to the gradient of K, whence the name of gradient algorithms. It can be shown that each step of these gradient algorithms requires the solution of a linear system with matrix A, which is equivalent to solving a Laplace equation per step. This explains why solving the Stokes system is expensive. The above strategy can be applied to (28) but not to (24) because its matrix A is no longer symmetric. In this case, a GMRES algorithm can be used, but this algorithm is expensive. For this reason, linearization by fully time lagging c may be preferable because the matrix A becomes symmetric. Of course, when Newton’s iterations are performed, as in (27), this option is not available because A is not symmetric. In this case, a splitting strategy may be useful.

Decoupling the Pressure and Velocity Let U be the vector of velocity unknowns, P that of pressure unknowns, and F the vector of data represented by .f ; vh /. Let A be the (symmetric positive definite) matrix of the discrete Laplace operator represented by .r uh ; r vh /, B the matrix of the discrete divergence operator represented by .qh ; div uh /, and C the matrix of the operator represented by ".ph ; qh /. Splitting Algorithms Owing to (21), the matrix B has maximal rank. With There is a wide variety of algorithms for splitting the this notation, (20) has the form: nonlinearity from the divergence constraint. Here is an

S

1416

Stokes or Navier-Stokes Flows

example where the divergence condition is enforced and therefore can be preconditioned by a Laplace once every other step. At step n, operator, while the second step is an implicit linearized 1. Knowing .uhn1 ; phn1 / 2 Xh  Qh , compute an in- system without constraint. termediate velocity .wnh ; phn / 2 Xh  Qh solution of

Numerical Experiments 8.vh ; qh / 2 Xh  Qh ; We present here two numerical experiments of benchmarks programmed with the software FreeFem++. More details including scripts and plots can be found  ".phn ; qh / D hf n ; vh i  .r uhn1 ; r vh / online at http://www.ljll.math.upmc.fr/ hecht/ftp/  c.uhn1 I un1 (33) ECM-2013. h ; v h /:

1 n .w  uhn1 ; vh /  .phn ; div vh /.qh ; div wnh / ıt h

2. Compute unh 2 Xh solution of 8vh 2 Xh ;

1 n .u  uhn1 ; vh / C .r unh ; r vh / ıt h

The Driven Cavity in a Square We use the Taylor-Hood P2  P1 scheme to solve the steady Navier-Stokes equations in the square cavity ˝ D0; 1Œ0; 1Œ with upper boundary 1 D0; 1Œf1g:

C c.wnh I unh ; vh / D hf n ; vh i C .phn ; div vh /: (34) The first step is fairly easy because it reduces to a “Laplace” operator with unknown boundary conditions



1 uC u r uC r p D 0 ; Re uj1 D .1; 0/ ;

div u D 0;

uj@˝n1 D .0; 0/;

Stokes or Navier-Stokes Flows, Fig. 1 From left to right: pressure at Re 9;000, adapted mesh at Re 8;000, stream function at Re 9;000. Observe the cascade of corner eddies

Stokes or Navier-Stokes Flows, Fig. 2 Von K´arm´an’s vortex street

Stokes or Navier-Stokes Flows

1417

with different values of Re ranging from 1 to 9;000. The discontinuity of the boundary values at the two upper corners of the cavity produces a singularity of the pressure. The nonlinearity is solved by Newton’s method, the initial guess being obtained by continuation on the Reynolds number, i.e., from the solution computed with the previous Reynolds number. The address of the script is cavityNewton.edp (Fig. 1).

´ an’s ´ Flow Past an Obstacle: Von Karm Vortex Street in a Rectangle The Taylor-Hood P2  P1 scheme in space and characteristics method in time are used to solve the time-dependent Navier-Stokes equations in a rectangle 2:2  0:41 m with a circular hole of diameter 0:1 m and the located near the inlet. The density  D 1:0 Kg m3 2

kinematic viscosity  D 103 ms . All the relevant data are taken from the benchmark case 2D-2 that can be found online at http://www.mathematik.tu-dortmund. de/lsiii/cms/papers/SchaeferTurek1996.pdf. The address of the script is at http://www.ljll.math.upmc. fr/ hecht/ftp/ECM-2013 is NSCaraCyl-100-mpi.edp or NSCaraCyl-100-seq.edp and func-max.idp (Fig. 2).

Bibilographical Notes The bibliography on Stokes and Navier-Stokes equations, theory and approximation, is very extensive and we have only selected a few references. A mechanical derivation of the Navier-Stokes equations can be found in the book by L.D. Landau and E.M. Lifshitz: Fluid Mechanics, Second Edition, Vol. 6 (Course of Theoretical Physics), Pergamon Press, 1959. The reader can also refer to the book by C. Truesdell and K.R. Rajagopal: An Introduction to the Mechanics of Fluids, Modeling and Simulation in Science, Engineering and Technology, Birkhauser, Basel, 2000. A thorough theory and description of finite element methods can be found in the book by P.G. Ciarlet: Basic error estimates for elliptic problems - Finite Element Methods, Part 1, in Handbook of Numerical Analysis, II, P.G. Ciarlet and J.L. Lions, eds., NorthHolland, Amsterdam, 1991.

The reader can also refer to the book by T. Oden and J.N. Reddy: An introduction to the mathematical theory of finite elements, Wiley, New-York, 1976. More computational aspects can be found in the book by A. Ern and J.L. Guermond: Theory and Practice of Finite Elements, AMS 159, Springer-Verlag, Berlin, 2004. The reader will find an introduction to the theory and approximation of the Stokes and steady NavierStokes equations, including a thorough discussion on the inf-sup condition, in the book by V. Girault and P.A. Raviart: Finite Element Methods for Navier-Stokes Equations. Theory and Algorithms, SCM 5, SpringerVerlag, Berlin, 1986. An introduction to the theory and approximation of the time-dependent Navier-Stokes problem is treated in the Lecture Notes by V. Girault and P.A. Raviart: Finite Element Approximation of the Navier-Stokes Equations, Lect. Notes in Math. 749, Springer-Verlag, Berlin, 1979. Non-conforming finite elements can be found in the reference by M. Crouzeix and P.A. Raviart: Conforming and non-conforming finite element methods for solving the stationary Stokes problem, RAIRO Anal. Num´er. 8 (1973), pp. 33–76. We also refer to the book by R. Temam: Navier-Stokes Equations, Theory and Numerical Analysis, North-Holland, Amsterdam, 1979. The famous Millennium Prize Problem is described at the URL: http://www.claymath.org/millennium/Navier-Stokes Equations. The reader will find a wide range of numerical methods for fluids in the book by O. Pironneau: Finite Element Methods for Fluids, Wiley, 1989. See also http://www.ljll.math.upmc.fr/ pironneau. We also refer to the course by R. Rannacher available online at the URL: http://numerik.iwr.uni-heidelberg.de/OberwolfachSeminar/CFD-Course.pdf. The book by R. Glowinski proposes a vey extensive collection of numerical methods, algorithms, and experiments for Stokes and Navier-Stokes equations: Finite Element Methods for Incompressible Viscous Flow, in Handbook of numerical analysis, IX, P.G. Ciarlet and J.L .Lions, eds., North-Holland, Amsterdam, 2003.

S

1418

Stratosphere and Its Coupling to the Troposphere and Beyond Edwin P. Gerber Center for Atmosphere Ocean Science, Courant Institute of Mathematical Sciences, New York University, New York, NY, USA

Synonyms Middle atmosphere

Glossary Mesosphere an atmospheric layer between approximately 50 and 100 km height Middle atmosphere a region of the atmosphere including the stratosphere and mesosphere Stratosphere an atmospheric layer between approximate 12 and 50 km Stratospheric polar vortex a strong, circumpolar jet that forms in the extratropical stratosphere during the winter season in each respective hemisphere Sudden stratospheric warming a rapid break down of the stratospheric polar vortex, accompanied by a sharp warming of the polar stratosphere Tropopause boundary between the troposphere and stratosphere, generally between 10 to 18 km. Troposphere lowermost layer of the atmosphere, extending from the surface to between 10 and 18 km. Climate engineering the deliberate modification of the Earth’s climate system, primarily aimed at reducing the impact of global warming caused by anthropogenic greenhouse gas emissions Geoengineering see climate engineering Quasi-biennial oscillation an oscillating pattern of easterly and westerly jets which propagates downward in the tropical stratosphere with a slightly varying period around 28 months Solar radiation management a form of climate engineering where the net incoming solar radiation to the surface is reduced to offset warming caused by greenhouse gases

Stratosphere and Its Coupling to the Troposphere and Beyond

Definition As illustrated in Fig. 1, the Earth’s atmosphere can be separated into distinct regions, or “spheres,” based on its vertical temperature structure. In the lowermost part of the atmosphere, the troposphere, the temperature declines steeply with height at an average rate of approximately 7 ı C per kilometer. At a distinct level, generally between 10–12 km in the extratropics and 16–18 km in the tropics (The separation between these regimes is rather abrupt and can be used as a dynamical indicator delineating the tropics and extratropics.) this steep descent of temperature abruptly shallows, transitioning to a layer of the atmosphere where temperature is initially constant with height, and then begins to rise. This abrupt change in the vertical temperature gradient, denoted the tropopause, marks the lower boundary of the stratosphere, which extends to approximately 50 km in height, at which point the temperature begins to fall with height again. The region above is denoted the mesosphere, extending to a second temperature minimum between 85 and 100 km. Together, the stratosphere and mesosphere constitute the middle atmosphere. The stratosphere was discovered at the dawn of the twentieth century. The vertical temperature gradient, or lapse rate, of the troposphere was established in the eighteenth century from temperature and pressure measurements taken on alpine treks, leading to speculation that the air temperature would approach absolute zero somewhere between 30 and 40 km: presumably the top of the atmosphere. Daring hot air balloon ascents in the late nineteenth century provided hints at a shallowing of the lapse rate – early evidence of the tropopause – but also led to the deaths of aspiring upper-atmosphere meteorologists. Teisserenc de Bort [15] and Assmann [2], working outside of Paris and Berlin, respectively, pioneered the first systematic, unmanned balloon observations of the upper atmosphere, establishing the distinct changes in the temperature structure that mark the stratosphere.

Overview The lapse rate of the atmosphere reflects the stability of the atmosphere to vertical motion. In the troposphere, the steep decline in temperature reflects near-neutral stability to moist convection. This is the turbulent

Stratosphere and Its Coupling to the Troposphere and Beyond 10−5

120 THERMOSPHERE

10−4 100

mesopause 10−3

MESOSPHERE 10−1

10−0

60

height (km)

pressure (hPa)

80 10−2

stratopause 40

101 STRATOSPHERE 102

103

20

tropopause TROPOSPHERE −80 −60 −40 −20

0

20

0

temperature (°C)

Stratosphere and Its Coupling to the Troposphere and Beyond, Fig. 1 The vertical temperature structure of the atmosphere. This sample profile shows the January zonal mean ˚ N from the Committee on Space Research temperature at 40A (COSPAR) International Reference Atmosphere 1986 model (CIRA-86). The changes in temperature gradients and hence stratification of the atmosphere reflect a difference in the dynamical and radiative processes active in each layer. The heights of the separation points (tropopause, stratopause, and mesopause) vary with latitude and season – and even on daily time scales due to dynamical variability – but are generally sharply defined in any given temperature profile

weather layer of the atmosphere where air is in close contact with the surface, with a turnover time scale on the order of days. In the stratosphere, the near-zero or positive lapse rates strongly stratify the flow. Here the air is comparatively isolated from the surface of the Earth, with a typical turnover time scale on the order of a year or more. This distinction in stratification and resulting impact on the circulation are reflected in the nomenclature: the “troposphere” and “stratosphere” were coined by Teisserenc de Bort, the former the “sphere of change” from the Greek tropos, to turn or whirl while the latter the “sphere of layers” from the Latin stratus, to spread out.

1419

In this sense, the troposphere can be thought of as a boundary layer in the atmosphere that is well connected to the surface. This said, it is important to note that the mass of the atmosphere is proportional to the pressure: the tropospheric “boundary” layer constitutes roughly 85 % of the mass of the atmosphere and contains all of our weather. The stratosphere contains the vast majority of the remaining atmospheric mass and the mesosphere and layers above just 0.1 %. Why Is There a Stratosphere? The existence of the stratosphere depends on the radiative forcing of the atmosphere by the Sun. As the atmosphere is largely transparent to incoming solar radiation, the bulk of the energy is absorbed at the surface. The presence of greenhouse gases, which absorb infrared light, allows the atmosphere to interact with radiation emitted by the surface. If the atmosphere were “fixed,” and so unable to convect (described as a radiative equilibrium), this would lead to an unstable situation, where the air near the surface is much warmer – and so more buoyant – than that above it. At height, however, temperature eventually becomes isothermal, given a fairly uniform distribution of the infrared absorber throughout the atmosphere (The simplest model for this is the so-called gray radiation scheme, where one assumes that all solar radiation is absorbed at the surface and a single infrared band from the Earth interacts with a uniformly distributed greenhouse gas.). If we allow the atmosphere to turn over in the vertical or convect in the nomenclature of atmospheric science, the circulation will produce to a well-mixed layer at the bottom with near-neutral stability: the troposphere. The energy available to the air at the surface is finite, however, only allowing it to penetrate so high into the atmosphere. Above the convection will sit the stratified isothermal layer that is closer to the radiative equilibrium: the stratosphere. This simplified view of a radiative-convective equilibrium obscures the role of dynamics in setting the stratification in both the troposphere and stratosphere but conveys the essential distinction between the layers. In this respect, “stratospheres” are found on other planets as well, marking the region where the atmosphere becomes more isolated from the surface. The increase in temperature seen in the Earth’s stratosphere (as seen in Fig. 1) is due to the fact that

S

1420

Stratosphere and Its Coupling to the Troposphere and Beyond

the atmosphere does interact with the incoming solar radiation through ozone. Ozone is produced by the interaction between molecular oxygen and ultraviolet radiation in the stratosphere [4] and takes over as the dominant absorber of radiation in this band. The decrease in density with height leads to an optimal level for net ultraviolet warming and hence the temperature maximum near the stratopause, which provides the demarcation for the mesosphere above. Absorption of ultraviolet radiation by stratospheric ozone protects the surface from high-energy radiation. The destruction of ozone over Antarctica by halogenated compounds has had significant health impacts, in addition to damaging all life in the biosphere. As described below, it has also had significant impacts on the tropospheric circulation in the Southern Hemisphere.

species, such as ozone. This exchange is critical for understanding the atmospheric chemistry in both the troposphere and stratosphere and has significant implications for tropospheric air quality, but will not be discussed. For further information, please see two review articles, [8] and [11]. The primary entry point for air into the stratosphere is through the tropics, where the boundary between the troposphere and stratosphere is less well defined. This region is known as the tropical tropopause layer and a review by [6] will provide the reader an introduction to research on this topic.

Compositional Differences The separation in the turnover time scale between the troposphere and stratosphere leads to distinct chemical or compositional properties of air in these two regions. Indeed, given a sample of air randomly taken from some point in the atmosphere, one can easily tell whether it came from the troposphere or the stratosphere. The troposphere is rich in water vapor and reactive organic molecules, such as carbon monoxide, which are generated by the biosphere and anthropogenic activity. Stratospheric air is extremely dry, with an average water vapor concentration of approximate 3–5 parts per billion, and comparatively rich in ozone. Ozone is a highly reactive molecule (causing lung damage when it is formed in smog at the surface) and does not exist for long in the troposphere. Scope and Limitations of this Entry Stratospheric research, albeit only a small part of the Earth system science, is a fairly mature field covering a wide range of topics. The remaining goal of this brief entry is to highlight the dynamical interaction between the stratosphere and the troposphere, with particular emphasis on the impact of the stratosphere on surface climate. In the interest of brevity, references have been kept to a minimum, focusing primarily on seminal historical papers and reviews. More detailed references can be found in the review articles listed in further readings. The stratosphere also interacts with the troposphere through the exchange of mass and trace chemical

Dynamical Coupling Between the Stratosphere and Troposphere The term “coupling” suggests interactions between independent components and so begs the question as to whether the convenient separation of the atmosphere into layers is merited in the first place. The key dynamical distinction between the troposphere and stratosphere lies in the differences in their stratification and the fact that moist processes (i.e., moist convection and latent heat transport) are restricted to the troposphere. The separation between the layers is partly historical, however, evolving in response to the development of weather forecasting and the availability of computational resources. Midlatitude weather systems are associated with large-scale Rossby waves, which owe their existence to gradients in the effective rotation, or vertical component of vorticity, of the atmosphere due to variations in the angle between the surface plain and the axis of rotation with latitude. Pioneering work by [5] and [10] showed that the dominant energy containing waves in the troposphere, wavenumber roughly 4–8, the so-called synoptic scales, cannot effectively propagate into the stratosphere due the presence of easterly winds in the summer hemisphere and strong westerly winds in the winter hemisphere. For the purposes of weather prediction, then, the stratosphere could largely be viewed as an upper-boundary condition. Models thus resolved the stratosphere as parsimoniously as possible in order to focus numerical resources on the troposphere. The strong winds in the winter stratosphere also impose a stricter Courant-Friedrichs-Lewy condition on the time step of the model, although more advanced numerical techniques have alleviated this problem.

Stratosphere and Its Coupling to the Troposphere and Beyond

1421

Despite the dynamical separation for weather system-scale waves, larger-scale Rossby waves (wavenumber 1–3, referred to as planetary scales) can penetrate into the winter stratosphere, allowing for momentum exchange between the layers. In addition, smaller-scale (on the order of 10–1,000 km) gravity waves (Gravity waves are generated in stratified fluids, where the restoring force is the gravitational acceleration of fluid parcels or buoyancy. They are completely distinct from relativistic gravity waves.) also transport momentum between the layers. As computation power increased, leading to a more accurate representation of tropospheric dynamics, it became increasingly clear that a better representation of the stratosphere was necessary to fully understand and simulate surface weather and climate.

is conversely associated with a poleward shift in the jet stream, although the onset of cold vortex events is not as abrupt. More significantly, the changes in the troposphere extend for up to 2–3 months on the slow time scale of the stratospheric recovery, while under normal conditions the chaotic nature of tropospheric flow restricts the time scale of jet variations to approximately 10 days. The associated changes in the stratospheric jet stream and tropospheric jet shift are conveniently described by the Northern Annular Mode (The NAM is also known as the Arctic Oscillation, although the annular mode nomenclature has become more prominent.) (NAM) pattern of variability. The mechanism behind this interaction is still an active area of research. It has become clear, however, that key lies in the fact that the lower stratosphere influences the formation and dissipation of synopticscale Rossby waves, despite the fact that these waves do not penetrate far into the stratosphere. A shift in the jet stream is associated with a largescale rearrangement of tropospheric weather patterns. In the Northern Hemisphere, where the stratosphere is more variable due to the stronger planetary wave activity (in short, because there are more continents), an equatorward shift in the jet stream following an SSW leads to colder, stormier weather over much of northern Europe and eastern North America. Forecast skill of temperature, precipitation, and wind anomalies at the surface increases in seasonal forecasts following an SSW. SSWs can be further differentiated into “vortex displacements” and “vortex splits,” depending on the dominant wavenumber (1 or 2, respectively) involved in the breakdown of the jet, and recent work has suggested this has an effect on the tropospheric impact of the warming. SSWs occur approximately every other year in the Northern Hemisphere, although there is strong intermittency: few events were observed in the 1990s, while they have been occurring in most years in the first decades of the twenty-first century. In the Southern Hemisphere, the winter westerlies are stronger and less variable – only one SSW has ever been observed, in 2002 – but predictability may be gained around the time of the “final warming,” when the stratosphere transitions to it’s summer state with easterly winds. Some years, this transition is accelerated by planetary wave dynamics, as in an SSW, while in other years it is gradual, associated with a slow radiative relaxation to the summer state.

Coupling on Daily to Intraseasonal Time Scales Weather prediction centers have found that the increased representation of the stratosphere improves tropospheric forecasts. On short time scales, however, much of the gain comes from improvements to the tropospheric initial condition. This stems from better assimilation of satellite temperature measurements which project onto both the troposphere and stratosphere. The stratosphere itself has a more prominent impact on intraseasonal time scales, due to the intrinsically longer time scales of variability in this region of the atmosphere. The gain in predictability, however, is conditional, depending on the state of the stratosphere. Under normal conditions, the winter stratosphere is very cold in the polar regions, associated with a strong westerly jet, or stratospheric polar vortex. As first observed in the 1950s [12], this strong vortex is sometimes disturbed by the planetary wave activity propagating below, leading to massive changes in temperature (up to 70 ı C in a matter of days) and a reversal of the westerly jet, a phenomenon known as a sudden stratospheric warming, or SSW. While the predictability of SSWs are limited by the chaotic nature of tropospheric dynamics, after an SSW the stratosphere remains in an altered state for up to 2–3 months as the polar vortex slowly recovers from the top down. Baldwin and Dunkerton [3] demonstrated the impact of these changes on the troposphere, showing that an abrupt warming of the stratosphere is followed by an equatorward shift in the tropospheric jet stream and associated storm track. An abnormally cold stratosphere

S

1422

Stratosphere and Its Coupling to the Troposphere and Beyond

Coupling on Interannual Time Scales On longer time scales, the impact of the stratosphere is often felt through a modulation of the intraseasonal coupling between the stratospheric and tropospheric jet streams. Stratospheric dynamics play an important role in internal modes of variability to the atmosphereocean system, such as El Ni˜no and the Southern Oscillation (ENSO), and in the response of the climate system to “natural” forcing by the solar cycle and volcanic eruptions. The quasi-biennial oscillation (QBO) is a nearly periodic oscillation of downward propagating easterly and westerly tropical jets in the tropical stratosphere, with a period of approximately 28 months. It is perhaps the most long-lived mode of variability intrinsic to the atmosphere alone. The QBO influences the surface by modulating the wave coupling between the troposphere and stratosphere in the Northern Hemisphere winter, altering the frequency and intensity of SSWs depending on the phase of the oscillation. Isolating the impact of the QBO has been complicated by the possible overlap with the ENSO, a coupled mode of atmosphere-ocean variability with a time scale of approximately 3–7 years. The relatively short observational record makes it difficult to untangle the signals from measurements alone, and models have only recently been able to simulate these phenomenon with reasonable accuracy. ENSO is driven by interaction between the tropical Pacific Ocean and the zonal circulation of the tropical atmosphere (the Walker circulation). Its impact on the extratropical circulation in the Northern Hemisphere, however, is in part effected through its influence on the stratospheric polar vortex. A warm phase of ENSO is associated with stronger planetary wave propagation into the stratosphere, hence a weaker polar vortex and equatorward shift in the tropospheric jet stream. Further complicating the statistical separation between the impacts of ENSO and the QBO is the influence of the 11-year solar cycle, associated with changes in the number of sunspots. While the overall intensity of solar radiation varies less than 0.1 % of its mean value over the cycle, the variation is stronger in the ultraviolet part of the spectrum. Ultraviolet radiation is primarily absorbed by ozone in the stratosphere, and it has been suggested that the associated changes in temperature structure alter the planetary wave propagation, along the lines of the influence of ENSO and QBO.

The role of the stratosphere in the climate response to volcanic eruptions is comparatively better understood. While volcanic aerosols are washed out of the troposphere on fairly short time scales by the hydrological cycle, sulfate particles in the stratosphere can last for 1–2 years. These particles reflect the incoming solar radiation, leading to a global cooling of the surface; following Pinatubo, the global surface cooled to 0.1– 0.2 K. The overturning circulation of the stratosphere lifts mass up into the tropical stratosphere, transporting it poleward where it descends in the extratropics. Thus, only tropical eruptions have a persistent, global impact. Sulfate aerosols warm the stratosphere, therefore modifying the planetary wave coupling. There is some evidence that the net result is a strengthening of the polar vortex which in turn drives a poleward shift in the tropospheric jets. Hence, eastern North America and northern Europe may experience warmer winters following eruptions, despite the overall cooling impact of the volcano. The Stratosphere and Climate Change Anthropogenic forcing has changed the stratosphere, with resulting impacts on the surface. While greenhouse gases warm the troposphere, they increase the radiative efficiency of the stratosphere, leading to a net cooling in this part of the atmosphere. The combination of a warming troposphere and cooling stratosphere leads to a rise in the tropopause and may be one of the most identifiable signatures of global warming on the atmospheric circulation. While greenhouse gases will have a dominant longterm impact on the climate system, anthropogenic emissions of halogenated compounds, such as chlorofluorocarbons (CFCs), have had the strongest impact on the stratosphere in recent decades. Halogens have caused some destruction of ozone throughout the stratosphere, but the extremely cold temperatures of the Antarctic stratosphere in winter permit the formation of polar stratospheric clouds, which greatly accelerate the production of Cl and Br atoms that catalyze ozone destruction (e.g., [14]). This led to the ozone hole, the effective destruction of all ozone throughout the middle and lower stratosphere over Antarctica. The effect of ozone loss on ultraviolet radiation was quickly appreciated, and the use of halogenated compounds regulated and phased out under the Montreal Protocol (which came into force in 1989) and subsequent agreements. Chemistry climate models suggest that

Stratosphere and Its Coupling to the Troposphere and Beyond

1423

the ozone hole should recover by the end of this century, assuming the ban on halogenated compounds is observed. It was not appreciated until the first decade of the twenty-first century, however, that the ozone hole also has impacted the circulation of the Southern Hemisphere. The loss of ozone leads to a cooling of the austral polar vortex in springtime and a subsequent poleward shift in the tropospheric jet stream. Note that this poleward shift in the tropospheric jet in response to a stronger stratospheric vortex mirrors the coupling associated with natural variability in the Northern Hemisphere. As reviewed by [16], this shift in the jet stream has had significant impacts on precipitation across much of the Southern Hemisphere. Stratospheric trends in water vapor also have the potential to affect the surface climate. Despite the minuscule concentration of water vapor in the stratosphere (just 3–5 parts per billion), the radiative impact of a greenhouse gases scales logarithmically, so relatively large changes in small concentrations can have a strong impact. Decadal variations in stratospheric water vapor can have an influence on surface climate comparable to decadal changes in greenhouse gas forcing, and there is evidence of a positive feedback of stratospheric water vapor on greenhouse gas forcing. The stratosphere has also been featured prominently in the discussion of climate engineering (or geoengineering), the deliberate alteration of the Earth system to offset the consequences of greenhouse-induced warming. Inspired by the natural cooling impact of volcanic aerosols, the idea is to inject hydrogen sulfide or sulfur dioxide into the stratosphere, where it will form sulfate aerosols. To date, this strategy of the so-called solar radiation management appears to be among the most feasible and cost-effective means of cooling the Earth’s surface, but it comes with many dangers. In particular, it does not alleviate ocean acidification, and the effect is short-lived – a maximum of two years – and so would require continual action ad infinitum or until greenhouse gas concentrations were returned to safer levels. (In saying this, it is important to note that the natural time scale for carbon dioxide removal is 100,000s of years, and there are no known strategies for accelerating CO2 removal that appear feasible, given current technology.) In addition, the impact of sulfate aerosols on stratospheric ozone and the potential regional effects due to changes in the

planetary wave coupling with the troposphere are not well understood.

Further Reading There are a number of review papers on stratospheretropospheric coupling in the literature. In particular, [13] provides a comprehensive discussion of stratosphere-troposphere coupling, while [7] highlights developments in the last decade. Andrews et al. [1] provide a classic text on the dynamics of the stratosphere, and [9] provides a wider perspective on the stratosphere, including the history of field.

References 1. Andrews, D.G., Holton, J.R., Leovy, C.B.: Middle Atmosphere Dynamics. Academic Press, Waltham, MA (1987) 2. Assmann, R.A.: u¨ ber die existenz eines w¨armeren Lufttromes in der H¨ohe von 10 bis 15 km. Sitzungsber K. Preuss. Akad. Wiss. 24, 495–504 (1902) 3. Baldwin, M.P., Dunkerton, T.J.: Stratospheric harbingers of anomalous weather regimes. Science 294, 581–584 (2001) 4. Chapman, S.: A theory of upper-atmosphere ozone. Mem. R. Meteor. Soc. 3, 103–125 (1930) 5. Charney, J.G., Drazin, P.G.: Propagation of planetary-scale disturbances from the lower into the upper atmosphere. J. Geophys. Res. 66, 83–109 (1961) 6. Fuegistaler, S., Dessler, A.E., Dunkerton, T.J., Folkins, I., Fu, Q., Mote, P.W.: Tropical tropopause layer. Rev. Geophys. 47, RG1004 (2009) 7. Gerber, E.P., Butler, A., Calvo, N., Charlton-Perez, A., Giorgetta, M., Manzini, E., Perlwitz, J., Polvani, L.M., Sassi, F., Scaife, A.A., Shaw, T.A., Son, S.W., Watanabe, S.: Assessing and understanding the impact of stratospheric dynamics and variability on the Earth system. Bull. Am. Meteor. Soc. 93, 845–859 (2012) 8. Holton, J.R., Haynes, P.H., McIntyre, M.E., Douglass, A.R., Rood, R.B., Pfister, L.: Stratosphere-troposphere exchange. Rev. Geophys. 33, 403–439 (1995) 9. Labitzke, K.G., Loon, H.V.: The Stratosphere: Phenomena, History, and Relevance. Springer, Berlin/New York (1999) 10. Matsuno, T.: Vertical propagation of stationary planetary waves in the winter Northern Hemisphere. J. Atmos. Sci. 27, 871–883 (1970) 11. Plumb, R.A.: Stratospheric transport. J. Meteor. Soc. Jpn. 80, 793–809 (2002) 12. Scherhag, R.: Die explosionsartige stratosph¨arenerwarmung des sp¨atwinters 1951/52. Ber Dtsch Wetterd. US Zone 6, 51–63 (1952)

S

1424 13. Shepherd, T.G.: Issues in stratosphere-troposphere coupling. J. Meteor. Soc. Jpn, 80, 769–792 (2002) 14. Solomon, S.: Stratospheric ozone depletion: a review of concepts and history. Rev. Geophys. 37(3), 275–316 (1999). doi:10.1029/1999RG900008 15. Teisserence de Bort, L.: Variations de la temperature d l’air libre dans la zone comprise entre 8 km et 13 km d’altitude. C. R. Acad. Sci. Paris 138, 42–45 (1902) 16. Thompson, D.W.J., Solomon, S., Kushner, P.J., England, M.H., Grise, K.M., Karoly, D.J.: Signatures of the Antarctic ozone hole in Southern Hemisphere surface climate change. Nature Geoscience 4, 741–749 (2011). doi:10.1038/NGEO1296

Structural Dynamics Roger Ohayon1 and Christian Soize2 1 Structural Mechanics and Coupled Systems Laboratory, LMSSC, Conservatoire National des Arts et M´etiers (CNAM), Paris, France 2 Laboratoire Mod´elisation et Simulation Multi-Echelle, MSME UMR 8208 CNRS, Universite Paris-Est, Marne-la-Vall´ee, France

Description The computational structural dynamics is devoted to the computation of the dynamical responses in time or in frequency domains of complex structures, submitted to prescribed excitations. The complex structure is constituted of a deformable medium constituted of metallic materials, heterogeneous composite materials, and more generally, of metamaterials. This chapter presents the linear dynamic analysis for complex structures, which is the most frequent case encountered in practice. For this situation, one of the most efficient modeling strategy is based on a formulation in the frequency domain (structural vibrations). There are many advantages to use a frequency domain formulation instead of a time domain formulation because the modeling can be adapted to the nature of the physical responses which are observed. This is the reason why the low-, the medium-, and the high-frequency ranges are introduced. The different types of vibration responses of a linear dissipative complex structure lead us to define the frequency ranges of analysis. Let uj .x; !/ be the Frequency Response Function (FRF) of a component j of the displacement u.x; !/, at a fixed

Structural Dynamics

point x of the structure and at a fixed circular frequency ! (in rad/s). Figure 1 represents the modulus juj .x; !/j in log scale and the unwrapped phase 'j .x; !/ of the FRF such that uj .x; !/ D juj .x; !/j expfi'j .x; !/g. The unwrapped phase is defined as a continuous function of ! obtained in adding multiples of ˙2 for jumps of the phase angle. The three frequency ranges can then be characterized as follows: 1. The low-frequency range (LF) is defined as the modal domain for which the modulus of the FRF exhibits isolated resonances due to a low modal density of elastic structural modes. The amplitudes of the resonances are driven by the damping and the phase rotates of  at the crossing of each isolated resonance (see Fig. 1). For the LF range, the strategy used consists in computing the elastic structural modes of the associated conservative dynamical system and then to construct a reduced-order model by the Ritz-Galerkin projection. The resulting matrix equation is solved in the time domain or in the frequency domain. It should be noted that substructuring techniques can also be introduced for complex structural systems. Those techniques consist in decomposing the structure into substructures and then in constructing a reduced-order model for each substructure for which the physical degrees of freedom on the coupling interfaces are kept. 2. The high-frequency range (HF) is defined as the range for which there is a high modal density which is constant on the considered frequency range. In this HF range the modulus of the FRF varies slowly as the function of the frequency and the phase is approximatively linear (see Fig. 1). Presently, this frequency range is relevant of various approaches such as Statistical Energy Analysis (SEA), diffusion of energy equation, and transport equation. However, due to the constant increase of computer power and advances in modeling of complex mechanical systems, this frequency domain becomes more and more accessible to the computational methods. 3. For complex structures (complex geometry, heterogeneous materials, complex junctions, complex boundary conditions, several attached equipments or mechanical subsystems, etc.), an intermediate frequency range, called the mediumfrequency range (MF), appears. This MF range does not exist for a simple structure (e.g., a simply supported homogeneous straight beam). This MF range is defined as the intermediate frequency range

1425

Unwrapped phase

FRF modulus (in log scale)

Structural Dynamics

LF

MF

HF Frequency (Hz)

LF

MF

HF Frequency (Hz)

Structural Dynamics, Fig. 1 Modulus (left) and unwrapped phase (right) of the FRF as a function of the frequency. Definition of the LF, MF, and HF ranges

for which the modal density exhibits large variations over the frequency band. Due to the presence of the damping which yields an overlapping of elastic structural modes, the frequency response functions do not exhibit isolated resonances, and the phase slowly varies as a function of the frequency (see Fig. 1). In this MF range, the responses are sensitive to damping modeling (for weakly dissipative structure), which is frequency dependent, and sensitive to uncertainties. For this MF range, the computational model is constructed as follows: The reduced-order computational model of the LF range can be used in (i) adapting the finite element discretization to the MF range, (ii) introducing appropriate damping models (due to dissipation in the structure and to transfer of mechanical energy from the structure to mechanical subsystems which are not taken into account in the computational model), and (iii) introducing uncertainty quantification for both the systemparameter uncertainties and the model uncertainties induced by the modeling errors. For sake of brevity, the case of nonlinear dynamical responses of structures (involving nonlinear constitutive equations, nonlinear geometrical effects, plays, etc.) is not considered in this chapter (see Bibliographical comments).

bounded connected domain of R3 , with a smooth boundary @& for which the external unit normal is denoted as n. The generic point of & is x D .x1 ; x2 ; x3 /. Let u.x; t/ D .u1 .x; t/; u2 .x; t/; u3 .x; t// be the displacement of a particle located at point x in & and at a time t. The structure is assumed to be free (0 D ;), a given surface force field G.x; t/ D .G1 .x; t/; G2 .x; t/; G3 .x; t// is applied to the total boundary  D @&, and a given body force field g.x; t/ D .g1 .x; t/; g2 .x; t/; g3 .x; t// is applied in &. It is assumed that these external forces are in equilibrium. Below, if w is any quantity depending on x, then w;j denotes the partial derivative of w with respect to xj . The classical convention for summations over repeated Latin indices is also used. The elastodynamic boundary value problem is written, in terms of u and at time t, as  @2t ui .x; t/  ij;j .x; t/ D gi .x; t/ ij .x; t/ nj .x/ D Gi .x; t/

in

&; (1)

on ;

(2)

ij;j .x;t/ D aij kh .x/ "kh .u/ C bij kh .x/"kh .@t u/; "kh .u/ D .uk;h C uh;k /=2:

(3)

In (1), .x/ is the mass density field, ij is the Cauchy stress tensor. The constitutive equation is defined by (3) exhibiting an elastic part defined by the tensor aij kh .x/ and a dissipative part defined by the tensor bij kh .x/, independent of t because the model is developed for the low-frequency range, and ".@t u/ is the linearized Formulation in the Low-Frequency Range for Complex Structures strain tensor. Let C D .H 1 .&//3 be the real Hilbert space We consider linear dynamics of a structure around a of the admissible displacement fields, x 7! v.x/, position of static equilibrium taken as the reference on &. Considering t as a parameter, the variational configuration, &, which is a three-dimensional formulation of the boundary value problem defined by

S

1426

Structural Dynamics

(1)–(3) consists, for fixed t, in finding u.:; t/ in C, such The finite element discretization with n degrees of freedom of (4) yields the following second-order difthat ferential equation on Rn : 8v 2 C; R P ŒM  U.t/ C ŒD U.t/ C ŒK U.t/ D F.t/; (8) (4) in which the bilinear form m is symmetric and positive and its Fourier transform, which corresponds to the definite, the bilinear forms d and k are symmetric, finite element discretization of (7), yields the complex positive semi-definite, and are such that matrix equation which is written as Z (9) .! 2 ŒM  C i ! ŒD C ŒK/ U.!/ D F.!/;  uj vj d x; m.u; v/ D m.@2t u; v/ C d.@t u; v/ C k.u; v/ D f .tI v/;

Z

&

in which ŒM  is the mass matrix which is a symmetric (5) positive definite .n  n/ real matrix and where ŒD and & Z ŒK are the damping and stiffness matrices which are bij kh "kh .u/ "ij .v/d x; d.u; v/ D symmetric positive semi-definite .n  n/ real matrices. & Case of a fixed structure. If the structure is fixed on a Z Z f .tI v/ D gj .t/ vj d x C Gj .t/ vj ds.x/: (6) part 0 of boundary @& (Dirichlet condition u D 0 on 0 ), then the given surface force field G.x; t/ is applied &  to the part  D @&n0 . The space C of the admissible The kernel of the bilinear forms k and d is the set of displacement fields must be replaced by the rigid body displacements, Crig  C of dimension 6. (10) C0 D fv 2 C; v D 0 on 0 g: Any displacement field urig in Crig is such that, for all x in &, urig .x/ D t C   x in which t and  are two The complex vector space C c must be replaced by the arbitrary constant vectors in R3 . c For the evolution problem with given Cauchy initial complex vector space C0 which is the complexified conditions u.:; 0/ D u0 and @t u.:; 0/ D v0 , the analysis vector space of C0 . The real matrices ŒD and ŒK are of the existence and uniqueness of a solution requires positive definite. the introduction of the following hypotheses:  is a positive bounded function on &; for all x in &, the fourth-order tensor aij kh .x/ (resp. bij kh .x/) is symmet- Associated Spectral Problem and ric, aij kh .x/ D aj i kh .x/ D aij hk .x/ D akhij .x/, and Structural Modes such that, for all second-order real symmetric tensor 2 ij , there is a positive constant c independent of x, Setting  D ! , the spectral problem, associated with such that aij kh .x/ kh ij  c ij ij I the functions the variational formulation defined by (4) or (7), is aij kh and bij kh are bounded on &; finally, g and G are stated as the following generalized eigenvalue probsuch that the linear form v 7! f .tI v/ is continuous lem. Find real   0 and u 6D 0 in C such that on C. Assuming that for all v in C, t 7! f .tI v/ k.u; v/ D  m.u; v/; 8v 2 C: (11) is a square integrable function on R. Let C c be the complexified vector space of C and let v be the complex conjugate of v. Then, introducing the Fourier trans- Rigid body modes (solutions for  D 0). Since the R i !t forms u.x; !/ D e u.x; t/ dt and f .!I v/ D dimension of Crig is 6, then  D 0 can be considR R i !t e f .tI v/ dt, the variational formulation defined ered as a “zero eigenvalue” of multiplicity 6, denoted R by (4) can be rewritten as follows: For all fixed real as 5 ; : : : ; 0 . Let u5 ; : : : ; u0 be the corresponding eigenfunctions which are constructed such that the ! 6D 0, find u.:; !/ with values in C c such that following orthogonality conditions are satisfied: for ˛ and ˇ in f5; : : : ; 0g, m.u˛ ; uˇ / D ˛ ı˛ˇ and  ! 2 m.u; v/ C i ! d.u; v/ C k.u; v/ k.u˛ ; uˇ / D 0. These eigenfunctions, called the rigid D f .!I v/; 8v 2 C c : (7) body modes, form a basis of Crig  C and any rigid k.u; v/ D

aij kh "kh .u/ "ij .v/d x;

Structural Dynamics

1427

body displacement urig in Crig can then be expanded as P urig D 0˛D5 q˛ u˛ . Elastic structural modes (solutions for  6D 0). We introduce the subset Celas D C n Crig . It can be shown that C D Crig ˚Celas which means that any displacement field u in C has the following unique decomposition u D urig C uelas with urig in Crig and uelas in Celas . Consequently, k.uelas ; velas / defined on Celas  Celas is positive definite and we then have k.uelas ; uelas / > 0 for all uelas 6D 0 2 Celas .

Z

Z u˛ .x/ .x/ d x D 0; &

x  u˛ .x/ .x/ d x D 0; &

(14)

which shows that the inertial center of the structure deformed under the elastic structural mode u˛ , coincides with the inertial center of the undeformed structure. Expansion of the displacement field using the rigid body modes and the elastic structural modes. Any displacement field u in C can then be written as u D P0 PC1 ˛D5 q˛ u˛ C ˛D1 q˛ u˛ .

Eigenvalue problem restricted to Celas . The eigenvalue problem restricted to Celas is written as follows: Find Terminology. In structural vibrations, !˛ > 0 is called the eigenfrequency of elastic structural mode u˛ (or  6D 0 and uelas 6D 0 in Celas such that the eigenmode or mode shape of vibration) whose normalization is defined by the generalized mass ˛ . k.uelas ; velas / D  m.uelas ; velas /; 8velas 2 Celas : (12) An elastic structural mode ˛ is defined by the three quantities f!˛ ; u˛ ; ˛ g. Countable number of positive eigenvalues. It can be proven that the eigenvalue problem, defined by (12), admits an increasing sequence of positive eigenvalues 0 < 1  2  : : :  ˛  : : :. In addition, any multiple positive eigenvalue has a finite multiplicity (which means that a multiple positive eigenvalue is repeated a finite number of times). Orthogonality of the eigenfunctions corresponding to the positive eigenvalues. The sequence of eigenfunctions fu˛ g˛ in Celas corresponding to the positive eigenvalues satisfies the following orthogonality conditions: m.u˛ ; uˇ / D ˛ ı˛ˇ ;

k.u˛ ; uˇ / D ˛ !˛2 ı˛ˇ ; (13)

p in which !˛ D ˛ and where ˛ is a positive real number depending on the normalization of eigenfunction u˛ . Completeness of the eigenfunctions corresponding to the positive eigenvalues. Let u˛ be the eigenfunction associated with eigenvalue ˛ > 0. It can be shown that eigenfunctions fu˛ g˛1 form a complete family in Celas and consequently, an arbitrary function uelas belonging PC1 to Celas can be expanded as uelas D ˛D1 q˛ u˛ in which fq˛ g˛ is a sequence of real numbers. These eigenfunctions are called the elastic structural modes.

Finite element discretization. The matrix equation of the generalized symmetric eigenvalue problem corresponding to the finite element discretization of (11) is written as ΠK  U D  ΠM  U: (15) For large computational model, this generalized eigenvalue problem is solved using iteration algorithms such as the Krylov sequence, the Lanczos method, and the subspace iteration method, which allows a prescribed number of eigenvalues and associated eigenvectors to be computed. Case of a fixed structure. If the structure is fixed on 0 , Crig is reduced to the empty set and admissible space Celas must be replaced by C0 . In this case the eigenvalues are strictly positive. In addition, the property given for the free structure concerning the inertial center of the structure does not hold.

Reduced-Order Computational Model in the Frequency Domain

In the frequency range, the reduced-order computational model is carried out using the Ritz-Galerkin projection. Let CS be the admissible function space Orthogonality between the elastic structural modes such that CS D Celas for a free structure and CS D C0 and the rigid body modes. We have k.u˛ ; urig / D 0 for a structure fixed on 0 . Let CS;N be the subspace and m.u˛ ; urig / D 0. Substituting urig .x/ D t C   x of CS , of dimension N  1, spanned by the finite family fu1 ; : : : ; uN g of elastic structural modes u˛ . into (13) yields

S

1428

Structural Dynamics

For all fixed !, the projection uN .!/ of u.!/ on the Bibliographical Comments complexified vector space of CS;N can be written as The mathematical aspects related to the variational formulation, existence and uniqueness, and finite elN X N q˛ .!/ u˛ .x/; (16) ement discretization of boundary value problems for u .x; !/ D elastodynamics can be found in Dautray and Lions [6], ˛D1 Oden and Reddy [11], and Hughes [9]. More details in which q.!/ D .q1 .!/; : : : ; qN .!// is the complex- concerning the finite element method can be found valued vector of the generalized coordinates which in Zienkiewicz and Taylor [14]. Concerning the time integration algorithms in nonlinear computational dyverifies the matrix equation on CN , namics, the readers are referred to Belytschko et al. [3], and Har and Tamma [8]. General mechanical formula2 .! ŒM  C i ! ŒD C Œ K / q.!/ D F .!/; (17) tions in computational structural dynamics, vibration, eigenvalue algorithms, and substructuring techniques in which ŒM , ŒD, and Œ K  are .N  N / real sym- can be found in Argyris and Mlejnek [1], Geradin metric positive definite matrices (for a free or a fixed and Rixen [7], Bathe and Wilson [2], and Craig and structure). Matrices ŒM and Œ K  are diagonal and Bampton [5]. For computational structural dynamics such that in the low- and the medium-frequency ranges and extensions to structural acoustics, we refer the reader ŒM ˛ˇ D m.uˇ ; u˛ / D ˛ ı˛ˇ ; to Ohayon and Soize [12]. Various formulations for the high-frequency range can be found in Lyon and Dejong Œ K ˛ˇ D k.uˇ ; u˛ / D ˛ !˛2 ı˛ˇ : (18) [10] for Statistical Energy Analysis, and in Chap. 4 of Bensoussan et al. [4] for diffusion of energy and transThe damping matrix ŒD is not sparse (fully populated) port equations. Concerning uncertainty quantification and the component F˛ of the complex-valued vector of (UQ) in computational structural dynamics, we refer the generalized forces F D .F1 ; : : : ; FN / are such that the reader to Soize [13]. ŒD˛ˇ D d.uˇ ; u˛ /;

F˛ .!/ D f .!I u˛ /:

(19)

The reduced-order model is defined by (16)–(19). Convergence of the solution constructed with the reduced-order model. For all real !, (17) has a unique solution uN .!/ which is convergent in CS when N goes to infinity. Quasi-static correction terms can be introduced to accelerate the convergence with respect to N . Remarks concerning the diagonalization of the damping operator. When damping operator is diagonalized by the elastic structural modes, matrix ŒD defined by (19), is an .N  N / diagonal matrix which can be written as ŒD˛ˇ D d.uˇ ; u˛ / D 2 ˛ !˛ ˛ ı˛ˇ in which ˛ and !˛ are defined by (18). The critical damping rate ˛ of elastic structural mode u˛ is a positive real number. A weakly damped structure is a structure such that 0 < ˛ 1 for all ˛ in f1; : : : ; N g. Several algebraic expressions exist for diagonalizing the damping bilinear form with the elastic structural modes.

References 1. Argyris, J., Mlejnek, H.P.: Dynamics of Structures. NorthHolland, Amsterdam (1991) 2. Bathe, K.J., Wilson, E.L.: Numerical Methods in Finite Element Analysis. Prentice-Hall, New York (1976) 3. Belytschko, T., Liu, W.K., Moran, B.: Nonlinear Finite Elements for Continua and Structures. Wiley, Chichester (2000) 4. Bensoussan, A., Lions, J.L., Papanicolaou, G.: Asymptotic Analysis for Periodic Structures. AMS Chelsea Publishing, Providence (2010) 5. Craig, R.R., Bampton, M.C.C.: Coupling of substructures for dynamic analysis. AIAA J. 6, 1313–1319 (1968) 6. Dautray, R., Lions, J.L.: Mathematical Analysis and Numerical Methods for Science and Technology. Springer, Berlin (2000) 7. Geradin, M., Rixen, D.: Mechanical Vibrations: Theory and Applications to Structural Dynamics, 2nd edn. Wiley, Chichester (1997) 8. Har, J., Tamma, K.: Advances in Computational Dynamics of Particles, Materials and Structures. Wiley, Chichester (2012)

Subdivision Schemes 9. Hughes, T.J.R.: The Finite Element Method: Linear Static and Dynamic Finite Element Analysis. Dover, New York (2000) 10. Lyon, R.H., Dejong, R.G.: Theory and Application of Statistical Energy Analysis, 2nd edn. Butterworth-Heinemann, Boston (1995) 11. Oden, J.T., Reddy, J.N.: An Introduction to the Mathematical Theory of Finite Elements. Dover, New York (2011) 12. Ohayon, R., Soize, C.: Structural Acoustics and Vibration. Academic, London (1998) 13. Soize, C.: Stochastic Models of Uncertainties in Computational Mechanics, vol. 2. Engineering Mechanics Institute (EMI) of the American Society of Civil Engineers (ASCE), Reston (2012) 14. Zienkiewicz, O.C., Taylor, R.L.: The Finite Element Method: The Basis, vol. 1, 5th edn. Butterworth- Heinemann, Oxford (2000)

1429

of refinable functions, which are instrumental in the construction of wavelets. The “classical” subdivision schemes are stationary and linear, applying the same linear refinement rule at each refinement level. The theory of these schemes is well developed and well understood; see [2, 7, 9, 15], and references therein. Nonlinear schemes were designed for the approximation of piecewise smooth functions (see, e.g., [1,4]), for taking into account the geometry of the initial points in the design of curves/surfaces (see, e.g., [5, 8, 11]), and for manifold-valued data (see, e.g., [12, 14, 16]). These schemes were studied at a later stage, and in many cases their analysis is based on their proximity to linear schemes.

Linear Schemes A linear refinement rule operates on a set of points in Subdivision Schemes Rd with topological relations among them, expressed by relating the points to the vertices of a regular grid in Nira Dyn Rs . In the design of 3D surfaces, d D 3 and s D 2. School of Mathematical Sciences, Tel-Aviv The refinement consists of a rule for refining the University, Tel-Aviv, Israel grid and a rule for defining the new points corresponding to the vertices of the refined grid. The most common refinement is binary, and the most common grid is Zs . Mathematics Subject Classification For a set of points P D fPi 2 Rd W Primary: 65D07; 65D10; 65D17. Secondary: 41A15; i 2 Zs g, related to 2k Zs , the binary refinement rule R generates points related to 2k1 Zs , of 41A25 the form

Definition .RP/i D A subdivision scheme is a method for generating a continuous function from discrete data, by repeated applications of refinement rules. A refinement rule operates on a set of data points and generates a denser set using local mappings. The function generated by a convergent subdivision scheme is the limit of the sequence of sets of points generated by the repeated refinements.

X

ai 2j Pj ; i 2 Zs ;

(1)

j 2Zs

with the point .RP/i related to the vertex i 2k1 . The set of coefficients fai 2 R W i 2 Zs g is called the mask of the refinement rule, and only a finite number of the coefficients are nonzero, reflecting the locality of the refinement. When the same refinement rule is applied in all refinement levels, the scheme is called stationary, while if different linear refinement rules are applied Description in different refinement levels, the scheme is called Subdivision schemes are efficient computational meth- nonstationary (see, e.g., [9]). A stationary scheme is defined to be convergent (in ods for the design, representation, and approximation of curves and surfaces in 3D and for the generation the L1 -norm) if for any initial set of points P, there

S

1430

Subdivision Schemes

exists a continuous function F defined on .Rs /d such representing surfaces of general topology, and a finite number of extraordinary points are required [13]. that The analysis on regular grids of the convergence of (2) lim sup jF .i 2k /  .Rk P/i j D 0; k!1 i 2Zs a stationary linear scheme, and of the smoothness of and if for at least one initial set of points, F 6 0. A the generated functions, is based on the coefficients similar definition of convergence is used for all other of the mask. It requires the computation of the joint types of subdivision schemes, with Rk replaced by the spectral radius of several finite dimensional matrices with mask coefficients as elements in specific positions appropriate product of refinement rules. in terms Although the refinement (1) is defined on all Zs , the (see, e.g., [2]) or an equivalent computation P i a finite support of the mask guarantees that the limit of of the Laurent polynomial a.z/ D s i 2Z i z (see, the subdivision scheme at a point is affected only by a e.g., [7]). When dealing with the design of surfaces, this finite number of initial points. When the initial points are samples of a smooth analysis applies only in all parts of the grids away function, the limit function of a convergent linear sub- from the extraordinary points. The analysis at these division scheme approximates the sampled function. points is local [13], but rather involved. It also dictates Thus, a convergent linear subdivision scheme is a linear changes in the refinement rules that have to be made near extraordinary points [13, 17]. approximation operator. As examples, we give two prototypes of stationary schemes for s D 1. Each is the simplest of its kind, converging to C 1 univariate functions. The first is the Chaikin scheme [3], called also “Corner cutting,” with References limit functions which “preserve the shape” of the initial 1. Amat, S., Dadourian, K., Liandart, J.: On a nonlinsets of points. The refinement rule is 3 1 Pi C Pi C1 ; 4 4 1 3 D Pi C Pi C1 ; i 2 Z: 4 4

.RP/2i D .RP/2i C1

(3)

The second is the 4-point scheme [6, 10], which interpolates the initial set of points (and all the sets of points generated by the scheme). It is defined by the refinement rule 9 .Pi C Pi C1 / 16 (4) 1  .Pi 1 C Pi C2 /; i 2 Z: 16

.RP/2i D Pi ; .RP/2i C1 D

2.

3.

4.

5.

6. 7.

While the limit functions of the Chaikin scheme can be written in terms of B-splines of degree 2, the limit 8. functions of the 4-point scheme for general initial sets of points have a fractal nature and are defined only 9. procedurally by (4). For the design of surfaces in 3D, s D 2 and the com- 10. mon grids are Z2 and regular triangulations. The latter are refined by dividing each triangle into four equal ones. Yet regular grids (with each vertex belonging to 11. four squares in the case of Z2 and to six triangles in the case of a regular triangulation) are not sufficient for

ear subdivision scheme avoiding Gibbs oscillations and converging towards C. Math. Comput. 80, 959–971 (2011) Cavaretta, A.S., Dahmen, W., Micchelli, C.A.: Stationary Subdivision. Memoirs of MAS, vol. 93, p. 186. American Mathematical Society, Providence (1991) Chaikin, G.M.: An algorithm for high speed curve generation. Comput. Graph. Image Process. 3, 346–349 (1974) Cohem, A., Dyn, N., Matei, B.: Quasilinear subdivision schemes with applications to ENO interpolation. Appl. Comput. Harmonic Anal. 15, 89–116 (2003) Deng, C., Wang, G.: Incenter subdivision scheme for curve interpolation. Comput. Aided Geom. Des. 27, 48–59 (2010) Dubuc, S.: Interpolation through an iterative scheme. J. Math. Anal. Appl. 114, 185–204 (1986) Dyn, N.: Subdivision schemes in computer-aided geometric design. In: Light, W. (ed.) Advances in Numerical Analysis – Volume II, Wavelets, Subdivision Algorithms and Radial Basis Functions, p. 36. Clarendon, Oxford (1992) Dyn, N., Hormann, K.: Geometric conditions for tangent continuity of interpolatory planar subdivision curves. Comput. Aided Geom. Des. 29, 332–347 (2012) Dyn, N., Levin, D.: Subdivision schemes in geometric modelling. Acta-Numer. 11, 73–144 (2002) Dyn, N., Gregory, J., Levin, D.: A 4-point interpolatory subdivision scheme for curve design. Comput. Aided Geom. Des. 4, 257–268 (1987) Dyn, N., Floater, M.S., Hormann, K.: Four-point curve subdivision based on iterated chordal and centripetal parametrizations. Comput. Aided Geom. Des. 26, 279–286 (2009)

Symbolic Computing 12. Grohs, P.: A general proximity analysis of nonlinear subdivision schemes. SIAM J. Math. Anal. 42, 729–750 (2010) 13. Peters, J., Reif, U.: Subdivision Surfaces, p. 204. Springer, Berlin/Heidelberg (2008) 14. Wallner, J., Navayazdani, E., Weinmann, A.: Convergence and smoothness analysis of subdivision rules in Riemannian and symmetric spaces. Adv. Comput. Math. 34, 201–218 (2011) 15. Warren, J., Weimer, H.: Subdivision Methods for Geometric Design, p. 299. Morgan Kaufmann, San Francisco (2002) 16. Xie, G., Yu, T.: Smoothness equivalence properties of general manifold-valued data subdivision schemes. Multiscale Model. Simul. 7, 1073–1100 (2010) 17. Zorin, D.: Smoothness of stationary subdivision on irregular meshes. Constructive Approximation. 16, 359–397 (2000)

1431

Symbolic computing Computing that uses symbols to manipulate and solve mathematical formulas and equations in order to obtain mathematically exact results.

Definition

Scientific computing can be generally divided into two subfields: numerical computation, which is based on finite precision arithmetic (usually single or double precision), and symbolic computing which uses symbols to manipulate and solve mathematical equations and formulas in order to obtain mathematically exact results. Symbolic computing, also called symbolic manipulation or computer algebra system (CAS), typically includes systems that can focus on well-defined comSymbolic Computing puting areas such as polynomials, matrices, abstract algebra, number theory, or statistics (symbolic), as ˇ ık1 , Mateusz Paprocki2, Aaron Meurer3, Ondˇrej Cert´ well as on calculus-like manipulation such as limits, Brian Granger4 , and Thilina Rathnayake5 differential equations, or integrals. A full-featured CAS 1 Los Alamos National Laboratory, Los Alamos, NM, should have all or most of the listed features. There USA are systems that focus only on one specific area like 2 refptr.pl, Wroclaw, Poland polynomials – those are often called CAS too. 3 Department of Mathematics, New Mexico State Some authors claim that symbolic computing and University, Las Cruces, NM, USA computer algebra are two views of computing with 4 Department of Physics, California Polytechnic State mathematical objects [1]. According to them, symbolic University, San Luis Obispo, CA, USA computation deals with expression trees and addresses 5 Department of Computer Science, University of problems of determination of expression equivalence, Moratuwa, Moratuwa, Sri Lanka simplification, and computation of canonical forms, while computer algebra is more centered around the computation of mathematical quantities in well-defined algebraic domains. The distinction between symbolic Synonyms computing and computer algebra is often not made; Computer algebra system; Symbolic computing; Sym- the terms are used interchangeably. We will do so in this entry as well. bolic manipulation

Keywords

History

Algorithms for Computer Algebra [2] provides a concise description about the history of symbolic computation. The invention of LISP in the early 1960s had a great impact on the development of symbolic computaGlossary/Definition Terms tion. FORTRAN and ALGOL which existed at the time were primarily designed for numerical computation. In Numerical computing Computing that is based on 1961, James Slagle at MIT (Massachusetts Institute of finite precision arithmetic. Technology) wrote a heuristics-based LISP program Symbolic computing; Computer algebra system

S

1432

Symbolic Computing

for Symbolic Automatic INTegration (SAINT) [3]. In 1963, Martinus J. G. Veltman developed Schoonschip [4, 5] for particle physics calculations. In 1966, Joel Moses (also from MIT) wrote a program called SIN [6] for the same purpose as SAINT, but he used a more efficient algorithmic approach. In 1968, REDUCE [7] was developed by Tony Hearn at Stanford University for physics calculations. Also in 1968, a specialized CAS called CAMAL [8] for handling Poisson series in celestial mechanics was developed by John Fitch and David Barton from the University of Cambridge. In 1970, a general purposed system called REDUCE 2 was introduced. In 1971 Macsyma [9] was developed with capabilities for algebraic manipulation, limit calculations, symbolic integration, and the solution of equations. In the late 1970s muMATH [10] was developed by the University of Hawaii and it came with its own programming language. It was the first CAS to run on widely available IBM PC computers. With the development of computing in 1980s, more modern CASes began to emerge. Maple [11] was introduced by the University of Waterloo with a small compiled kernel and a large mathematical library, thus allowing it to be used powerfully on smaller platforms. In 1988, Mathematica [12] was developed by Stephen Wolfram with better graphical capabilities and integration with graphical user interfaces. In the 1980s more and more CASes were developed like Macaulay [13], PARI [14], GAP [15], and CAYLEY [16] (which later became Magma [17]). With the popularization of open-source software in the past decade, many open-source CASes were developed like Sage [18], SymPy [19], etc. Also, many of the existing CASes were later open sourced;

for example, Macsyma became Maxima [20]; Scratchpad [21] became Axiom [22].

Overview A common functionality of all computer algebra systems typically includes at least the features mentioned in the following subsections. We use SymPy 0.7.5 and Mathematica 9 as examples of doing the same operation in two different systems, but any other full-featured CAS can be used as well (e.g., from the Table 1) and it should produce the same results functionally. To run the SymPy examples in a Python session, execute the following first: from sympy import * x, y, z, n, m = symbols(’x, y, z, n, m’) f = Function(’f’) To run the Mathematica examples, just execute them in a Mathematica Notebook. Arbitrary Formula Representation One can represent arbitrary expressions (not just polynomials). SymPy: In [1]: Out[1]: In [2]: Out[2]:

(1+1/x)**x (1 + 1/x)**x sqrt(sin(x))/z sqrt(sin(x))/z

Mathematica: In[1]:= (1+1/x)ˆx Out[1]= (1+1/x)ˆx

Symbolic Computing, Table 1 Implementation details of various computer algebra systems Program

License

Internal implementation language

CAS language

Mathematica [12] Maple [11] Symbolic MATLAB toolbox [23] Axioma [22] SymPy [19] Maxima [20] Sage [18] Giac/Xcas [24]

Commercial Commercial Commercial BSD BSD GPL GPL GPL

C/C++ C/C++ C/C++ Lisp Python Lisp C++/Cython/Lisp C++

Custom custom Custom Custom Python Custom Pythonb Custom

a

The same applies to its two forks FriCAS [25] and OpenAxiom [26] The default environment in Sage actually extends the Python language using a preparser that converts things like 2ˆ3 into Integer(2)**Integer(3), but the preparser can be turned off and one can use Sage from a regular Python session as well b

Symbolic Computing

In[2]:= Sqrt[Sin[x]]/z Out[2]= Sqrt[Sin[x]]/z Limits SymPy:

1433

Mathematica: In[1]:= Factor[xˆ2 y+xˆ2 z+x yˆ2+2 x y z+x zˆ2+yˆ2 z+y zˆ2] Out[1]= (x+y) (x+z) (y+z)

In [1]: Out[1]: In [2]: Out[2]:

limit(sin(x)/x, x, 0) 1 Algebraic and Differential Equation Solvers limit((2-sqrt(x))/(4-x),x,4) Algebraic equations, SymPy: 1/4 In [1]: solve(x**4+x**2+1, x) Mathematica: Out[1]: [-1/2 - sqrt(3)*I/2, -1/2 In[1]:= Limit[Sin[x]/x,x->0] + sqrt(3)*I/2, Out[1]= 1 1/2 - sqrt(3)*I/2, 1/2 In[2]:= Limit[(2-Sqrt[x])/(4-x),x->4] + sqrt(3)*I/2] Out[2]= 1/4

Mathematica: Differentiation SymPy: In [1]: Out[1]: In [1]: Out[1]:

diff(sin(2*x), x) 2*cos(2*x) diff(sin(2*x), x, 10) -1024*sin(2*x)

Mathematica: In[1]:= Out[1]= In[2]:= Out[2]=

D[Sin[2 x],x] 2 Cos[2 x] D[Sin[2 x],{x,10}] -1024 Sin[2 x]

Integration SymPy: In [1]: Out[1]: In [1]: Out[1]:

integrate(1/(x**2+1), x) atan(x) integrate(1/(x**2+3), x) sqrt(3)*atan(sqrt(3)*x/3)/3

Mathematica: In[1]:= Out[1]= In[2]:= Out[2]=

Integrate[1/(xˆ2+1),x] ArcTan[x] Integrate[1/(xˆ2+3),x] ArcTan[x/Sqrt[3]]/Sqrt[3]

Polynomial Factorization SymPy: In [1]: factor(x**2*y + x**2*z + x*y**2 + 2*x*y*z + x*z**2 + y**2*z + y*z**2) Out[1]: (x + y)*(x + z)*(y + z)

In[1]:= Reduce[1+xˆ2+xˆ4==0,x] Out[1]= x==-(-1)ˆ(1/3)||x ==(-1)ˆ(1/3)|| x==-(-1)ˆ(2/3)||x ==(-1)ˆ(2/3) and differential equations, SymPy: In [1]: dsolve(f(x).diff(x, 2) +f(x), f(x)) Out[1]: f(x) == C1*sin(x) + C2*cos(x) In [1]: dsolve(f(x).diff(x, 2) +9*f(x), f(x)) Out[1]: f(x) == C1*sin(3*x) + C2*cos(3*x) Mathematica: In[1]:= DSolve[f’’[x]+f[x]==0,f[x],x] Out[1]= {{f[x]->C[1] Cos[x] +C[2] Sin[x]}} In[2]:= DSolve[f’’[x]+9 f[x] ==0,f[x],x] Out[2]= {{f[x]->C[1] Cos[3 x] +C[2] Sin[3 x]}}

Formula Simplification Simplification is not a well-defined operation (i.e., there are many ways how to define the complexity of an expression), but typically the CAS is able to simplify, for example, the following expressions in an expected way, SymPy: In [1]: simplify(-1/(2*(x**2 + 1)) - 1/(4*(x + 1))+1/(4*(x - 1)) - 1/(x**4-1))

S

1434

Symbolic Computing

Out[1]: 0

Mathematica:

Mathematica: In[1]:= Simplify[-1/(2(xˆ2+1)) -1/(4(x+1))+1/(4(x-1)) -1/(xˆ4-1)] Out[1]= 0 or, SymPy:

In[1]:= Out[1]= In[2]:= Out[2]= In[3]:= Out[3]=

Sum[n,{n,1,m}] 1/2 m (1+m) Sum[1/nˆ6,{n,1,Infinity}] Piˆ6/945 Sum[1/nˆ5,{n,1,Infinity}] Zeta[5]

In [1]: simplify((x - 1)/(x**2 - 1)) Out[1]: 1/(x + 1)

Mathematica:

Software

In[1]:= Simplify[(x-1)/(xˆ2-1)] Out[1]= 1/(1+x)

Numerical Evaluation p Exact expressions like 2, constants, sums, integrals, and symbolic expressions can be evaluated to a desired accuracy using a CAS. For example, in SymPy: In [1]: N(sqrt(2),30) Out[1]: 1.41421356237309504880168872421 In [2]: N(Sum(1/n**n, (n,1,oo)),30) Out[2]: 1.29128599706266354040728259060 Mathematica: In[1]:= N[Sqrt[2],30] Out[1]= 1.41421356237309504880168872421 In[2]:= N[Sum[1/nˆn,{n,1, Infinity}],30] Out[2]= 1.29128599706266354040728259060

Symbolic Summation There are circumstances where it is mathematically impossible to get an explicit formula for a given sum. When an explicit formula exists, getting the exact result is usually desirable. SymPy: In [1]: Out[1]: In [2]: Out[2]: In [3]: Out[3]:

Sum(n, (n, 1, m)).doit() m**2/2 + m/2 Sum(1/n**6,(n,1,oo)).doit() pi**6/945 Sum(1/n**5,(n,1,oo)).doit() zeta(5)

A computer algebra system (CAS) is typically composed of a high-level (usually interpreted) language that the user interacts with in order to perform calculations. Many times the implementation of such a CAS is a mix of the high-level language together with some low-level language (like C or C++) for efficiency reasons. Some of them can easily be used as a library in user’s programs; others can only be used from the custom CAS language. A comprehensive list of computer algebra software is at [27]. Table 1 lists features of several established computer algebra systems. We have only included systems that can handle at least the problems mentioned in the Overview section. Besides general full-featured CASes, there exist specialized packages, for Example, Singular [28] for very fast polynomial manipulation or GiNaC [29] that can handle basic symbolic manipulation but does not have integration, advanced polynomial algorithms, or limits. Pari [14] is designed for number theory computations and Cadabra [30] for field theory calculations with tensors. Magma [17] specializes in algebra, number theory, algebraic geometry, and algebraic combinatorics. Finally, a CAS also usually contains a notebook like interface, which can be used to enter commands or programs, plot graphs, and show nicely formatted equations. For Python-based CASes, one can use IPython Notebook [31] or Sage Notebook [18], both of which are interactive web applications that can be used from a web browser. C++ CASes can be wrapped in Python, for example, GiNaC has several Python wrappers: Swiginac [32], Pynac [33], etc. These can then be used from Python-based notebooks. Mathematica and Maple also contain a notebook interface, which accepts the given CAS high-level language.

Symbolic Computing

Applications of Symbolic Computing Symbolic computing has traditionally had numerous applications. By 1970s, many CASes were used for celestial mechanics, general relativity, quantum electrodynamics, and other applications [34]. In this section we present a few such applications in more detail, but necessarily our list is incomplete and is only meant as a starting point for the reader. Many of the following applications and scientific advances related to them would not be possible without symbolic computing.

1435

Code Generation One of the frequent use of a CAS is to derive some symbolic expression and then generate C or Fortran code that numerically evaluates it in a production highperformance code. For example, to obtain the best rational function approximation (of orders 8, 8) to a modified Bessel function of the first kind of halfinteger argument I9=2 .x/ on an interval Œ4; 10, one can use (in Mathematica):

In[1]:= Needs["FunctionApproximations‘"] In[2]:= FortranForm[HornerForm[MiniMaxApproximation[ BesselI[9/2, x]*Sqrt[Pi*x/2]/Exp[x], {x,{4,10},8,8},WorkingPrecision->30][[2,1]]]] Out[2]//FortranForm= (0.000395502959013236968661582656143 + x*(-0.001434648369704841686633794071 + x*(0.00248783474583503473135143644434 + x*(-0.00274477921388295929464613063609 + x*(0.00216275018107657273725589740499 + x*(-0.000236779926184242197820134964535 + x*(0.0000882030507076791807159699814428 + (-4.62078105288798755556136693122e-6 + 8.23671374777791529292655504214e-7*x)*x))))))) / (1. + x*(0.504839286873735708062045336271 + x*(0.176683950009401712892997268723 + x*(0.0438594911840609324095487447279 + x*(0.00829753062428409331123592322788 + x*(0.00111693697900468156881720995034 + x*(0.000174719963536517752971223459247 + (7.22885338737473776714257581233e-6 + 1.64737453771748367647332279826e-6*x)*x)))))))

The result can be readily used in a Fortran code (we designed for this task was Schoonschip [4, 5], and in reformatted the white space in the output Out[2] to 1984 FORM [35] was created as a successor. FORM better fit into the page). has built-in features for manipulating formulas in particle physics, but it can also be used as a general purpose system (it keeps all expressions in expanded form, so it cannot do factorization; it also does not have more Particle Physics The application of symbolic computing in particle advanced features like series expansion, differential physics typically involves generation and then calcu- equations, or integration). Many of the scientific results in particle physics lation of Feynman diagrams (among other things that involves doing fast traces of Dirac gamma matrices would not be possible without a powerful CAS; and other tensor operations). The first CAS that was Schoonschip was used for calculating properties of the

S

1436

Symbolic Computing

W boson in the 1960s and FORM is still maintained and used to this day. Another project is FeynCalc [36], originally written for Macsyma (Maxima) and later Mathematica. It is a package for algebraic calculations in elementary particle physics, among other things; it can do tensor and Dirac algebra manipulation, Lorentz index contraction, and generation of Feynman rules from a Lagrangian, Fortran code generation. There are hundreds of publications that used FeynCalc to perform calculations. Similar project is FeynArts [37], which is also a Mathematica package that can generate and visualize Feynman diagrams and amplitudes. Those can then be calculated with a related project FormCalc [38], built on top of FORM. PyDy PyDy, short for Python Dynamics, is a work flow that utilizes an array of scientific tools written in the Python programming language to study multi-body dynamics [39]. SymPy mechanics package is used to generate symbolic equations of motion in complex multi-body systems, and several other scientific Python packages are used for numerical evaluation (NumPy [40]), visualization (Matplotlib [41]), etc. First, an idealized version of the system is represented (geometry, configuration, constraints, external forces). Then the symbolic equations of motion (often very long) are generated using the mechanics package and solved (integrated after setting numerical values for the parameters) using differential equation solvers in SciPy [42]. These solutions can then be used for simulations and visualizations. Symbolic equation generation guarantees no mistakes in the calculations and makes it easy to deal with complex systems with a large number of components. General Relativity In general relativity the CASes have traditionally been used to symbolically represent the metric tensor g and then use symbolic derivation to derive various tensors (Riemann and Ricci tensor, curvature, : : :) that are present in the Einstein’s equations [34]: 1 8G R  g R C g  D 4 T : 2 c Those can then be solved for simple systems.

(1)

Celestial Mechanics The equations in celestial mechanics are solved using perturbation theory, which requires very efficient manipulation of a Poisson series [8, 34, 43–50]: X

P .a; b; c; : : : ; h/

sin .u C v C    C  z/ (2) cos

where P .a; b; c; : : : ; h/ is a polynomial and each term contains either sin or cos. Using trigonometric relations, it can be shown that this form is closed to addition, subtraction, multiplication, differentiation, and restricted integration. One of the earliest specialized CASes for handling Poisson series is CAMAL [8]. Many others were since developed, for example, TRIP [43]. Quantum Mechanics Quantum mechanics is well known for many tedious calculations, and the use of a CAS can aid in doing them. There have been several books published with many worked-out problems in quantum mechanics done using Mathematica and other CASes [51, 52]. There are specialized packages for doing computations in quantum mechanics, for example, SymPy has extensive capabilities for symbolic quantum mechanics in the sympy.physics.quantum subpackage. At the base level, this subpackage has Python objects to represent the different mathematical objects relevant in quantum theory [53]: states (bras and kets), operators (unitary, Hermitian, etc.), and basis sets as well as operations on these objects such as tensor products, inner products, outer products, commutators, anticommutators, etc. The base objects are designed in the most general way possible to enable any particular quantum system to be implemented by subclassing the base operators to provide system specific logic. There is a general purpose qapply function that is capable of applying operators to states symbolically as well as simplifying a wide range of symbolic expressions involving different types of products and commutator/anticommutators. The state and operator objects also have a rich API for declaring their representation in a particular basis. This includes the ability to specify a basis for a multidimensional system using a complete set of commuting Hermitian operators. On top of this base set of objects, a number of specific quantum systems have been implemented. First, there is traditional algebra for quantum angular

Symbolic Computing

momentum [54]. This allows the different spin operators (Sx , Sy , Sz ) and their eigenstates to be represented in any basis and for any spin quantum number. Facilities for Clebsch-Gordan coefficients, Wigner coefficients, rotations, and angular momentum coupling are also present in their symbolic and numerical forms. Other examples of particular quantum systems that are implemented include second quantization, the simple harmonic oscillator (position/momentum and raising/lowering forms), and continuous position/momentum-based systems. Second there is a full set of states and operators for symbolic quantum computing [55]. Multidimensional qubit states can be represented symbolically and as vectors. A full set of one (X , Y , Z, H , etc.) and two qubit (CNOT, SWAP, CPHASE, etc.) gates (unitary operators) are provided. These can be represented as matrices (sparse or dense) or made to act on qubits symbolically without representation. With these gates, it is possible to implement a number of basic quantum circuits including the quantum Fourier transform, quantum error correction, quantum teleportation, Grover’s algorithm, dense coding, etc. There are other packages that specialize in quantum computing, for example, [56].

1437

Teaching Calculus and Other Classes Computer algebra systems are extremely useful for teaching calculus [64] as well as other classes where tedious symbolic algebra is needed, such as many physics classes (general relativity, quantum mechanics and field theory, symbolic solutions to partial differential equations, e.g., in electromagnetism, fluids, plasmas, electrical circuits, etc.) [65]. Experimental Mathematics One field that would not be possible at all without computer algebra systems is called “experimental mathematics” [66], where CASes and related tools are used to “experimentally” verify or suggest mathematical relations. For example, the famous Bailey–Borwein– Plouffe (BBP) formula  1  X 2 1 4  D 16k 8k C 1 8k C 4 kD0  1 1   8k C 5 8k C 6

(3)

was first discovered experimentally (using arbitraryprecision arithmetic and extensive searching using an integer relation algorithm), only then proved rigorously [67]. Another example is in [68] where the authors first Number Theory numerically discovered and then proved that for ratioNumber theory provides an important base for modern nal x, y, the 2D Poisson potential function satisfies computing, especially in cryptography and coding theory [57]. For example, LLL [58] algorithm is used in 1 X cos.ax/ cos.by/ 1 .x; y/ D D log ˛ 2 2 2 integer programming; primality testing and factoring  a Cb  a;b odd algorithms are used in cryptography [59]. CASes are (4) heavily used in these calculations. Riemann hypothesis [60, 61] which implies results where ˛ is algebraic (a root of an integer polynomial). about the distribution of prime numbers has important applications in computational mathematics since it can Acknowledgements This work was performed under the auspices of the US Department of Energy by Los Alamos National be used to estimate how long certain algorithms take to Laboratory. run [61]. Riemann hypothesis states that all nontrivial zeros of the Riemann zeta function, defined for complex variable s defined in the half-plane < .s/ > 1 by P1 s References the absolutely convergent series  .s/ D nD1 n , have real part equal to 12 . In 1986, this was proven 1. Watt, S.M.: Making computer algebra more symbolic. In: for the first 1,500,000,001 nontrivial zeros using comDumas J.-G. (ed.) Proceedings of Transgressive Computing 2006: A Conference in Honor of Jean Della Dora, Facultad putational methods [62]. Sebastian Wedeniwski using de Ciencias, Universidad de Granada, pp 43–49, April 2006 ZettaGrid (a distributed computing project to find roots 2. Geddes, K.O., Czapor, S.R., Labahn, G.: Algorithms for of the zeta function) verified the result for the first 400 computer algebra. In: Introduction to Computer Algebra. billion zeros in 2005 [63]. Kluwer Academic, Boston (1992)

S

1438 3. Slagle, J.R.: A heuristic program that solves symbolic integration problems in freshman calculus. J. ACM 10(4), 507–520 (1963) 4. Veltman, M.J.G.: From weak interactions to gravitation. Int. J. Mod. Phys. A 15(29), 4557–4573 (2000) 5. Strubbe, H.: Manual for SCHOONSCHIP a CDC 6000/7000 program for symbolic evaluation of algebraic expressions. Comput. Phys. Commun. 8(1), 1–30 (1974) 6. Moses, J.: Symbolic integration. PhD thesis, MIT (1967) 7. REDUCE: A portable general-purpose computer algebra system. http://reduce-algebra.sourceforge.net/ (2013) 8. Fitch, J.: CAMAL 40 years on – is small still beautiful? Intell. Comput. Math., Lect. Notes Comput. Sci. 5625, 32–44 (2009) 9. Moses, J.: Macsyma: a personal history. J. Symb. Comput. 47(2), 123–130 (2012) 10. Shochat, D.D.: Experience with the musimp/mumath-80 symbolic mathematics system. ACM SIGSAM Bull. 16(3), 16–23 (1982) 11. Bernardin, L., Chin, P. DeMarco, P., Geddes, K.O., Hare, D.E.G., Heal, K.M., Labahn, G., May, J.P., McCarron, J., Monagan, M.B., Ohashi, D., Vorkoetter, S.M.: Maple Programming Guide. Maplesoft, Waterloo (2011) 12. Wolfram, S.: The Mathematica Book. Wolfram Research Inc., Champaign (2000) 13. Grayson, D.R., Stillman, M.E.: Macaulay 2, a software system for research in algebraic geometry. Available at http://www.math.uiuc.edu/Macaulay2/ 14. The PARI Group, Bordeaux: PARI/GP version 2.7.0 . Available from http://pari.math.u-bordeaux.fr/ (2014) 15. The GAP Group: GAP – groups, algorithms, and programming, version 4.4. http://www.gap-system.org (2003) 16. Cannon, J.J.: An introduction to the group theory language cayley. Comput. Group Theory 145, 183 (1984) 17. Bosma, W., Cannon, J., Playoust, C.: The Magma algebra system. I. The user language. J. Symb. Comput. 24(3– 4), 235–265 (1997). Computational Algebra and Number Theory. London (1993) 18. Stein, W.A., et al.: Sage Mathematics Software (Version 6.4.1). The Sage Development Team (2014). http://www. sagemath.org 19. SymPy Development Team: SymPy: Python library for symbolic mathematics. http://www.sympy.org (2013) 20. Maxima: Maxima, a computer algebra system. Version 5.34.1. http://maxima.sourceforge.net/ (2014) 21. Jenks, R.D., Sutor, R.S., Watt, S.M.: Scratchpad ii: an abstract datatype system for mathematical computation. In: Janßen, R. (ed.) Trends in Computer Algebra. Volume 296 of Lecture Notes in Computer Science, pp. 12–37. Springer, Berlin/Heidelberg (1988) 22. Jenks, R.D., Sutor, R.S.: Axiom – The Scientific Computation System. Springer, New York/Berlin etc. (1992) 23. MATLAB: Symbolic math toolbox. http://www.mathworks. com/products/symbolic/ (2014) 24. Parisse, B.: Giac/xcas, a free computer algebra system. Technical report, Technical report, University of Grenoble (2008) 25. FriCAS, an advanced computer algebra system: http://fricas. sourceforge.net/ (2015) 26. OpenAxiom, The open scientific computation platform: http://www.open-axiom.org/ (2015)

Symbolic Computing 27. Wikipedia: List of computer algebra systems. http:// en.wikipedia.org/wiki/List of computer algebra systems (2013) 28. Decker, W., Greuel, G.-M., Pfister, G., Sch¨onemann, H.: SINGULAR 3-1-6 – a computer algebra system for polynomial computations. http://www.singular.uni-kl.de (2012) 29. Bauer, C., Frink, A., Kreckel, R.: Introduction to the ginac framework for symbolic computation within the c++ programming language. J. Symb. Comput. 33(1), 1–12 (2002) 30. Peeters, K.: Introducing cadabra: a symbolic computer algebra system for field theory problems. arxiv:hep-th/0701238; A field-theory motivated approach to symbolic computer algebra. Comput. Phys. Commun. 176, 550 (2007). [arXiv:cs/0608005]. - 14 31. P´erez, F., Granger, B.E.: IPython: a system for interactive scientific computing. Comput. Sci. Eng. 9(3), 21–29 (2007) 32. Swiginac, Python interface to GiNaC: http://sourceforge. net/projects/swiginac.berlios/ (2015) 33. Pynac, derivative of GiNaC with Python wrappers: http:// pynac.org/ (2015) 34. Barton, D., Fitch, J.P.: Applications of algebraic manipulation programs in physics. Rep. Prog. Phys. 35(1), 235–314 (1972) 35. Vermaseren, J.A.M.: New features of FORM. Math. Phys. e-prints (2000). ArXiv:ph/0010025 36. Mertig, R., B¨ohm, M., Denner, A.: Feyn calc – computeralgebraic calculation of feynman amplitudes. Comput. Phys. Commun. 64(3), 345–359 (1991) 37. Hahn, T.: Generating feynman diagrams and amplitudes with feynarts 3. Comput. Phys. Commun. 140(3), 418–431 (2001) 38. Hahn, T., P´erez-Victoria, M.: Automated one-loop calculations in four and d dimensions. Comput. Phys. Commun. 118(2–3), 153–165 (1999) 39. Gede, G., Peterson, D.L., Nanjangud, A.S., Moore, J.K., Hubbard, M.: Constrained multibody dynamics with python: from symbolic equation generation to publication. In: ASME 2013 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, pp. V07BT10A051–V07BT10A051. American Society of Mechanical Engineers, San Diego (2013) 40. NumPy: The fundamental package for scientific computing in Python. http://www.numpy.org/ (2015) 41. Hunter, J.D.: Matplotlib: a 2d graphics environment. Comput. Sci. Eng. 9(3), 90–95 (2007) 42. Jones, E., Oliphant, T., Peterson, P., et al.: SciPy: open source scientific tools for Python (2001–). http://www.scipy. org/ 43. Gastineau, M., Laskar, J.: Development of TRIP: fast sparse multivariate polynomial multiplication using burst tries. In: Computational Science – ICCS 2006, University of Reading, pp. 446–453 (2006) 44. Biscani, F.: Parallel sparse polynomial multiplication on modern hardware architectures. In: Proceedings of the 37th International Symposium on Symbolic and Algebraic Computation – ISSAC ’12, (1), Grenoble, pp. 83–90 (2012) 45. Shelus, P.J., Jefferys III, W.H.: A note on an attempt at more efficient Poisson series evaluation. Celest. Mech. 11(1), 75– 78 (1975) 46. Jefferys, W.H.: A FORTRAN-based list processor for Poisson series. Celest. Mech. 2(4), 474–480 (1970)

Symmetric Methods 47. Broucke, R., Garthwaite, K.: A programming system for analytical series expansions on a computer. Celest. Mech. 1(2), 271–284 (1969) 48. Fateman, R.J.: On the multiplication of poisson series. Celest. Mech. 10(2), 243–247 (1974) 49. Danby, J.M.A., Deprit, A., Rom, A.R.M.: The symbolic manipulation of poisson series. In: SYMSAC ’66 Proceedings of the First ACM Symposium on Symbolic and Algebraic Manipulation, New York, pp. 0901–0934 (1965) 50. Bourne, S.R.: Literal expressions for the co-ordinates of the moon. Celest. Mech. 6(2), 167–186 (1972) 51. Feagin, J.M.: Quantum Methods with Mathematica. Springer, New York (2002) 52. Steeb, W.-H., Hardy, Y.: Quantum Mechanics Using Computer Algebra: Includes Sample Programs in C++, SymbolicC++, Maxima, Maple, and Mathematica. World Scientific, Singapore (2010) 53. Sakurai, J.J., Napolitano, J.J.: Modern Quantum Mechanics. Addison-Wesley, Boston (2010) 54. Zare, R.N.: Angular Momentum: Understanding Spatial Aspects in Chemistry and Physics. Wiley, New York (1991) 55. Nielsen, M.A., Chuang, I.L.: Quantum Computation and Quantum Information. Cambridge University Press, Cambridge (2011) 56. Gerdt, V.P., Kragler, R., Prokopenya, A.N.: A Mathematica Package for Simulation of Quantum Computation. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 5743 LNCS, pp. 106–117. Springer, Berlin/Heidelberg (2009) 57. Shoup, V.: A Computational Introduction to Number Theory and Algebra. Cambridge University Press, Cambridge (2009) 58. Lenstra, A.K., Lenstra, H.W., Lov´asz, L.: Factoring polynomials with rational coefficients. Mathematische Annalen 261(4), 515–534 (1982) 59. Cohen, H.: A Course in Computational Algebraic Number Theory, vol. 138. Springer, Berlin/New York (1993) 60. Titchmarsh, E.C.: The Theory of the Riemann ZetaFunction, vol. 196. Oxford University Press, Oxford (1951) 61. Mazur, B., Stein, W.: Prime Numbers and the Riemann Hypothesis. Cambridge University Press, Cambridge (2015) 62. van de Lune, J., Te Riele, H.J.J., Winter, D.T.: On the zeros of the riemann zeta function in the critical strip. iv. Math. Comput. 46(174), 667–681 (1986) 63. Wells, D.: Prime Numbers: The Most Mysterious Figures in Math. Wiley, Hoboken (2011) 64. Palmiter, J.R.: Effects of computer algebra systems on concept and skill acquisition in calculus. J. Res. Math. Educ. 22, 151–156 (1991) 65. Bing, T.J., Redish, E.F.: Symbolic manipulators affect mathematical mindsets. Am. J. Phys. 76(4), 418–424 (2008) 66. Bailey, D.H., Borwein, J.M.: Experimental mathematics: examples, methods and implications. Not. AMS 52, 502– 514 (2005) 67. Bailey, D.H., Borwein, P., Plouffe, S.: On the rapid computation of various polylogarithmic constants. Math. Comput. 66(218), 903–913 (1996) 68. Bailey, D.H., Borwein, J.M.: Lattice sums arising from the Poisson equation. J. Phys. A: Math. Theor. 3, 1–18 (2013)

1439

Symmetric Methods Philippe Chartier INRIA-ENS Cachan, Rennes, France

Synonyms Time reversible

Definition This entry is concerned with symmetric methods for solving ordinary differential equations (ODEs) of the form yP D f .y/ 2 Rn ; y.0/ D y0 : (1) Throughout this article, we denote by 't;f .y0 / the flow of equation (1) with vector field f , i.e., the exact solution at time t with initial condition y.0/ D y0 , and we assume that the conditions for its well definiteness and smoothness for .y0 ; jtj/ in an appropriate subset ˝ of Rn  RC are satisfied. Numerical methods for (1) implement numerical flows ˚h;f which, for small enough stepsizes h, approximate 'h;f . Of central importance in the context of symmetric methods is the concept of adjoint method.  Definition 1 The adjoint method ˚h;f is the inverse of ˚t;f with reversed time step h:  1 WD ˚h;f ˚h;f

(2)

S A numerical method ˚h is then said to be symmetric if  . ˚h;f D ˚h;f

Overview Symmetry is an essential property of numerical methods with regard to the order of accuracy and geometric properties of the solution. We briefly discuss the implications of these two aspects and refer to the corresponding sections for a more involved presentation: • A method ˚h;f is said to be of order p if ˚h;f .y/ D 'h;f .y/ C O.hpC1 /;

1440

Rn

Symmetric Methods f

Rn

Rn

ρ

ρ

ρ

Rn

Rn

Rn

't;f

−1 't;f

(−f)

Rn

Rn

ρ

ρ

Rn

Rn

©h;f

Rn ρ

−1 ©h;f

Rn

Symmetric Methods, Fig. 1 -reversibility of f , 't;f and ˚h;f

with a Hamiltonian function H.q; p/ satisfying H.y; z/ D H.y; z/ are -reversible for .y; z/ D .y; z/. Definition 2 A method ˚h , applied to a -reversible ordinary differential equation, is said to be -reversible if 1 ı :  ı ˚h;f D ˚h;f Note that if ˚h;f is symmetric, it is -reversible if and only if the following condition holds:

and, if the local error has the following first-term expansion

 ı ˚h;f D ˚h;f ı :

(4)

Besides, if (4) holds for an invertible , then ˚h;f is -reversible if and only if it is symmetric.

˚h;f .y/ D 'h;f .y/ C hpC1 C.y/ C O.hpC2 /;

Example 2 The trapezoidal rule, whose flow is defined then straightforward application of the implicit func- by the implicit equation tion theorem leads to   1 1 y C ˚ .y/ D y C hf .y/ ; (5) ˚ h;f h;f  ˚h;f .y/ D 'h;f .y/  .h/pC1 C.y/ C O.hpC2 /: 2 2 This implies that a symmetric method is necessarily  of even order p D 2q, since ˚h;f .y/ D ˚h;f .y/ pC1 means that .1 C .1/ /C.y/ D 0. This property plays a key role in the construction of composition methods by triple jump techniques (see section on “Symmetric Methods Obtained by Composition”), and this is certainly no coincidence that Runge-Kutta methods of optimal order (Gauss methods) are symmetric (see section on “Symmetric Methods of Runge-Kutta Type”). It also explains why symmetric methods are used in conjunction with (Richardson) extrapolation techniques. • The exact flow 't;f is itself symmetric owing to the group property 'sCt;f D 's;f ı 't;f . Consider now an isomorphism  of the vector space Rn (the phase space of (1)) and assume that the vector field f satisfies the relation  ı f D f ı  (see Fig. 1). Then, 't;f is said to be -reversible, that is it to say the following equality holds: 1  ı 't;f D 't;f ı

Example 1 Hamiltonian systems yP D

@H .y; z/ @z

zP D 

@H .y; z/ @y

(3)

is symmetric and is -reversible when applied to -reversible f . Since most numerical methods satisfy relation (4), symmetry is the required property for numerical methods to share with the exact flow not only time reversibility but also -reversibility. This illustrates that a symmetric method mimics geometric properties of the exact flow. Modified differential equations sustain further this assertion (see next section) and allow for the derivation of deeper results for integrable reversible systems such as the preservation of invariants and the linear growth of errors by symmetric methods (see section on “Reversible Kolmogorov-Arnold– Moser Theory”).

Modified Equations for Symmetric Methods

Constant stepsize backward error analysis. Considering a numerical method ˚h (not necessarily symmetric) and the sequence of approximations obtained by application of the formula ynC1 D ˚h;f .yn /; n D 0; 1; 2; : : :, from the initial value y0 , the idea of backward error analysis consists in searching for a modified vector field fhN such that 'h;f N .y0 / D ˚h;f .y0 / C O.hN C2 /; h

(6)

Symmetric Methods

1441

where the modified vector field, uniquely defined by a Taylor expansion of (6), is of the form

Remark 1 The expansion (7) of the modified vector field fhN can be computed explicitly at any order N with the substitution product of B-series [2].

fhN .y/ D f .y/ C hf1 .y/ C h2 f2 .y/ C : : : C hN fN .y/: Example 3 Consider the Lotka-Volterra equations in (7) Poisson form Theorem 1 The modified vector field of a symmetric      ru H.u; v/ 0 uv uP method ˚h;f has an expansion in even powers of h, ; D rv H.u; v/ uv 0 vP i.e., f2j C1 0 for j D 0; 1; : : : Moreover, if f and ˚h;f are -reversible, then fhN is -reversible as well for any N  0.

H.u; v/ D log.u/ C log.v/  u  v;

i.e., y 0 D f .y/ with f .y/ D .u.1  v/; v.u  1//T . Proof. Reversing the time step h in (6) and taking the Note that  ı f D f ı  with .u; v/ D .v; u/. 2 inverse of both sides, we obtain for the implicit Euler The modified vector fields fh;iE 2 method and fh;mr for the implicit midpoint rule read .'h;f N /1 .y0 / D .˚h;f /1 .y0 / C O.hN C2 /: (with N D 2) h Now, the group property of exact flows implies that .'h;f N /1 .y0 / D 'h;f N .y0 /, so that h

h

'h;f N .y0 / D h

 ˚h;f

N C2

.y0 / C O.h

1 h2 h2 2 D f C hf 0 f C f 00 .f; f / C f 0 f 0 f fh;iE 2 12 3 2 Df  and fh;mr

/;

h2 00 h2 f .f; f / C f 0 f 0 f: 24 12

The exact solutions of the modified ODEs are plotted N and by uniqueness, .fhN / D fh . This proves the on Fig. 2 together with the corresponding numerical first statement. Assume now that f is -reversible, so solution. Though the modified vector fields are trunN N that (4) holds. It follows from fh D fh that cated only at second order, the agreement is excellent. The difference of the behavior of the two solutions is O.hN C2 / D  ı ˚h;f  ı 'h;f N D  ı 'h;f N also striking: only the symmetric method captures the h h periodic nature of the solution. (The good behavior of O.hN C2 / the midpoint rule cannot be attributed to its symplecticD ˚h;f ı  D 'h;f N ı ; h ity since the system is a noncanonical Poisson system.) where the second and last equalities are valid up to This will be further explored in the next section. O.hN C2 /-error terms. Yet the group property then implies that  ı 'nh;f N D 'nh;f N ı  C On .hN C2 / h h where the constant in the On -term depends on n and an interpolation argument shows that for fixed N and small jtj  ı 't;f N D 't;f N ı  C O.hN C1 /; h

h

where the O-term depends smoothly on t and on N . Finally, differentiating with respect to t, we obtain  ı fhN

ˇ ˇ d  ı 't;f N ˇˇ D h dt

D t D0

d ' N dt t;fh

ˇ ˇ ı ˇˇ

ynC1 D ˚s.yn ;/;f .yn /;

n D 0; : : : :

A remarkable feature of this algorithm is that it preserves the symmetry of the exact solution as soon as ˚h;f is symmetric and s satisfies the relation

t D0

CO.hN C2 / D fhN ı  C O.hN C1 /; and consequently  ı fhN D fhN ı .

Variable stepsize backward error analysis. In practice, it is often fruitful to resort to variable stepsize implementations of the numerical flow ˚h;f . In accordance with [17], we consider stepsizes that are proportional to a function s.y; / depending only on the current state y and of a parameter  prescribed by the user and aimed at controlling the error. The approximate solution is then given by

t u

s.˚s.y;/;f .y/; / D s.y; /

S

1442

Symmetric Methods Implicit Euler

Implicit midpoint rule

2.5

3 2.5

2

2 v

v

1.5 1.5

1 1 0.5

0.5

0

0 0

0.5

1 u

1.5

2

0

0.5

1

1.5

2

2.5

3

u

Symmetric Methods, Fig. 2 Exact solutions of modified equations (red lines) versus numerical solutions by implicit Euler and midpoint rule (blue points)

and preserves the -reversibility as soon as ˚h;f is An interesting instance is the case of completely inte-reversible and satisfies the relation grable Hamiltonian systems: s.1 ı ˚s.y;/;f .y/; / D s.y; /:

yP D

A result similar to Theorem 1 then holds with h replaced by . Remark 2 A recipe to construct such a function s, suggested by Stoffer in [17], consists in requiring that the local error estimate is kept constantly equal to a tolerance parameter. For the details of the implementation, we refer to the original paper or to Chap. VIII.3 of [10].

@H .y; z/; @z

zP D 

@H .y; z/; @y

with first integrals Ij ’s in involution (That is to say such that .ry Ii /  .rz Ij / D .rz Ii /  .ry Ij / for all i; j .) such that Ij ı  D Ij . In the conditions where ArnoldLiouville theorem (see Chap. X.1.3. of [10]) can be applied, then, under the additional assumption that 9.y  ; 0/ 2 f.y; z/; 8j; Ij .y; z/ D Ij .y0 ; z0 /g; (9)

Reversible Kolmogorov-Arnold-Moser Theory

such a system is reversible integrable. In this situation, -reversible methods constitute a very interesting way around symplectic method, as the following result The theory of integrable Hamiltonian systems has its shows: counterpart for reversible integrable ones. A reversible Theorem 2 Let ˚h;.f;g/ be a reversible numerical system method of order p applied to an integrable reversible system (8) with real-analytic f and g. Consider yP D f .y; z/; zP D g.y; z/ where  ı .f; g/ a D .I1 .y ; z /; : : : ; Id .y ; z //: If the condition D .f; g/ ı 

with

.y; z/ D .y; z/;

(8)

is reversible integrable if it can be brought, through a reversible transformation .a; / D .I.y; z/; ‚.y; z//, to the canonical equations aP D 0;

P D !.a/:



8k 2 Z =f0g; jk  !.a /j   d

d X

! jki j

i D1

is satisfied for some positive constants  and , then there exist positive C , c, and h0 such that the following assertion holds:

Symmetric Methods

1443

8h  h0 ; 8.x0 ; y0 / such that max jIj .y0 ; z0 /  a j  cj log hj1 ; ( p

8t D nh  h

;

(10)

j D1;:::;d

n .x0 ; y0 /  .y.t/; z.t//k  C thp k˚h;.f;g/ n jIj .˚h;.f;g/ .y0 ; z0 //  Ij .y0 ; z0 /j  C hp for all j:

Analogously to symplectic methods, -reversible methods thus preserve invariant tori Ij D cst over long intervals of times, and the error growth is linear in t. Remarkably and in contrast with symplectic methods, this result remains valid for reversible variable stepsize implementations (see Chap. X.I.3 of [10]). However, it is important to note that for a Hamiltonian reversible system, the Hamiltonian ceases to be preserved when condition (9) is not fulfilled. This situation is illustrated on Fig. 3 for the Hamiltonian system with H.q; p/ D 12 p 2 C cos.q/ C 15 sin.2q/, an example borrowed from [4].

c1 :: :

a1;1 :: :

:::

a1;s :: :

cs

as;1 b1

::: :::

as;s bs

(13)

Runge-Kutta methods automatically satisfy the -compatibility condition (4): changing h into h in (11) and (12), we have indeed by linearity of  and by using  ı f D f ı  .Yi / D .y/  h

s X

 ai;j f .Yj / ;

i D 1; : : : ; s

j D1

Symmetric Methods of Runge-Kutta Type .y/ Q D .y/  h

s X

 bj f .Yj / :

Runge-Kutta methods form a popular class of numerj D1 ical integrators for (1). Owing to their importance in applications, we consider general systems (1) and By construction, this is .˚h;f .y// and by previous subsequently partitioned systems. definition ˚h;f ..y//. As a consequence, -reversible Methods for general systems. We start with the Runge-Kutta methods coincide with symmetric methods. Nevertheless, symmetry requires an additional following: algebraic condition stated in the next theorem: Definition 3 Consider a matrix A D .ai;j / 2 Rs  Rs and a vector b D .bj / 2 Rs . The Runge-Kutta method Theorem 3 A Runge-Kutta method .A; b/ is symmetric if denoted .A; b/ is defined by

Yi D y C h

s X

PA C AP D eb T and b D P b; ai;j f .Yj /;

i D 1; : : : ; s

where e D .1; : : : ; 1/T 2 Rs and P is the permutation matrix defined by pi;j D ıi;sC1j .

j D1

yQ D y C h

s X

(14)

(11)

bj f .Yj /:

(12)

T

Proof. Denoting Y D Y1T ; : : : ; YsT and F .Y / D

T Note that strictly speaking, the method is properly def .Y1 /T ; : : : ; f .Ys /T , a more compact form fined only for small jhj. In this case, the corresponding for (11) and (12) is numerical flow ˚h;f maps y to y. Q Vector Yi approximates the solution at intermediate point t0 Cci h, where P Y D e ˝ y C h.A ˝ I /F .Y /; (15) ci D j ai;j , and it is customary since [1] to represent (16) yQ D y C h.b T ˝ I /F .Y /: a method by its tableau: j D1

S

1444

Symmetric Methods

4

1.071 q0 = 0 and p0 =2.034 q0 = 0 and p0 =2.033

3 1.0705 Hamiltonian

2

p

1 0 −1 −2

1.07

1.0695

1.069

−3 −10

−5

0 q

5

10

15

1.0685

0

2000

4000

6000 Time

8000

10000

Symmetric Methods, Fig. 3 Level sets of H (left) and evolution of H w.r.t. time for two different initial values

On the one hand, premultiplying (15) by P ˝ I and noticing that

 .P ˝ I /F .Y / D F .P ˝ I /Y ;

p 1 2

 1 2

1 2

C

p

15 10

p

15 10

5 36 5 36

C

5 36

C 5 18

p

15 24 p 15 30

2 9

 2 9

2 9

C

15 15

p

15 15

4 9

p 5 36



5 36



15 30 p 15 24

(17)

5 36 5 18

it is straightforward to see that ˚h;f can also be Methods for partitioned systems. For systems of the defined by coefficients PAP T and P b. On the other form hand, exchanging h and h, y, and y, Q it appears that   T ˚h;f is defined by coefficients A D eb  A and yP D f .z/; zP D g.y/; (18) b  D b. The flow ˚h;f is thus symmetric as soon as eb T  A D PAP and b D P b, which is nothing but condition (14). t it is natural to apply two different Runge-Kutta methu ods to variables y and z: Written in compact form, a partitioned Runge-Kutta method reads: Remark 3 For methods without redundant stages, condition (14) is also necessary.

Y D e ˝ y C h.A ˝ I /F .Z/;

Example 4 The implicit midpoint rule, defined by A D Z D e ˝ y C h.AO ˝ I /G.Y /; 1 and b D 1, is a symmetric method of order 2. More 2 generally, the s-stage Gauss collocation method based yQ D y C h.b T ˝ I /F .Z/; on the roots of the sth shifted Legendre polynomial is zQ D y C h.bO T ˝ I /G.Y /; a symmetric method of order 2s. For instance, the 2stage and 3-stage Gauss methods of orders 4 and 6 have O the following coefficients: O b/ and the method is symmetric if both .A; b/ and .A; are. An important feature of partitioned Runge-Kutta p p method is that they can be symmetric and explicit for 3 3 1 1 1 2  6 4 4  6 systems of the form (18). p p 3 3 1 1 1 C 6 C 6 2 4 4 Example 5 The Verlet method is defined by the fol1 2

1 2

lowing two Runge-Kutta tableaux:

Symmetric Methods

1445

0

0

0

1

1 2 1 2

1 2 1 2

and

1 2

1 2

0

1 2

1 2 1 2

0 1 2

as soon as 1 C : : : C s D 1, ‰h;f is of order at least p C 1 if (19) pC1 1 C : : : C spC1 D 0:

This observation is the key to triple jump compositions, as proposed by a series of authors [3,5,18,21]: Starting The method becomes explicit owing to the special from a symmetric method ˚h;f of (even) order 2q, the structure of the partitioned system: new method obtained for Z1 D z0 C h2 f .Y1 /; Y1 D y0 ; Y2 D y0 C hg.Z1 /; Z2 D Z1 ;

 y1 D Y2 ; z1 D z0 C h2 f .Y1 /Cf .Y2 /

1 D 3 D 2 D

1 2  21=.2qC1/

and

21=.2qC1/ 2  21=.2qC1/

The Verlet method is the most elementary method of is symmetric the class of partitioned Runge-Kutta methods known  as Lobatto IIIA-IIIB. Unfortunately, methods of higher ‰h;f D ˚1 h;f ı ˚2 h;f ı ˚3 h;f orders within this class are no longer explicit in genD ˚3 h;f ı ˚2 h;f ı ˚1 h;f D ‰h;f eral, even for the equations of the form (18). It is nevertheless possible to construct symmetric explicit and of order at least 2q C 1. Since the order of a Runge-Kutta methods, which turn out to be equivalent symmetric method is even, ‰h;f is in fact of order to compositions of Verlet’s method and whose intro2q C 2. The procedure can then be repeated recursively duction is for this reason postponed to the next section. to construct arbitrarily high-order symmetric methods Note that a particular instance of partitioned sys- of orders 2q C 2, 2q C 4, 2q C 6, : : :., with respectively tems are second-order differential equations of the 3, 9, 27, : : :, compositions of the original basic method ˚h;f . However, the construction is far from being the form most efficient, for the combined coefficients become yP D z; zP D g.y/; (20) large, some of which being negatives. A partial remedy is to envisage compositions with s D 5. We hereby give which covers many situations of practical interest (for the coefficients obtained by Suzuki [18]: instance, mechanical systems governed by Newton’s 1 law in absence of friction). and 1 D 2 D 4 D 5 D 1=.2qC1/ 44 41=.2qC1/ 3 D  4  41=.2qC1/ Symmetric Methods Obtained by

Composition Another class of symmetric methods is constituted of symmetric compositions of low-order methods. The idea consists in applying a basic method ˚h;f with a sequence of prescribed stepsizes: Given s real numbers 1 ; : : : ; s , its composition with stepsizes 1 h; : : : ; s h gives rise to a new method: ‰h;f D ˚s h;f ı : : : ı ˚1 h;f :

(21)

Noticing that the local error of ‰h;f , defined by ‰h;f .y/  'h;f .y/, is of the form pC1

.1

C : : : C spC1 /hpC1 C.y/ C O.hpC2 /;

which give rise to very efficient methods for q D 1 and q D 2. The most efficient high-order composition methods are nevertheless obtained by solving the full system of order conditions, i.e., by raising the order directly from 2 to 8, for instance, without going through the intermediate steps described above. This requires much more effort though, first to derive the order conditions and then to solve the resulting polynomial system. It is out of the scope of this article to describe the two steps involved, and we rather refer to the paper [15] on the use of 1B-series for order conditions and to Chap. V.3.2. of [10] for various examples and numerical comparisons. An excellent method of order 6 with 9 stages has been obtained by Kahan and Li [12] and we reproduce here its coefficients:

S

1446

Symmetric Methods 100 Kahan and Li Triple jump

10−1

Error at time t=100

10−2 10−3 10−4 10−5 10−6 10−7 10−8 10−9

105 Number of Verlet steps

1 D 9 D 0:3921614440073141;

qR D rVf ast .q/  rVslow .q/

(22)

2 D 8 D 0:3325991367893594; 3 D 7 D 0:7062461725576393; 4 D 6 D 0:0822135962935508; 5 D 0:7985439909348299:

where Vf ast and Vslow are two potentials acting on different time scales, typically such that r 2 Vf ast is positive semi-definite and kr 2 Vf ast k >> kr 2 Vslow k. Explicit standard methods suffer from severe stability restrictions due to the presence of high oscillations at the slow time scale and necessitate small steps and many evaluations of the forces. Since slow forces rVslow are in many applications much more expensive to evaluate than fast ones, efficient methods in this context are thus devised to require significantly fewer evaluations per step of the slow force.

For the sake of illustration, we have computed the solution of Kepler’s equations with this method and the method of order 6 obtained by the triple jump technique. In both cases, the basic method is Verlet’s scheme. The gain offered by the method of Kahan and Li is impressive (it amounts to two digits of accuracy on this example). Other methods can be found, for Example 6 In applications to molecular dynamics, for instance, fast forces deriving from Vf ast (short-range instance, in [10, 14]. interactions) are much cheaper to evaluate than slow Remark 4 It is also possible to consider symmetric forces deriving from Vslow (long-range interactions). compositions of nonsymmetric methods. In this situaOther examples of applications are presented in [11]. tion, raising the order necessitates to compose the basic method and its adjoint. Methods for general problems with nonlinear fast potentials. Introducing the variable p D qP in (22), the equation reads

Symmetric Methods for Highly Oscillatory Problems In this section, we present methods aimed at solving problems of the form

      p 0 qP D C 0 rq Vf ast .q/ pP ƒ‚ … „ƒ‚… „ƒ‚… „ yP

fK .y/

fF .y/

Symmetric Methods

1447

 0 C : rq Vslow .q/ „ ƒ‚ … 

(and/or symplectic) and allows for larger stepsizes. However, it still suffers from resonances and a better option is given by the mollified impulse methods, fS .y/ which considers the mollified potential VNslow .q/ D 0 The usual Verlet method [20] would consist in compos- Vslow .a.q// in loco of Vslow .q/, where a.q/ and a .q/ are averaged values given by ing the flows 'h;.fF CfS / and 'h;fK as follows: Z Z 1 h 1 h 0 ' h ;.fF CfS / ı 'h;fK ı ' h ;.fF CfS / x.s/ds; a .q/ D X.s/ds a.q/ D 2 2 h 0 h 0 or, if necessary, numerical approximations thereof and where would typically be restricted to very small stepsizes. The impulse method [6,8,19] combines the three pieces xR D rVf ast .x/; x.0/ D q; x.0/ P D p; of the vector field differently: 2 XR D r Vf ast .x/X; X.0/ D I; XP .0/ D 0: (23) ' h ;fS ı 'h;.fK CfF / ı ' h ;fS : 2 2 The resulting method uses the mollified force a0 .q/T .rq Vslow /.a.q// and is still symmetric (and/or Note that 'h;fS is explicit symplectic) provided (23) is solved with a symmetric     (and/or symplectic) method. q q D 'h;fS p  hrq Vslow .q/ p Methods for problems with quadratic fast potenwhile 'h;.fK CfF / may require to be approximated by a tials. In many applications of practical importance, numerical method ˚h;.fK CfF / which uses stepsizes that the potential Vf ast is quadratic of the form Vf ast .q/ D 1 T 2 are fractions of h. If ˚h;.fK CfF / is symmetric (and/or 2 q ˝ q. In this case, the mollified impulse method symplectic), the overall method is symmetric as well falls into the class of trigonometric symmetric methods of the form

0     1 @ p p  h D R.h˝/ ˆh q q 2

0 .h˝/rVslow

 .h˝/q0 C

1 .h˝/rVslow

 h .h˝/rVslow .h˝/q0

1 .h˝/q1 A

S

where R.h˝/ is the block matrix given by  R.h˝/ D

cos.h˝/ ˝ sin.h˝/ ˝ 1 sin.h˝/ cos.h˝/

and the functions , such that

,

sin.z/ 1 .z/; z .0/ D .0/ D 1:

0

and

1

interesting ones are



or

.z/ D

sin3 .z/ z3

sin .z/ , .z/ z2 sin.z/ (see [7]). z

.z/ D

, .z/ D

2

D 1 (see [9])

are even functions

Conclusion

This entry should be regarded as an introduction to the subject of symmetric methods. Several topics have not been exposed here, such as symmetric projection for ODEs on manifolds, DAEs of index 1 or 2, symmetVarious choices of functions and are possible ric multistep methods, symmetric splitting methods, and documented in the literature. Two particularly and symmetric Lie-group methods, and we refer the .z/ D

0 .z/

D cos.z/

1 .z/;

and

1448

Symmetries and FFT

interested reader to [10, 13, 16] for a comprehensive 20. Verlet, L.: Computer “experiments” on classical fluids. i. Thermodynamical properties of Lennard-Jones molecules. presentation of these topics.

Phys. Rev. 159, 98–103 (1967) 21. Yoshida, H.: Construction of higher order symplectic integrators. Phys. Lett. A 150, 262–268 (1990)

References 1. Butcher, J.C.: Implicit Runge-Kutta processes. Math. Comput. 18, 50–64 (1964) 2. Chartier, P., Hairer, E., Vilmart, G.: Algebraic structures of B-series. Found. Comput. Math. 10(4), 407–427 (2010) 3. Creutz, M., Gocksch, A.: Higher-order hybrid Monte Carlo algorithms. Phys. Rev. Lett. 63, 9–12 (1989) 4. Faou, E., Hairer, E., Pham, T.L.: Energy conservation with non-symplectic methods: examples and counter-examples. BIT 44(4), 699–709 (2004) 5. Forest, E.: Canonical integrators as tracking codes. AIP Conf. Proc. 184, 1106–1136 (1989) 6. Garc´ıa-Archilla, B., Sanz-Serna, J.M., Skeel, R.D.: Longtime-step methods for oscillatory differential equations. SIAM J. Sci. Comput. 20, 930–963 (1999) 7. Grimm, V., Hochbruck, M.: Error analysis of exponential integrators for oscillatory second-order differential equations. J. Phys. A 39, 5495–5507 (2006) 8. Grubm¨uller, H., Heller, H., Windemuth, A., Tavan, P.: Generalized Verlet algorithm for efficient molecular dynamics simulations with long-range interactions. Mol. Sim. 6, 121– 142 (1991) 9. Hairer, E., Lubich, C.: Long-time energy conservation of numerical methods for oscillatory differential equations. SIAM J. Numer. Anal. 38, 414–441 (2000) 10. Hairer, E., Lubich, C., Wanner, G.: Geometric Numerical Integration: Structure-Preserving Algorithms for Ordinary Differential Equations, 2nd edn. Springer Series in Computational Mathematics, vol. 31. Springer, Berlin (2006) 11. Jia, Z., Leimkuhler, B.: Geometric integrators for multiple time-scale simulation. J. Phys. A 39, 5379–5403 (2006) 12. Kahan, W., Li, R.C.: Composition constants for raising the orders of unconventional schemes for ordinary differential equations. Math. Comput. 66, 1089–1099 (1997) 13. Leimkuhler, B., Reich, S.: Simulating Hamiltonian Dynamics. Cambridge Monographs on Applied and Computational Mathematics, vol. 14. Cambridge University Press, Cambridge (2004) 14. McLachlan, R.I., Quispel, G.R.W.: Splitting methods. Acta Numer. 11, 341–434 (2002) 15. Murua, A., Sanz-Serna, J.M.: Order conditions for numerical integrators obtained by composing simpler integrators. Philos. Trans. R. Soc. Lond. A 357, 1079–1100 (1999) 16. Sanz-Serna, J.M., Calvo, M.P.: Numerical Hamiltonian Problems. Chapman & Hall, London (1994) 17. Stoffer, D.: On reversible and canonical integration methods. Technical Report SAM-Report No. 88-05, ETH-Z¨urich (1988) 18. Suzuki, M.: Fractal decomposition of exponential operators with applications to many-body theories and monte carlo simulations. Phys. Lett. A 146, 319–323 (1990) 19. Tuckerman, M., Berne, B.J., Martyna, G.J.: Reversible multiple time scale molecular dynamics. J. Chem. Phys. 97, 1990–2001 (1992)

Symmetries and FFT Hans Z. Munthe-Kaas Department of Mathematics, University of Bergen, Bergen, Norway

Synonyms Fourier transform; Group theory; Representation theory; Symmetries

Synopsis The fast Fourier transform (FFT), group theory, and symmetry of linear operators are mathematical topics which all connect through group representation theory. This entry provides a brief introduction, with emphasis on computational applications.

Symmetric FFTs The finite Fourier transform maps functions on a periodic lattice to a dual Fourier domain. Formally, let Zn D Zn1  Zn2      Zn` be a finite abelian (commutative) group, where Znj is the cyclic group Znj D f0; 1; : : : ; nj  1g with group operation C. mod nj / and n D .n1 ; : : : ; n` / is a multi-index with jnj D n1 n2    n` . Let CZn denote the linear space of complex-valued functions on Zn . The primal domain Zn is an `-dimensional lattice periodic in all directions, and the Fourier domain is b Zn D Zn (this is the Pontryagin dual group [12]). For infinite abelian groups, the primal and Fourier domains in general differ, such as Fourier series on the circle where R=Z D Z. The discrete Fourier transform (DFT) is an (up to Zn . Letting F .f /

scaling) unitary map F W CZn ! Cb b f , we have

b

Symmetries and FFT

1449

P such that 2R d2 D jGj. For example, the cyclic group Zn D f0; 1; : : : ; r  1g with group operation j2Zn C.modn/ has exactly n 1-dimensional irreps given as bn , for k D .k1 ; : : : ; k` / 2 Z (1) k .j / D exp.2 i kj=n/. A matrix A is equivariant with respect to a representation R of a group G if

 k` j` k1 j1 AR.g/ D R.g/A for all g 2 G. Any representation 1 Xb 2 i n CC n 1 ` f .j/ D ; f .k/e R can be block diagonalized, with irreducible reprejnj k2b Zn sentations on the diagonal. This provides a change of 1 is block diagonal for j D .j1 ; : : : ; j` / 2 Zn . (2) basis matrix F such that FAF for any R-equivariant A. This result underlies most of computational Fourier analysis and will be exemplified A symmetry for a function f 2 CZn is an R-linear by convolutional operators. map S W CZn ! CZn such that Sf D f . As examples, consider real symmetry SR f .j/ D f .j/, even symConvolutions in the Group Algebra metry Se f .j/ D f .j/ and odd symmetry So f .j/ D b has an adjoint The group algebra CG is the complex jGj-dimensional f .j/. If f has a symmetry S , then f vector space where the elements of G are the basis vecsymmetry b S D F S F 1 , example Sbe D Se , Sbo D So tors; equivalently CG consists of all complex-valued b b c and S R f .k/ D f .k/. The set of all symmetries functions on G. The product in G extends linearly to of f forms a group, i.e., the set of symmetries is the convolution product W CG  CG ! CG, given 1 closed under composition and inversion. Equivalently, as .a b/.g/ D P h2G a.h/b.h g/ for all g 2 G. the symmetries can be specified by defining an abstract The right regular representation of G on CG is, for group G and a map RW G ! LinR .CZn /, which for every h 2 G, a linear map R.h/W CG ! CG given each g 2 G defines an R-linear map R.g/ on CZn , as right translation R.h/a.g/ D a.gh/. A linear map such that R.gg0 / D R.g/R.g 0 / for all g; g 0 2 G. R is AW CG ! CG is convolutional (i.e., there exists an an example of a real representation of G. a 2 CG such that Ab D a b for all b 2 CG) if and The DFT on Zn is computed by the fast Fourier only if A is equivariant with respect to the right regular transform (FFT), costing O.jnj log.jnj// floating point representation. operations. It is possible to exploit may symmetries in the computation of the FFT, and for large classes of The Generalized Fourier Transform (GFT) symmetry groups, savings a factor jGj can be obtained The generalized Fourier transform [6, 10] and the compared to the nonsymmetric FFT. inverse are given as b.k/ D f

X



f .j/e

2 i

k` j` k1 j1 n1 CC n`



;

Equivariant Linear Operators and the GFT Representations Let G be a finite group with jGj elements. A dR dimensional unitary representation of G is a map RW G ! U.dR / such that R.gh/ D R.g/R.h/ for all g; h 2 G, where U.dR / is the set of dR  dR unitary matrices. More generally, a representation is a linear action of a group on a vector space. Two e are equivalent if there exists representations R and R e a matrix X such that R.g/ D XR.g/X 1 for all g 2 G. A representation R is reducible if it is equivalent to a block diagonal representation; otherwise it is irreducible. For any finite group G, there exists a complete list of nonequivalent irreducible representations R D f1 ; 2 ; : : : ; n g, henceforth called irreps,

b a./ D

X

a.g/.g/ 2 Cd d ; for all  2 R

(3)

g2G

S

  1 X a.g/ D d trace .g 1 /b a./ ; for all g 2 G. jGj 2R (4)

1

From the convolution formula a b./ D b a./b b./, we conclude: The GFT block-diagonalizes convolutional operators on CG. The blocks are of size d , the dimensions of the irreps. Equivariant Linear Operators More generally, consider a linear operator AW V ! V where V is a finite-dimensional vector space and A is equivariant with respect to a linear right action of G

1450

on V. If the action is free and transitive, then A is convolutional on CG. If the action is not transitive, then V splits in s separate orbits under the action of G, and A is a block-convolutional operator. In this case, the GFT block diagonalizes A with blocks of size approximately sd . The theory generalizes to infinite compact Lie groups via the Peter–Weyl theorem and to certain important non-compact groups (unimodular groups), such as the group of Euclidean transformations acting on Rn ; see [13].

Symmetries and FFT

imate symmetries) of the problem and employ the irreducible characters of the symmetry group and the GFT to (approximately) block diagonalize the operators. Such techniques are called domain reduction techniques [7]. Block diagonalization of linear operators has applications in solving linear systems, eigenvalue problems, and computation of matrix exponentials. The GFT has also applications in image processing, image registration, and computational statistics [1– 5, 8].

Applications Symmetric FFTs appear in many situations, such as real sine and cosine transforms. The 1-dim real cosine transform has four symmetries generated by SR and Se , and it can be computed four times faster than the full complex FFT. This transform is central in Chebyshev approximations. More generally, multivariate Chebyshev polynomials possess symmetries of kaleidoscopic reflection groups acting upon Zn [9, 11, 14]. Diagonalization of equivariant linear operators is essential in signal and image processing, statistics, and differential equations. For a cyclic group Zn , an equivariant A is a Toeplitz circulant, and the GFT is given by the discrete Fourier transform, which can be computed fast by the FFT algorithm. More generally, a finite abelian group Zn has jnj one-dimensional irreps, given by the exponential functions. Zn -equivariant matrices are block Toeplitz circulant matrices. These are diagonalized by multidimensional FFTs. Linear differential operators with constant coefficients typically lead to discrete linear operators which commute with translations acting on the domain. In the case of periodic boundary conditions, this yields block circulant matrices. For more complicated boundaries, block circulants may provide useful approximations to the differential operators. More generally, many computational problems possess symmetries given by a (discrete or continuous) group acting on the domain. For example, the Laplacian operator commutes with any isometry of the domain. This can be discretized as an equivariant discrete operator. If the group action on the discretized domain is free and transitive, the discrete operator is a convolution in the group algebra. More generally, it is a blockconvolutional operator. For computational efficiency, it is important to identify the symmetries (or approx-

References ˚ 1. Ahlander, K., Munthe-Kaas, H.: Applications of the generalized Fourier transform in numerical linear algebra. BIT Numer. Math. 45(4), 819–850 (2005) 2. Allgower, E., B¨ohmer, K., Georg, K., Miranda, R.: Exploiting symmetry in boundary element methods. SIAM J. Numer. Anal. 29, 534–552 (1992) 3. Bossavit, A.: Symmetry, groups, and boundary value problems. A progressive introduction to noncommutative harmonic analysis of partial differential equations in domains with geometrical symmetry. Comput. Methods Appl. Mech. Eng. 56(2), 167–215 (1986) 4. Chirikjian, G., Kyatkin, A.: Engineering Applications of Noncommutative Harmonic Analysis. CRC, Boca Raton (2000) 5. Diaconis, P.: Group Representations in Probability and Statistics. Institute of Mathematical Statistics, Hayward (1988) 6. Diaconis, P., Rockmore, D.: Efficient computation of the Fourier transform on finite groups. J. Am. Math. Soc. 3(2), 297–332 (1990) 7. Douglas, C., Mandel, J.: An abstract theory for the domain reduction method. Computing 48(1), 73–96 (1992) 8. F¨assler, A., Stiefel, E.: Group Theoretical Methods and Their Applications. Birkhauser, Boston (1992) 9. Hoffman, M., Withers, W.: Generalized Chebyshev polynomials associated with affine Weyl groups. Am. Math. Soc. 308(1), 91–104 (1988) 10. Maslen, D., Rockmore, D.: Generalized FFTs—a survey of some recent results. Publisher: In: Groups and Computation II, Am. Math. Soc. vol. 28, pp. 183–287 (1997) 11. Munthe-Kaas, H.Z., Nome, M., Ryland, B.N.: Through the Kaleidoscope; symmetries, groups and Chebyshev approximations from a computational point of view. In: Foundations of Computational Mathematics. Cambridge University Press, Cambridge (2012) 12. Rudin, W.: Fourier Analysis on Groups. Wiley, New York (1962) 13. Serre, J.: Linear Representations of Finite Groups, vol 42. Springer, New York (1977) 14. Ten Eyck, L.: Crystallographic fast Fourier transforms. Acta Crystallogr Sect. A Crystal Phys. Diffr. Theor. General Crystallogr. 29(2), 183–191 (1973)

Symplectic Methods

1451

Symplectic Methods J.M. Sanz-Serna Departamento de Matem´atica Aplicada, Universidad de Valladolid, Valladolid, Spain

Definition This entry, concerned with the practical task of integrating numerically Hamiltonian systems, follows up the entry  Hamiltonian Systems and keeps the notation and terminology used there. Each one-step numerical integrator is specified by a smooth map tH that advances the numerical nC1 ;tn solution from a time level tn to the next tnC1 .p nC1 ; q nC1 / D tH .p n ; q n /I nC1 ;tn

(1)

the superscript H refers to the Hamiltonian function H.p; qI t/ of the system being integrated. For instance for the explicit Euler rule  .p nC1 ; q nC1 / D .p n ; q n / C .tnC1  tn / f .p n ; q n I tn /;  g.p n ; q n I tn / I

of their order  – replace ˚ H by a nonsymplectic mapping  H . This is illustrated in Fig. 1 that corresponds to the Euler rule as applied to the harmonic oscillator pP D q, qP D p. The (constant) step size is tnC1  tn D 2=12. We have taken as a family of initial conditions the points of a circle centered at p D 1, q D 0 and seen the evolution after 1, 2, . . . , 12 steps. Clearly the circle, which should move clockwise without changing area, gains area as the integration proceeds: The numerical  H is not symplectic. As a result, the origin, a center in the true dynamics, is turned by the discretization procedure into an unstable spiral point, i.e., into something that cannot arise in Hamiltonian dynamics. For the implicit Euler rule, the corresponding integration loses area and gives rise to a family of smaller and smaller circles that spiral toward the origin. Again, such a stable focus is incompatible with Hamiltonian dynamics. This failure of well-known methods in mimicking Hamiltonian dynamics motivated the consideration of integrators that generate a symplectic mapping  H when applied to a Hamiltonian problem. Such methods are called symplectic or canonical. Since symplectic transformations also preserve volume, symplectic integrators applied to Hamiltonian problems are automatically volume preserving. On the other hand, while many important symplectic integrators are time-

here and later f and g denote the d -dimensional real vectors with entries @H=@qi , @H=@pi (d is the number of degrees of freedom) so that .f; g/ is the canonical vector field associated with H (in simpler words: the right-hand side of Hamilton’s equations). has to approxFor the integrator to make sense, tH nC1 ;tn imate the solution operator ˚tHnC1 ;tn that advances the true solution from its value at tn to its value at tnC1 :    p.tnC1 /; q.tnC1 / D ˚tHnC1 ;tn p.tn /; q.tn / :

4 3 2 1 p



5

0 −1

For a method of (consistency) order , tH differs   nC1 ;tn −2 from ˚tHnC1 ;tn in terms of magnitude O .tnC1  tn /C1 . The solution map ˚tHnC1 ;tn is a canonical (symplec−3 tic) transformation in phase space, an important fact −4 that substantially constrains the dynamics of the true   solution p.t/; q.t/ . If we wish the approximation −5 −4 −2 0 2 4  H to retain the “Hamiltonian” features of ˚ H , we q H should insist on  also being a symplectic transformation. However, most standard numerical integrators Symplectic Methods, Fig. 1 The harmonic oscillator inte– including explicit Runge–Kutta methods, regardless grated by the explicit Euler method

S

1452

reversible (symmetric), reversibility is neither sufficient nor necessary for a method to be symplectic ([8], Remark 6.5). Even though early examples of symplectic integration may be traced back to the 1950s, the systematic exploration of the subject started with the work of Feng Kang (1920–1993) in the 1980s. An early short monograph is [8] and later books are the comprehensive [5] and the more applied [6]. Symplectic integration was the first step in the larger endeavor of developing structure-preserving integrators, i.e., of what is now often called, following [7], geometric integration. Limitations of space restrict this entry to one-step methods and canonical Hamiltonian problems. For noncanonical Hamiltonian systems and multistep integrators the reader is referred to [5], Chaps. VII and XV.

Symplectic Methods

Runge–Kutta Methods Symplecticness Conditions When the Runge–Kutta (RK) method with s stages specified by the tableau

a11 :: : as1 b1

 :: :  

a1s :: : ass bs

(2)

is applied to the integration of the Hamiltonian system with Hamiltonian function H , the relation (1) takes the form p nC1 D p n C hnC1

s X

bi f .Pi ; Qi I tn C ci hnC1 /;

i D1

q nC1 D q n C hnC1

Integrators Based on Generating Functions

s X

bi g.Pi ; Qi I tn C ci hnC1 /;

i D1

P where ci D j aij are the abscissae, hnC1 D tnC1  tn The earliest systematic approaches by Feng Kang and is the step size and Pi , Qi , i D 1; : : : ; s are the internal others to the construction of symplectic integrators (see stage vectors defined through the system [5], Sect. VI.5.4 and [8], Sect. 11.2) exploited the fols X lowing well-known result of the canonical formalism: n D p C h aij f .Pj ; Qj I tn C cj hnC1 /; P i nC1 The canonical transformation ˚tHnC1 ;tn possesses a genj D1 erating function S2 that solves an initial value problem (3) for the associated Hamilton–Jacobi equation. It is then s X possible, by Taylor expanding that equation, to obtain n Q D q C h aij g.Pj ; Qj I tn C cj hnC1 /: H i nC1 e an approximation S 2 to S2 . The transformation tnC1 ;tn j D1 generated by e S 2 will automatically be canonical and (4) therefore will define a symplectic integrator. If e S2   differs from S2 by terms O .tnC1  tn /C1 , the inteLasagni, Sanz-Serna, and Suris proved that if the grator will be of order . Generally speaking, the highcoefficients of the method in (2) satisfy order methods obtained by following this procedure are more difficult to implement than those derived by the bi aij C bj aj i  bi bj D 0; i; j D 1; : : : ; s; (5) techniques discussed in the next two sections.

Runge–Kutta and Related Integrators

then the method is symplectic. Conversely ([8], Sect. 6.5), the relations (5) are essentially necessary for the method to be symplectic. Furthermore for symplectic RK methods the transformation (1) is in fact exact symplectic ([8], Remark 11.1).

In 1988, Lasagni, Sanz-Serna, and Suris (see [8], Chap. 6) discovered independently that some Order Conditions well-known families of numerical methods contain Due to symmetry considerations, the relations (5) impose s.s C 1/=2 independent equations on the s 2 C s symplectic integrators.

Symplectic Methods

elements of the RK tableau (2), so that there is no shortage of symplectic RK methods. The available free parameters may be used to increase the accuracy of the method. It is well known that the requirement that an RK formula has a target order leads to a set of nonlinear relations (order conditions) between the elements of the corresponding tableau (2). For order   there is an order condition associated with each rooted tree with   vertices and, if the aij and bi are free parameters, the order conditions are mutually independent. For symplectic methods however the tableau coefficients are constrained by (5), and Sanz-Serna and Abia proved in 1991 that then there are redundancies between the order conditions ([8], Sect. 7.2). In fact to ensure order   when (5) holds it is necessary and sufficient to impose an order condition for each so-called nonsuperfluous (nonrooted) tree with   vertices. Examples of Symplectic Runge–Kutta Methods Setting j D i in (5) shows that explicit RK methods (with aij D 0 for i  j ) cannot be symplectic. Sanz-Serna noted in 1988 ([8], Sect. 8.1) that the Gauss method with s stages, s D 1; 2; : : : , (i.e., the unique method with s stages that attains the maximal order 2s) is symplectic. When s D 1 the method is the familiar implicit midpoint rule. Since for all Gauss methods the matrix .aij / is full, the computation of the stage vectors Pi and Qi require, at each step, the solution of the system (3) and (4) that comprises s  2d scalar equations. In non-stiff situations this system is readily solved by functional iteration, see [8] Sects. 5.4 and 5.5 and [5] Sect. VIII.6, and then the Gauss methods combine the advantages of symplecticness, easy implementation, and high order with that of being applicable to all canonical Hamiltonian systems. If the system being solved is stiff (e.g., it arises through discretization of the spatial variables of a Hamiltonian partial differential equation), Newton iteration has to be used to solve the stage equations (3) and (4), and for high-order Gauss methods the cost of the linear algebra may be prohibitive. It is then of interest to consider the possibility of diagonally implicit symplectic RK methods, i.e., methods where aij D 0 for i < j and therefore (3) and (4) demand the successive solution of s systems of dimension 2d , rather than that of a single .s  2d /–dimensional system. It turns out ([8], Sect. 8.2) that such methods are necessarily composition methods (see below)

1453

obtained by concatenating implicit midpoint sub-steps of lengths b1 hnC1 , . . . , bs hnC1 . The determination of the free parameters bi is a task best accomplished by means of the techniques used to analyze composition methods. The B-series Approach In 1994, Calvo and Sanz-Serna ([5], Sect. VI.7.2) provided an indirect technique for the derivation of the symplecticness conditions (5). The first step is to identify conditions for the symplecticness of the associated B-series (i.e., the series that expands the transformation (1)) in powers of the step size. Then the conditions (on the B-series) obtained in this way are shown to be equivalent to (5). This kind of approach has proved to be very powerful in the theory of geometric integration, where extensive use is made of formal power series.

Partitioned Runge–Kutta Methods Partitioned Runge–Kutta (PRK) methods differ from standard RK integrators in that they use two tableaux of coefficients of the form (2): one to advance p and the other to advance q. Most developments of the theory of symplectic RK methods are easily adapted to cover the partitioned situation, see e.g., [8], Sects. 6.3, 7.3, and 8.4. The main reason ([8], Sect. 8.4) to consider the class of PRK methods is that it contains integrators that are both explicit and symplectic when applied to separable Hamiltonian systems with H.p; qI t/ D T .p/CV .qI t/, a format that often appears in the applications. It turns out ([8], Remark 8.1, [5], Sect. VI.4.1, Theorem 4.7) that such explicit, symplectic PRK methods may always be viewed as splitting methods (see below). Moreover it is advantageous to perform their analysis by interpreting them as splitting algorithms. ¨ Methods Runge–Kutta–Nystrom In the special but important case where the (separable) Hamiltonian is of the form H D .1=2/p T M 1 p C V .qI t/ (M a positive-definite symmetric matrix) the canonical equations d p D rV .qI t/; dt lead to

d q D M 1 p dt

d2 q D M 1 rV .qI t/; dt 2

(6)

S

1454

Symplectic Methods

a second-order system whose right-hand side is independent of .d=dt/q. Runge–Kutta–Nystr¨om (RKN) methods may then be applied to the second-order form and are likely to improve on RK integrations of the original first-order system (6). There are explicit, symplectic RKN integrators ([8], Sect. 8.5). However their application (see [8], Remark 8.5) is always equivalent to the application of an explicit, symplectic PRK method to the firstorder equations (6) and therefore – in view of a consideration made above – to the application of a splitting algorithm.

A particular case of (7) of great practical significance is provided by the separable Hamiltonian H.p; q/ D T .p/ C V .q/ with H1 D T , H2 D V ; the flows associated with H1 and H2 are respectively given by         p; q 7! p; q C trT .p/ ; p; q 7! p  trV .q/; q : Thus, in this particular case the scheme (7) reads p nC1 D p n  hrV .q nC1 /;

q nC1 D q n C hrT .p n /; (8)

and it is sometimes called the symplectic Euler rule (it is obviously possible to interchange the roles of p and q). Alternatively, (8) may be considered as a one-stage, explicit, symplectic PRK integrator as in [8], Sect. 8.4.3. As a second example of splitting, one may consider (nonseparable) formats H D H1 .p; q/CV  .q/, where the Hamiltonian system associated with H1 can be integrated in closed form. For instance, H1 may correspond to a set of uncoupled harmonic oscillators and V  .q/ represent the potential energy of the interactions between oscillators. Or H1 may correspond to the Keplerian motion of a point mass attracted to a fixed gravitational center and V  be a potential describing some sort of perturbation.

Integrators Based on Splitting and Composition

The related ideas of splitting and composition are extremely fruitful in deriving practical symplectic integrators in many fields of application. The corresponding methods are typically ad hoc for the problem at hand and do not enjoy the universal off-the-shelf applicability of, say, Gaussian RK methods; however, when applicable, they may be highly efficient. In order to simplify the exposition, we assume hereafter that the Hamiltonian H is time-independent H D H.p; q/; we write hHnC1 and hHnC1 rather than ˚tHnC1 ;tn and tH . Furthermore, we shall denote the time step nC1 ;tn by h omitting the possible dependence on the step Strang Splitting number n. With the notation in (7), the symmetric Strang formula ([8], Sect. 12.4.3, [5], Sect. II.5) Splitting Simplest Splitting The easiest possibility of splitting occurs when the Hamiltonian H may be written as H1 C H2 and the Hamiltonian systems associated with H1 and H2 may be explicitly integrated. If the corresponding flows are denoted by tH1 and tH2 , the recipe (Lie–Trotter splitting, [8], Sect. 12.4.2, [5], Sect. II.5) H h

D

hH2

ı

hH1

N hH D H2 ı H1 ı H2 h=2 h h=2

defines a time-reversible, second-order symplectic integrator N hH that improves on the first order (7). In the separable Hamiltonian case H D T .p/ C V .q/, (9) leads to p nC1=2 D p n 

(7)

(9)

h rV .q n /; 2

q nC1 D q n C hrT .p nC1=2 /;

defines the map (1) of a first-order integrator that h p nC1 D p nC1=2  rV .q nC1 /: is symplectic (the mappings being composed in the 2 right-hand side are Hamiltonian flows and therefore symplectic). Splittings of H in more than two pieces This is the St¨ormer–Leapfrog–Verlet method that plays a key role in molecular dynamics [6]. It is also possible are feasible but will not be examined here.

Symplectic Methods

1455

to regard this integrator as an explicit, symplectic PRK with two stages ([8], Sect. 8.4.3). More Sophisticated Formulae A further generalization of (7) is H2 H1 2 ˇHs2h ı ˛Hs 1h ı ˇHs1 h ı    ı ˇ1 h ı ˛1 h

within the field of geometric integration, where it is frequently not difficult to write down first- or secondorder integrators with good conservation properties. A useful example, due to Suzuki, Yoshida, and others (see [8], Sect. 13.1), is as follows. Let hH be a time-reversible integrator that we shall call the basic bH (10) method and define the composition method h by

P P where the coefficients ˛i and ˇi , i ˛i D 1, i ˇi D 1, are chosen so as to boost the order  of the method. A systematic treatment based on trees of the required order conditions was given by Murua and Sanz-Serna in 1999 ([5], Sect. III.3). There has been much recent activity in the development of accurate splitting coefficients ˛i , ˇi and the reader is referred to the entry  Splitting Methods in this encyclopedia. In the particular case where the splitting is given by H D T .p/ C V .q/, the family (10) provides the most general explicit, symplectic PRK integrator. Splitting Combined with Approximations In (7), (9), or (10) use is made of the exact solution flows tH1 and tH2 . Even if one or both of these flows are not available, it is still possible to employ the idea of splitting to construct symplectic integrators. A simple example will be presented next, but many others will come easily to mind. Assume that we wish to use a Strang-like method but tH1 is not available. We may then advance the numerical solution via H2 H2 1 h=2 ı bH h ı h=2 ;

(11)

1 where bH h denotes a consistent method for the integration of the Hamiltonian problem associated with H1 . 1 If bH h is time-reversible, the composition (11) is also time-reversible and hence of order  D 2 (at least). 1 And if bH h is symplectic, (11) will define a symplectic method.

Composition A step of a composition method ([5], Sect. II.4) consists of a concatenation of a number of sub-steps performed with one or several simpler methods. Often the aim is to create a high-order method out of loworder integrators; the composite method automatically inherits the conservation properties shared by the methods being composed. The idea is of particular appeal

bH h D

H ˛h

ı

H .12˛/h

ı

H ˛h I

if the basic method is symplectic, then bH h will obviously be a symplectic method. It may be proved that, if ˛ D .1=3/.2 C 21=3 C 21=3 /, then bH h will have order  D 4. By using this idea one may perform symplectic, fourth-order accurate integrations while really implementing a simpler second-order integrator. The approach is particularly attractive when the direct application of a fourth-order method (such as the two-stage Gauss method) has been ruled out on implementation grounds, but a suitable basic method (for instance the implicit midpoint rule or a scheme derived by using Strang splitting) is available. If the (time-reversible) basic method is of order 2 1  then bH and ˛ D 221=.2C1/ h will have order  D 2C2; the recursive application of this idea shows that it is possible to reach arbitrarily high orders starting from a method of order 2. For further possibilities, see the entry  Composition Methods and [8], Sect. 13.1, [5], Sects. II.4 and III.3.

The Modified Hamiltonian The properties of symplectic integrators outlined in the next section depend on the crucial fact that, when a symplectic integrator is used, a numerical solution of the Hamiltonian system with Hamiltonian H may be viewed as an (almost) exact solution of a Hamiltonian e (the so-called system whose Hamiltonian function H modified Hamiltonian) is a perturbation of H . An example. Consider the application of the symplectic Euler rule (8) to a one-degree-of-freedom system with separable Hamiltonian H D T .p/ C V .q/. In order to describe the behavior of the points .p n ; q n / computed by the algorithm, we could just say that they approximately behave like the solutions .p.tn /; q.tn // of the Hamiltonian system S being integrated. This would not be a very precise description because the

S

1456

Symplectic Methods 2.5 2

d q D g.p/ dt (12)

∗

∗ 0.5 0

∗

∗

∗

∗

∗

∗

−1

∗

∗

−1.5

∗

∗

∗

−2 −2.5

where we recognize the Hamiltonian system with (hdependent!) Hamiltonian

∗

∗ 1

−0.5

h d p D f .q/ C g.p/f 0 .q/; dt 2 h  g 0 .p/f .q/; 2

∗

∗

1.5

p

true flow hH and its numerical approximation hH differ in O.h2 / terms. Can we find another differential system S2 (called a modified system) so that (8) is consistent of the second order with S2 ? The points .p n ; q n / would then be closer to the solutions of S2 than to the solutions of the system S we want to integrate. Straightforward Taylor expansions ([8], Sect. 10.1) lead to the following expression for S2 (recall that f D @H=@q, g D @H=@p)

−2

−1

0 q

1

2

Symplectic Methods, Fig. 2 Computed points, true trajectory (dotted line) and modified trajectory (dash–dot line)

e h2 D T .p/ C V .q/ C h T 0 .p/V 0 .q/ D H C O.h/: H 2 (13) Figure 2 corresponds to the pendulum equations g.p/ D p, f .q/ D  sin q with initial condition p.0/ D 0, q.0/ D 2. The stars plot the numerical solution with h D 0:5. The dotted line H D constant provides the true pendulum solution. The dash–dot line e h D constant gives the solution of the modified H 2 system (12). The agreement of the computed points with the modified trajectory is very good. The origin is a center of the modified system (recall that a small Hamiltonian perturbation of a Hamiltonian center is still a center); this matches the fact that, in the plot, the computed solution does not spiral in or out. On the other hand, the analogous modified system for the (nonsymplectic) integration in 1 is found not be a Hamiltonian system, but rather a system with negative dissipation: This agrees with the spiral behavior observed there. By adding extra O.h2 / terms to the right-hand sides of (12), it is possible to construct a (more accurate) modified system S3 so that (8) is consistent of the third order with S3 ; thus, S3 would provide an even better description of the numerical solution. The procedure may be iterated to get modified systems S4 , S5 , . . . and all of them turn out to be Hamiltonian.

General case. Given an arbitrary Hamiltonian system with a smooth Hamiltonian H , a consistent symplectic integrator hH and an arbitrary integer  > 0, it is possible ([8], Sect. 10.1) to construct a modified e h , Hamiltonian system S with Hamiltonian function H e Hh such that hH differs from the flow h  in O.hC1 / e h may be chosen as a polynomial of terms. In fact, H degree <  in h; the term independent of h coincides with H (cf. (13)) and for a method of order  the terms in h, . . . , h1 vanish. e h ,  D 2; 3; : : : are the The polynomials in h H partial sums of a series in powers of h. Unfortunately this series does not in general converge for fixed h, e Hh so that, in particular, the modified flows h  cannot converge to hH as  " 1. Therefore, in general, it Hh e h such that e is impossible to find a Hamiltonian H h coincides exactly with the integrator hH . Neishtadt ([8], Sect. 10.1) proved that by retaining for each h > 0 a suitable number N D N.h/ of terms of the e h such series it is possible to obtain a Hamiltonian H h H that he differs from hH in an exponentially small quantity. Here is the conclusion for the practitioner: For a symplectic integrator applied to an autonomous

Symplectic Methods

Hamiltonian system, modified autonomous Hamiltonian problems exist so that the computed points lie “very approximately” on the exact trajectories of the modified problems. This makes possible a backward error interpretation of the numerical results: The computed solutions are solving “very approximately” a nearby Hamiltonian problem. In a modeling situation where the exact form of the Hamiltonian H may be in doubt, or some coefficients in H may be the result of experimental measurements, the fact that integrating the model numerically introduces perturbations to H comparable to the uncertainty in H inherent in the model is the most one can hope for. On the other hand, when a nonsymplectic formula is used the modified systems are not Hamiltonian: The process of numerical integration perturbs the model in such a way as to take it out of the Hamiltonian class. Variable steps. An important point to be noted is as follows: The backward error interpretation only holds if the numerical solution after n steps is computed by iterating n times one and the same symplectic map. If, alternatively, one composes n symplectic maps (one from t0 to t1 , a different one from t1 to t2 , etc.) the backward error interpretation is lost, because the modified system changes at each step ([8], Sect. 10.1.3). As a consequence, most favorable properties of symplectic integrators (and of other geometric integrators) are lost when they are naively implemented with variable step sizes. For a complete discussion of this difficulty and of ways to circumvent it, see [5], Sects. VIII 1–4. Finding explicitly the modified Hamiltonians. The existence of a modified Hamiltonian system is a general result that derives directly from the symplecticness of the transformation hH ([8], Sect. 10.1) and does not require any hypothesis on the particular nature of such a transformation. However, much valuable information may be derived from the explicit construction of the modified Hamiltonians. For RK and related methods, e h ’s was first a way to compute systematically the H described by Hairer in 1994 and then by Calvo, Murua, and Sanz-Serna ([5], Sect. IX.9). For splitting and e h ’s may be obtained by composition integrators, the H use of the Baker–Campbell–Hausdorff formula ([8], Sect. 12.3, [5], Sect. III.4) that provides a means to express as a single flow the composition of two flows.

1457

This kind of research relies very much on concepts and techniques from the theory of Lie algebras.

Properties of Symplectic Integrators We conclude by presenting an incomplete list of favorable properties of symplectic integrators. Note that the advantage of symplecticness become more prominent as the integration interval becomes longer. Conservation of energy. For autonomous Hamiltonians, the value of H is of course a conserved quantity and the invariance of H usually expresses conservation of physical energy. Ge and Marsden proved in 1988 ([8], Sect. 10.3.2) that the requirements of symplecticness and exact conservation of H cannot be met simultaneously by a bona fide numerical integrator. Nevertheless, symplectic integrators have very good energy behavior ([5], Sect. IX.8): Under very general hypotheses, for a symplectic integrator of order : H.p n ; q n / D H.p 0 ; q 0 / C O.h /, where the constant implied in the O notation is independent of n over exponentially long time intervals nh   exp h0 =.2h/ . Linear error growth in integrable systems. For a Hamiltonian problem that is integrable in the sense of the Liouville–Arnold theorem, it may be proved ([5], Sect. X.3) that, in (long) time intervals of length proportional to h , the errors in the action variables are of magnitude O.h / and remain bounded, while the errors in angle variables are O.h / and exhibit a growth that is only linear in t. By implication the error growth in the components of p and q will be O.h / and grow, at most, linearly. Conventional integrators, including explicit Runge–Kutta methods, typically show quadratic error growth in this kind of situation and therefore cannot be competitive in a sufficiently long integration. KAM theory. When the system is closed to integrable, the KAM theory ([5], Chap. X) ensures, among other things, the existence of a number of invariant tori that contribute to the stability of the dynamics (see [8], Sect. 10.4 for an example). On each invariant torus the motion is quasiperiodic. Symplectic integrators ([5], Chap. X, Theorem 6.2) possess invariant tori O.h / close to those of the system being integrated and

S

1458

Systems Biology, Minimalist vs Exhaustive Strategies

furthermore the dynamics on each invariant torus is Systems Biology, Minimalist vs conjugate to its exact counterpart. Exhaustive Strategies Linear error growth in other settings. Integrable systems are not the only instance where symplectic Jeremy Gunawardena integrators lead to linear error growth. Other cases Department of Systems Biology, Harvard Medical include, under suitable hypotheses, periodic orbits, School, Boston, MA, USA solitons, relative equilibria, etc., see, among others, [1–4].

Introduction

Cross-References  B-Series  Composition Methods  Euler Methods, Explicit, Implicit, Symplectic  Gauss Methods  Hamiltonian Systems  Molecular Dynamics  Nystr¨om Methods  One-Step Methods, Order, Convergence  Runge–Kutta Methods, Explicit, Implicit  Symmetric Methods

References 1. Cano, B., Sanz-Serna, J.M.: Error growth in the numerical integration of periodic orbits, with application to Hamiltonian and reversible systems. SIAM J. Numer. Anal. 34, 1391–1417 (1997) 2. de Frutos, J., Sanz-Serna, J.M.: Accuracy and conservation properties in numerical integration: the case of the Korteweg-deVries equation. Numer. Math. 75, 421–445 (1997) 3. Duran, A., Sanz-Serna, J.M.: The numerical integration of relative equilibrium solution. Geometric theory. Nonlinearity 11, 1547–1567 (1998) 4. Duran, A., Sanz-Serna, J.M.: The numerical integration of relative equilibrium solutions. The nonlinear Schroedinger equation. IMA. J. Numer. Anal. 20, 235–261 (1998) 5. Hairer, E., Lubich, Ch., Wanner, G.: Geometric Numerical Integration, 2nd edn. Springer, Berlin (2006) 6. Leimkuhler, B., Reich, S.: Simulating Hamiltonian Dynamics. Cambridge University Press, Cambridge (2004) 7. Sanz-Serna, J.M.: Geometric integration. In: Duff, I.S., Watson, G.A. (eds.) The State of the Art in Numerical Analysis. Clarendon Press, Oxford (1997) 8. Sanz-Serna, J.M., Calvo, M.P.: Numerical Hamiltonian Problems. Chapman & Hall, London (1994)

Systems biology may be defined as the study of how physiology emerges from molecular interactions [11]. Physiology tells us about function, whether at the organismal, tissue, organ or cellular level; molecular interactions tell us about mechanism. How do we relate mechanism to function? This has always been one of the central problems of biology and medicine but it attains a particular significance in systems biology because the molecular realm is the base of the biological hierarchy. Once the molecules have been identified, there is nowhere left to go but up. This is an enormous undertaking, encompassing, among other things, the development of multicellular organisms from their unicellular precursors, the hierarchical scales from molecules to cells, tissues, and organs, and the nature of malfunction, disease, and repair. Underlying all of this is evolution, without which biology can hardly be interpreted. Organisms are not designed to perform their functions, they have evolved to do so—variation, transfer, drift, and selection have tinkered with them over 3:5  109 years—and this has had profound implications for how their functions have been implemented at the molecular level [12]. The mechanistic viewpoint in biology has nearly always required a strongly quantitative perspective and therefore also a reliance on quantitative models. If this trend seems unfamiliar to those who have been reared on molecular biology, it is only because our historical horizons have shrunk. The quantitative approach would have seemed obvious to physiologists, geneticists, and biochemists of an earlier generation. Moreover, quantitative methods wax and wane within an individual discipline as new experimental techniques emerge and the focus shifts between the descriptive and the functional. The great Santiago Ram´on y Cajal, to whom we owe the conception of the central nervous system as a network of neurons, classified “theorists” with “contemplatives, bibliophiles and polyglots,

Systems Biology, Minimalist vs Exhaustive Strategies

megalomaniacs, instrument addicts, misfits” [3]. Yet, when Cajal died in 1934, Alan Hodgkin was already starting down the road that would lead to the HodgkinHuxley equations. In a similar way, the qualitative molecular biology of the previous era is shifting, painfully and with much grinding of gears, to a quantitative systems biology. What kind of mathematics will be needed to support this? Here, we focus on the level of abstraction for modelling cellular physiology at the molecular level. This glosses over many other relevant and hard problems but allows us to bring out some of the distinctive challenges of molecularity. We may caricature the current situation in two extreme views. One approach, mindful of the enormous complexity at the molecular level, strives to encompass that complexity, to dive into it, and to be exhaustive; the other, equally mindful of the complexity but with a different psychology, strives to abstract from it, to rise above it, and to be minimalist. Here, we examine the requirements and the implications of both strategies.

Models as Dynamical Systems Many different kinds of models are available for describing molecular systems. It is convenient to think of each as a dynamical system, consisting of a description of the state of the system along with a description of how that state changes in time. The system state typically amalgamates the states of various molecular components, which may be described at various levels of abstraction. For instance, Boolean descriptions are often used by experimentalists when discussing gene expression: this gene is ON, while that other is OFF. Discrete dynamic models can represent time evolution as updates determined by Boolean functions. At the other end of the abstraction scale, gene expression may be seen as a complex stochastic process that takes place at an individual promoter site on DNA: states may be described by the numbers of mRNA molecules and the time evolution may be described by a stochastic master equation. In a different physiological context, Ca2C ions are a “second messenger” in many key signalling networks and show complex spatial and temporal behaviour within an individual cell. The state may need to be described as a concentration that varies in space and time and the time evolution by a partial differential equation. In the most widely-used form

1459

of dynamical system, the state is represented by the (scalar) concentrations of the various molecular components in specific cellular compartments and the time evolution by a system of coupled ordinary differential equations (ODEs). What is the right kind of model to use? That depends entirely on the biological context, the kind of biological question that is being asked and on the experimental capabilities that can be brought to bear on the problem. Models in biology are not objective descriptions of reality; they are descriptions of our assumptions about reality [2]. For our purposes, it will be simplest to discuss ODE models. Much of what we say applies to other classes of models. Assume, therefore, that the system is described by the concentrations within specific cellular compartments of n components, x1 ;    ; xn , and the time evolution is given, in vector form, by dx=dt D f .xI a/. Here, a 2 Rm is a vector of parameters. These may be quantities like proportionality constants in rate laws. They have to take numerical values before the dynamics on the state space can be fully defined and thereby arises the “parameter problem” [6]. In any serious model, most of the parameter values are not known, nor can they be readily determined experimentally. (Even if some of them can, there is always a question of whether an in-vitro measurement reflects the in-vivo context.) The dynamical behaviour of a system may depend crucially on the specific parameter values. As these values change through a bifurcation, the qualitative “shape” of the dynamics may alter drastically; for instance, steady states may alter their stability or appear or disappear [22]. In between bifurcations, the shape of the dynamics only alters in a quantitative way while the qualitative portrait remains the same. The geography of parameter space therefore breaks up into regions; within each region the qualitative portrait of the dynamics remains unaltered, although its quantitative details may change, while bifurcations take place between regions resulting in qualitative changes in the dynamics (Fig. 1).

Parameterology We see from this that parameter values matter. They are typically determined by fitting the model to experimental data, such as time series for the concentrations of

S

1460

Systems Biology, Minimalist vs Exhaustive Strategies a1x1(1 – x1)

b

a2x1

a dx1 dt

= f1(x1)

0

= a1x1(1 – x1) – a2x1

x1

1 – a2/a1 f1(x1)

c

a1 ≤ a2 a1 a1 > a2 a2

0

x1

0

1 – a2/a1

x1

Systems Biology, Minimalist vs Exhaustive Strategies, Fig. 1 The geography of parameter space. (a) A dynamical system with one state variable, x1 and two parameters, a1 , a2 . (b) Graphs of f1 .x1 / (dashed curve) and of the terms in it (solid curves), showing the steady states, where dx1 =dt D f1 .x1 / D 0. (c) Parameter space breaks up into two regions: (1) a1  a2 , in which the state space has a single stable steady state at x1 D 0 to which any positive initial condition tends (arrow); and (2)

a1 > a2 , in which there are two steady states, a positive stable state at x1 D 1  a2 =a1 , to which any positive initial conditions tends (arrows), and an unstable state at x1 D 0. Here, the state space is taken to be the nonnegative real line. A magenta dot indicates a stable state and a cyan dot indicates an unstable state. The dynamics in the state space undergoes a transcritical bifurcation at a1 D a2 [22]

some of the components. The fitting may be undertaken by minimizing a suitable measure of discrepancy between the calculated and the observed data, for which several nonlinear optimization algorithms are available [10]. Empirical studies on models with many (>10) parameters have revealed what could be described as a “80/20” rule [9, 19]. Roughly speaking, 20 % of the parameters are well constrained by the data or “stiff”: They cannot be individually altered by much without significant discrepancy between calculated and observed data. On the other hand, 80 % of the parameters are poorly constrained or “sloppy,” they can be individually altered by an order of magnitude or more, without a major impact on the discrepancy. The minimization landscape, therefore, does not have a single deep hole but a flat valley with rather few dimensions orthogonal to the valley. The fitting should have localized the valley within one of the parameter regions. At present, no theory accounts for the emergence of these valleys. Two approaches can be taken to this finding. On the one hand, one might seek to constrain the parameters

further by acquiring more data. This raises an interesting problem of how best to design experiments to efficiently constrain the data. Is it better to get more of the same data or to get different kinds of data? On the other hand, one might seek to live with the sloppiness, to acknowledge that the fitted parameter values may not reflect the actual ones but nevertheless seek to draw testable conclusions from them. For instance, the stiff parameters may suggest experimental interventions whose effects are easily observable. (Whether they are also biological interesting is another matter.) There may also be properties of the system that are themselves insensitive, or “robust,” to the parameter differences. One can, in any case, simply draw conclusions based on the fits and seek to test these experimentally. A successful test may provide some encouragement that the model has captured aspects of the mechanism that are relevant to the question under study. However, models are working hypotheses, not explanations. The conclusion that is drawn may be correct but that may

Systems Biology, Minimalist vs Exhaustive Strategies

be an accident of the sloppy parameter values or the particular assumptions made. It may be correct for the wrong reasons. Molecular complexity is such that there may well be other models, based on different assumptions, that lead to the same conclusions. Modellers often think of models as finished entities. Experimentalists know better. A model is useful only as a basis for making a better model. It is through repeated tests and revised assumptions that a firmly grounded, mechanistic understanding of molecular behaviour slowly crystallizes. Sometimes, one learns more when a test is not successful because that immediately reveals a problem with the assumptions and stimulates a search to correct them. However, it may be all too easy, because of the sloppiness in the fitting, to refit the model using the data that invalidated the conclusions and then claim that the newly fitted model “accounts for the new data.” From this, one learns nothing. It is better to follow Popperian principles and to specify in advance how a model is to be rejected. If a model cannot be rejected, it cannot tell you anything. In other areas of experimental science, it is customary to set aside some of the data to fit the model and to use another part of the data, or newly acquired data, to assess the quality of the model. In this way, a rejection criterion can be quantified and one can make objective comparisons between different models. The kind of approaches sketched above only work well when modelling and experiment are intimately integrated [18,21]. As yet, few research groups are able to accomplish this, as both aspects require substantial, but orthogonal, expertise as well as appropriate integrated infrastructure for manipulating and connecting data and models.

Model Simplification As pointed out above, complexity is a relative matter. Even the most complex model has simplifying assumptions: components have been left out; posttranslational modification states collapsed; complex interactions aggregated; spatial dimensions ignored; physiological context not made explicit. And these are just some of the things we know about, the “known unknowns.” There are also the “unknown unknowns.” We hope that what has been left out is not relevant to the question

1461

being asked. As always, that is a hypothesis, which may or may not turn out to be correct. The distinction between exhaustive and minimal models is therefore more a matter of scale than of substance. However, having decided upon a point on the scale, and created a model of some complexity, there are some systematic approaches to simplifying it. Here, we discuss just two. One of the most widely used methods is separation of time scales. A part of the system is assumed to be working significantly faster than the rest. If the faster part is capable of reaching a (quasi) steady state, then the slower part is assumed to see only that steady state and not the transient states that led to it. In some cases, this allows variables within the faster part to be eliminated from the dynamics. Separation of time scales appears in Michaelis and Menten’s pioneering study of enzyme-catalysed reactions. Their famous formula for the rate of an enzyme arises by assuming that the intermediate enzymesubstrate complex is in quasi-steady state [8]. Although the enzyme-substrate complex plays an essential role, it has been eliminated from the formula. The KingAltman procedure formalizes this process of elimination for complex enzyme mechanisms with multiple intermediates. This is an instance of a general method of linear elimination underlying several well-known formulae in enzyme kinetics, in protein allostery and in gene transcription, as well as more modern simplifications arising in chemical reaction networks and in multienzyme posttranslational modification networks [7]. An implicit assumption is often made that, after elimination, the behaviour of the simplified dynamical system approximates that of the original system. The mathematical basis for confirming this is through a singular perturbation argument and Tikhonov’s Theorem [8], which can reveal the conditions on parameter values and initial conditions under which the approximation is valid. It must be said that, aside from the classical Michaelis–Menten example, few singular perturbation analyses have been undertaken. Biological intuition can be a poor guide to the right conditions. In the Michaelis–Menten case, for example, the intuitive basis for the quasi-steady state assumption is that under in vitro conditions, substrate, S , is in excess over enzyme, E: St ot  Et ot . However, singular perturbation reveals a broader region, St ot C KM  Et ot , where KM is the Michaelis–Menten constant, in

S

1462

which the quasi-steady state approximation remains valid [20]. Time-scale separation has been widely used but cannot be expected to provide dramatic reductions in complexity; typically, the number of components are reduced twofold, not tenfold. The other method of simplification is also based on an old trick: linearization in the neighbourhood of a steady state. The Hartman–Grobman Theorem for a dynamical system states that, in the local vicinity of a hyperbolic steady state—that is, one in which none of the eigenvalues of the Jacobian have zero real part— the nonlinear dynamics is qualitatively similar to the dynamics of the linearized system, dy=dt D .Jf /ss y, where y D x  xss is the offset relative to the steady state, xss , and .Jf /ss is the Jacobian matrix for the nonlinear system, dx=dt D f .x/, evaluated at the steady state. Linearization simplifies the dynamics but does not reduce the number of components. Straightforward linearization has not been particularly useful for analysing molecular networks, because it loses touch with the underlying network structure. However, control engineering provides a systematic way to interrogate the linearized system and, potentially, to infer a simplified network. Such methods were widely used in physiology in the cybernetic era [5], and are being slowly rediscovered by molecular systems biologists. They are likely to be most useful when the steady state is homeostatically maintained. That is, when the underlying molecular network acts like a thermostat to maintain some internal variable within a narrow range, despite external fluctuations. Cells try to maintain nutrient levels, energy levels, pH, ionic balances, etc., fairly constant, as do organisms in respect of Claude Bernard’s “milieu int´erieure”; chemotaxing E. coli return to a constant tumbling rate after perturbation by attractants or repellents [23]; S. cerevisiae cells maintain a constant osmotic pressure in response to external osmotic shocks [16]. The internal structure of a linear control system can be inferred from its frequency response. If a stable linear system is subjected to a sinusoidal input, its steady-state output is a sinuosoid of the same frequency but possibly with a different amplitude and phase. The amplitude gain and the phase shifts, plotted as functions of frequency—the so-called Bode plots, after Hendrik Bode, who developed frequency analysis at Bell Labs—reveal a great deal about the structure of the system [1]. More generally, the art of systems

Systems Biology, Minimalist vs Exhaustive Strategies

engineering lies in designing a linear system whose frequency response matches specified Bode plots. The technology is now available to experimentally measure approximate cellular frequency responses in the vicinity of a steady state, at least under simple conditions. Provided the amplitude of the forcing is not too high, so that a linear approximation is reasonable, and the steady state is homeostatically maintained, reverse engineering of the Bode plots can yield a simplified linear control system that may be an useful abstraction of the complex nonlinear molecular network responsible for the homeostatic regulation [15]. Unlike timescale separation, the reduction in complexity can be dramatic. As always, this comes at the price of a more abstract representation of the underlying biology but, crucially, one in which some of the control structure is retained. However, at present, we have little idea how to extend such frequency analysis to large perturbations, where the nonlinearities become significant, or to systems that are not homeostatic. Frequency analysis, unlike separation of time scales, relies on data, reinforcing the point made previously that integrating modelling with experiments and data can lead to powerful synergies.

Looking Behind the Data Experimentalists have learned the hard way to develop their conceptual understanding from experimental data. As the great Otto Warburg advised, “Solutions usually have to be found by carrying out innumerable experiments without much critical hesitation” [13]. However, sometimes the data you need is not the data you get, in which case conceptual interpretation can become risky. For instance, signalling in mammalian cells has traditionally relied on grinding up 106 cells and running Western blots with antibodies against specific molecular states. Such data has told us a great deal, qualitatively. However, a molecular network operates in a single cell. Quantitative data aggregated over a cell population is only meaningful if the distribution of responses in the population is well represented by its average. Unfortunately, that is not always the case, most notoriously when responses are oscillatory. The averaged response may look like a damped oscillation, while individual cells actually have regular oscillations but at different frequencies and phases [14, 17]. Even when the response is not oscillatory single-cell analysis

Systems Biology, Minimalist vs Exhaustive Strategies

may reveal a bimodal response, with two apparently distinct sub-populations of cells [4]. In both cases, the very concept of an “average” response is a statistical fiction that may be unrelated to the behaviour of any cell in the population. One moral of this story is that one should always check whether averaged data is representative of the individual, whether individual molecules, cells, or organisms. It is surprising how rarely this is done. The other moral is that data interpretation should always be mechanistically grounded. No matter how intricate the process through which it is acquired, the data always arises from molecular interactions taking place in individual cells. Understanding the molecular mechanism helps us to reason correctly, to know what data are needed and how to interpret the data we get. Mathematics is an essential tool in this, just as it was for Michaelis and Menten. Perhaps one of the reasons that biochemists of the Warburg generation were so successful, without “critical hesitation,” was because Michaelis and others had already provided a sound mechanistic understanding of how individual enzymes worked. These days, systems biologists confront extraordinarily complex multienzyme networks and want to know how they give rise to cellular physiology. We need all the mathematical help we can get.

References ˚ om, K.J., Murray, R.M.: Feedback systems. An Intro1. Astr¨ duction for Scientists and Engineers. Princeton University Press, Princeton (2008) 2. Black, J.: Drugs from emasculated hormones: the principles of syntopic antagonism. In: Fr¨angsmyr, T. (ed.) Nobel Lectures, Physiology or Medicine 1981–1990. World Scientific, Singapore (1993) 3. Cajal, S.R.: Advice for a Young Investigator. MIT Press, Cambridge (2004) 4. Ferrell, J.E., Machleder, E.M.: The biochemical basis of an all-or-none cell fate switch in Xenopus oocytes. Science 280, 895–898 (1998) 5. Grodins, F.S.: Control Theory and Biological Systems. Columbia University Press, New York (1963) 6. Gunawardena, J.: Models in systems biology: the parameter problem and the meanings of robustness. In: Lodhi, H., Muggleton, S. (eds.) Elements of Computational Systems Biology. Wiley Book Series on Bioinformatics. Wiley, Hoboken (2010)

1463 7. Gunawardena, J.: A linear elimination framework. http:// arxiv.org/abs/1109.6231 (2011) 8. Gunawardena, J.: Modelling of interaction networks in the cell: theory and mathematical methods. In: Egelmann, E. (ed.) Comprehensive Biophysics, vol. 9. Elsevier, Amsterdam (2012) 9. Gutenkunst, R.N., Waterfall, J.J., Casey, F.P., Brown, K.S., Myers, C.R., Sethna, J.P.: Universally sloppy parameter sensitivities in systems biology models. PLoS Comput. Biol. 3, 1871–1878 (2007) 10. Kim, K.A., Spencer, S.L., Ailbeck, J.G., Burke, J.M., Sorger, P.K., Gaudet, S., Kim, D.H.: Systematic calibration of a cell signaling network model. BMC Bioinformatics. 11, 202 (2010) 11. Kirschner, M.: The meaning of systems biology. Cell 121, 503–504 (2005) 12. Kirschner, M.W., Gerhart, J.C.: The Plausibility of Life. Yale University Press, New Haven (2005) 13. Krebs, H.: Otto Warburg: Cell Physiologist, Biochemist and Eccentric. Clarendon, Oxford (1981) 14. Lahav, G., Rosenfeld, N., Sigal, A., Geva-Zatorsky, N., Levine, A.J., Elowitz, M.B., Alon, U.: Dynamics of the p53Mdm2 feedback loop in individual cells. Nat. Genet. 36, 147–150 (2004) 15. Mettetal, J.T., Muzzey, D., G´omez-Uribe, C., van Oudenaarden, A.: The frequency dependence of osmoadaptation in Saccharomyces cerevisiae. Science 319, 482–484 (2008) 16. Muzzey, D., G´omez-Uribe, C.A., Mettetal, J.T., van Oudenaarden, A.: A systems-level analysis of perfect adaptation in yeast osmoregulation. Cell 138, 160–171 (2009) 17. Nelson, D.E., Ihekwaba, A.E.C., Elliott, M., Johnson, J.R., Gibney, C.A., Foreman, B.E., Nelson, G., See, V., Horton, C.A., Spiller, D.G., Edwards, S.W., McDowell, H.P., Unitt, J.F., Sullivan, E., Grimley, R., Benson, N., Broomhead, D., Kell, D.B., White, M.R.H.: Oscillations in NF-B control the dynamics of gene expression. Science 306, 704–708 (2004) 18. Neumann, L., Pforr, C., Beaudoin, J., Pappa, A., Fricker, N., Krammer, P.H., Lavrik, I.N., Eils, R.: Dynamics within the CD95 death-inducing signaling complex decide life and death of cells. Mol. Syst. Biol. 6, 352 (2010) 19. Rand, D.A.: Mapping the global sensitivity of cellular network dynamics: sensitivity heat maps and a global summation law. J. R. Soc Interface. 5(Suppl 1), S59–S69 (2008) 20. Segel, L.: On the validity of the steady-state assumption of enzyme kinetics. Bull. Math. Biol. 50, 579–593 (1988) 21. Spencer, S.L., Gaudet, S., Albeck, J.G., Burke, J.M., Sorger, P.K.: Non-genetic origins of cell-to-cell variability in TRAIL-induced apoptosis. Nature 459, 428–433 (2009) 22. Strogatz, S.H.: Nonlinear Dynamics and Chaos: with Applications to Physics, Biology, Chemistry and Engineering. Perseus Books, Cambridge (2001) 23. Yi, T.M., Huang, Y., Simon, M.I., Doyle, J.: Robust perfect adaptation in bacterial chemotaxis through integral feedback control. Proc. Natl. Acad. Sci. U.S.A. 97, 4649–4653 (2000)

S

T

Taylor Series Methods Roberto Barrio Departamento de Matem´atica Aplicada and IUMA, University of Zaragoza, Zaragoza, Spain

Mathematics Subject Classification 65L05; 65L80; 65L07; 65Lxx

complete introduction of the method. Besides, it has been rediscovered several times with different names. The main idea behind Taylor series method is to approximate the solution of a differential system by means of a Taylor polynomial approximation. To state the method, we fix the concept for solving an initial value problem of the form d y.t/ D f .t; y.t//; dt

y.t0 / D y 0 ;

y 2 Rs : (1)

Now, the value of the solution at ti C1 D ti C hi (i.e., y.ti C1 /) is approximated from the nth degree Taylor series of y.t/ about ti and evaluated at hi (it Recurrent power series method; Taylor methods; TS; is understood that the function f has to be smooth enough). TSM

Synonyms

Short Definition

y.t0 /  y 0 ; y.ti C1 / ' y.ti / C

The Taylor series method term stands for any method to solve differential equations based on the classical Taylor series expansion. Numerical versions of these methods can be applied to different kinds of differential systems, especially to ordinary differential equations. They are very well suited for solving low-medium dimensional systems of differential equations at any specified degree of accuracy.

d y.ti / hi C : : : dt

1 d n y.ti / n hi nŠ dt n ' y i C f .ti ; y i / hi C : : : C

C

(2)

1 d n1 f .ti ; y i / n hi  y i C1 : nŠ dt n1

From the formulation of the Taylor series method (TSM and TSM(n) for the nth degree TSM), the probDescription lem is reduced to compute the Taylor coefficients. If the computation is based on numerical schemes the TSM The Taylor series method has a long history, and in the will be a numerical method (there are also symbolic works of Euler [5] [663 E342] we can see a nice and versions of the TSM). © Springer-Verlag Berlin Heidelberg 2015 B. Engquist (ed.), Encyclopedia of Applied and Computational Mathematics, DOI 10.1007/978-3-540-70529-1

1466

Taylor Series Methods

member f of the two-body problem xP D X; yP D Basics TSMs are frequently used in literature as first examples Y; XP D x=.x 2 C y 2 /3=2 ; YP D y=.x 2 C y 2 /3=2 , when introducing numerical methods for ODEs, and in we can decompose it as shown in Fig. 1. This decomposition of the function can be used, particular, the simplest method, Euler’s method together with the chain rule, to evaluate the derivatives of the function f and any optimization at this point y i C1 ' y i C f .ti ; y i / hi ; will give faster TSMs. Thus, the next step is to obtain a list of rules to compute the coefficients of the Taylor which corresponds to TSM(1). series. If we denote by f Œj  .t/ D f .j / .t/=j Š the j th normalized Taylor coefficient of f .t/ at t, some basic Order of the Method The order of TSM(n) is obtained in a straightforward rules (for a complete list see [9]) are as follows: way from the definition (2) since the Local Truncation • If h.t/ D f .t/ ˙ g.t/ then Error (LTE) at step i C 1 is given by the remainder hŒm .t/ D f Œm .t/ ˙ g Œm .t/:

1 hnC1 y .nC1/ .ti / C O.hnC2 / i .n C 1/Š i 1  hnC1 f .n/ .ti ; y i / C O.hinC2 /: .n C 1/Š i

LTE D

• If h.t/ D f .t/  g.t/ then

Therefore, the TSM(n) of degree n is also of order n. hŒm .t/  TIMES.f; g; m/ D Computation of the Taylor Coefficients The main point to convert the TSM in a practical numerical method consists of providing efficient formulas to evaluate the Taylor coefficients. Classically, this is solved with the recursive differentiation of the function f , but this approach is not affordable in real situations. In contrast, this can be done quite efficiently by using Automatic Differentiation (AD) techniques [6]. To obtain the successive derivatives of a function, first we have to decompose it into a sequence of arithmetic operations and calls to standard unary or binary functions. This part is the trickiest step of the TSM, since it has to be done manually for each ODE or automatically by using some of the nowadays available codes. For instance, if we want to evaluate the second Taylor Series Methods, Fig. 1 Left: Decomposition of the two-body problem in unary and binary functions. Right: AD rules

m X

f Œmi  .t/ g Œi  .t/:

i D0

• If h.t/ D f .t/˛ with ˛, .f Œ0 .t//˛ 2 Rnf0g then hŒ0 .t/ D .f Œ0 .t//˛ ; hŒm .t/  POWER.f; h; ˛; m/ D



m1 X

1 m f Œ0 .t/

  m˛  i.˛ C 1/ f Œmi  .t/ hŒi  .t/:

i D0

• If g.t/ D cos.f .t// and h.t/ D sin.f .t// then AD rules

f1 = X

f3 = Times

f4 = Times

f2 = Y −x

−y

s4 = Power

[m] f1

= X

[m] f2

= Y

[m]

s1 −3/2

s3 = Plus

s1 = Times

x

s2 = Times

x

y

y

[m] s2 [m] s3 [m] s4

[m] [m]

= TIMES(x, x, m) = TIMES(y, y, m) [m]

[m]

= s1 + s2

= POWER (s3, s4, −3/2, m)

[m] f3

= - TIMES(x, s4, m)

[m] f4

= - TIMES(y, s4, m)

Taylor Series Methods

1467

  g Œ0 .t/ D cos f Œ0 .t/ ; g Œm .t/  COS.f; h; m/ m 1 X Œmi  D ih .t/ f Œi  .t/; m i D1   hŒ0 .t/ D sin f Œ0 .t/ ; hŒm .t/  SIN.f; g; m/ m 1 X Œmi  D ig .t/ f Œi  .t/: m i D1 The set of formulas may be easily increased with the recurrences of other elementary functions like exp; log; tan; cosh; sinh; arccos; arcsin, and so on. Note that some functions have to be evaluated in groups, like sin and cos. Once we have the way to construct, order by order, the Taylor coefficients of the second member of (1), we easily obtain the coefficients of the Taylor series solution f Œi  .y Œ0 ; : : : ; y Œi  / y Œi C1 D : .i C 1/ Domain of Stability The stability function of TSM(n) is

Rn .z/ D 1 C z C D e 1. In order to define a finite energy, it is, in fact, enough to assume that VC D maxfV; 0g 2 L5=2 .R3 / C L1 .R3 / 5=2 and V D minfV; 0g 2 Lloc .R3 /. Let us briefly explain the different terms in the functional. The third term in EVTF is the Coulomb selfenergy of the classical charge distribution . In the context of using Thomas–Fermi theory to describe a quantum gas, it should be considered as an approximation to the electrostatic energy of the particles. The second term in EVTF is the energy due to the interaction of the particles with the exterior potential. Finally, the source of the first term in EVTF is motivated by the semiclassical integral .2/3 q

Z jpjF

1 2 p dp D physical 5=3 ; 2

T

1472

Thomas–Fermi Type Theories (and Their Relation to Exact Models)

R if F  0 is chosen such that  D q.2/3 jpjF 1dp (the factor q in front of the two integrals is due to the internal degrees of freedom). In other words, the Fermi gas is assumed to fill a ball (the Fermi sphere) in momentum space. The density of the gas is (q times) the volume of the ball, and the kinetic energy density of the gas is (q times) the momentum integral of the kinetic energy 12 p 2 over the ball. The parameter F that we introduced is the Fermi momentum. As already stated above, Thomas–Fermi theory was introduced by Thomas and Fermi in the 1920s as a model for atoms and molecules. The first rigorous results were due to Hille [6], but it was not until 1973 that Lieb and Simon [13, 14] did a complete rigorous analysis of the model. Unless otherwise stated, the results given in this review are from these papers and can also be found in the detailed review [10] (see also [8]). The basic properties of Thomas–Fermi theory are collected in the following theorem. Theorem 1 (Basic properties of the Thomas–Fermi variational problem) If V 2 L5=2 .R3 / C L1 .R3 / and tends to zero at infinity, then the energy EVTF is a convex, nonincreasing function of particle number. There exists a critical particle number Nc  0 (possibly infinity) such that the variational problem (1) has a unique minimizer  if and only if N  Nc . Moreover, there exists a unique chemical potential   0 such that the minimizing  satisfies the Thomas–Fermi equation 5 .x/2=3 D ŒV .x/   jxj1  C ; 3

Thomas–Fermi Theory for Atoms and Molecules In the case of a molecule consisting of K atoms, i.e., with K nuclei which we assume to have charges Z1 ; : : : ; ZK >0 (in our units the physical nuclei have integer charges, but it is not necessary to make this assumption) and to be situated at points R1 ; : : : ; RK 2 R3 , we have

V .x/ D

K X i D1

Zk ; jx  Rk j

UD

X 1k 0 there exists N1 ; N2  0 such that N D N1 CN2 This theorem is proved by standard functional analytic and methods using the strict convexity of E TF to show that for all N  0, E TF .N; Z; R/  E TF .N1 ; Z 1 ; R/ C E TF .N2 ; Z 2 ; R/: ˇ n   ˇ EVTF .N / D inf EVTF ./ ˇ 0   2 L5=3 R3 ; (Note that we allow some components of Z 1 , Z 2 to Z o vanish, which simply means that the molecule has .x/dx  N fewer nuclei. In particular, the energy does not depend R3 on the corresponding components of R1 , R2 .) and that the minimizer for this problem exists and is In this theorem, the presence of the nuclear repulsion unique for all N  0. For the Thomas–Fermi model, term U is important. In the Thomas–Fermi theory, the inequality in the no-binding theorem is, in fact, strict, the energy EVTF .N / D EVTF .Nc / if N  Nc .

Thomas–Fermi Type Theories (and Their Relation to Exact Models)

but this fails in some of the generalizations discussed below. The minimal energy and the minimizing density in Thomas–Fermi theory satisfy the exact scaling relations that for all > 0: E TF . N; Z; 1=3 R/ D 7=3 E TF .N; Z; R/

1473

 .x/ D N

q X s1 D1

Z



q X sN D1

j .x; s1 ; x2 ; s2 ; : : : ; xN ; sN /j2 dx2 : : : dxN :



The quantum energy is then defined as

and TF . 1=3 x; N; Z; 1=3 R/ D 2 TF .x; N; Z; R/:

N n ^ E Q .N; Z; R/ D inf h ; HN iL2 j 2 H 2 .R3 I Cq /; o k k2L2 D 1 : (4)

For positive ions, i.e., if N < Z1 C : : : C ZK , the density TF .x; N; Z; R/ has compact support. In the neutral case, N D Nc D Z1 C : : : C ZK , the density The following asymptotic exactness of Thomas–Fermi theory was proved in [13]. satisfies the large x asymptotics 

5  .x; Nc ; Z; R/ 27 3 TF

3

Theorem 4 (Large Z asymptotic exactness of TF theory) As ! 1, we have

6

jxj :

In particular, it follows that we have a limit of an infinite molecule TF

lim  .x; Nc ; Z;

!1

1=3



5 R/ D 27 3

3

D 7=3 E TF .N; Z; R/   Co 7=3 :

jxj6 :

Validity of Thomas–Fermi Theory as an Approximation As already, stated Thomas–Fermi theory is motivated by a semiclassical calculation for the kinetic energy. It is therefore natural to guess that it will be a good approximation to the exact quantum model in a semiclassical regime, i.e., when the average particle distance is small compared to the scale on which the density and the potential vary. From the Thomas–Fermi scaling, we see that this is the case in the large nuclear charge limit. To make this precise, consider the molecular quantum Hamiltonian

HN D

    E Q N; Z; 1=3 R D E TF N; Z; 1=3 R   Co 7=3

 N  X X 1 1 CU  i  V .xi / C 2 jxi  xj j i D1 1i Z1 C: : :CZK . A natural question is what are the physically correct values of the parameters CD and CW . Dirac [2] suggested that CD D .3=2/4=3 .2q/1=3 which was confirmed to give the correct asymptotics for the highdensity uniform gas in [5]. It is not entirely clear how Z Z to choose the constant CW . One possibility is to note, D 5=3 .x/ dx  V .x/.x/dx E ./ D  see [10], that it may be chosen in such a way that the R3 R3 ZZ TFW energy reproduces the leading 2 correction, also 1 .x/.y/ C dxdy called the Scott correction, to the energy asymptotics 2 R3 R3 jx  yj in (5). Z 4=3 An interesting observation is that as a consequence CD .x/ dx C U R3 of the Lieb-Thirring inequality [15, 16] and the Lieb exchange estimate [9, 11], the TFD functional gives an 2. The Thomas–Fermi-von Weizs¨acker (TFW) exact lower bound to the expected quantum energy. theory [21]: in which a correction to the kinetic Theorem 5 (Thomas–Fermi-Dirac functional as an energy has been added (CW > 0) exact lower bound) There exist positive values for the Z V p constants  and CD such that if 2 N H 2 .R3 I Cq / TFW 2 E ./ D CW .r .x// dx is a fermionic wave function with density  and HN R3 Z Z denotes the quantum Hamiltonian (3), then .x/5=3 dx  V .x/.x/dx C

Generalizations of Thomas–Fermi Theory All density functional theories can be thought of as generalizations of the Thomas–Fermi model. The three simplest are 1. The Thomas–Fermi-Dirac (TFD) theory [2]: in which an exchange correlation term has been added (CD > 0)

R3

1 C 2

R3

ZZ R3 R3

.x/.y/ dxdy C U: jx  yj

h ; HN i  E TFD . /: The famous Lieb-Thirring conjecture [16] suggests that we may choose  D physical here.

3. The combined Thomas–Fermi-Dirac-von Weizs¨aCombining this lower bound with the no-binding Thecker (TFDW) theory: orem shows that the quantum energy is bounded below Z p by a sum of atomic TFD energies. This indeed proves TFDW 2 E ./ D CW .r .x/ dx stability of matter, i.e., that the energy is bounded R3 Z Z proportionally to the number of nuclei [3, 12, 15]. C .x/5=3 dx V .x/.x/dx R3

ZZ

R3

Magnetic Thomas–Fermi Theory In the presence of a strong magnetic field, a modificaR3 R3 tion of Thomas–Fermi theory is needed, in particular, Z because of the interaction of the electron spin with the 4=3 CD .x/ dx C U magnetic field. In the case of a homogeneous magnetic 3 R field of strength B, the appropriate modification is to The energies in these models are defined similarly replace the kinetic energy function 5=3 by the Legento the Thomas–Fermi energy with the appropriate dre transform supV 0 .V  PB .V // of the pressure of changes to the domain of the functionals. For the the free Landau gas TFD and TFW theories, the energies are convex ! 1 X and nonincreasing. In the molecular case, we have 3=2 2 1 3=2 C2 ŒV  2BC ; for the TFD model, as for the TF model that PB .V / D .3 / B V D1 Nc D Z1 C : : : C ZK , and the no-binding theorem 2 holds. For the TFW model, Nc > Z1 C : : : C ZK , and P0 .v/ D lim PB .V / D  V 5=2 : B!0C 15 2 the no-binding theorem does not hold. For these results 1 C 2

.x/.y/ dxdy jx  yj

Tight Frames and Framelets

The corresponding Thomas–Fermi model was studied in [17, 22]. It again satisfies Nc D Z1 C : : : C ZK and the no-binding Theorem. Moreover, it was shown that the magnetic Thomas–Fermi energy approximates the exact quantum energy (as in Theorem 4) if B=Z 3 ! 0 as Z ! 1.

1475 19. Teller, E.: On the stability of molecules in the Thomas– Fermi theory. Rev. Mod. Phys. 34, 627–631 (1962) 20. Thomas, L.H.: The calculation of atomic fields. Proc. Camb. Philops. Soc. 23, 542–548 (1927) 21. von Weizs¨acker, C.F.: Zur theorie der Kernmassen. Z. Phys. 96, 431–458 (1935) 22. Yngvason, J.: Thomas–Fermi theory for matter in a magnetic field as a limit of quantum mechanics. Lett. Math. Phys. 22, 107–117 (1991)

References 1. Benguria, R.: The von-Weisz¨acker and exchange corrections in Thomas–Fermi theory. Ph.D. thesis, Princeton University (unpublished 1979) 2. Dirac, P.A.M.: Note on exchange phenomena in the Thomas–Fermi atom. Proc. Camb. Philos. Soc. 26, 376–385 (1930) 3. Dyson, F.J., Lenard, A.: Stability of matter. I and II. J. Math. Phys. 8, 423–434, (1967); ibid. J. Math. Phys. 9, 698–711 (1968) 4. Fermi, E.: Un metodo statistico per la determinazione di alcune prioriet´a del atomo. Rend. Accad. Nat. Lincei 6, 602–607 (1927) 5. Graf, G.M., Solovej, J.P.: A correlation estimate with applications to quantum systems with Coulomb interactions. Rev. Math. Phys. 6, 977–997 (1994). Special issue dedicated to Elliott H. Lieb 6. Hille, E.: On the Thomas–Fermi equation. Proc. Nat. Acad. Sci. U S A 62, 7–10 (1969) 7. Le Bris, C.: Some results on the Thomas–Fermi-Dirac-von Weizs¨acker model. Differ. Integral Equ. 6, 337–353 (1993) 8. Le Bris, C., Lions, P.-L.: From atoms to crystals: a mathematical J. Bull. Am. Math. Soc. 42, 291–363 (2005) 9. Lieb, E.H.: A lower bound for Coulomb energies. Phys. Lett. A 70, 444–446 (1979) 10. Lieb, E.H.: Thomas–Fermi and related theories of atoms and molecules. Rev. Mod. Phys. 53, 603–642 (1981) 11. Lieb, E.H., Oxford, S.: An improved lower bound on the indirect Coulomb energy. Int. J. Quantum Chem. 19, 427–439 (1981) 12. Lieb, E.H., Seiringer, R.: The Stability of Matter in Quantum Mechanics. Cambridge University Press, New York (2010) 13. Lieb, E.H., Simon, B.: Thomas–Fermi theory revisited. Phys. Rev. Lett. 31, 681–683 (1973) 14. Lieb, E.H., Simon, B.: The Thomas–Fermi theory of atoms molecules and solids. Adv. Math. 23, 22–116 (1977) 15. Lieb, E.H., Thirring, W.E.: Bound for the kinetic energy of fermions which proves the stability of matter. Phys. Rev. Lett. 35, 687–689 (1975) 16. Lieb, E.H., Thirring, W.E.: Inequalities for the moments of the eigenvalues of the Schr¨odinger Hamiltonian and their relation to sobolev inequalities. In: Lieb, E., Simon, B., Wightman, A. (eds.) Studies in Mathematical Physics, pp. 269–303. Princeton University Press, Princeton (1976) 17. Lieb, E.H., Solovej J.P., Yngvason, J.: Asymptotics of heavy atoms in high magnetic fields. II. Semiclassical regions. Comm. Math. Phys. 161, 77–124 (1995) 18. Lions, P.-L.: Solutions of Hartree-Fock equations for Coulomb systems. Comm. Math. Phys. 109, 33–97 (1987)

Tight Frames and Framelets Raymond Chan Department of Mathematics, The Chinese University of Hong Kong, Shatin, Hong Kong

The construction of compactly supported (bi-)orthonormal wavelet bases of arbitrarily high smoothness has been widely studied since Ingrid Daubechies’ celebrated works [3, 4]. Tight frames generalize orthonormal systems and give more flexibility in filter designs. A system X  L2 .R/ is called a tight frame of L2 .R/ if X jhf; hij2 D kf k2 ; h2X

holds for all f 2 L2 .R/, where h; i and k  k D h; i1=2 are the inner product and norm of L2 .R/; see [5, Chapter 5]. This is equivalent to X

hf; hih D f;

f 2 L2 .R/:

h2X

Hence, like an orthonormal system, one can use the same system X for both the decomposition and reconstruction processes. The tight frame system X is usually linear dependent in order to get more flexibility. Tight framelet systems are of particular interest. A tight framelet system is constructed from a refinable function. Let 2 L2 .R/ be a refinable function whose refinement equation is D2

X

h0 Œk .2  k/:

k2Z

The sequence h0 is called refinement mask or lowpass filter. Let hi , i D 1; : : : ; m be high-pass filters satisfying

T

1476

Tight Frames and Framelets m X

hbi .!/hbi .!/ D 1

Then the perfect reconstruction formula (1) holds. The framelets corresponding to the high-pass filters (3) are:

and

i D0 m X

hbi .!/hbi .! C / D 0;

8! 2 Œ; ;

(1)

i D0

1 1 D p .2x C 1/  p .2x  1/ 2 2 1 1 2 .x/ D  .2x C 1/ C 1  .2x/  .2x  1/ 2 2 1 .x/

P where hbi .!/ WD k2Z hi Œke i k! is the Fourier series of hi . Equation (1) is called the perfect reconstruction The system obtained by dilation and translation k=2 k formula, and fhi gm i .2  j / W k; j 2 ZI i D 1; 2g is the piecewise i D1 are called framelet masks. Define f2

WD f 1 ; : : : ; m g with linear tight framelet system. There is a general process for constructing highX pass filters for B-splines; see [6]. Here we give the hi Œk .2  k/: i D2 filters for the piecewise cubic tight framelet system k2Z which, like the piecewise linear one, is also used very Then X . / D f2j=2 i .2j  k/ W k; j 2 ZI i D often in image processing; see [1]: 1; : : : ; mg is a tight frame of L2 .R/, and f i gm i D1 are 1 1 called framelets. Œ1; 4; 6; 4; 1I h1 D Œ1; 2; 0; 2; 1I h0 D As an example, consider the piecewise linear B16 8 p spline (the hat function): 1 6 h2 D Œ1; 0; 2; 0; 1I h3 D Œ1; 2; 0; 2; 1I  16 8 1 C x; 1  x  0; .x/ D 1 1  x; 0  x  1: Œ1; 4; 6; 4; 1: h4 D 16 Its refinement equation is .x/ D

1 1 .2x C 1/ C 1  .2x/ C .2x  1/: 2 2

Similar to the orthonormal wavelet system, we also have analysis and synthesis tight frame transform and multi-resolution analysis. The forward (or analysis) tight frame transform is obtained by

Thus, the low-pass filter is h0 D

2

1 Œ1; 2; 1 4

3 H0 T D 4 H1 5 H2

(2)

and the corresponding Fourier series is

where Hi are filter matrices that correspond to the filters. For example,

1 1 1 hb0 .!/ D e i ! C C e i ! : 4 2 4

2

If we define the high-pass filters as p 2 Œ1; 1; h1 D 4

h0 D h2 D

1 Œ1; 2; 1 4

with Fourier series p p 2 i! 2 i ! b e  e h1 .!/ D 4 4

and

1 1 1 hb2 .!/ D  e i ! C  e i ! : 4 2 4

(3)

1 Œ1; 2; 1 4

2 61 16 6 ! H0 D 6 46 40 1

1 0 2 1 :: :: :: : : : 1 2 0 1

3 1 07 7 7 7 7 15 2

where we have used the periodic extension at the boundary. The backward (or synthesis) tight frame transform is obtained by taking the transpose of the forward transform:   T t D H0 t H1 t H2 t :

Time Reversal, Applications and Experiments

1477

Notice that the perfect reconstruction formula (1) guar- References antees that T t T D I, but T T t may not be equal to I. 1. Cai, J., Chan, R., Shen, L., Shen, Z.: Convergence analysis In fact, T t T D I , H0t H0 C H1t H1 C H2t H2 D I

2.

To obtain multi-resolution analysis, define h0 at 3. level ` to be 2 3

4.

1 17 61 .`/ h0 D 4 ; 0; : : : ; 0; ; 0; : : : ; 0; 5 : 4 „ ƒ‚ … 2 „ ƒ‚ … 4 2.`1/ 1 .`/

2.`1/ 1

.`/

5. .`/

The masks h1 and h2 can be given similarly. Let Hi .`/ be the matrix corresponding to hi . Then 2

QL1 `D0

.L`/

H0

6.

of tight framelet approach for missing data recovery. Adv. Comput. Math. 31, 87–113 (2009) Cai, J., Dong, B., Osher, S., Shen, Z.: Image restoration: total variation; wavelet frames, and beyond (submitted). http:// math.arizona.edu/dongbin/Publications/CDOS.pdf Daubechies, I.: Orthogonal bases of compactly supported wavelets. Commun. Pure Appl. Math. 41, 909–996 (1988) Daubechies, I.: Ten Lectures on Wavelets. Volume 61 of CBMS Conference Series in Applied Mathematics. SIAM, Philadelphia (1992) Mallat, S.: A Wavelet Tour of Signal Processing. Academic, San Diego (1998) Ron, A., Shen, Z.: Affine system in L2 .Rd /: the analysis of the analysis operator. J. Funct. Anal. 148, 408–447 (1997)

3

6 7 2 3 6 H .L/ QL1 H .L`/ 7 AL 6 1 7 `D1 0 6 .L/ QL1 .L`/ 7 6 7 6H 7 6 7 `D1 H0 2 76 7; AD6 6 6 7 7 :: 6 7 4 AH 5 : 6 7 6 7 .1/ 4 5 H1 .1/ H2

Time Reversal, Applications and Experiments Mathias Fink Institut Langevin, ESPCI ParisTech, Paris, France

and we also have the perfect reconstruction property: At A D AtL AL C AtH AH D I . In image processing where the problems are twodimensional, we use tensor products of the univariate tight frames to produce tight framelet systems in L2 .R2 /, i.e., the filters are given by hij D hti hj , where hi are the filters from the univariate tight framelet system. As an example, the filters for the two-dimensional piecewise linear tight framelet system are as follows:

Taking advantage of the time-reversal invariance of the acoustic wave equation, the concept of time-reversal mirror has been developed and several devices have been built which illustrated the efficiency of this concept [1–3]. In such a device, an acoustic source, located inside a lossless medium, radiates a brief transient pulse that propagates and is potentially distorted by the medium. Time reversal as described above would entail the reversal, at some instant, of every particle velocity in the medium. As an alternative, the acoustic 2 2 2 3 3 3 field could be measured on every point of an enclosing 121 1 0 1 1 2 1 p surface (acoustic retina) and retransmitted in time2 4 1 4 1 4 2 4 25; 2 0 25; 16 2 4 25; 16 16 reversed order; then, the wave will travel back to its 121 1 0 1 1 2 1 source; see Fig. 1. From an experimental point of view, 3 3 3 2 2 2 a closed TRM consists of a two-dimensional piezoelec1 2 1 1 0 1 p 1 2 1 p 2 4 5 ; 1 4 0 0 0 5; 2 4 0 0 0 5; tric transducer array that samples the wave field over a 0 0 0 16 8 16 closed surface. An array pitch of the order of /2 where 1 2 1 1 0 1 1 2 1 2 2 2 3 3 3 is the smallest wavelength of the pressure field is 1 2 1 1 0 1 1 2 1 p needed to insure the recording of all the information 2 4 1 4 5; 5; 1 42 4 25: 2 4 2 2 0 2 16 16 16 on the wave field. Each transducer is connected to its 1 2 1 1 0 1 1 2 1 own electronic circuitry that consists of a receiving amplifier, an A/D converter, a storage memory, and a A good reference for the material mentioned programmable transmitter able to synthesize a timereversed version of the stored signal. In practice, closed here is [2].

T

1478

a

Time Reversal, Applications and Experiments Heterogeneous Medium

Elementary transducers

ACOUSTIC SOURCE

RAMs

p(r,t)

b

p(r,T-t)

Time Reversal, Applications and Experiments, Fig. 1 (a) Recording step: a closed surface is filled with transducer elements. A point-like source generates a wave front which is distorted by heterogeneities. The distorted pressure field is recorded

on the cavity elements. (b) Time-reversed or reconstruction step: the recorded signals are time reversed and reemitted by the cavity elements. The time-reversed pressure field back-propagates and refocuses exactly on the initial source

Time Reversal, Applications and Experiments, Fig. 2 One part of the transducers is replaced by reflecting boundaries. In (a) the wave radiated by the source is recorded by a set of

transducers through the reverberation inside the cavity. In (b), the recorded signals are time reversed and reemitted by the transducers

TRMs are difficult to realize and the TR operation is usually performed on a limited angular area, thus apparently limiting focusing quality. A TRM consists typically of a small number of elements or timereversal channels. The major interest of TRM, compared to classical focusing devices (lenses and beam forming) is certainly the relation between the medium complexity and the size of the focal spot. A TRM acts as an antenna that uses complex environments to appear wider than it is, resulting in a refocusing quality that does not depend of the TRM aperture. It is generally difficult to use acoustic arrays that completely surround the area of interest, so the closed cavity is usually replaced by a TRM of finite angular aperture. However, wave propagation in media with

complex boundaries or random scattering medium can increase the apparent aperture of the TRM, resulting in a focal spot size smaller than that predicted by classical formulas. The basic idea is to replace one part of the transducers needed to sample a closed timereversal surface by reflecting boundaries that redirect one part of the incident wave towards the TRM aperture (see Fig. 2). When a source radiates a wave field inside a closed cavity or in a waveguide, multiple reflections along the medium boundaries can significantly increase the apparent aperture of the TRM. Such a concept is strongly related to a kaleidoscopic effect that appears, thanks to the multiple reverberations on the waveguide boundaries. Waves emitted by each transducer are multiply reflected, creating at each

Time Reversal, Applications and Experiments

1479

reflection “virtual” transducers that can be observed from the desired focal point. Thus, we create a large virtual array from a limited number of transducers and a small number of transducers is multiplied to create a “kaleidoscopic” transducer array. Three different experiments illustrating this concept will be presented (a waveguide, a chaotic cavity, and a multiply scattering medium).

conducted in the ultrasonic regime by Roux and Fink [3] showed clearly this effect with a TRM made of a 1D transducer array located in a rectangular ultrasonic waveguide (see Fig. 3a). For an observer, located in the waveguide, the TRM seems to be escorted by a periodic set of virtual images related to multipath propagation and effective aperture 10 times larger than the real aperture was observed. The experiment was conducted in a waveguide whose interfaces (water-air or water-steel interfaces) are plane and parallel. The length of the guide was Time Reversal in Acoustic Waveguide L 800 mm, on the order of 20 times the water depth The simplest boundaries that can give rise to such a of H 40 mm. A subwavelength ultrasonic source is kaleidoscopic effect are plane boundaries as in rect- located at one end of the waveguide. On the other angular waveguides or cavities. The first experiment end, a 1D time-reversal mirror made of a 96-element

a x reflecting boundaries H water

vertical transducer array

S

O

b

L

y

40

0

−40 0 0

Amplitude

c

80 µs

40

T

1 0.5 0 −0.5 −1

0

20

Time Reversal, Applications and Experiments, Fig. 3 (a) Schematic of the acoustic waveguide: the guide length ranges from 40 to 80 cm and the water depth from 1 to 5 cm. The central acoustic wavelength ( ) is 0.5 mm. The array element spacing is 0.42 mm. The TRM is always centered at the middle of the

40 Time (µs)

60

80

water depth. (b) Spatial-temporal representation of the incident acoustic field received by the TRM; the amplitude of the field is in dB. (c) Temporal evolution of the signal measured on one transducer of the array

1480

Time Reversal, Applications and Experiments

array spanned the waveguide. The transducers had a center frequency of 3.5 MHz and 50 % bandwidth. Due to experimental imitations, the array pitch was greater than =2. A time-reversal experiment was then performed in the following way: (1) the point source emits a pulsed wave (1 s duration); (2) the TRM receives, selects a time-reversal window and time reverses, and retransmits the field; (3) after back propagation the time-reversed field is scanned in the plane of the source. Figure 3b shows the incident field recorded by the array after forward propagation through the channel. After the arrival of the first wave front corresponding to the direct path, we observe a set of signals, due to multiple reflections of the incident wave between the interfaces that spread over 100 s. Figure 3c represents the signal received on one transducer of the TRM. After retransmission and propagation of the timereversed signals recorded by the array during a window of 100 s, we observe a remarkable temporal compression at the source location (see Fig. 4). This means that multipath effects are fully compensated. It shows that the time-reversed signal observed at the source is nearly identical to the one received in a time-reversed experiment conducted in free space. The peak signal exceeds its temporal side lobes by 45 dB. 1 Amplitude

Time Reversal, Applications and Experiments, Fig. 4 Time-reversed signal measured at the point source

The spatial focusing of the time-reversed field is also of interest. Figure 5 shows the directivity pattern of the time-reversed field observed in the source plane. The time-reversed field is focused on a spot which is much smaller than the one obtained with the same TRM working in free space. In our experiment, the 6 dB lateral resolution is improved by a factor of 9. This can be easily interpreted by the images theorem in a medium bounded by two mirrors. For an observer, located at the source point, the 40-mm TRM appears to be accompanied by a set of virtual images related to multipath reverberation. Acoustic waveguides are currently found in underwater acoustic, especially in shallow water, and TRMs can compensate for the multipath propagation in oceans that limits the capacity of underwater communication systems. The problem arises because acoustic transmissions in shallow water bounce off the ocean surface and floor, so that a transmitted pulse gives rise to multiple copies of itself that arrive at the receiver. Underwater acoustic experiments have been conducted by W. Kuperman and his group from San Diego University in a seawater channel of 120 m depth, with a 24-element TRM working at 500 Hz and 3.5 kHz. They observed focusing with super-resolution and multipath compensation at

0.5 0 −0.5 −1 −40

−20

0

20

40

Time (µs)

−20 0 Amplitude (dB)

Time Reversal, Applications and Experiments, Fig. 5 Directivity pattern of the time-reversed field in the plane of source: dotted line corresponds to free space, full line to the waveguide

−10 −20 −30 −40 −50

−15

−10

−5

mm 0

5

10

15

20

Time Reversal, Applications and Experiments

1481

a distance up to 30 kms [4]. Such properties open the other modes, we have only to deal with one field, the field of new discrete communication systems in un- flexural-scalar field. derwater applications as it was experimentally demonThe experiment is a “two-step process” as described strated by different groups [5]. above: In the first step, one of the transducers, located at point A, transmits a short omnidirectional signal of duration 0.5 s. Another transducer, located at B, observes a long random-looking signal that results Time Reversal in Chaotic Cavities from multiple reflections along the boundaries of the In this paragraph, we are interested in another aspect of cavity. It continues for more than 50 ms corresponding multiply reflected waves: waves confined in closed re- to some hundred reflections at the boundaries. Then, flecting cavities with nonsymmetrical geometry. With a portion T of the signal is selected, time reversed, closed boundary conditions, no information can escape and reemitted by point B. As the time-reversed wave is from the system and a reverberant acoustic field is a flexural wave that induces vertical displacements of created. If, moreover, the geometry of the cavity shows the silicon surface, it can be observed using the optical ergodic and mixing properties, one may hope to collect interferometer that scans the surface around point A all information at only one point. Ergodicity means (see Fig. 6). For time-reversal windows of sufficiently longthat, due to the boundary geometry, any acoustic ray radiated by a point source and multiply reflected would durationT , one observes both an impressive time pass every location in the cavity. Therefore, all the recompression at point A and a refocusing of the timeinformation about the source can be redirected towards reversed wave around the origin (see Fig. 7a, b for a single time-reversal transducer. This is the regime of T D 1 ms), with a focal spot whose radial dimension fully diffuse wave fields that can be also defined as in is equal to half the wavelength of the flexural wave. room acoustics as an uncorrelated and isotropic mix of Using reflections at the boundaries, the time-reversed plane waves of all propagation directions. Draeger and wave field converges towards the origin from all Fink [6] showed experimentally and theoretically that directions and gives a circular spot, like the one that in this particular case, a time reversal focusing with could be obtained with a closed time-reversal cavity /2 spot can be obtained using only one TR channel covered with transducers. A complete study of the dependence of the spatiotemporal side lobes around operating in a closed cavity. The first experiments were made with elastic waves the origin shows a major result: a time-duration T propagating in a 2D cavity with negligible absorption. of nearly 1 ms is enough to obtain good focusing. They were carried out using guided elastic waves in For values of T larger than 1 ms, the side lobes’ a monocrystalline D-shaped silicon wafer known to shape and the signal-to-noise ratio (focal peak/side have chaotic ray trajectories. This property eliminates lobes) do not improve further. There is a saturation the effective regular gratings of the previous section. regime. Once the saturation regime is reached, point Silicon was selected also for its weak absorption. B will receive redundant information. The saturation Elastic waves in such a plate are akin to Lamb waves. regime is reached after a time heisenberg called the An aluminum cone coupled to a longitudinal trans- Heisenberg time. It is the minimum time needed to ducer generated waves at one point of the cavity. resolve the eigenmodes in the cavity. It can also be A second transducer was used as a receiver. The interpreted as the time it takes for all single rays to central frequency of the transducers was 1 MHz, and reach the vicinity of any point in the cavity within their bandwidth was 100 %. At this frequency, only a distance /2. This guarantees enough interference three propagating modes are possible (one flexural, between all the multiply reflected waves to build each one quasi-extensional, one quasi-shear). The source of the eigenmodes in the cavity. The mean distance was considered point-like and isotropic because the ! between the eigenfrequencies is related to the 1 . cone tip is much smaller than the central wavelength. Heisenberg time; Heisenberg D ! The success of this time-reversal experiment in A heterodyne laser interferometer measures the displacement field as a function of time at different points closed chaotic cavity is particularly interesting with on the cavity. Assuming that there is no mode conver- respect to two aspects. Firstly, it proves the feasibilsion at the boundaries between the flexural mode and ity of acoustic time reversal in cavities of complex

T

1482

Time Reversal, Applications and Experiments Chaotic cavity

a

k r0

c

rtrm

Recorded signal

b rtrm

r0

1000 microseconds

Time-reversal Coupling tips Transducers

Time Reversal, Applications and Experiments, Fig. 6 (a) Geometry of the chaotic cavity. (b) Time-reversal experiment conducted in a chaotic cavity with flexural waves. In a first step, a point transducer located at point r0 transmits a 1 s long signal.

The signal is recorded at point rtrm by a second transducer. The signal spreads on more than 30 ms due to reverberation. In the second step of the experiment, a 1 ms portion of the recorded signal is time reversed and retransmitted back in the cavity

geometry that give rise to chaotic ray dynamics. Paradoxically, in the case of one-channel time reversal, chaotic dynamics is not only harmless but even useful, as it guarantees ergodicity and mixing. Secondly, using a source of vanishing aperture, there is an almost perfect focusing quality. The procedure approaches the performance of a closed TRM, which has an aperture of 360ı. Hence, a one-point time reversal in a chaotic cavity produces better results than a limited aperture TRM in an open system. Using reflections at the edge, focusing quality is not aperture limited; the timereversed collapsing wave front approaches the focal spot from all directions. Although one obtains excellent focusing, a onechannel time reversal is not perfect, as residual fluctuations can be observed. Residual temporal and spatial side lobes persist even for time-reversal windows of duration larger than the Heisenberg time. These are due to multiple reflections passing over the locations of the TR transducer and have been expressed in closed form by Draeger and Fink. Using an eigenmode analysis of

the wave field, they explain that, for long time-reversal windows, there is a minimum signal-to-noise ratio (SNR) even after the Heisenberg time. Time reversal in reverberant cavities at audible frequencies has been shown to be an efficient localizing technique in solid objects. The idea consists in detecting acoustic waves in solid objects (e.g., a table or a glass plate) generated by a slight finger knock. As in a reverberating object, a one-channel TRM has the memory of many distinct source locations, and the information location of an unknown source can then be extracted from a simulated time-reversal experiment in a computer. Any action, turn on the light or a compact disk player, for example, can be associated with each source location. Thus, the system transforms solid objects into interactive interfaces. Compared to the existing acoustic techniques, it presents the great advantage of being simple and easily applicable to inhomogeneous objects whatever their shapes. The number of possible touch locations at the surface of objects is directly related to the number of independent

Time Reversal, Applications and Experiments

1483

a

b

wafer scanned region

15 mm 15 mm

(a) tR = -8.0 ms

(b) tR = -2.2 ms

(c) tR = -1.4 ms

(d) tR = -0.5 ms

T (e) tR = +0.0 ms

(f) tR = +0.9 ms

Time Reversal, Applications and Experiments, Fig. 7 (a) Time-reversed signal observed at point r0 . The observed signal is 210 s long. (b) Time-reversed wave field observed at different times around point r0 on a square of 15  15 mm

time-reversed focal spots that can be obtained. For example, a virtual keyboard can be drawn on the surface of an object; the sound made by fingers when a text is captured is used to localize impacts. Then, the corresponding letters are displayed on a computer screen [7].

Time Reversal in Open Systems: Random Medium The ability to focus with a one-channel time-reversal mirror is not only limited to experiments conducted inside closed cavity. Similar results have also been

1484

Time Reversal, Applications and Experiments

observed in time-reversal experiments conducted in open random medium with multiple scattering [8]. Derode et al. carried out the first experimental demonstration of the reversibility of an acoustic wave propagating through a random collection of scatterers with strong multiple-scattering contributions. A multiplescattering sample is immersed between the source and a TRM array made of 128 elements. The scattering medium consists of 2,000 randomly distributed parallel steel rods (diameter 0.8 mm) arrayed over a region of thickness L D 40 mm with average distance between rods 2.3 mm. The elastic mean free path in this sample was found to be 4 mm (see Fig. 8). A source 30 cm from the 128-element TRM transmitted a short (1 s) ultrasonic pulse (3 cycles of a 3.5 MHz). Figure 9a shows one part of the waveform received by one element of the TRM. It spread over more than 200 ms, i.e., 200 times the initial pulse duration. After the arrival of a first wave front corresponding to the ballistic wave, a long diffuse wave is observed due to the multiple scattering. In the second step of the experiment, any number of signals (between 1 and 128) is time reversed and transmitted, and a hydrophone measures the time-reversed wave in the vicinity of the source. For a TRM of 128 elements, with Time Reversal, Applications and Experiments, Fig. 8 Time-reversal focusing through a random medium. In the first step, the source r0 transmits a short pulse that propagates through the rods. The scattered waves are recorded on a 128-element array. In the second step, N elements of the array (0 < N < 128) retransmit the time-reversed signals through the rods. The piezoelectric element located at r0 is now used as a detector and measures the signal reconstructed at the source position. It can also be translated along the x-axis, while the same time-reversed signals are transmitted by the array, in order to measure the directivity pattern

a time-reversal window of 300 s, the time-reversed signal received on the source is represented in Fig. 9b: an impressive compression is observed, since the received signal lasts about 1 s, against over 300 s for the scattered signals. The directivity pattern of the TR field is also plotted on Fig. 10. It shows that the resolution (i.e., the beam width around the source) is significantly finer than it is in the absence of scattering: the resolution is 30 times finer, and the background level is below 20 dB. Moreover, Fig. 11 shows that the resolution is independent of the array aperture: even with only one transducer doing the time-reversal operation, the quality of focusing is quite good and the resolution remains approximately the same as with an aperture 128 times larger. This is clearly the same effect as observed with the closed cavity. High transverse spatial frequencies that would have been lost in a homogeneous medium are redirected by the scatterers towards the array. In conclusion, these experiments illustrated the fact that in the presence of multiple reflections or multiple scattering, a small-size time-reversal mirror manages to focus a pulse back to the source with a spatial resolution that beats the diffraction limit. The resolution is no more dependent on the mirror aperture

a

r0

TRM array

b

r

TRM array

Time Reversal, Applications and Experiments

1485

a

Time Reversal, Applications and Experiments, Fig. 9 Experimental results. (a) Signal transmitted through the sample (L D 40 mm) and recorded by the array element nı 64 and (b) signal recreated at the source after time reversal

0

25

50

b

75 time (ms)

100

125

150

Normal TR, L=40 mm 1

0.5 0

−25

−50

25

50

−0.5 −1

0

T

−5 −10 dB

Time Reversal, Applications and Experiments, Fig. 10 Directivity pattern of the time-reversed waves around the source position, in water (thick line) and through the rods (thin line), with a 16-element aperture. The sample thickness is L D 40 mm. The 6 dB widths are 0.8 and 22 mm, respectively

time (ms)

−15 −20 −25 −3

−2

−1 0 1 Distance from the source (cm)

2

3

1486

0 −5 −10 dB

Time Reversal, Applications and Experiments, Fig. 11 Directivity pattern of the time-reversed waves around the source position through L D 40 mm, with N D 128 transducers (thin line) and N D 1 transducer (thick line). The 6 dB resolutions are 0.84 and 0.9 mm, respectively

Toeplitz Matrices: Computation

−15 −20 −25 −30

−10

−5 5 0 Distance from the source (mm)

10

size, but it is only limited by the spatial correlation 8. Derode, A., Roux, P., Fink, M.: Robust acoustic time reversal with high-order multiple scattering. Phys. Rev. Lett. 75(23), of the wave field. In these media, due to a sort of 4206–4209 (1995) kaleidoscopic effect that creates virtual transducers, the TRM appears to have an effective aperture that is much larger that it’s physical size. Various applications of these concepts have been developed as acoustic tactile screens and underwater acoustic telecommunication systems. Toeplitz Matrices: Computation

References 1. Fink, M.: Time reversal of ultrasonic fields – part I: basic principles. IEEE Trans. Ultrason. Ferroelec. Freq. Control 39(5), 555–566 (1992) 2. Fink, M.: Time reversed acoustics. Phys. Today 50(3), 34–40 (1997) 3. Roux, P., Roman, B., Fink, M.: Time-reversal in an ultrasonic waveguide. Appl. Phys. Lett. 70(14), 1811–1813 (1997) 4. Kuperman, W.A., Hodgkiss, W.S., Song, H.C., Akal, T., Ferla, T., Jackson, D.: Phase conjugation in the ocean: experimental demonstration of a time reversal mirror. J. Acoust. Soc. Am. 103, 25–40 (1998) 5. Edelmann, G.F., Akal, T., Hodgkiss, W.S., Kim, S., Kuperman, W.A., Song, H.C.: An initial demonstration of underwater acoustic communication using time reversal. IEEE J. Ocean. Eng. 27(3), 602–609 (2002) 6. Draeger, C., Fink, M.: One-channel time reversal of elastic waves in a chaotic 2D-silicon cavity. Phys. Rev. Lett. 79(3), 407–410 (1997) 7. Ing, R.K., Quieffin, N., Catheline, S., Fink, M.: In solid localization of finger impacts using acoustic time-reversal process. Appl. Phys. Lett. 87(20), Art. No. 204104 (2005)

Michael Kwok-Po Ng Department of Mathematics, Hong Kong Baptist University, Kowloon, Hong Kong

Description Structured matrices have been around for a long time and are encountered in various fields of application: Toeplitz matrices, matrices with constant diagonals, i.e., ŒT i;j D ti j for all 1  i; j  n: 2

t0 t1 :: :

6 6 6 T D6 6 6 4 tn2

3 t1    t2n t1n t0 t1 t2n 7 7 :: 7 :: : : 7 t1 t0 7: 7 :: :: : : t 5

tn1 tn2    t1

1

t0

(1)

Toeplitz Matrices: Computation

1487

Circulant matrices: Toeplitz matrices where each col- system. The third criterion (iii) comes from the fact that umn is a circular shift of its preceding column: the more well condition or clustered the eigenvalues are, the faster the convergence of the method will be. 2 3 c0 cn1    c2 c1 Strang and Olkin independently proposed using 6 c1 c0 cn1 7 c circulant matrices as preconditioners for Toeplitz sys2 6 7 6 :: 7 tems. Several other circulant preconditioners have then 6 c2 c1 c0 : 7 6 7 (2) been proposed and analyzed. The main important reC D6 : : 7: :: ::: ::: ::: 6 :: 7 sult of this methodology is that the complexity of 6 7 6 7 :: :: solving a large class of n  n Toeplitz systems can 4 cn2 5 : : cn1 be reduced to O.n log n/ operations, provided that a cn1 cn2    c2 c1 c0 suitable preconditioner is used. Besides the reduction The name Toeplitz originates from the work of Otto of the arithmetic complexity, there are important types Toeplitz [6] in the early 1900s on bilinear forms related of Toeplitz matrix where the fast direct Toeplitz solvers to Laurent series. More details about his work can are notoriously unstable, for example, indefinite and be found in [2]. Toeplitz systems are sets of linear certain non-Hermitian Toeplitz matrices. Therefore, equations with coefficient matrices having a Toeplitz iterative methods provide alternatives for solving these structure. These systems arise in a variety of applica- Toeplitz systems; see [4]. Recent research in this area is to find good precontions in mathematics and engineering. In fact, Toeplitz ditioners for Toeplitz-related systems with large disstructure was one of the first structures analyzed in placement rank arising from image processing. Good signal processing; see, for instance, [4]. For a disexamples are Toeplitz-plus-band systems and weighted cussion of direct Toeplitz solvers, refer to the book Toeplitz least squares problems [4]. Direct Toeplitzby Kailath and Sayed [3]. Fast direct Toeplitz solvers of complexity O.n log2 n/ were developed for n-by-n like solvers cannot be employed because of the large displacement rank. However, iterative methods are atToeplitz systems. In the 1970s, researchers have considered the de- tractive since the involved coefficient matrix-vector velopment of solely iterative methods for Toeplitz products can be computed efficiently at each iteration. systems; see, for instance, [4]. In the 1980s, the idea Some recent results using splitting-type preconditionof using the preconditioned conjugate gradient method ers and approximate inverse-type preconditioners can as an iterative method for solving Toeplitz systems be found in [1] and [5], respectively. has brought much attention. In each iteration, the Toeplitz matrix-vector multiplication can be reduced to a convolution and can be computed via Fast Fourier Transform in O.n log n/ operations. To speed up the convergence of the conjugate gradient method, one can precondition the system. Instead of solving Tn x D b, we solve the preconditioned system Pn1 Tn x D Pn1 b. The preconditioner Pn should be chosen according to the following criteria: (i) Pn should be constructed within O.n log n/ operations; (ii) Pn v D y should be solved in O.n log n/ operations for any vector y; (iii) The spectrum of Pn1 Tn should be clustered and/or the condition number of the preconditioned matrix should be close to 1. The first two criteria (i) and (ii) are to keep the operation count per iteration within O.n log n/, as it is the count for the non-preconditioned

References 1. Benzi, M., Ng, M.: Preconditioned iterative methods for weighted Toeplitz least squares problems. SIAM J. Matrix Anal. Appl. 27, 1106–1124 (2006) 2. Grenander, U., Szeg¨o, G.: Toeplitz Forms and Their Applications, 2nd edn. Chelsea, New York (1984) 3. Kailath, T., Sayed, A.: Fast Reliable Algorithms for Matrices with Structure. SIAM, Philadelphia (1998) 4. Ng, M.: Iterative Methods for Toeplitz Systems. Oxford University Press, Oxford (2004) 5. Ng, M., Pan, J.: Approximate inverse circulant-plus-diagonal preconditioners for Toeplitz-plus-diagonal matrices. SIAM J. Sci. Comput. 32, 1442–1464 (2010) 6. Toeplitz, O.: Zur Theorie der quadratischen und bilinearen Formen von unendlichvielen Ver¨anderlichen. I. Teil: Theorie der L-Formen. Math. Annal. 70, 351–376 (1911)

T

1488

Tomography, Photoacoustic, and Thermoacoustic Peter Kuchment1 and Otmar Scherzer2;3 1 Mathematics Department, Texas A&M University, College Station, TX, USA 2 Computational Science Center, University of Vienna, Vienna, Austria 3 Johann Radon Institute for Computational and Applied Mathematics (RICAM), Austrian Academy of Sciences, Linz, Austria

Synonyms CT Computerized Tomography; PAT Photoacoustic Tomography; QPAT Quantitative Photoacoustic Tomography

Description of the Modality Photoacoustic imaging is one of the recent hybrid imaging (recently this also runs under the name Coupled Physics Imaging [6]) techniques that attempts to visualize the distribution of the electromagnetic absorption coefficient inside a biological object. In photoacoustic experiments, the medium is exposed to a short pulse of an electromagnetic (EM) wave. The exposed medium absorbs a fraction of the EM energy, heats up, and reacts with thermoelastic expansion. This consequently induces acoustic waves, which can be recorded outside the object and are used to determine the electromagnetic absorption coefficient. The combination of EM and ultrasound waves (which explains the usage of the term “hybrid”) allows one to combine high contrast in the EM absorption coefficient with high resolution of ultrasound. In fact, what is commonly called PAT, only recovers the distribution of an intermediate quantity, namely, of the absorbed EM energy. The consequent reconstruction of the true EM absorption coefficient is an interesting problem by itself and is usually called the quantitative PAT (QPAT). The PAT technique has demonstrated great potential for biomedical applications, including functional brain imaging of animals [109], soft-tissue characterization, and early-stage cancer diagnostics [52], as well

Tomography, Photoacoustic, and Thermoacoustic

as imaging of vasculature [116]. In comparison with the X-ray CT, photoacoustics is non-ionizing. Its further advantage is that soft biological tissues display high contrasts in their ability to absorb electromagnetic waves in certain frequency ranges. For instance, for radiation in the near-infrared domain, as produced by a Nd:YAG laser, the absorption coefficient in human soft tissues varies in the range of 0:1–0:5=cm [19]. The contrast is also known to be high between healthy and cancerous cells, which makes photoacoustics a promising early cancer detection technique. Another application arises in biology: multispectral photoacoustic technique is capable of high-resolution visualization of fluorescent proteins deep within highly light-scattering living organisms [71,90]. In contrast, the current fluorescence microscopy techniques are limited to the depth of several hundred micrometers, due to intense light scattering. Mathematically, the problem of multispectral optoacoustic tomography was considered in [10]. Different terms are often used to indicate different excitation sources: optoacoustics refers to illumination in the visible light spectrum, photoacoustics is associated with excitations in the visible and infrared range, and thermoacoustics corresponds to excitations in the microwave or radio-frequency range. In fact, the carrier frequency of the illuminating pulse is varying, which is usually not taken into account in mathematical modeling. Since the corresponding mathematical models are equivalent, in the mathematics literature, the terms opto-, photo-, and thermoacoustics are used interchangeably. In this entry, we will be addressing only the photoacoustic tomographic technique PAT (which is mathematically equivalent to the thermoacoustic tomography TAT, although the situation changes when moving to QPAT). Various kinds of photoacoustic imaging techniques have been implemented. One should distinguish between photoacoustic microscopy (PAM) and photoacoustic tomography (PAT). In microscopy, the object is scanned pixel by pixel (voxel by voxel). The measured pressure data provides an image of the electromagnetic absorption coefficient [115]. Tomography, on the other hand, measures pressure waves with detectors surrounding completely or partially the object. Then the internal distribution of the absorption coefficients is reconstructed using mathematical inversion techniques (see the sections below).

Tomography, Photoacoustic, and Thermoacoustic

1489

The underlying mathematical equation of PAT is the measured, and the main task is to reconstruct the wave equation for the pressure initial pressure u from these data. While the excitation principle is always as described above and thus (5) dj 1 @2 p holds, the specific type of data measured depends on .x; t/  p.x; t/ D .t/u.x/; x 2 R3 ; t > 0; 2 2 the type of transducers used and thus influences the vs @t dt (1) mathematical model. We address this issue in the next where u.x/ is the absorption density and vs denotes the section. speed of sound. In the most general case of Maxwell’s equation (see [13]),

Mathematical Models for PAT with Various u.x/ D O .x/jE.x/j2 :

(2) Detector Types

Here O denotes the absorption coefficient and E is In the following we describe several detector setups the amplitude of a time-harmonic wave e i!t E.x/ and used in PAT. solves Maxwell’s equations Point Detectors In the initial experimental realization of Kruger et al. r  r  E C .! 2 n.x/ C i! O .x//E D 0 ; in ˝ ; [52], as well as in many other experimental setups, N  E D f in @˝ ; small piezocrystal ultrasound detectors (transducers) (3) are placed along an observation surface S surrounding with n being the refractive index and N the normal the object and measure pressure over a period of time. vector to @˝. We assume that detectors are sufficiently small (In fact, In [13] it is assumed that the support of the specimen transducers have a finite size, which can be taken into of interest is compact and contained in ˝ and the flux account (e.g., [110]) and a finite bandwidth, which has of the electric field f is applied. Also note that due been taken into account in [39]) to be considered as to the initialization with a time-harmonic wave, the points. Then the measurement operator maps the initial equation is independent of time. The reconstruction of pressure u.x/ in (5) to the values of pressure p on S the refractive n and absorption coefficient O are typical over time: problems of QPAT. The assumption that there is no acoustic pressure M W u./ 7! g.x; t/ WD p.x; t/; x 2 S; t  0: (6) before the object is illuminated at time t D 0 is expressed by Thus, the PAT inverse problem consists in inverting the operator M, i.e., reconstructing the initial value u of p.x; t/ D 0 ; t 0 is the trust region bound updated from iteration to iteration.

Description

Variable metric algorithms are a class of algorithms for unconstrained optimization. Consider the unconstrained optimization problem History min f .x/:

(1)

The first variable metric algorithm is the DavidonFletcher-Powell (DFP) method which was invented by Line search type variable metric algorithms have the Davidon [1] and reformulated by Fletcher and Powell following form: [3]. The DFP method updates the matrix Hk by the following formula: xkC1 D xk  ˛k Hk gk ; (2) x2 0 is a steplength obtained by some line search techniques and Hk 2 0. Please notice that the condition skT yk > 0 is always true for strictly convex function. Even for non-convex functions, skT yk > 0 also holds if ˛k is computed by certain line search techniques. Perhaps the most famous variable metric algorithm is the BFGS method, which was discovered by 1 Qk .d / D f .xk / C d T rf .xk / C d T Bk d; (3) Broyden [4], Fletcher [5], Goldfarb [6], and Shanno 2 [7] independently. The BFGS method updates the quasi-Newton matrices by where Bk D Hk1 . Normally, we require the matrix Bk to satisfy the quasi-Newton condition: Bk sk s T Bk yk y T C T k; (9) BkC1 D Bk  T k sk Bk sk sk yk (4) Bk .xk  xk1 / D gk  gk1 : Hk yk skT C sk ykT Hk Thus, variable metric algorithms are special quasiHkC1 D Hk  ykT sk Newton methods when the quasi-Newton matrices Bk  are positive definite. y T Hk yk  sk skT C 1C kT : (10) Variable metric algorithm with trust region techsk yk skT yk nique was first studied by Powell [2]. In a trust region type variable metric algorithm, we compute the trial The BFGS update can be obtained by interchanging H step dk by solving the trust region subproblem: and B and interchanging s and y in the DFP update. Hence, BF GS is called as the dual update of DFP . 1 (5) Numerical results indicate that for most problems, the minn gkT d C d T Bk d d 2< 2 BFGS method is much better than the DFP method, s. t. jjd jj2  k ; (6) and it is widely believed that it is very difficult to find

Variable Metric Algorithms

1517

a variable metric method which is much better than the BFGS method. There are many variable metric methods. For example, Broyden [4] gives a family of quasi-Newton updates: BkC1 ./ D Bk 

Bk sk skT Bk yk ykT C T C wk wTk ; (11) skT Bk sk sk yk

least change from the previous one among all the matrices satisfying the current quasi-Newton condition. For example, we have the following result (see [8]): Theorem 1 Assume that Bk is symmetric, skT yk > 0. Let M be any symmetric non-singular matrix M that satisfies M 2 sk D yk ; the solution of problem kB  Bk kM;F

min

where

Bsk D yk ; B D B

s. t. wk D

q skT Bk sk

 y Bk sk  k ;  T T sk yk sk Bk sk

 2 1=.1  ˇk k /, BkC1 ./ defined by (11) is positive definite, as long as Bk is positive definite and skT yk > 0. Thus, in this case, it leads to a variable metric algorithm. In particularly, if we restrict  2 Œ0; 1, we obtain an important special family of variable metric algorithms, which is called the Broyden convex family.

Properties of Variable Metric Algorithms Variable metric algorithms have very nice properties. First of all, it has the invariance property, which says that variable metric algorithms is invariance under linear transformations. Another important property of the variable metric algorithms is quadratic termination. Namely, variable metric algorithms with exact line searches can find the unique minimizer of a strict convex quadratic function after at most n iterations. Variable metric algorithms have a so-called least change property. Namely, the quasi-Newton updates usually imply that the new quasi-Newton matrix is a

lim

(15)

Many important results on convergence properties of quasi-Newton methods are given by Dennis and Mor´e [8]. One of them is given as follows. Theorem 3 Assume that f .x/ is twice continuously differentiable and r 2 f .x/ is Lipschitz continuous. Let x  be a minimizer of f .x/ and r 2 f .x  / positive definite. Then the DFP method and the BFGS method with ˛k D 1 converges locally Q-superlinearly if x1 is sufficiently close to x  and B1 is sufficiently close to r 2 f .x  /. For the global convergence of inexact line search quasi-Newton methods, the following pioneer result was obtained by Powell [10]. Theorem 4 Assume that f .x/ is convex and twice continuously differentiable in fxI f .x/  f .x1 /g. For any positive definite B1 , BFGS method with Wolfe line search xk either terminates finitely or lim kgk k2 D 0:

k!1

holds.

(16)

V

1518

Variable Metric Algorithms

The above result can be extended from BFGS Let k D 1=skT yk and Vk D .I  k yk skT /; repeatedly method to all methods in the Broyden convex family applying (17) gives the following formula: except DFP [11]. Theorem 5 Under the assumptions of the above theorem, let BkC1 be given by Broyden convex family, namely, BkC1 D BkC1 .k /. If there exists ı > 0, such that 1  k  ı, then either fxk g terminates finitely or (16) holds.

T HkC1 D .VkT    Vki /Hki .Vki    Vk /

C

i X

ki Cj

j D0

 i j Y1

 T T ski Cj ski Vkl Cj

lD0

 i j 1

T Y The proof of the above theorem cannot be extended T Vkl : (18) to the DFP method, which lacks the “self-correction” lD0 T T Bk sk sk Bk =sk Bk sk . Therefore, the convergence of the DFP method with inexact line search remains as an open problem [12]: Thus, the m C 1 step limited memory BFGS method uses the following update: Open Question Assume that the objective function f .x/ is uniformly convex and twice continuously dif.0/ T ferentiable. Does the DFP method with Wolfe inexact HkC1 D VkT    Vkm Hk Vkm    Vk line search converge to the unique minimizer of f .x/ m  mj  X Y1 for any starting point x1 and any positive definite initial T T skmCj skmCj C kmCj Vkl matrix B1 ? j D0 lD0 Numerical performances indicate that variable met mj T Y1 ric methods can also be applied to non-convex miniT V : (19) kl mizations. However, there are no theoretical results that lD0 ensure the global convergence of inexact line search type variable metric algorithms. Indeed, Dai [13] gives a 2-dimensional example in which the BFGS method .0/ Assume we know Hk ; we only need to store with Wolfe line search cycles near 6 points and kgk k .0/ si ; yi .i D k  m;    ; k/. One particular Hk is are bounded away from 0. T s y .0/ Hk D kyk kk2 I . Other choices of H .0/ can be found in k 2 Liu and Nocedal [14]. Another approach for reducing storage and comLimited Memory and Subspace puting cost is subspace technique. Subspace quasiTechnique Newton methods are based on the following result For middle and small problems, variable metric algo- (e.g., see [15]). rithms are very efficient, and BFGS method is one of Theorem 6 Consider the BFGS method applied to most widely used algorithm for solving middle and a general nonlinear function. If H D I. > 1 small problems. However, for large-scale problems 0/, then the search direction d 2 G D SPAN k k (when n is very large), variable metric algorithms fg ; g ; : : : ; g g for all k. Moreover, if z 2 G and 1 2 k k need huge storage and the linear algebra computation w 2 G ? , then H z D G and H w D w. k k k k cost per iteration is very high. One approach is the The above theorem is also true if BFGS method is limited memory quasi-Newton methods, which try to replaced by any method in the Broyden family. Due to reduce the memory requirement while maintaining this subspace property, in iteration k, we can obtain the certain quasi-Newton property. Let us write the BFGS search direction dk by solving a subproblem defined in update (10) in the following form: the lower dimensional subspace Gk instead of working in the whole space