Artificial Intelligence: A Modern Approach (Pearson Series in Artifical Intelligence) [4 ed.] 0134610997, 9780134610993

Citation preview

Artificial Intelligence A Modern Approach Fourth Edition

PEARSON SERIES

IN ARTIFICIAL INTELLIGENCE

Stuart Russell and Peter Norvig, Editors

Artificial Intelligence A Modern Approach Fourth Edition

Stuart J. Russell and Peter Norvig Contributing writers:

Ming-Wei Chang

Jacob Devlin

Anca Dragan

David Forsyth

Ian Goodfellow

Jitendra M. Malik

Vikash Mansinghka

Judea Pearl

Michael Wooldridge

Copyright © 2021, 2010, 2003 by Pearson Education, Inc. or its affiliates, 221 River Street, Hoboken, NJ 07030. All Rights Reserved. Manufactured in the United States of America. This publication is protected by copyright, and permission should be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise. For information regarding permissions, request forms, and the appropriate contacts within the Pearson Education Global Rights and Permissions department, please visit www.pearsoned.com/permissions/.

Acknowledgments of third-party content appear on the appropriate page within the text.

Cover Images:

Alan Turing – Science History Images/Alamy Stock Photo

Statue of Aristotle – Panos Karas/Shutterstock

Ada Lovelace – Pictorial Press Ltd/Alamy Stock Photo

Autonomous cars – Andrey Suslov/Shutterstock

Atlas Robot – Boston Dynamics, Inc.

Berkeley Campanile and Golden Gate Bridge – Ben Chu/Shutterstock

Background ghosted nodes – Eugene Sergeev/Alamy Stock Photo

Chess board with chess figure – Titania/Shutterstock

Mars Rover – Stocktrek Images, Inc./Alamy Stock Photo

Kasparov – KATHY WILLENS/AP Images

PEARSON, ALWAYS LEARNING is an exclusive trademark owned by Pearson Education, Inc. or its affiliates in the U.S. and/or other countries.

Unless otherwise indicated herein, any third-party trademarks, logos, or icons that may appear in this work are the property of their respective owners, and any references to thirdparty trademarks, logos, icons, or other trade dress are for demonstrative or descriptive purposes only. Such references are not intended to imply any sponsorship, endorsement, authorization, or promotion of Pearson’s products by the owners of such marks, or any relationship between the owner and Pearson Education, Inc., or its affiliates, authors, licensees, or distributors.

Library of Congress Cataloging-in-Publication Data

Names: Russell, Stuart J. (Stuart Jonathan), author. | Norvig, Peter, author.

Title: Artificial intelligence : a modern approach / Stuart J. Russell and Peter Norvig.

Description: Fourth edition. | Hoboken : Pearson, [2021] | Series: Pearson series in artificial intelligence | Includes bibliographical references and index. | Summary: “Updated edition of popular textbook on Artificial Intelligence.”— Provided by publisher.

Identifiers: LCCN 2019047498 | ISBN 9780134610993 (hardcover)

Subjects: LCSH: Artificial intelligence.

Classification: LCC Q335 .R86 2021 | DDC 006.3–dc23

LC record available at https://lccn.loc.gov/2019047498

ScoutAutomatedPrintCode

ISBN-10:

0-13-461099-7

ISBN-13: 978-0-13-461099-3

For Loy, Gordon, Lucy, George, and Isaac — S.J.R.

For Kris, Isabella, and Juliet — P.N.

Preface Artificial Intelligence (AI) is a big field, and this is a big book. We have tried to explore the full breadth of the field, which encompasses logic, probability, and continuous mathematics; perception, reasoning, learning, and action; fairness, trust, social good, and safety; and applications that range from microelectronic devices to robotic planetary explorers to online services with billions of users.

The subtitle of this book is “A Modern Approach.” That means we have chosen to tell the story from a current perspective. We synthesize what is now known into a common framework, recasting early work using the ideas and terminology that are prevalent today. We apologize to those whose subfields are, as a result, less recognizable.

New to this edition This edition reflects the changes in AI since the last edition in 2010:

We focus more on machine learning rather than hand-crafted knowledge engineering, due to the increased availability of data, computing resources, and new algorithms. Deep learning, probabilistic programming, and multiagent systems receive expanded coverage, each with their own chapter. The coverage of natural language understanding, robotics, and computer vision has been revised to reflect the impact of deep learning. The robotics chapter now includes robots that interact with humans and the application of reinforcement learning to robotics. Previously we defined the goal of AI as creating systems that try to maximize expected utility, where the specific utility information—the objective—is supplied by the human designers of the system. Now we no longer assume that the objective is fixed and known by the AI system; instead, the system may be uncertain about the true objectives of the humans on whose behalf it operates. It must learn what to maximize and must function appropriately even while uncertain about the objective.

We increase coverage of the impact of AI on society, including the vital issues of ethics, fairness, trust, and safety. We have moved the exercises from the end of each chapter to an online site. This allows us to continuously add to, update, and improve the exercises, to meet the needs of instructors and to reflect advances in the field and in AI-related software tools. Overall, about 25% of the material in the book is brand new. The remaining 75% has been largely rewritten to present a more unified picture of the field. 22% of the citations in this edition are to works published after 2010.

Overview of the book The main unifying theme is the idea of an intelligent agent. We define AI as the study of agents that receive percepts from the environment and perform actions. Each such agent implements a function that maps percept sequences to actions, and we cover different ways to represent these functions, such as reactive agents, real-time planners, decision-theoretic systems, and deep learning systems. We emphasize learning both as a construction method for competent systems and as a way of extending the reach of the designer into unknown environments. We treat robotics and vision not as independently defined problems, but as occurring in the service of achieving goals. We stress the importance of the task environment in determining the appropriate agent design.

Our primary aim is to convey the ideas that have emerged over the past seventy years of AI research and the past two millennia of related work. We have tried to avoid excessive formality in the presentation of these ideas, while retaining precision. We have included mathematical formulas and pseudocode algorithms to make the key ideas concrete; mathematical concepts and notation are described in Appendix A  and our pseudocode is described in Appendix B .

This book is primarily intended for use in an undergraduate course or course sequence. The book has 28 chapters, each requiring about a week’s worth of lectures, so working through the whole book requires a two-semester sequence. A one-semester course can use selected chapters to suit the interests of the instructor and students. The book can also be used in a graduate-level course (perhaps with the addition of some of the primary sources suggested in the bibliographical notes), or for self-study or as a reference.

Throughout the book, important points are marked with a triangle icon in the margin. Wherever a new term is defined, it is also noted in the margin. Subsequent significant uses of the term are in bold, but not in the margin. We have included a comprehensive index and an extensive bibliography.

Term

The only prerequisite is familiarity with basic concepts of computer science (algorithms, data structures, complexity) at a sophomore level. Freshman calculus and linear algebra are useful for some of the topics.

Online resources Online resources are available through pearsonhighered.com/cs-resources or at the book’s Web site, aima.cs.berkeley.edu. There you will find: Exercises, programming projects, and research projects. These are no longer at the end of each chapter; they are online only. Within the book, we refer to an online exercise with a name like “Exercise 6.NARY.” Instructions on the Web site allow you to find exercises by name or by topic. Implementations of the algorithms in the book in Python, Java, and other programming languages (currently hosted at github.com/aimacode). A list of over 1400 schools that have used the book, many with links to online course materials and syllabi. Supplementary material and links for students and instructors. Instructions on how to report errors in the book, in the likely event that some exist.

Book cover The cover depicts the final position from the decisive game 6 of the 1997 chess match in which the program Deep Blue defeated Garry Kasparov (playing Black), making this the first

time a computer had beaten a world champion in a chess match. Kasparov is shown at the top. To his right is a pivotal position from the second game of the historic Go match between former world champion Lee Sedol and DeepMind’s ALPHAGO program. Move 37 by ALPHAGO violated centuries of Go orthodoxy and was immediately seen by human experts as an embarrassing mistake, but it turned out to be a winning move. At top left is an Atlas humanoid robot built by Boston Dynamics. A depiction of a self-driving car sensing its environment appears between Ada Lovelace, the world’s first computer programmer, and Alan Turing, whose fundamental work defined artificial intelligence. At the bottom of the chess board are a Mars Exploration Rover robot and a statue of Aristotle, who pioneered the study of logic; his planning algorithm from De Motu Animalium appears behind the authors’ names. Behind the chess board is a probabilistic programming model used by the UN Comprehensive Nuclear-Test-Ban Treaty Organization for detecting nuclear explosions from seismic signals.

Acknowledgments It takes a global village to make a book. Over 600 people read parts of the book and made suggestions for improvement. The complete list is at aima.cs.berkeley.edu/ack.html; we are grateful to all of them. We have space here to mention only a few especially important contributors. First the contributing writers: Judea Pearl (Section 13.5 , Causal Networks); Vikash Mansinghka (Section 15.3 , Programs as Probability Models); Michael Wooldridge (Chapter 18 , Multiagent Decision Making); Ian Goodfellow (Chapter 21 , Deep Learning); Jacob Devlin and Mei-Wing Chang (Chapter 24 , Deep Learning for Natural Language); Jitendra Malik and David Forsyth (Chapter 25 , Computer Vision); Anca Dragan (Chapter 26 , Robotics).

Then some key roles:

Cynthia Yeung and Malika Cantor (project management); Julie Sussman and Tom Galloway (copyediting and writing suggestions);

Omari Stephens (illustrations); Tracy Johnson (editor); Erin Ault and Rose Kernan (cover and color conversion); Nalin Chhibber, Sam Goto, Raymond de Lacaze, Ravi Mohan, Ciaran O’Reilly, Amit Patel, Dragomir Radiv, and Samagra Sharma (online code development and mentoring); Google Summer of Code students (online code development).

Stuart would like to thank his wife, Loy Sheflott, for her endless patience and boundless wisdom. He hopes that Gordon, Lucy, George, and Isaac will soon be reading this book after they have forgiven him for working so long on it. RUGS (Russell’s Unusual Group of Students) have been unusually helpful, as always.

Peter would like to thank his parents (Torsten and Gerda) for getting him started, and his wife (Kris), children (Bella and Juliet), colleagues, boss, and friends for encouraging and tolerating him through the long hours of writing and rewriting.

About the Authors STUART RUSSELL was born in 1962 in Portsmouth, England. He received his B.A. with first-class honours in physics from Oxford University in 1982, and his Ph.D. in computer science from Stanford in 1986. He then joined the faculty of the University of California at Berkeley, where he is a professor and former chair of computer science, director of the Center for Human-Compatible AI, and holder of the Smith–Zadeh Chair in Engineering. In 1990, he received the Presidential Young Investigator Award of the National Science Foundation, and in 1995 he was cowinner of the Computers and Thought Award. He is a Fellow of the American Association for Artificial Intelligence, the Association for Computing Machinery, and the American Association for the Advancement of Science, an Honorary Fellow of Wadham College, Oxford, and an Andrew Carnegie Fellow. He held the Chaire Blaise Pascal in Paris from 2012 to 2014. He has published over 300 papers on a wide range of topics in artificial intelligence. His other books include The Use of Knowledge in Analogy and Induction, Do the Right Thing: Studies in Limited Rationality (with Eric Wefald), and Human Compatible: Artificial Intelligence and the Problem of Control.

PETER NORVIG is currently a Director of Research at Google, Inc., and was previously the director responsible for the core Web search algorithms. He co-taught an online AI class that signed up 160,000 students, helping to kick off the current round of massive open online classes. He was head of the Computational Sciences Division at NASA Ames Research Center, overseeing research and development in artificial intelligence and robotics. He received a B.S. in applied mathematics from Brown University and a Ph.D. in computer science from Berkeley. He has been a professor at the University of Southern California and a faculty member at Berkeley and Stanford. He is a Fellow of the American Association for Artificial Intelligence, the Association for Computing Machinery, the American Academy of Arts and Sciences, and the California Academy of Science. His other books are Paradigms of AI Programming: Case Studies in Common Lisp, Verbmobil: A Translation System for Face-to-Face Dialog, and Intelligent Help Systems for UNIX.

The two authors shared the inaugural AAAI/EAAI Outstanding Educator award in 2016.

Contents I Artificial Intelligence  1 Introduction 1  1.1 What Is AI? 1  1.2 The Foundations of Artificial Intelligence 5  1.3 The History of Artificial Intelligence 17  1.4 The State of the Art 27  1.5 Risks and Benefits of AI 31  Summary 34  Bibliographical and Historical Notes 35  2 Intelligent Agents 36  2.1 Agents and Environments 36  2.2 Good Behavior: The Concept of Rationality 39  2.3 The Nature of Environments 42  2.4 The Structure of Agents 47  Summary 60  Bibliographical and Historical Notes 60  II Problem-solving  3 Solving Problems by Searching 63  3.1 Problem-Solving Agents 63  3.2 Example Problems 66  3.3 Search Algorithms 71  3.4 Uninformed Search Strategies 76  3.5 Informed (Heuristic) Search Strategies 84 

3.6 Heuristic Functions 97  Summary 104  Bibliographical and Historical Notes 106  4 Search in Complex Environments 110  4.1 Local Search and Optimization Problems 110  4.2 Local Search in Continuous Spaces 119  4.3 Search with Nondeterministic Actions 122  4.4 Search in Partially Observable Environments 126  4.5 Online Search Agents and Unknown Environments 134  Summary 141  Bibliographical and Historical Notes 142  5 Adversarial Search and Games 146  5.1 Game Theory 146  5.2 Optimal Decisions in Games 148  5.3 Heuristic Alpha–Beta Tree Search 156  5.4 Monte Carlo Tree Search 161  5.5 Stochastic Games 164  5.6 Partially Observable Games 168  5.7 Limitations of Game Search Algorithms 173  Summary 174  Bibliographical and Historical Notes 175  6 Constraint Satisfaction Problems 180  6.1 Defining Constraint Satisfaction Problems 180  6.2 Constraint Propagation: Inference in CSPs 185  6.3 Backtracking Search for CSPs 191 

6.4 Local Search for CSPs 197  6.5 The Structure of Problems 199  Summary 203  Bibliographical and Historical Notes 204  III Knowledge, reasoning, and planning  7 Logical Agents 208  7.1 Knowledge-Based Agents 209  7.2 The Wumpus World 210  7.3 Logic 214  7.4 Propositional Logic: A Very Simple Logic 217  7.5 Propositional Theorem Proving 222  7.6 Effective Propositional Model Checking 232  7.7 Agents Based on Propositional Logic 237  Summary 246  Bibliographical and Historical Notes 247  8 First-Order Logic 251  8.1 Representation Revisited 251  8.2 Syntax and Semantics of First-Order Logic 256  8.3 Using First-Order Logic 265  8.4 Knowledge Engineering in First-Order Logic 271  Summary 277  Bibliographical and Historical Notes 278  9 Inference in First-Order Logic 280  9.1 Propositional vs. First-Order Inference 280  9.2 Unification and First-Order Inference 282 

9.3 Forward Chaining 286  9.4 Backward Chaining 293  9.5 Resolution 298  Summary 309  Bibliographical and Historical Notes 310  10 Knowledge Representation 314  10.1 Ontological Engineering 314  10.2 Categories and Objects 317  10.3 Events 322  10.4 Mental Objects and Modal Logic 326  10.5 Reasoning Systems for Categories 329  10.6 Reasoning with Default Information 333  Summary 337  Bibliographical and Historical Notes 338  11 Automated Planning 344  11.1 Definition of Classical Planning 344  11.2 Algorithms for Classical Planning 348  11.3 Heuristics for Planning 353  11.4 Hierarchical Planning 356  11.5 Planning and Acting in Nondeterministic Domains 365  11.6 Time, Schedules, and Resources 374  11.7 Analysis of Planning Approaches 378  Summary 379  Bibliographical and Historical Notes 380 

IV Uncertain knowledge and reasoning  12 Quantifying Uncertainty 385  12.1 Acting under Uncertainty 385  12.2 Basic Probability Notation 388  12.3 Inference Using Full Joint Distributions 395  12.4 Independence 397  12.5 Bayes’ Rule and Its Use 399  12.6 Naive Bayes Models 402  12.7 The Wumpus World Revisited 404  Summary 407  Bibliographical and Historical Notes 408  13 Probabilistic Reasoning 412  13.1 Representing Knowledge in an Uncertain Domain 412  13.2 The Semantics of Bayesian Networks 414  13.3 Exact Inference in Bayesian Networks 427  13.4 Approximate Inference for Bayesian Networks 435  13.5 Causal Networks 449  Summary 453  Bibliographical and Historical Notes 454  14 Probabilistic Reasoning over Time 461  14.1 Time and Uncertainty 461  14.2 Inference in Temporal Models 465  14.3 Hidden Markov Models 473  14.4 Kalman Filters 479  14.5 Dynamic Bayesian Networks 485 

Summary 496  Bibliographical and Historical Notes 497  15 Probabilistic Programming 500  15.1 Relational Probability Models 501  15.2 Open-Universe Probability Models 507  15.3 Keeping Track of a Complex World 514  15.4 Programs as Probability Models 519  Summary 523  Bibliographical and Historical Notes 524  16 Making Simple Decisions 528  16.1 Combining Beliefs and Desires under Uncertainty 528  16.2 The Basis of Utility Theory 529  16.3 Utility Functions 532  16.4 Multiattribute Utility Functions 540  16.5 Decision Networks 544  16.6 The Value of Information 547  16.7 Unknown Preferences 553  Summary 557  Bibliographical and Historical Notes 557  17 Making Complex Decisions 562  17.1 Sequential Decision Problems 562  17.2 Algorithms for MDPs 572  17.3 Bandit Problems 581  17.4 Partially Observable MDPs 588  17.5 Algorithms for Solving POMDPs 590 

Summary 595  Bibliographical and Historical Notes 596  18 Multiagent Decision Making 599  18.1 Properties of Multiagent Environments 599  18.2 Non-Cooperative Game Theory 605  18.3 Cooperative Game Theory 626  18.4 Making Collective Decisions 632  Summary 645  Bibliographical and Historical Notes 646  V Machine Learning  19 Learning from Examples 651  19.1 Forms of Learning 651  19.2 Supervised Learning 653  19.3 Learning Decision Trees 657  19.4 Model Selection and Optimization 665  19.5 The Theory of Learning 672  19.6 Linear Regression and Classification 676  19.7 Nonparametric Models 686  19.8 Ensemble Learning 696  19.9 Developing Machine Learning Systems 704  Summary 714  Bibliographical and Historical Notes 715  20 Learning Probabilistic Models 721  20.1 Statistical Learning 721  20.2 Learning with Complete Data 724 

20.3 Learning with Hidden Variables: The EM Algorithm 737  Summary 746  Bibliographical and Historical Notes 747  21 Deep Learning 750  21.1 Simple Feedforward Networks 751  21.2 Computation Graphs for Deep Learning 756  21.3 Convolutional Networks 760  21.4 Learning Algorithms 765  21.5 Generalization 768  21.6 Recurrent Neural Networks 772  21.7 Unsupervised Learning and Transfer Learning 775  21.8 Applications 782  Summary 784  Bibliographical and Historical Notes 785  22 Reinforcement Learning 789  22.1 Learning from Rewards 789  22.2 Passive Reinforcement Learning 791  22.3 Active Reinforcement Learning 797  22.4 Generalization in Reinforcement Learning 803  22.5 Policy Search 810  22.6 Apprenticeship and Inverse Reinforcement Learning 812  22.7 Applications of Reinforcement Learning 815  Summary 818  Bibliographical and Historical Notes 819 

VI Communicating, perceiving, and acting  23 Natural Language Processing 823  23.1 Language Models 823  23.2 Grammar 833  23.3 Parsing 835  23.4 Augmented Grammars 841  23.5 Complications of Real Natural Language 845  23.6 Natural Language Tasks 849  Summary 850  Bibliographical and Historical Notes 851  24 Deep Learning for Natural Language Processing 856  24.1 Word Embeddings 856  24.2 Recurrent Neural Networks for NLP 860  24.3 Sequence-to-Sequence Models 864  24.4 The Transformer Architecture 868  24.5 Pretraining and Transfer Learning 871  24.6 State of the art 875  Summary 878  Bibliographical and Historical Notes 878  25 Computer Vision 881  25.1 Introduction 881  25.2 Image Formation 882  25.3 Simple Image Features 888  25.4 Classifying Images 895  25.5 Detecting Objects 899 

25.6 The 3D World 901  25.7 Using Computer Vision 906  Summary 919  Bibliographical and Historical Notes 920  26 Robotics 925  26.1 Robots 925  26.2 Robot Hardware 926  26.3 What kind of problem is robotics solving? 930  26.4 Robotic Perception 931  26.5 Planning and Control 938  26.6 Planning Uncertain Movements 956  26.7 Reinforcement Learning in Robotics 958  26.8 Humans and Robots 961  26.9 Alternative Robotic Frameworks 968  26.10 Application Domains 971  Summary 974  Bibliographical and Historical Notes 975  VII Conclusions  27 Philosophy, Ethics, and Safety of AI 981  27.1 The Limits of AI 981  27.2 Can Machines Really Think? 984  27.3 The Ethics of AI 986  Summary 1005  Bibliographical and Historical Notes 1006  28 The Future of AI 1012 

28.1 AI Components 1012  28.2 AI Architectures 1018  A Mathematical Background 1023  A.1 Complexity Analysis and O() Notation 1023  A.2 Vectors, Matrices, and Linear Algebra 1025  A.3 Probability Distributions 1027  Bibliographical and Historical Notes 1029  B Notes on Languages and Algorithms 1030  B.1 Defining Languages with Backus–Naur Form (BNF) 1030  B.2 Describing Algorithms with Pseudocode 1031  B.3 Online Supplemental Material 1032  Bibliography 1033  Index 1069 

I Artificial Intelligence

Chapter 1 Introduction In which we try to explain why we consider artificial intelligence to be a subject most worthy of study, and in which we try to decide what exactly it is, this being a good thing to decide before embarking.

We call ourselves Homo sapiens—man the wise—because our intelligence is so important to us. For thousands of years, we have tried to understand how we think and act—that is, how our brain, a mere handful of matter, can perceive, understand, predict, and manipulate a world far larger and more complicated than itself. The field of artificial intelligence, or AI, is concerned with not just understanding but also building intelligent entities—machines that can compute how to act effectively and safely in a wide variety of novel situations.

Intelligence

Artificial Intelligence

Surveys regularly rank AI as one of the most interesting and fastest-growing fields, and it is already generating over a trillion dollars a year in revenue. AI expert Kai-Fu Lee predicts that its impact will be “more than anything in the history of mankind.” Moreover, the intellectual frontiers of AI are wide open. Whereas a student of an older science such as physics might feel that the best ideas have already been discovered by Galileo, Newton, Curie, Einstein, and the rest, AI still has many openings for full-time masterminds.

AI currently encompasses a huge variety of subfields, ranging from the general (learning, reasoning, perception, and so on) to the specific, such as playing chess, proving

mathematical theorems, writing poetry, driving a car, or diagnosing diseases. AI is relevant to any intellectual task; it is truly a universal field.

1.1 What Is AI? We have claimed that AI is interesting, but we have not said what it is. Historically, researchers have pursued several different versions of AI. Some have defined intelligence in terms of fidelity to human performance, while others prefer an abstract, formal definition of intelligence called rationality—loosely speaking, doing the “right thing.” The subject matter itself also varies: some consider intelligence to be a property of internal thought processes and reasoning, while others focus on intelligent behavior, an external characterization.1

1 In the public eye, there is sometimes confusion between the terms “artificial intelligence” and “machine learning.” Machine learning is a subfield of AI that studies the ability to improve performance based on experience. Some AI systems use machine learning methods to achieve competence, but some do not.

Rationality

From these two dimensions—human vs. rational2 and thought vs. behavior—there are four possible combinations, and there have been adherents and research programs for all four. The methods used are necessarily different: the pursuit of human-like intelligence must be in part an empirical science related to psychology, involving observations and hypotheses about actual human behavior and thought processes; a rationalist approach, on the other hand, involves a combination of mathematics and engineering, and connects to statistics, control theory, and economics. The various groups have both disparaged and helped each other. Let us look at the four approaches in more detail.

2 We are not suggesting that humans are “irrational” in the dictionary sense of “deprived of normal mental clarity.” We are merely conceding that human decisions are not always mathematically perfect.

1.1.1 Acting humanly: The Turing test approach

Turing test

The Turing test, proposed by Alan Turing (1950), was designed as a thought experiment that would sidestep the philosophical vagueness of the question “Can a machine think?” A computer passes the test if a human interrogator, after posing some written questions, cannot tell whether the written responses come from a person or from a computer. Chapter 27  discusses the details of the test and whether a computer would really be intelligent if it passed. For now, we note that programming a computer to pass a rigorously applied test provides plenty to work on. The computer would need the following capabilities:

natural language processing to communicate successfully in a human language; knowledge representation to store what it knows or hears; automated reasoning to answer questions and to draw new conclusions; machine learning to adapt to new circumstances and to detect and extrapolate patterns.

Natural language processing

Knowledge representation

Automated reasoning

Machine learning

Total Turing test

Turing viewed the physical simulation of a person as unnecessary to demonstrate intelligence. However, other researchers have proposed a total Turing test, which requires interaction with objects and people in the real world. To pass the total Turing test, a robot will need

computer vision and speech recognition to perceive the world; robotics to manipulate objects and move about.

Computer vision

Robotics

These six disciplines compose most of AI. Yet AI researchers have devoted little effort to passing the Turing test, believing that it is more important to study the underlying principles of intelligence. The quest for “artificial flight” succeeded when engineers and inventors stopped imitating birds and started using wind tunnels and learning about aerodynamics. Aeronautical engineering texts do not define the goal of their field as making “machines that fly so exactly like pigeons that they can fool even other pigeons.”

1.1.2 Thinking humanly: The cognitive modeling approach To say that a program thinks like a human, we must know how humans think. We can learn about human thought in three ways:

introspection—trying to catch our own thoughts as they go by; psychological experiments—observing a person in action; brain imaging—observing the brain in action.

Introspection

Psychological experiments

Brain imaging

Once we have a sufficiently precise theory of the mind, it becomes possible to express the theory as a computer program. If the program’s input–output behavior matches corresponding human behavior, that is evidence that some of the program’s mechanisms could also be operating in humans.

For example, Allen Newell and Herbert Simon, who developed GPS, the “General Problem Solver” (Newell and Simon 1961), were not content merely to have their program solve problems correctly. They were more concerned with comparing the sequence and timing of its reasoning steps to those of human subjects solving the same problems. The interdisciplinary field of cognitive science brings together computer models from AI and experimental techniques from psychology to construct precise and testable theories of the human mind.

Cognitive science

Cognitive science is a fascinating field in itself, worthy of several textbooks and at least one encyclopedia (Wilson and Keil 1999). We will occasionally comment on similarities or differences between AI techniques and human cognition. Real cognitive science, however, is necessarily based on experimental investigation of actual humans or animals. We will leave that for other books, as we assume the reader has only a computer for experimentation.

In the early days of AI there was often confusion between the approaches. An author would argue that an algorithm performs well on a task and that it is therefore a good model of human performance, or vice versa. Modern authors separate the two kinds of claims; this

distinction has allowed both AI and cognitive science to develop more rapidly. The two fields fertilize each other, most notably in computer vision, which incorporates neurophysiological evidence into computational models. Recently, the combination of neuroimaging methods combined with machine learning techniques for analyzing such data has led to the beginnings of a capability to “read minds”—that is, to ascertain the semantic content of a person’s inner thoughts. This capability could, in turn, shed further light on how human cognition works.

1.1.3 Thinking rationally: The “laws of thought” approach The Greek philosopher Aristotle was one of the first to attempt to codify “right thinking”— that is, irrefutable reasoning processes. His syllogisms provided patterns for argument structures that always yielded correct conclusions when given correct premises. The canonical example starts with Socrates is a man and all men are mortal and concludes that Socrates is mortal. (This example is probably due to Sextus Empiricus rather than Aristotle.) These laws of thought were supposed to govern the operation of the mind; their study initiated the field called logic.

Syllogisms

Logicians in the 19th century developed a precise notation for statements about objects in the world and the relations among them. (Contrast this with ordinary arithmetic notation, which provides only for statements about numbers.) By 1965, programs could, in principle, solve any solvable problem described in logical notation. The so-called logicist tradition within artificial intelligence hopes to build on such programs to create intelligent systems.

Logicist

Logic as conventionally understood requires knowledge of the world that is certain—a condition that, in reality, is seldom achieved. We simply don’t know the rules of, say,

politics or warfare in the same way that we know the rules of chess or arithmetic. The theory of probability fills this gap, allowing rigorous reasoning with uncertain information. In principle, it allows the construction of a comprehensive model of rational thought, leading from raw perceptual information to an understanding of how the world works to predictions about the future. What it does not do, is generate intelligent behavior. For that, we need a theory of rational action. Rational thought, by itself, is not enough.

Probability

1.1.4 Acting rationally: The rational agent approach

Agent

An agent is just something that acts (agent comes from the Latin agere, to do). Of course, all computer programs do something, but computer agents are expected to do more: operate autonomously, perceive their environment, persist over a prolonged time period, adapt to change, and create and pursue goals. A rational agent is one that acts so as to achieve the best outcome or, when there is uncertainty, the best expected outcome.

Rational agent

In the “laws of thought” approach to AI, the emphasis was on correct inferences. Making correct inferences is sometimes part of being a rational agent, because one way to act rationally is to deduce that a given action is best and then to act on that conclusion. On the other hand, there are ways of acting rationally that cannot be said to involve inference. For

example, recoiling from a hot stove is a reflex action that is usually more successful than a slower action taken after careful deliberation.

All the skills needed for the Turing test also allow an agent to act rationally. Knowledge representation and reasoning enable agents to reach good decisions. We need to be able to generate comprehensible sentences in natural language to get by in a complex society. We need learning not only for erudition, but also because it improves our ability to generate effective behavior, especially in circumstances that are new.

The rational-agent approach to AI has two advantages over the other approaches. First, it is more general than the “laws of thought” approach because correct inference is just one of several possible mechanisms for achieving rationality. Second, it is more amenable to scientific development. The standard of rationality is mathematically well defined and completely general. We can often work back from this specification to derive agent designs that provably achieve it—something that is largely impossible if the goal is to imitate human behavior or thought processes.

For these reasons, the rational-agent approach to AI has prevailed throughout most of the field’s history. In the early decades, rational agents were built on logical foundations and formed definite plans to achieve specific goals. Later, methods based on probability theory and machine learning allowed the creation of agents that could make decisions under uncertainty to attain the best expected outcome. In a nutshell, AI has focused on the study and construction of agents that do the right thing. What counts as the right thing is defined by the objective that we provide to the agent. This general paradigm is so pervasive that we might call it the standard model. It prevails not only in AI, but also in control theory, where a controller minimizes a cost function; in operations research, where a policy maximizes a sum of rewards; in statistics, where a decision rule minimizes a loss function; and in economics, where a decision maker maximizes utility or some measure of social welfare.

Do the right thing

Standard model

We need to make one important refinement to the standard model to account for the fact that perfect rationality—always taking the exactly optimal action—is not feasible in complex environments. The computational demands are just too high. Chapters 5  and 17  deal with the issue of limited rationality—acting appropriately when there is not enough time to do all the computations one might like. However, perfect rationality often remains a good starting point for theoretical analysis.

Limited rationality

1.1.5 Beneficial machines The standard model has been a useful guide for AI research since its inception, but it is probably not the right model in the long run. The reason is that the standard model assumes that we will supply a fully specified objective to the machine.

For an artificially defined task such as chess or shortest-path computation, the task comes with an objective built in—so the standard model is applicable. As we move into the real world, however, it becomes more and more difficult to specify the objective completely and correctly. For example, in designing a self-driving car, one might think that the objective is to reach the destination safely. But driving along any road incurs a risk of injury due to other errant drivers, equipment failure, and so on; thus, a strict goal of safety requires staying in the garage. There is a tradeoff between making progress towards the destination and incurring a risk of injury. How should this tradeoff be made? Furthermore, to what extent can we allow the car to take actions that would annoy other drivers? How much should the car moderate its acceleration, steering, and braking to avoid shaking up the passenger? These kinds of questions are difficult to answer a priori. They are particularly problematic in the general area of human–robot interaction, of which the self-driving car is one example.

The problem of achieving agreement between our true preferences and the objective we put into the machine is called the value alignment problem: the values or objectives put into

the machine must be aligned with those of the human. If we are developing an AI system in the lab or in a simulator—as has been the case for most of the field’s history—there is an easy fix for an incorrectly specified objective: reset the system, fix the objective, and try again. As the field progresses towards increasingly capable intelligent systems that are deployed in the real world, this approach is no longer viable. A system deployed with an incorrect objective will have negative consequences. Moreover, the more intelligent the system, the more negative the consequences.

Value alignment problem

Returning to the apparently unproblematic example of chess, consider what happens if the machine is intelligent enough to reason and act beyond the confines of the chessboard. In that case, it might attempt to increase its chances of winning by such ruses as hypnotizing or blackmailing its opponent or bribing the audience to make rustling noises during its opponent’s thinking time.3 It might also attempt to hijack additional computing power for itself. These behaviors are not “unintelligent” or “insane”; they are a logical consequence of defining winning as the sole objective for the machine.

3 In one of the first books on chess, Ruy Lopez (1561) wrote, “Always place the board so the sun is in your opponent’s eyes.”

It is impossible to anticipate all the ways in which a machine pursuing a fixed objective might misbehave. There is good reason, then, to think that the standard model is inadequate. We don’t want machines that are intelligent in the sense of pursuing their objectives; we want them to pursue our objectives. If we cannot transfer those objectives perfectly to the machine, then we need a new formulation—one in which the machine is pursuing our objectives, but is necessarily uncertain as to what they are. When a machine knows that it doesn’t know the complete objective, it has an incentive to act cautiously, to ask permission, to learn more about our preferences through observation, and to defer to human control. Ultimately, we want agents that are provably beneficial to humans. We will return to this topic in Section 1.5 .

Provably beneficial

1.2 The Foundations of Artificial Intelligence In this section, we provide a brief history of the disciplines that contributed ideas, viewpoints, and techniques to AI. Like any history, this one concentrates on a small number of people, events, and ideas and ignores others that also were important. We organize the history around a series of questions. We certainly would not wish to give the impression that these questions are the only ones the disciplines address or that the disciplines have all been working toward AI as their ultimate fruition.

1.2.1 Philosophy Can formal rules be used to draw valid conclusions? How does the mind arise from a physical brain? Where does knowledge come from? How does knowledge lead to action?

Aristotle (384–322

BCE)

was the first to formulate a precise set of laws governing the rational

part of the mind. He developed an informal system of syllogisms for proper reasoning, which in principle allowed one to generate conclusions mechanically, given initial premises.

Ramon Llull (c. 1232–1315) devised a system of reasoning published as Ars Magna or The Great Art (1305). Llull tried to implement his system using an actual mechanical device: a set of paper wheels that could be rotated into different permutations.

Around 1500, Leonardo da Vinci (1452–1519) designed but did not build a mechanical calculator; recent reconstructions have shown the design to be functional. The first known calculating machine was constructed around 1623 by the German scientist Wilhelm Schickard (1592–1635). Blaise Pascal (1623–1662) built the Pascaline in 1642 and wrote that it “produces effects which appear nearer to thought than all the actions of animals.” Gottfried Wilhelm Leibniz (1646–1716) built a mechanical device intended to carry out operations on concepts rather than numbers, but its scope was rather limited. In his 1651 book Leviathan, Thomas Hobbes (1588–1679) suggested the idea of a thinking machine, an “artificial animal” in his words, arguing “For what is the heart but a spring; and the nerves, but so many strings; and the joints, but so many wheels.” He also suggested that reasoning

was like numerical computation: “For ‘reason’ ... is nothing but ‘reckoning,’ that is adding and subtracting.”

It’s one thing to say that the mind operates, at least in part, according to logical or numerical rules, and to build physical systems that emulate some of those rules. It’s another to say that the mind itself is such a physical system. René Descartes (1596–1650) gave the first clear discussion of the distinction between mind and matter. He noted that a purely physical conception of the mind seems to leave little room for free will. If the mind is governed entirely by physical laws, then it has no more free will than a rock “deciding” to fall downward. Descartes was a proponent of dualism. He held that there is a part of the human mind (or soul or spirit) that is outside of nature, exempt from physical laws. Animals, on the other hand, did not possess this dual quality; they could be treated as machines.

Dualism

An alternative to dualism is materialism, which holds that the brain’s operation according to the laws of physics constitutes the mind. Free will is simply the way that the perception of available choices appears to the choosing entity. The terms physicalism and naturalism are also used to describe this view that stands in contrast to the supernatural.

Given a physical mind that manipulates knowledge, the next problem is to establish the source of knowledge. The empiricism movement, starting with Francis Bacon’s (1561–1626) Novum Organum,4 is characterized by a dictum of John Locke (1632–1704): “Nothing is in the understanding, which was not first in the senses.”

4 The Novum Organum is an update of Aristotle’s Organon, or instrument of thought.

Empiricism

Induction

David Hume’s (1711–1776) A Treatise of Human Nature (Hume, 1739) proposed what is now known as the principle of induction: that general rules are acquired by exposure to repeated associations between their elements.

Building on the work of Ludwig Wittgenstein (1889–1951) and Bertrand Russell (1872– 1970), the famous Vienna Circle [2017], a group of philosophers and mathematicians meeting in Vienna in the 1920s and 1930s, developed the doctrine of logical positivism. This doctrine holds that all knowledge can be characterized by logical theories connected, ultimately, to observation sentences that correspond to sensory inputs; thus logical positivism combines rationalism and empiricism.

Logical positivism

Observation sentence

The confirmation theory of Rudolf Carnap (1891–1970) and Carl Hempel (1905–1997) attempted to analyze the acquisition of knowledge from experience by quantifying the degree of belief that should be assigned to logical sentences based on their connection to observations that confirm or disconfirm them. Carnap’s book The Logical Structure of the World (1928) was perhaps the first theory of mind as a computational process.

Confirmation theory

The final element in the philosophical picture of the mind is the connection between knowledge and action. This question is vital to AI because intelligence requires action as well as reasoning. Moreover, only by understanding how actions are justified can we understand how to build an agent whose actions are justifiable (or rational).

Aristotle argued (in De Motu Animalium) that actions are justified by a logical connection between goals and knowledge of the action’s outcome:

But how does it happen that thinking is sometimes accompanied by action and sometimes not, sometimes by motion, and sometimes not? It looks as if almost the same thing happens as in the case of reasoning and making inferences about unchanging objects. But in that case the end is a speculative proposition … whereas here the conclusion which results from the two premises is an action. … I need covering; a cloak is a covering. I need a cloak. What I need, I have to make; I need a cloak. I have to make a cloak. And the conclusion, the “I have to make a cloak,” is an action.

In the Nicomachean Ethics (Book III. 3, 1112b), Aristotle further elaborates on this topic, suggesting an algorithm:

We deliberate not about ends, but about means. For a doctor does not deliberate whether he shall heal, nor an orator whether he shall persuade, … They assume the end and consider how and by what means it is attained, and if it seems easily and best produced thereby; while if it is achieved by one means only they consider how it will be achieved by this and by what means this will be achieved, till they come to the first cause, … and what is last in the order of analysis seems to be first in the order of becoming. And if we come on an impossibility, we give up the search, e.g., if we need money and this cannot be got; but if a thing appears possible we try to do it.

Aristotle’s algorithm was implemented 2300 years later by Newell and Simon in their General Problem Solver program. We would now call it a greedy regression planning system (see Chapter 11 ). Methods based on logical planning to achieve definite goals dominated the first few decades of theoretical research in AI.

Utility

Thinking purely in terms of actions achieving goals is often useful but sometimes inapplicable. For example, if there are several different ways to achieve a goal, there needs to be some way to choose among them. More importantly, it may not be possible to achieve a goal with certainty, but some action must still be taken. How then should one decide? Antoine Arnauld (1662), analyzing the notion of rational decisions in gambling, proposed a quantitative formula for maximizing the expected monetary value of the outcome. Later, Daniel Bernoulli (1738) introduced the more general notion of utility to capture the internal, subjective value of an outcome. The modern notion of rational decision making under uncertainty involves maximizing expected utility, as explained in Chapter 16 .

In matters of ethics and public policy, a decision maker must consider the interests of multiple individuals. Jeremy Bentham (1823) and John Stuart Mill (1863) promoted the idea of utilitarianism: that rational decision making based on maximizing utility should apply to all spheres of human activity, including public policy decisions made on behalf of many individuals. Utilitarianism is a specific kind of consequentialism: the idea that what is right and wrong is determined by the expected outcomes of an action.

Utilitarianism

In contrast, Immanuel Kant, in 1875 proposed a theory of rule-based or deontological ethics, in which “doing the right thing” is determined not by outcomes but by universal social laws that govern allowable actions, such as “don’t lie” or “don’t kill.” Thus, a utilitarian could tell a white lie if the expected good outweighs the bad, but a Kantian would be bound not to, because lying is inherently wrong. Mill acknowledged the value of rules, but understood them as efficient decision procedures compiled from first-principles reasoning about consequences. Many modern AI systems adopt exactly this approach.

Deontological ethics

1.2.2 Mathematics What are the formal rules to draw valid conclusions? What can be computed? How do we reason with uncertain information?

Philosophers staked out some of the fundamental ideas of AI, but the leap to a formal science required the mathematization of logic and probability and the introduction of a new branch of mathematics: computation.

The idea of formal logic can be traced back to the philosophers of ancient Greece, India, and China, but its mathematical development really began with the work of George Boole (1815–1864), who worked out the details of propositional, or Boolean, logic (Boole, 1847). In 1879, Gottlob Frege (1848–1925) extended Boole’s logic to include objects and relations, creating the first-order logic that is used today.5 In addition to its central role in the early period of AI research, first-order logic motivated the work of Gödel and Turing that underpinned computation itself, as we explain below.

5 Frege’s proposed notation for first-order logic—an arcane combination of textual and geometric features—never became popular.

Formal logic

The theory of probability can be seen as generalizing logic to situations with uncertain information—a consideration of great importance for AI. Gerolamo Cardano (1501–1576) first framed the idea of probability, describing it in terms of the possible outcomes of gambling events. In 1654, Blaise Pascal (1623–1662), in a letter to Pierre Fermat (1601– 1665), showed how to predict the future of an unfinished gambling game and assign average payoffs to the gamblers. Probability quickly became an invaluable part of the quantitative sciences, helping to deal with uncertain measurements and incomplete theories. Jacob Bernoulli (1654–1705, uncle of Daniel), Pierre Laplace (1749–1827), and others advanced the theory and introduced new statistical methods. Thomas Bayes (1702–1761) proposed a rule for updating probabilities in the light of new evidence; Bayes’ rule is a crucial tool for AI systems.

Probability

Statistics

The formalization of probability, combined with the availability of data, led to the emergence of statistics as a field. One of the first uses was John Graunt’s analysis of London census data in 1662. Ronald Fisher is considered the first modern statistician (Fisher, 1922). He brought together the ideas of probability, experiment design, analysis of data, and computing—in 1919, he insisted that he couldn’t do his work without a mechanical calculator called the MILLIONAIRE (the first calculator that could do multiplication), even though the cost of the calculator was more than his annual salary (Ross, 2012).

The history of computation is as old as the history of numbers, but the first nontrivial algorithm is thought to be Euclid’s algorithm for computing greatest common divisors. The word algorithm comes from Muhammad ibn Musa al-Khwarizmi, a 9th century mathematician, whose writings also introduced Arabic numerals and algebra to Europe. Boole and others discussed algorithms for logical deduction, and, by the late 19th century, efforts were under way to formalize general mathematical reasoning as logical deduction.

Algorithm

Kurt Gödel (1906–1978) showed that there exists an effective procedure to prove any true statement in the first-order logic of Frege and Russell, but that first-order logic could not capture the principle of mathematical induction needed to characterize the natural numbers. In 1931, Gödel showed that limits on deduction do exist. His incompleteness theorem showed that in any formal theory as strong as Peano arithmetic (the elementary theory of natural numbers), there are necessarily true statements that have no proof within the theory.

Incompleteness theorem

This fundamental result can also be interpreted as showing that some functions on the integers cannot be represented by an algorithm—that is, they cannot be computed. This motivated Alan Turing (1912–1954) to try to characterize exactly which functions are computable—capable of being computed by an effective procedure. The Church–Turing thesis proposes to identify the general notion of computability with functions computed by a Turing machine (Turing, 1936). Turing also showed that there were some functions that no Turing machine can compute. For example, no machine can tell in general whether a given program will return an answer on a given input or run forever.

Computability

Although computability is important to an understanding of computation, the notion of tractability has had an even greater impact on AI. Roughly speaking, a problem is called intractable if the time required to solve instances of the problem grows exponentially with the size of the instances. The distinction between polynomial and exponential growth in complexity was first emphasized in the mid-1960s (Cobham, 1964; Edmonds, 1965). It is important because exponential growth means that even moderately large instances cannot be solved in any reasonable time.

Tractability

The theory of NP-completeness, pioneered by Cook (1971) and Karp (1972), provides a basis for analyzing the tractability of problems: any problem class to which the class of NPcomplete problems can be reduced is likely to be intractable. (Although it has not been proved that NP-complete problems are necessarily intractable, most theoreticians believe

it.) These results contrast with the optimism with which the popular press greeted the first computers—“Electronic Super-Brains” that were “Faster than Einstein!” Despite the increasing speed of computers, careful use of resources and necessary imperfection will characterize intelligent systems. Put crudely, the world is an extremely large problem instance!

NP-completeness

1.2.3 Economics How should we make decisions in accordance with our preferences? How should we do this when others may not go along? How should we do this when the payoff may be far in the future?

The science of economics originated in 1776, when Adam Smith (1723–1790) published An Inquiry into the Nature and Causes of the Wealth of Nations. Smith proposed to analyze economies as consisting of many individual agents attending to their own interests. Smith was not, however, advocating financial greed as a moral position: his earlier (1759) book The Theory of Moral Sentiments begins by pointing out that concern for the well-being of others is an essential component of the interests of every individual.

Most people think of economics as being about money, and indeed the first mathematical analysis of decisions under uncertainty, the maximum-expected-value formula of Arnauld (1662), dealt with the monetary value of bets. Daniel Bernoulli (1738) noticed that this formula didn’t seem to work well for larger amounts of money, such as investments in maritime trading expeditions. He proposed instead a principle based on maximization of expected utility, and explained human investment choices by proposing that the marginal utility of an additional quantity of money diminished as one acquired more money.

Léon Walras (pronounced “Valrasse”) (1834–1910) gave utility theory a more general foundation in terms of preferences between gambles on any outcomes (not just monetary outcomes). The theory was improved by Ramsey (1931) and later by John von Neumann

and Oskar Morgenstern in their book The Theory of Games and Economic Behavior (1944). Economics is no longer the study of money; rather it is the study of desires and preferences.

Decision theory, which combines probability theory with utility theory, provides a formal and complete framework for individual decisions (economic or otherwise) made under uncertainty—that is, in cases where probabilistic descriptions appropriately capture the decision maker’s environment. This is suitable for “large” economies where each agent need pay no attention to the actions of other agents as individuals. For “small” economies, the situation is much more like a game: the actions of one player can significantly affect the utility of another (either positively or negatively). Von Neumann and Morgenstern’s development of game theory (see also Luce and Raiffa, 1957) included the surprising result that, for some games, a rational agent should adopt policies that are (or least appear to be) randomized. Unlike decision theory, game theory does not offer an unambiguous prescription for selecting actions. In AI, decisions involving multiple agents are studied under the heading of multiagent systems (Chapter 18 ).

Decision theory

Economists, with some exceptions, did not address the third question listed above: how to make rational decisions when payoffs from actions are not immediate but instead result from several actions taken in sequence. This topic was pursued in the field of operations research, which emerged in World War II from efforts in Britain to optimize radar installations, and later found innumerable civilian applications. The work of Richard Bellman (1957) formalized a class of sequential decision problems called Markov decision processes, which we study in Chapter 17  and, under the heading of reinforcement learning, in Chapter 22 .

Operations research

Work in economics and operations research has contributed much to our notion of rational agents, yet for many years AI research developed along entirely separate paths. One reason was the apparent complexity of making rational decisions. The pioneering AI researcher Herbert Simon (1916–2001) won the Nobel Prize in economics in 1978 for his early work showing that models based on satisficing—making decisions that are “good enough,” rather than laboriously calculating an optimal decision—gave a better description of actual human behavior (Simon, 1947). Since the 1990s, there has been a resurgence of interest in decisiontheoretic techniques for AI.

Satisficing

1.2.4 Neuroscience How do brains process information?

Neuroscience is the study of the nervous system, particularly the brain. Although the exact way in which the brain enables thought is one of the great mysteries of science, the fact that it does enable thought has been appreciated for thousands of years because of the evidence that strong blows to the head can lead to mental incapacitation. It has also long been known that human brains are somehow different; in about 335 BCE Aristotle wrote, “Of all the animals, man has the largest brain in proportion to his size.”6 Still, it was not until the middle of the 18th century that the brain was widely recognized as the seat of consciousness. Before then, candidate locations included the heart and the spleen.

6 It has since been discovered that the tree shrew and some bird species exceed the human brain/body ratio.

Neuroscience

Paul Broca’s (1824–1880) investigation of aphasia (speech deficit) in brain-damaged patients in 1861 initiated the study of the brain’s functional organization by identifying a localized

area in the left hemisphere—now called Broca’s area—that is responsible for speech production.7 By that time, it was known that the brain consisted largely of nerve cells, or neurons, but it was not until 1873 that Camillo Golgi (1843–1926) developed a staining technique allowing the observation of individual neurons (see Figure 1.1 ). This technique was used by Santiago Ramon y Cajal (1852–1934) in his pioneering studies of neuronal organization.8 It is now widely accepted that cognitive functions result from the electrochemical operation of these structures. That is, a collection of simple cells can lead to thought, action, and consciousness. In the pithy words of John Searle (1992), brains cause minds.

7 Many cite Alexander Hood (1824) as a possible prior source.

8 Golgi persisted in his belief that the brain’s functions were carried out primarily in a continuous medium in which neurons were embedded, whereas Cajal propounded the “neuronal doctrine.” The two shared the Nobel Prize in 1906 but gave mutually antagonistic acceptance speeches.

Figure 1.1

The parts of a nerve cell or neuron. Each neuron consists of a cell body, or soma, that contains a cell nucleus. Branching out from the cell body are a number of fibers called dendrites and a single long fiber called the axon. The axon stretches out for a long distance, much longer than the scale in this diagram indicates. Typically, an axon is 1 cm long (100 times the diameter of the cell body), but can reach up to 1 meter. A neuron makes connections with 10 to 100,000 other neurons at junctions called synapses. Signals are propagated from neuron to neuron by a complicated electrochemical reaction. The signals control brain activity in the short term and also enable long-term changes in the connectivity of neurons. These mechanisms are thought to form the basis for learning in the brain. Most information processing

goes on in the cerebral cortex, the outer layer of the brain. The basic organizational unit appears to be a column of tissue about 0.5 mm in diameter, containing about 20,000 neurons and extending the full depth of the cortex (about 4 mm in humans).

Neuron

We now have some data on the mapping between areas of the brain and the parts of the body that they control or from which they receive sensory input. Such mappings are able to change radically over the course of a few weeks, and some animals seem to have multiple maps. Moreover, we do not fully understand how other areas can take over functions when one area is damaged. There is almost no theory on how an individual memory is stored or on how higher-level cognitive functions operate.

The measurement of intact brain activity began in 1929 with the invention by Hans Berger of the electroencephalograph (EEG). The development of functional magnetic resonance imaging (fMRI) (Ogawa et al., 1990; Cabeza and Nyberg, 2001) is giving neuroscientists unprecedentedly detailed images of brain activity, enabling measurements that correspond in interesting ways to ongoing cognitive processes. These are augmented by advances in single-cell electrical recording of neuron activity and by the methods of optogenetics (Crick, 1999; Optogenetics Zemelman et al., 2002; Han and Boyden, 2007), which allow both measurement and control of individual neurons modified to be light-sensitive.

Optogenetics

The development of brain–machine interfaces (Lebedev and Nicolelis, 2006) for both sensing and motor control not only promises to restore function to disabled individuals, but also sheds light on many aspects of neural systems. A remarkable finding from this work is that the brain is able to adjust itself to interface successfully with an external device, treating it in effect like another sensory organ or limb.

Brain–machine interface

Brains and digital computers have somewhat different properties. Figure 1.2  shows that computers have a cycle time that is a million times faster than a brain. The brain makes up for that with far more storage and interconnection than even a high-end personal computer, although the largest supercomputers match the brain on some metrics. Futurists make much of these numbers, pointing to an approaching singularity at which computers reach a superhuman level of performance (Vinge, 1993; Kurzweil, 2005; Doctorow and Stross, 2012), and then rapidly improve themselves even further. But the comparisons of raw numbers are not especially informative. Even with a computer of virtually unlimited capacity, we still require further conceptual breakthroughs in our understanding of intelligence (see Chapter 28 ). Crudely put, without the right theory, faster machines just give you the wrong answer faster.

Singularity

Figure 1.2

A crude comparison of a leading supercomputer, Summit (Feldman, 2017); a typical personal computer of 2019; and the human brain. Human brain power has not changed much in thousands of years, whereas supercomputers have improved from megaFLOPs in the 1960s to gigaFLOPs in the 1980s, teraFLOPs in the 1990s, petaFLOPs in 2008, and exaFLOPs in 2018 (1 exaFLOP = 1018 floating point operations per second).

1.2.5 Psychology

How do humans and animals think and act?

The origins of scientific psychology are usually traced to the work of the German physicist Hermann von Helmholtz (1821–1894) and his student Wilhelm Wundt (1832–1920). Helmholtz applied the scientific method to the study of human vision, and his Handbook of Physiological Optics has been described as “the single most important treatise on the physics and physiology of human vision” (Nalwa, 1993, p.15). In 1879, Wundt opened the first laboratory of experimental psychology, at the University of Leipzig. Wundt insisted on carefully controlled experiments in which his workers would perform a perceptual or associative task while introspecting on their thought processes. The careful controls went a long way toward making psychology a science, but the subjective nature of the data made it unlikely that experimenters would ever disconfirm their own theories.

Biologists studying animal behavior, on the other hand, lacked introspective data and developed an objective methodology, as described byH. S. Jennings (1906) in his influential work Behavior of the Lower Organisms. Applying this viewpoint to humans, the behaviorism movement, led by John Watson (1878–1958), rejected any theory involving mental processes on the grounds that introspection could not provide reliable evidence. Behaviorists insisted on studying only objective measures of the percepts (or stimulus) given to an animal and its resulting actions (or response). Behaviorism discovered a lot about rats and pigeons but had less success at understanding humans.

Behaviorism

Cognitive psychology, which views the brain as an information-processing device, can be traced back at least to the works of William James (1842–1910). Helmholtz also insisted that perception involved a form of unconscious logical inference. The cognitive viewpoint was largely eclipsed by behaviorism in the United States, but at Cambridge’s Applied Psychology Unit, directed by Frederic Bartlett (1886–1969), cognitive modeling was able to flourish. The Nature of Explanation, by Bartlett’s student and successor Kenneth Craik (1943), forcefully reestablished the legitimacy of such “mental” terms as beliefs and goals, arguing that they

are just as scientific as, say, using pressure and temperature to talk about gases, despite gasses being made of molecules that have neither.

Cognitive psychology

Craik specified the three key steps of a knowledge-based agent: (1) the stimulus must be translated into an internal representation, (2) the representation is manipulated by cognitive processes to derive new internal representations, and (3) these are in turn retranslated back into action. He clearly explained why this was a good design for an agent:

If the organism carries a “small-scale model” of external reality and of its own possible actions within its head, it is able to try out various alternatives, conclude which is the best of them, react to future situations before they arise, utilize the knowledge of past events in dealing with the present and future, and in every way to react in a much fuller, safer, and more competent manner to the emergencies which face it. (Craik, 1943)

After Craik’s death in a bicycle accident in 1945, his work was continued by Donald Broadbent, whose book Perception and Communication (1958) was one of the first works to model psychological phenomena as information processing. Meanwhile, in the United States, the development of computer modeling led to the creation of the field of cognitive science. The field can be said to have started at a workshop in September 1956 at MIT—just two months after the conference at which AI itself was “born.”

At the workshop, George Miller presented The Magic Number Seven, Noam Chomsky presented Three Models of Language, and Allen Newell and Herbert Simon presented The Logic Theory Machine. These three influential papers showed how computer models could be used to address the psychology of memory, language, and logical thinking, respectively. It is now a common (although far from universal) view among psychologists that “a cognitive theory should be like a computer program” (Anderson, 1980); that is, it should describe the operation of a cognitive function in terms of the processing of information.

For purposes of this review, we will count the field of human–computer interaction (HCI) under psychology. Doug Engelbart, one of the pioneers of HCI, championed the idea of intelligence augmentation—IA rather than AI. He believed that computers should augment human abilities rather than automate away human tasks. In 1968, Engelbart’s “mother of all demos” showed off for the first time the computer mouse, a windowing system, hypertext, and video conferencing—all in an effort to demonstrate what human knowledge workers could collectively accomplish with some intelligence augmentation.

Intelligence augmentation

Today we are more likely to see IA and AI as two sides of the same coin, with the former emphasizing human control and the latter emphasizing intelligent behavior on the part of the machine. Both are needed for machines to be useful to humans.

1.2.6 Computer engineering How can we build an efficient computer?

The modern digital electronic computer was invented independently and almost simultaneously by scientists in three countries embattled in World War II. The first operational computer was the electromechanical Heath Robinson,9 built in 1943 by Alan Turing’s team for a single purpose: deciphering German messages. In 1943, the same group developed the Colossus, a powerful general-purpose machine based on vacuum tubes.10 The first operational programmable computer was the Z-3, the invention of Konrad Zuse in Germany in 1941. Zuse also invented floating-point numbers and the first high-level programming language, Plankalkül. The first electronic computer, the ABC, was assembled by John Atanasoff and his student Clifford Berry between 1940 and 1942 at Iowa State University. Atanasoff’s research received little support or recognition; it was the ENIAC, developed as part of a secret military project at the University of Pennsylvania by a team including John Mauchly and J. Presper Eckert, that proved to be the most influential forerunner of modern computers.

9 A complex machine named after a British cartoonist who depicted whimsical and absurdly complicated contraptions for everyday tasks such as buttering toast.

10 In the postwar period, Turing wanted to use these computers for AI research—for example, he created an outline of the first chess program (Turing et al., 1953) —but the British government blocked this research.

Moore’s law

Since that time, each generation of computer hardware has brought an increase in speed and capacity and a decrease in price—a trend captured in Moore’s law. Performance doubled every 18 months or so until around 2005, when power dissipation problems led manufacturers to start multiplying the number of CPU cores rather than the clock speed. Current expectations are that future increases in functionality will come from massive parallelism—a curious convergence with the properties of the brain. We also see new hardware designs based on the idea that in dealing with an uncertain world, we don’t need 64 bits of precision in our numbers; just 16 bits (as in the bfloat16 format) or even 8 bits will be enough, and will enable faster processing.

We are just beginning to see hardware tuned for AI applications, such as the graphics processing unit (GPU), tensor processing unit (TPU), and wafer scale engine (WSE). From the 1960s to about 2012, the amount of computing power used to train top machine learning applications followed Moore’s law. Beginning in 2012, things changed: from 2012 to 2018 there was a 300,000-fold increase, which works out to a doubling every 100 days or so (Amodei and Hernandez, 2018). A machine learning model that took a full day to train in 2014 takes only two minutes in 2018 (Ying et al., 2018). Although it is not yet practical, quantum computing holds out the promise of far greater accelerations for some important subclasses of AI algorithms.

Quantum computing

Of course, there were calculating devices before the electronic computer. The earliest automated machines, dating from the 17th century, were discussed on 6. The first programmable machine was a loom, devised in 1805 by Joseph Marie Jacquard (1752–1834), that used punched cards to store instructions for the pattern to be woven.

In the mid-19th century, Charles Babbage (1792–1871) designed two computing machines, neither of which he completed. The Difference Engine was intended to compute mathematical tables for engineering and scientific projects. It was finally built and shown to work in 1991 (Swade, 2000). Babbage’s Analytical Engine was far more ambitious: it included addressable memory, stored programs based on Jacquard’s punched cards, and conditional jumps. It was the first machine capable of universal computation.

Babbage’s colleague Ada Lovelace, daughter of the poet Lord Byron, understood its potential, describing it as “a thinking or ... a reasoning machine,” one capable of reasoning about “all subjects in the universe” (Lovelace, 1843). She also anticipated AI’s hype cycles, writing, “It is desirable to guard against the possibility of exaggerated ideas that might arise as to the powers of the Analytical Engine.” Unfortunately, Babbage’s machines and Lovelace’s ideas were largely forgotten.

AI also owes a debt to the software side of computer science, which has supplied the operating systems, programming languages, and tools needed to write modern programs (and papers about them). But this is one area where the debt has been repaid: work in AI has pioneered many ideas that have made their way back to mainstream computer science, including time sharing, interactive interpreters, personal computers with windows and mice, rapid development environments, the linked-list data type, automatic storage management, and key concepts of symbolic, functional, declarative, and object-oriented programming.

1.2.7 Control theory and cybernetics How can artifacts operate under their own control?

Ktesibios of Alexandria (c. 250 BCE) built the first self-controlling machine: a water clock with a regulator that maintained a constant flow rate. This invention changed the definition of what an artifact could do. Previously, only living things could modify their behavior in response to changes in the environment. Other examples of self-regulating feedback control systems include the steam engine governor, created by James Watt (1736–1819), and the

thermostat, invented by Cornelis Drebbel (1572–1633), who also invented the submarine. James Clerk Maxwell (1868) initiated the mathematical theory of control systems.

A central figure in the post-war development of control theory was Norbert Wiener (1894– 1964). Wiener was a brilliant mathematician who worked with Bertrand Russell, among others, before developing an interest in biological and mechanical control systems and their connection to cognition. Like Craik (who also used control systems as psychological models), Wiener and his colleagues Arturo Rosenblueth and Julian Bigelow challenged the behaviorist orthodoxy (Rosenblueth et al., 1943). They viewed purposive behavior as arising from a regulatory mechanism trying to minimize “error”—the difference between current state and goal state. In the late 1940s, Wiener, along with Warren McCulloch, Walter Pitts, and John von Neumann, organized a series of influential conferences that explored the new mathematical and computational models of cognition. Wiener’s book Cybernetics (1948) became a bestseller and awoke the public to the possibility of artificially intelligent machines.

Control theory

Cybernetics

Meanwhile, in Britain, W. Ross Ashby pioneered similar ideas (Ashby, 1940). Ashby, Alan Turing, Grey Walter, and others formed the Ratio Club for “those who had Wiener’s ideas before Wiener’s book appeared.” Ashby’s Design for a Brain 1948, 1952 elaborated on his idea that intelligence could be created by the use of homeostatic devices containing appropriate feedback loops to achieve stable adaptive behavior.

Homeostatic

Modern control theory, especially the branch known as stochastic optimal control, has as its goal the design of systems that maximize a cost function over time. This roughly matches the standard model of AI: designing systems that behave optimally. Why, then, are AI and control theory two different fields, despite the close connections among their founders? The answer lies in the close coupling between the mathematical techniques that were familiar to the participants and the corresponding sets of problems that were encompassed in each world view. Calculus and matrix algebra, the tools of control theory, lend themselves to systems that are describable by fixed sets of continuous variables, whereas AI was founded in part as a way to escape from these perceived limitations. The tools of logical inference and computation allowed AI researchers to consider problems such as language, vision, and symbolic planning that fell completely outside the control theorist’s purview.

Cost function

1.2.8 Linguistics How does language relate to thought?

In 1957, B. F. Skinner published Verbal Behavior. This was a comprehensive, detailed account of the behaviorist approach to language learning, written by the foremost expert in the field. But curiously, a review of the book became as well known as the book itself, and served to almost kill off interest in behaviorism. The author of the review was the linguist Noam Chomsky, who had just published a book on his own theory, Syntactic Structures. Chomsky pointed out that the behaviorist theory did not address the notion of creativity in language—it did not explain how children could understand and make up sentences that they had never heard before. Chomsky’s theory—based on syntactic models going back to the Indian linguist Panini (c. 350 BCE)—could explain this, and unlike previous theories, it was formal enough that it could in principle be programmed.

Computational linguistics

Modern linguistics and AI, then, were “born” at about the same time, and grew up together, intersecting in a hybrid field called computational linguistics or natural language processing. The problem of understanding language turned out to be considerably more complex than it seemed in 1957. Understanding language requires an understanding of the subject matter and context, not just an understanding of the structure of sentences. This might seem obvious, but it was not widely appreciated until the 1960s. Much of the early work in knowledge representation (the study of how to put knowledge into a form that a computer can reason with) was tied to language and informed by research in linguistics, which was connected in turn to decades of work on the philosophical analysis of language.

1.3 The History of Artificial Intelligence One quick way to summarize the milestones in AI history is to list the Turing Award winners: Marvin Minsky (1969) and John McCarthy (1971) for defining the foundations of the field based on representation and reasoning; Ed Feigenbaum and Raj Reddy (1994) for developing expert systems that encode human knowledge to solve real-world problems; Judea Pearl (2011) for developing probabilistic reasoning techniques that deal with uncertainty in a principled manner; and finally Yoshua Bengio, Geoffrey Hinton, and Yann LeCun (2019) for making “deep learning” (multilayer neural networks) a critical part of modern computing. The rest of this section goes into more detail on each phase of AI history.

1.3.1 The inception of artificial intelligence (1943–1956) The first work that is now generally recognized as AI was done by Warren McCulloch and Walter Pitts (1943). Inspired by the mathematical modeling work of Pitts’s advisor Nicolas (1936, 1938), they drew on three sources: knowledge of the basic physiology and function of neurons in the brain; a formal analysis of propositional logic due to Russell and Whitehead; and Turing’s theory of computation. They proposed a model of artificial neurons in which each neuron is characterized as being “on” or “off,” with a switch to “on” occurring in response to stimulation by a sufficient number of neighboring neurons. The state of a neuron was conceived of as “factually equivalent to a proposition which proposed its adequate stimulus.” They showed, for example, that any computable function could be computed by some network of connected neurons, and that all the logical connectives (AND, OR, NOT,

etc.) could be implemented by simple network structures. McCulloch and Pitts also

suggested that suitably defined networks could learn. Donald Hebb (1949) demonstrated a simple updating rule for modifying the connection strengths between neurons. His rule, now called Hebbian learning, remains an influential model to this day.

Hebbian learning

Two undergraduate students at Harvard, Marvin Minsky (1927–2016) and Dean Edmonds, built the first neural network computer in 1950. The SNARC, as it was called, used 3000 vacuum tubes and a surplus automatic pilot mechanism from a B-24 bomber to simulate a network of 40 neurons. Later, at Princeton, Minsky studied universal computation in neural networks. His Ph.D. committee was skeptical about whether this kind of work should be considered mathematics, but von Neumann reportedly said, “If it isn’t now, it will be someday.”

There were a number of other examples of early work that can be characterized as AI, including two checkers-playing programs developed independently in 1952 by Christopher Strachey at the University of Manchester and by Arthur Samuel at IBM. However, Alan Turing’s vision was the most influential. He gave lectures on the topic as early as 1947 at the London Mathematical Society and articulated a persuasive agenda in his 1950 article “Computing Machinery and Intelligence.” Therein, he introduced the Turing test, machine learning, genetic algorithms, and reinforcement learning. He dealt with many of the objections raised to the possibility of AI, as described in Chapter 27 . He also suggested that it would be easier to create human-level AI by developing learning algorithms and then teaching the machine rather than by programming its intelligence by hand. In subsequent lectures he warned that achieving this goal might not be the best thing for the human race.

In 1955, John McCarthy of Dartmouth College convinced Minsky, Claude Shannon, and Nathaniel Rochester to help him bring together U.S. researchers interested in automata theory, neural nets, and the study of intelligence. They organized a two-month workshop at Dartmouth in the summer of 1956. There were 10 attendees in all, including Allen Newell and Herbert Simon from Carnegie Tech,11 Trenchard More from Princeton, Arthur Samuel from IBM, and Ray Solomonoff and Oliver Selfridge from MIT. The proposal states:12

11 Now Carnegie Mellon University (CMU).

12 This was the first official usage of McCarthy’s term artificial intelligence. Perhaps “computational rationality” would have been more precise and less threatening, but “AI” has stuck. At the 50th anniversary of the Dartmouth conference, McCarthy stated that he resisted the terms “computer” or “computational” in deference to Norbert Wiener, who was promoting analog cybernetic devices rather than digital computers.

We propose that a 2 month, 10 man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College in Hanover, New Hampshire. The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so

precisely described that a machine can be made to simulate it. An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer.

Despite this optimistic prediction, the Dartmouth workshop did not lead to any breakthroughs. Newell and Simon presented perhaps the most mature work, a mathematical theorem-proving system called the Logic Theorist (LT). Simon claimed, “We have invented a computer program capable of thinking non-numerically, and thereby solved the venerable mind–body problem.”13 Soon after the workshop, the program was able to prove most of the theorems in Chapter 2  of Russell and Whitehead’s Principia Mathematica. Russell was reportedly delighted when told that LT had come up with a proof for one theorem that was shorter than the one in Principia. The editors of the Journal of Symbolic Logic were less impressed; they rejected a paper coauthored by Newell, Simon, and Logic Theorist.

13 Newell and Simon also invented a list-processing language, IPL, to write LT. They had no compiler and translated it into machine code by hand. To avoid errors, they worked in parallel, calling out binary numbers to each other as they wrote each instruction to make sure they agreed.

1.3.2 Early enthusiasm, great expectations (1952–1969) The intellectual establishment of the 1950s, by and large, preferred to believe that “a

X.” (See Chapter 27  for a long list of X’s gathered by Turing.) AI researchers naturally responded by demonstrating one X after another. They focused in

machine can never do

particular on tasks considered indicative of intelligence in humans, including games, puzzles, mathematics, and IQ tests. John McCarthy referred to this period as the “Look, Ma, no hands!” era.

Newell and Simon followed up their success with LT with the General Problem Solver, or GPS. Unlike LT, this program was designed from the start to imitate human problem-solving protocols. Within the limited class of puzzles it could handle, it turned out that the order in which the program considered subgoals and possible actions was similar to that in which humans approached the same problems. Thus, GPS was probably the first program to embody the “thinking humanly” approach. The success of GPS and subsequent programs as models of cognition led Newell and Simon (1976) to formulate the famous physical symbol system hypothesis, which states that “a physical symbol system has the necessary and

sufficient means for general intelligent action.” What they meant is that any system (human or machine) exhibiting intelligence must operate by manipulating data structures composed of symbols. We will see later that this hypothesis has been challenged from many directions.

Physical symbol system

At IBM, Nathaniel Rochester and his colleagues produced some of the first AI programs. Herbert Gelernter (1959) constructed the Geometry Theorem Prover, which was able to prove theorems that many students of mathematics would find quite tricky. This work was a precursor of modern mathematical theorem provers.

Of all the exploratory work done during this period, perhaps the most influential in the long run was that of Arthur Samuel on checkers (draughts). Using methods that we now call reinforcement learning (see Chapter 22 ), Samuel’s programs learned to play at a strong amateur level. He thereby disproved the idea that computers can do only what they are told to: his program quickly learned to play a better game than its creator. The program was demonstrated on television in 1956, creating a strong impression. Like Turing, Samuel had trouble finding computer time. Working at night, he used machines that were still on the testing floor at IBM’s manufacturing plant. Samuel’s program was the precursor of later systems such as TD-GAMMON (Tesauro, 1992), which was among the world’s best backgammon players, and ALPHAGO (Silver et al., 2016), which shocked the world by defeating the human world champion at Go (see Chapter 5 ).

In 1958, John McCarthy made two important contributions to AI. In MIT AI Lab Memo No. 1, he defined the high-level language Lisp, which was to become the dominant AI programming language for the next 30 years. In a paper entitled Programs with Common Sense, he advanced a conceptual proposal for AI systems based on knowledge and reasoning. The paper describes the Advice Taker, a hypothetical program that would embody general knowledge of the world and could use it to derive plans of action. The concept was illustrated with simple logical axioms that suffice to generate a plan to drive to the airport. The program was also designed to accept new axioms in the normal course of operation, thereby allowing it to achieve competence in new areas without being

reprogrammed. The Advice Taker thus embodied the central principles of knowledge representation and reasoning: that it is useful to have a formal, explicit representation of the world and its workings and to be able to manipulate that representation with deductive processes. The paper influenced the course of AI and remains relevant today.

Lisp

1958 also marked the year that Marvin Minsky moved to MIT. His initial collaboration with McCarthy did not last, however. McCarthy stressed representation and reasoning in formal logic, whereas Minsky was more interested in getting programs to work and eventually developed an anti-logic outlook. In 1963, McCarthy started the AI lab at Stanford. His plan to use logic to build the ultimate Advice Taker was advanced by J. A. Robinson’s discovery in 1965 of the resolution method (a complete theorem-proving algorithm for first-order logic; see Chapter 9 ). Work at Stanford emphasized general-purpose methods for logical reasoning. Applications of logic included Cordell Green’s question-answering and planning systems (Green, 1969b) and the Shakey robotics project at the Stanford Research Institute (SRI). The latter project, discussed further in Chapter 26 , was the first to demonstrate the complete integration of logical reasoning and physical activity.

Microworld

At MIT, Minsky supervised a series of students who chose limited problems that appeared to require intelligence to solve. These limited domains became known as microworlds. James Slagle’s SAINT program (1963) was able to solve closed-form calculus integration problems typical of first-year college courses. Tom Evans’s ANALOGY program (1968) solved geometric analogy problems that appear in IQ tests. Daniel Bobrow’s STUDENT program (1967) solved algebra story problems, such as the following:

If the number of customers Tom gets is twice the square of 20 percent of the number of advertisements he runs, and the number of advertisements he runs is 45, what is the number of customers Tom gets?

The most famous microworld is the blocks world, which consists of a set of solid blocks placed on a tabletop (or more often, a simulation of a tabletop), as shown in Figure 1.3 . A typical task in this world is to rearrange the blocks in a certain way, using a robot hand that can pick up one block at a time. The blocks world was home to the vision project of David Huffman (1971), the vision and constraint-propagation work of David Waltz (1975), the learning theory of Patrick Winston (1970), the natural-language-understanding program of Terry Winograd (1972), and the planner of Scott Fahlman (1974).

Figure 1.3

A scene from the blocks world. SHRDLU (Winograd, 1972) has just completed the command “Find a block which is taller than the one you are holding and put it in the box.”

Blocks world

Early work building on the neural networks of McCulloch and Pitts also flourished. The work of Shmuel Winograd and Jack Cowan (1963) showed how a large number of elements could collectively represent an individual concept, with a corresponding increase in robustness and parallelism. Hebb’s learning methods were enhanced by Bernie Widrow (Widrow and Hoff, 1960; Widrow, 1962), who called his networks adalines, and by Frank Rosenblatt (1962) with his perceptrons. The perceptron convergence theorem (Block et al., 1962) says that the learning algorithm can adjust the connection strengths of a perceptron to match any input data, provided such a match exists.

1.3.3 A dose of reality (1966–1973) From the beginning, AI researchers were not shy about making predictions of their coming successes. The following statement by Herbert Simon in 1957 is often quoted:

It is not my aim to surprise or shock you—but the simplest way I can summarize is to say that there are now in the world machines that think, that learn and that create. Moreover, their ability to do these things is going to increase rapidly until—in a visible future—the range of problems they can handle will be coextensive with the range to which the human mind has been applied.

The term “visible future” is vague, but Simon also made more concrete predictions: that within 10 years a computer would be chess champion and a significant mathematical theorem would be proved by machine. These predictions came true (or approximately true) within 40 years rather than 10. Simon’s overconfidence was due to the promising performance of early AI systems on simple examples. In almost all cases, however, these early systems failed on more difficult problems.

There were two main reasons for this failure. The first was that many early AI systems were based primarily on “informed introspection” as to how humans perform a task, rather than on a careful analysis of the task, what it means to be a solution, and what an algorithm would need to do to reliably produce such solutions.

The second reason for failure was a lack of appreciation of the intractability of many of the problems that AI was attempting to solve. Most of the early problem-solving systems worked by trying out different combinations of steps until the solution was found. This strategy worked initially because microworlds contained very few objects and hence very few possible actions and very short solution sequences. Before the theory of computational

complexity was developed, it was widely thought that “scaling up” to larger problems was simply a matter of faster hardware and larger memories. The optimism that accompanied the development of resolution theorem proving, for example, was soon dampened when researchers failed to prove theorems involving more than a few dozen facts. The fact that a program can find a solution in principle does not mean that the program contains any of the mechanisms needed to find it in practice.

The illusion of unlimited computational power was not confined to problem-solving programs. Early experiments in machine evolution (now called genetic programming) (Fried-Machine evolutionberg, 1958; Friedberg et al., 1959) were based on the undoubtedly correct belief that by making an appropriate series of small mutations to a machine-code program, one can generate a program with good performance for any particular task. The idea, then, was to try random mutations with a selection process to preserve mutations that seemed useful. Despite thousands of hours of CPU time, almost no progress was demonstrated.

Machine evolution

Failure to come to grips with the “combinatorial explosion” was one of the main criticisms of AI contained in the Lighthill report (Lighthill, 1973), which formed the basis for the decision by the British government to end support for AI research in all but two universities. (Oral tradition paints a somewhat different and more colorful picture, with political ambitions and personal animosities whose description is beside the point.)

A third difficulty arose because of some fundamental limitations on the basic structures being used to generate intelligent behavior. For example, Minsky and Papert’s book Perceptrons (1969) proved that, although perceptrons (a simple form of neural network) could be shown to learn anything they were capable of representing, they could represent very little. In particular, a two-input perceptron could not be trained to recognize when its two inputs were different. Although their results did not apply to more complex, multilayer networks, research funding for neural-net research soon dwindled to almost nothing. Ironically, the new back-propagation learning algorithms that were to cause an enormous

resurgence in neural-net research in the late 1980s and again in the 2010s had already been developed in other contexts in the early 1960s (Kelley, 1960; Bryson, 1962).

1.3.4 Expert systems (1969–1986) The picture of problem solving that had arisen during the first decade of AI research was of a general-purpose search mechanism trying to string together elementary reasoning steps to find complete solutions. Such approaches have been called weak methods because, although general, they do not scale up to large or difficult problem instances. The alternative to weak methods is to use more powerful, domain-specific knowledge that allows larger reasoning steps and can more easily handle typically occurring cases in narrow areas of expertise. One might say that to solve a hard problem, you have to almost know the answer already.

Weak method

The DENDRAL program (Buchanan et al., 1969) was an early example of this approach. It was developed at Stanford, where Ed Feigenbaum (a former student of Herbert Simon), Bruce Buchanan (a philosopher turned computer scientist), and Joshua Lederberg (a Nobel laureate geneticist) teamed up to solve the problem of inferring molecular structure from the information provided by a mass spectrometer. The input to the program consists of the elementary formula of the molecule (e.g., C6 H13 NO2 ) and the mass spectrum giving the masses of the various fragments of the molecule generated when it is bombarded by an electron beam. For example, the mass spectrum might contain a peak at

m = 15,

corresponding to the mass of a methyl (CH3 ) fragment.

The naive version of the program generated all possible structures consistent with the formula, and then predicted what mass spectrum would be observed for each, comparing this with the actual spectrum. As one might expect, this is intractable for even moderatesized molecules. The DENDRAL researchers consulted analytical chemists and found that they worked by looking for well-known patterns of peaks in the spectrum that suggested common substructures in the molecule. For example, the following rule is used to recognize a ketone (C=O) subgroup (which weighs 28):

M is the mass of the whole molecule and there are two peaks at x1 and x2 such that (a) x1 + x2 = M + 28; (b) x1 − 28 is a high peak; (c) x2 − 28 is a high peak; and (d) At least one of x1 and x2 is high then there is a ketone subgroup. if

Recognizing that the molecule contains a particular substructure reduces the number of possible candidates enormously. According to its authors, DENDRAL was powerful because it embodied the relevant knowledge of mass spectroscopy not in the form of first principles but in efficient “cookbook recipes” (Feigenbaum et al., 1971). The significance of DENDRAL was that it was the first successful knowledge-intensive system: its expertise derived from large numbers of special-purpose rules. In 1971, Feigenbaum and others at Stanford began the Heuristic Programming Project (HPP) to investigate the extent to which the new methodology of expert systems could be applied to other areas.

Expert systems

The next major effort was the MYCIN system for diagnosing blood infections. With about 450 rules, MYCIN was able to perform as well as some experts, and considerably better than junior doctors. It also contained two major differences from DENDRAL. First, unlike the DENDRAL rules, no general theoretical model existed from which the MYCIN rules could be deduced. They had to be acquired from extensive interviewing of experts. Second, the rules had to reflect the uncertainty associated with medical knowledge. MYCIN incorporated a calculus of uncertainty called certainty factors (see Chapter 13 ), which seemed (at the time) to fit well with how doctors assessed the impact of evidence on the diagnosis.

Certainty factor

The first successful commercial expert system, R1, began operation at the Digital Equipment Corporation (McDermott, 1982). The program helped configure orders for new computer systems; by 1986, it was saving the company an estimated $40 million a year. By 1988,

DEC’s AI group had 40 expert systems deployed, with more on the way. DuPont had 100 in use and 500 in development. Nearly every major U.S. corporation had its own AI group and was either using or investigating expert systems.

The importance of domain knowledge was also apparent in the area of natural language understanding. Despite the success of Winograd’s SHRDLU system, its methods did not extend to more general tasks: for problems such as ambiguity resolution it used simple rules that relied on the tiny scope of the blocks world.

Several researchers, including Eugene Charniak at MIT and Roger Schank at Yale, suggested that robust language understanding would require general knowledge about the world and a general method for using that knowledge. (Schank went further, claiming, “There is no such thing as syntax,” which upset a lot of linguists but did serve to start a useful discussion.) Schank and his students built a series of programs (Schank and Abelson, 1977; Wilensky, 1978; Schank and Riesbeck, 1981) that all had the task of understanding natural language. The emphasis, however, was less on language per se and more on the problems of representing and reasoning with the knowledge required for language understanding.

The widespread growth of applications to real-world problems led to the development of a wide range of representation and reasoning tools. Some were based on logic—for example, the Prolog language became popular in Europe and Japan, and the PLANNER family in the United States. Others, following Minsky’s idea of frames (1975), adopted a more structured approach, assembling facts about particular object and event types and arranging the types into a large taxonomic hierarchy analogous to a biological taxonomy.

Frames

In 1981, the Japanese government announced the “Fifth Generation” project, a 10-year plan to build massively parallel, intelligent computers running Prolog. The budget was to exceed a $1.3 billion in today’s money. In response, the United States formed the Microelectronics and Computer Technology Corporation (MCC), a consortium designed to assure national competitiveness. In both cases, AI was part of a broad effort, including chip design and

human-interface research. In Britain, the Alvey report reinstated the funding removed by the Lighthill report. However, none of these projects ever met its ambitious goals in terms of new AI capabilities or economic impact.

Overall, the AI industry boomed from a few million dollars in 1980 to billions of dollars in 1988, including hundreds of companies building expert systems, vision systems, robots, and software and hardware specialized for these purposes.

Soon after that came a period called the “AI winter,” in which many companies fell by the wayside as they failed to deliver on extravagant promises. It turned out to be difficult to build and maintain expert systems for complex domains, in part because the reasoning methods used by the systems broke down in the face of uncertainty and in part because the systems could not learn from experience.

1.3.5 The return of neural networks (1986–present) In the mid-1980s at least four different groups reinvented the back-propagation learning algorithm first developed in the early 1960s. The algorithm was applied to many learning problems in computer science and psychology, and the widespread dissemination of the results in the collection Parallel Distributed Processing (Rumelhart and McClelland, 1986) caused great excitement.

These so-called connectionist models were seen by some as direct competitors both to the symbolic models promoted by Newell and Simon and to the logicist approach of McCarthy and others. It might seem obvious that at some level humans manipulate symbols—in fact, the anthropologist Terrence Deacon’s book The Symbolic Species (1997) suggests that this is the defining characteristic of humans. Against this, Geoff Hinton, a leading figure in the resurgence of neural networks in the 1980s and 2010s, has described symbols as the “luminiferous aether of AI”—a reference to the non-existent medium through which many 19th-century physicists believed that electromagnetic waves propagated. Certainly, many concepts that we name in language fail, on closer inspection, to have the kind of logically defined necessary and sufficient conditions that early AI researchers hoped to capture in axiomatic form. It may be that connectionist models form internal concepts in a more fluid and imprecise way that is better suited to the messiness of the real world. They also have the capability to learn from examples—they can compare their predicted output value to the

true value on a problem and modify their parameters to decrease the difference, making them more likely to perform well on future examples.

connectionist

1.3.6 Probabilistic reasoning and machine learning (1987– present) The brittleness of expert systems led to a new, more scientific approach incorporating probability rather than Boolean logic, machine learning rather than hand-coding, and experimental results rather than philosophical claims.14 It became more common to build on existing theories than to propose brand-new ones, to base claims on rigorous theorems or solid experimental methodology (Cohen, 1995) rather than on intuition, and to show relevance to real-world applications rather than toy examples.

14 Some have characterized this change as a victory of the neats—those who think that AI theories should be grounded in mathematical rigor—over the scruffies—those who would rather try out lots of ideas, write some programs, and then assess what seems to be working. Both approaches are important. A shift toward neatness implies that the field has reached a level of stability and maturity. The present emphasis on deep learning may represent a resurgence of the scruffies.

Shared benchmark problem sets became the norm for demonstrating progress, including the UC Irvine repository for machine learning data sets, the International Planning Competition for planning algorithms, the LibriSpeech corpus for speech recognition, the MNIST data set for handwritten digit recognition, ImageNet and COCO for image object recognition, SQUAD for natural language question answering, the WMT competition for machine translation, and the International SAT Competitions for Boolean satisfiability solvers.

AI was founded in part as a rebellion against the limitations of existing fields like control theory and statistics, but in this period it embraced the positive results of those fields. As David McAllester (1998) put it:

In the early period of AI it seemed plausible that new forms of symbolic computation, e.g., frames and semantic networks, made much of classical theory obsolete. This led to a form of isolationism in which AI became largely separated from the rest of computer science. This isolationism is currently

being abandoned. There is a recognition that machine learning should not be isolated from information theory, that uncertain reasoning should not be isolated from stochastic modeling, that search should not be isolated from classical optimization and control, and that automated reasoning should not be isolated from formal methods and static analysis.

The field of speech recognition illustrates the pattern. In the 1970s, a wide variety of different architectures and approaches were tried. Many of these were rather ad hoc and fragile, and worked on only a few carefully selected examples. In the 1980s, approaches using hidden Markov models (HMMs) came to dominate the area. Two aspects of HMMs are relevant. First, they are based on a rigorous mathematical theory. This allowed speech researchers to build on several decades of mathematical results developed in other fields. Second, they are generated by a process of training on a large corpus of real speech data. This ensures that the performance is robust, and in rigorous blind tests HMMs improved their scores steadily. As a result, speech technology and the related field of handwritten character recognition made the transition to widespread industrial and consumer applications. Note that there was no scientific claim that humans use HMMs to recognize speech; rather, HMMs provided a mathematical framework for understanding and solving the problem. We will see in Section 1.3.8 , however, that deep learning has rather upset this comfortable narrative.

Hidden Markov models

1988 was an important year for the connection between AI and other fields, including statistics, operations research, decision theory, and control theory. Judea Pearl’s (1988) Probabilistic Reasoning in Intelligent Systems led to a new acceptance of probability and decision theory in AI. Pearl’s development of Bayesian networks yielded a rigorous and efficient formalism for representing uncertain knowledge as well as practical algorithms for probabilistic reasoning. Chapters 12  to 16  cover this area, in addition to more recent developments that have greatly increased the expressive power of probabilistic formalisms; Chapter 20  describes methods for learning Bayesian networks and related models from data.

Bayesian network

A second major contribution in 1988 was Rich Sutton’s work connecting reinforcement learning—which had been used in Arthur Samuel’s checker-playing program in the 1950s— to the theory of Markov decision processes (MDPs) developed in the field of operations research. A flood of work followed connecting AI planning research to MDPs, and the field of reinforcement learning found applications in robotics and process control as well as acquiring deep theoretical foundations.

One consequence of AI’s newfound appreciation for data, statistical modeling, optimization, and machine learning was the gradual reunification of subfields such as computer vision, robotics, speech recognition, multiagent systems, and natural language processing that had become somewhat separate from core AI. The process of reintegration has yielded significant benefits both in terms of applications—for example, the deployment of practical robots expanded greatly during this period—and in a better theoretical understanding of the core problems of AI.

1.3.7 Big data (2001–present) Remarkable advances in computing power and the creation of the World Wide Web have facilitated the creation of very large data sets—a phenomenon sometimes known as big data. These data sets include trillions of words of text, billions of images, and billions of hours of speech and video, as well as vast amounts of genomic data, vehicle tracking data, clickstream data, social network data, and so on.

Big data

This has led to the development of learning algorithms specially designed to take advantage of very large data sets. Often, the vast majority of examples in such data sets are unlabeled; for example, in Yarowsky’s (1995) influential work on word-sense disambiguation, occurrences of a word such as “plant” are not labeled in the data set to indicate whether they

refer to flora or factory. With large enough data sets, however, suitable learning algorithms can achieve an accuracy of over 96% on the task of identifying which sense was intended in a sentence. Moreover, Banko and Brill (2001) argued that the improvement in performance obtained from increasing the size of the data set by two or three orders of magnitude outweighs any improvement that can be obtained from tweaking the algorithm.

A similar phenomenon seems to occur in computer vision tasks such as filling in holes in photographs—holes caused either by damage or by the removal of ex-friends. Hays and Efros (2007) developed a clever method for doing this by blending in pixels from similar images; they found that the technique worked poorly with a database of only thousands of images but crossed a threshold of quality with millions of images. Soon after, the availability of tens of millions of images in the ImageNet database (Deng et al., 2009) sparked a revolution in the field of computer vision.

The availability of big data and the shift towards machine learning helped AI recover commercial attractiveness (Havenstein, 2005; Halevy et al., 2009). Big data was a crucial factor in the 2011 victory of IBM’s Watson system over human champions in the Jeopardy! quiz game, an event that had a major impact on the public’s perception of AI.

1.3.8 Deep learning (2011–present) The term deep learning refers to machine learning using multiple layers of simple, adjustable computing elements. Experiments were carried out with such networks as far back as the 1970s, and in the form of convolutional neural networks they found some success in handwritten digit recognition in the 1990s (LeCun et al., 1995). It was not until 2011, however, that deep learning methods really took off. This occurred first in speech recognition and then in visual object recognition.

Deep learning

In the 2012 ImageNet competition, which required classifying images into one of a thousand categories (armadillo, bookshelf, corkscrew, etc.), a deep learning system created in Geoffrey Hinton’s group at the University of Toronto (Krizhevsky et al., 2013) demonstrated

a dramatic improvement over previous systems, which were based largely on handcrafted features. Since then, deep learning systems have exceeded human performance on some vision tasks (and lag behind in some other tasks). Similar gains have been reported in speech recognition, machine translation, medical diagnosis, and game playing. The use of a deep network to represent the evaluation function contributed to ALPHAGO’s victories over the leading human Go players (Silver et al., 2016, 2017, 2018).

These remarkable successes have led to a resurgence of interest in AI among students, companies, investors, governments, the media, and the general public. It seems that every week there is news of a new AI application approaching or exceeding human performance, often accompanied by speculation of either accelerated success or a new AI winter.

Deep learning relies heavily on powerful hardware. Whereas a standard computer CPU can do 109 or 1010 operations per second. a deep learning algorithm running on specialized hardware (e.g., GPU, TPU, or FPGA) might consume between 1014 and 1017 operations per second, mostly in the form of highly parallelized matrix and vector operations. Of course, deep learning also depends on the availability of large amounts of training data, and on a few algorithmic tricks (see Chapter 21 ).

1.4 The State of the Art

AI Index

Stanford University’s One Hundred Year Study on AI (also known as AI100) convenes panels of experts to provide reports on the state of the art in AI. Their 2016 report (Stone et al., 2016; Grosz and Stone, 2018) concludes that “Substantial increases in the future uses of AI applications, including more self-driving cars, healthcare diagnostics and targeted treatment, and physical assistance for elder care can be expected” and that “Society is now at a crucial juncture in determining how to deploy AI-based technologies in ways that promote rather than hinder democratic values such as freedom, equality, and transparency.” AI100 also produces an AI Index at aiindex.org to help track progress. Some highlights from the 2018 and 2019 reports (comparing to a year 2000 baseline unless otherwise stated):

Publications: AI papers increased 20-fold between 2010 and 2019 to about 20,000 a year. The most popular category was machine learning. (Machine learning papers in arXiv.org doubled every year from 2009 to 2017.) Computer vision and natural language processing were the next most popular. Sentiment: About 70% of news articles on AI are neutral, but articles with positive tone increased from 12% in 2016 to 30% in 2018. The most common issues are ethical: data privacy and algorithm bias. Students: Course enrollment increased 5-fold in the U.S. and 16-fold internationally from a 2010 baseline. AI is the most popular specialization in Computer Science. Diversity: AI Professors worldwide are about 80% male, 20% female. Similar numbers hold for Ph.D. students and industry hires. Conferences: Attendance at NeurIPS increased 800% since 2012 to 13,500 attendees. Other conferences are seeing annual growth of about 30%. Industry: AI startups in the U.S. increased 20-fold to over 800. Internationalization: China publishes more papers per year than the U.S. and about as many as all of Europe. However, in citation-weighted impact, U.S. authors are 50%

ahead of Chinese authors. Singapore, Brazil, Australia, Canada, and India are the fastest growing countries in terms of the number of AI hires. Vision: Error rates for object detection (as achieved in LSVRC, the Large-Scale Visual Recognition Challenge) improved from 28% in 2010 to 2% in 2017, exceeding human performance. Accuracy on open-ended visual question answering (VQA) improved from 55% to 68% since 2015, but lags behind human performance at 83%. Speed: Training time for the image recognition task dropped by a factor of 100 in just the past two years. The amount of computing power used in top AI applications is doubling every 3.4 months. Language: Accuracy on question answering, as measured by F1 score on the Stanford Question Answering Dataset (SQUAD), increased from 60 to 95 from 2015 to 2019; on the SQUAD 2 variant, progress was faster, going from 62 to 90 in just one year. Both scores exceed human-level performance. Human benchmarks: By 2019, AI systems had reportedly met or exceeded human-level performance in chess, Go, poker, Pac-Man, Jeopardy!, ImageNet object detection, speech recognition in a limited domain, Chinese-to-English translation in a restricted domain, Quake III, Dota 2, StarCraft II, various Atari games, skin cancer detection, prostate cancer detection, protein folding, and diabetic retinopathy diagnosis.

When (if ever) will AI systems achieve human-level performance across a broad variety of tasks? Ford (2018) interviews AI experts and finds a wide range of target years, from 2029 to 2200, with a mean of 2099. In a similar survey (Grace et al., 2017) 50% of respondents thought this could happen by 2066, although 10% thought it could happen as early as 2025, and a few said “never.” The experts were also split on whether we need fundamental new breakthroughs or just refinements on current approaches. But don’t take their predictions too seriously; as Philip Tetlock (2017) demonstrates in the area of predicting world events, experts are no better than amateurs.

How will future AI systems operate? We can’t yet say. As detailed in this section, the field has adopted several stories about itself—first the bold idea that intelligence by a machine was even possible, then that it could be achieved by encoding expert knowledge into logic, then that probabilistic models of the world would be the main tool, and most recently that machine learning would induce models that might not be based on any well-understood theory at all. The future will reveal what model comes next.

What can AI do today? Perhaps not as much as some of the more optimistic media articles might lead one to believe, but still a great deal. Here are some examples:

ROBOTIC VEHICLES: The history of robotic vehicles stretches back to radio-controlled cars of the 1920s, but the first demonstrations of autonomous road driving without special guides occurred in the 1980s (Kanade et al., 1986; Dickmanns and Zapp, 1987). After successful demonstrations of driving on dirt roads in the 132-mile DARPA Grand Challenge in 2005 (Thrun, 2006) and on streets with traffic in the 2007 Urban Challenge, the race to develop self-driving cars began in earnest. In 2018, Waymo test vehicles passed the landmark of 10 million miles driven on public roads without a serious accident, with the human driver stepping in to take over control only once every 6,000 miles. Soon after, the company began offering a commercial robotic taxi service.

In the air, autonomous fixed-wing drones have been providing cross-country blood deliveries in Rwanda since 2016. Quadcopters perform remarkable aerobatic maneuvers, explore buildings while constructing 3-D maps, and self-assemble into autonomous formations.

Legged locomotion: BigDog, a quadruped robot by Raibert et al. (2008), upended our notions of how robots move—no longer the slow, stiff-legged, side-to-side gait of Hollywood movie robots, but something closely resembling an animal and able to recover when shoved or when slipping on an icy puddle. Atlas, a humanoid robot, not only walks on uneven terrain but jumps onto boxes and does backflips (Ackerman and Guizzo, 2016).

AUTONOMOUS PLANNING AND SCHEDULING: A hundred million miles from Earth, NASA’s Remote Agent program became the first on-board autonomous planning program to control the scheduling of operations for a spacecraft (Jonsson et al., 2000). Remote Agent generated plans from high-level goals specified from the ground and monitored the execution of those plans—detecting, diagnosing, and recovering from problems as they occurred. Today, the EUROPA planning toolkit (Barreiro et al., 2012) is used for daily operations of NASA’s Mars rovers and the SEXTANT system (Winternitz, 2017) allows autonomous navigation in deep space, beyond the global GPS system.

During the Persian Gulf crisis of 1991, U.S. forces deployed a Dynamic Analysis and Replanning Tool, DART (Cross and Walker, 1994), to do automated logistics planning and

scheduling for transportation. This involved up to 50,000 vehicles, cargo, and people at a time, and had to account for starting points, destinations, routes, transport capacities, port and airfield capacities, and conflict resolution among all parameters. The Defense Advanced Research Project Agency (DARPA) stated that this single application more than paid back DARPA’s 30-year investment in AI.

Every day, ride hailing companies such as Uber and mapping services such as Google Maps provide driving directions for hundreds of millions of users, quickly plotting an optimal route taking into account current and predicted future traffic conditions.

MACHINE TRANSLATION: Online machine translation systems now enable the reading of documents in over 100 languages, including the native languages of over 99% of humans, and render hundreds of billions of words per day for hundreds of millions of users. While not perfect, they are generally adequate for understanding. For closely related languages with a great deal of training data (such as French and English) translations within a narrow domain are close to the level of a human (Wu et al., 2016b).

SPEECH RECOGNITION: In 2017, Microsoft showed that its Conversational Speech Recognition System had reached a word error rate of 5.1%, matching human performance on the Switchboard task, which involves transcribing telephone conversations (Xiong et al., 2017). About a third of computer interaction worldwide is now done by voice rather than keyboard; Skype provides real-time speech-to-speech translation in ten languages. Alexa, Siri, Cortana, and Google offer assistants that can answer questions and carry out tasks for the user; for example the Google Duplex service uses speech recognition and speech synthesis to make restaurant reservations for users, carrying out a fluent conversation on their behalf.

RECOMMENDATIONS: Companies such as Amazon, Facebook, Netflix, Spotify, YouTube, Walmart, and others use machine learning to recommend what you might like based on your past experiences and those of others like you. The field of recommender systems has a long history (Resnick and Varian, 1997) but is changing rapidly due to new deep learning methods that analyze content (text, music, video) as well as history and metadata (van den Oord et al., 2014; Zhang et al., 2017). Spam filtering can also be considered a form of recommendation (or dis-recommendation); current AI techniques filter out over 99.9% of

spam, and email services can also recommend potential recipients, as well as possible response text.

Game playing: When Deep Blue defeated world chess champion Garry Kasparov in 1997, defenders of human supremacy placed their hopes on Go. Piet Hut, an astrophysicist and Go enthusiast, predicted that it would take “a hundred years before a computer beats humans at Go—maybe even longer.” But just 20 years later, ALPHAGO surpassed all human players (Silver et al., 2017). Ke Jie, the world champion, said, “Last year, it was still quite human-like when it played. But this year, it became like a god of Go.” ALPHAGO benefited from studying hundreds of thousands of past games by human Go players, and from the distilled knowledge of expert Go players that worked on the team.

A followup program, ALPHAZERO, used no input from humans (except for the rules of the game), and was able to learn through self-play alone to defeat all opponents, human and machine, at Go, chess, and shogi (Silver et al., 2018). Meanwhile, human champions have been beaten by AI systems at games as diverse as Jeopardy! (Ferrucci et al., 2010), poker (Bowling et al., 2015; Moravčík et al., 2017; Brown and Sandholm, 2019), and the video games Dota 2 (Fernandez and Mahlmann, 2018), StarCraft II (Vinyals et al., 2019), and Quake III (Jaderberg et al., 2019).

IMAGE UNDERSTANDING: Not content with exceeding human accuracy on the challenging ImageNet object recognition task, computer vision researchers have taken on the more difficult problem of image captioning. Some impressive examples include “A person riding a motorcycle on a dirt road,” “Two pizzas sitting on top of a stove top oven,” and “A group of young people playing a game of frisbee” (Vinyals et al., 2017b). Current systems are far from perfect, however: a “refrigerator filled with lots of food and drinks” turns out to be a no-parking sign partially obscured by lots of small stickers.

MEDICINE: AI algorithms now equal or exceed expert doctors at diagnosing many conditions, particularly when the diagnosis is based on images. Examples include Alzheimer’s disease (Ding et al., 2018), metastatic cancer (Liu et al., 2017; Esteva et al., 2017), ophthalmic disease (Gulshan et al., 2016), and skin diseases (Liu et al., 2019c). A systematic review and meta-analysis (Liu et al., 2019a) found that the performance of AI programs, on average, was equivalent to health care professionals. One current emphasis in medical AI is in facilitating human–machine partnerships. For example, the LYNA system achieves 99.6%

overall accuracy in diagnosing metastatic breast cancer—better than an unaided human expert—but the combination does better still (Liu et al., 2018; Steiner et al., 2018)..

The widespread adoption of these techniques is now limited not by diagnostic accuracy but by the need to demonstrate improvement in clinical outcomes and to ensure transparency, lack of bias, and data privacy (Topol, 2019). In 2017, only two medical AI applications were approved by the FDA, but that increased to 12 in 2018, and continues to rise.

CLIMATE SCIENCE: A team of scientists won the 2018 Gordon Bell Prize for a deep learning model that discovers detailed information about extreme weather events that were previously buried in climate data. They used a supercomputer with specialized GPU hardware to exceed the exaop level (1018 operations per second), the first machine learning program to do so (Kurth et al., 2018). Rolnick et al. (2019) present a 60-page catalog of ways in which machine learning can be used to tackle climate change.

These are just a few examples of artificial intelligence systems that exist today. Not magic or science fiction—but rather science, engineering, and mathematics, to which this book provides an introduction.

1.5 Risks and Benefits of AI Francis Bacon, a philosopher credited with creating the scientific method, noted in The Wisdom of the Ancients (1609) that the “mechanical arts are of ambiguous use, serving as well for hurt as for remedy.” As AI plays an increasingly important role in the economic, social, scientific, medical, financial, and military spheres, we would do well to consider the hurts and remedies—in modern parlance, the risks and benefits—that it can bring. The topics summarized here are covered in greater depth in Chapters 27  and 28 .

To begin with the benefits: put simply, our entire civilization is the product of our human intelligence. If we have access to substantially greater machine intelligence, the ceiling on our ambitions is raised substantially. The potential for AI and robotics to free humanity from menial repetitive work and to dramatically increase the production of goods and services could presage an era of peace and plenty. The capacity to accelerate scientific research could result in cures for disease and solutions for climate change and resource shortages. As Demis Hassabis, CEO of Google DeepMind, has suggested: “First solve AI, then use AI to solve everything else.”

Long before we have an opportunity to “solve AI,” however, we will incur risks from the misuse of AI, inadvertent or otherwise. Some of these are already apparent, while others seem likely based on current trends:

LETHAL AUTONOMOUS WEAPONS: These are defined by the United Nations as weapons that can locate, select, and eliminate human targets without human intervention. A primary concern with such weapons is their scalability: the absence of a requirement for human supervision means that a small group can deploy an arbitrarily large number of weapons against human targets defined by any feasible recognition criterion. The technologies needed for autonomous weapons are similar to those needed for self-driving cars. Informal expert discussions on the potential risks of lethal autonomous weapons began at the UN in 2014, moving to the formal pre-treaty stage of a Group of Governmental Experts in 2017. SURVEILLANCE AND PERSUASION: While it is expensive, tedious, and sometimes legally questionable for security personnel to monitor phone lines, video camera feeds, emails, and other messaging channels, AI (speech recognition, computer vision, and

natural language understanding) can be used in a scalable fashion to perform mass surveillance of individuals and detect activities of interest. By tailoring information flows to individuals through social media, based on machine learning techniques, political behavior can be modified and controlled to some extent—a concern that became apparent in elections beginning in 2016. BIASED DECISION MAKING: Careless or deliberate misuse of machine learning algorithms for tasks such as evaluating parole and loan applications can result in decisions that are biased by race, gender, or other protected categories. Often, the data themselves reflect pervasive bias in society. IMPACT ON EMPLOYMENT: Concerns about machines eliminating jobs are centuries old. The story is never simple: machines do some of the tasks that humans might otherwise do, but they also make humans more productive and therefore more employable, and make companies more profitable and therefore able to pay higher wages. They may render some activities economically viable that would otherwise be impractical. Their use generally results in increasing wealth but tends to have the effect of shifting wealth from labor to capital, further exacerbating increases in inequality. Previous advances in technology—such as the invention of mechanical looms—have resulted in serious disruptions to employment, but eventually people find new kinds of work to do. On the other hand, it is possible that AI will be doing those new kinds of work too. This topic is rapidly becoming a major focus for economists and governments around the world. SAFETY-CRITICAL APPLICATIONS: As AI techniques advance, they are increasingly used in high-stakes, safety-critical applications such as driving cars and managing the water supplies of cities. Fatal accidents have already occurred and highlight the difficulty of formal verification and statistical risk analysis for systems developed using machine learning techniques. The field of AI will need to develop technical and ethical standards at least comparable to those prevalent in other engineering and healthcare disciplines where people’s lives are at stake. CYBERSECURITY: AI techniques are useful in defending against cyberattack, for example by detecting unusual patterns of behavior, but they will also contribute to the potency, survivability, and proliferation capability of malware. For example, reinforcement learning methods have been used to create highly effective tools for automated, personalized blackmail and phishing attacks.

We will revisit these topics in more depth in Section 27.3 . As AI systems become more capable, they will take on more of the societal roles previously played by humans. Just as humans have used these roles in the past to perpetrate mischief, we can expect that humans may misuse AI systems in these roles to perpetrate even more mischief. All of the examples given above point to the importance of governance and, eventually, regulation. At present, the research community and the major corporations involved in AI research have developed voluntary self-governance principles for AI-related activities (see Section 27.3 ). Governments and international organizations are setting up advisory bodies to devise appropriate regulations for each specific use case, to prepare for the economic and social impacts, and to take advantage of AI capabilities to address major societal problems.

What of the longer term? Will we achieve the long-standing goal: the creation of intelligence comparable to or more capable than human intelligence? And, if we do, what then?

Human-level AI

Artificial general intelligence (AGI)

For much of AI’s history, these questions have been overshadowed by the daily grind of getting AI systems to do anything even remotely intelligent. As with any broad discipline, the great majority of AI researchers have specialized in a specific subfield such as gameplaying, knowledge representation, vision, or natural language understanding—often on the assumption that progress in these subfields would contribute to the broader goals of AI. Nils Nilsson (1995), one of the original leaders of the Shakey project at SRI, reminded the field of those broader goals and warned that the subfields were in danger of becoming ends in themselves. Later, some influential founders of AI, including John McCarthy (2007), Marvin Minsky (2007), and Patrick Winston (Beal and Winston, 2009), concurred with Nilsson’s warnings, suggesting that instead of focusing on measurable performance in specific applications, AI should return to its roots of striving for, in Herb Simon’s words, “machines that think, that learn and that create.” They called the effort human-level AI or HLAI—a

machine should be able to learn to do anything a human can do. Their first symposium was in 2004 (Minsky et al., 2004). Another effort with similar goals, the artificial general intelligence (AGI) movement (Goertzel and Pennachin, 2007), held its first conference and organized the Journal of Artificial General Intelligence in 2008.

Artificial superintelligence (ASI)

At around the same time, concerns were raised that creating artificial superintelligence or ASI—intelligence that far surpasses human ability—might be a bad idea (Yudkowsky, 2008; Omohundro, 2008). Turing (1996) himself made the same point in a lecture given in Manchester in 1951, drawing on earlier ideas from Samuel Butler (1863):15

15 Even earlier, in 1847, Richard Thornton, editor of the Primitive Expounder, railed against mechanical calculators: “Mind ... outruns itself and does away with the necessity of its own existence by inventing machines to do its own thinking. ... But who knows that such machines when brought to greater perfection, may not think of a plan to remedy all their own defects and then grind out ideas beyond the ken of mortal mind!”

It seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. ... At some stage therefore we should have to expect the machines to take control, in the way that is mentioned in Samuel Butler’s Erewhon.

These concerns have only become more widespread with recent advances in deep learning, the publication of books such as Superintelligence by Nick Bostrom (2014), and public pronouncements from Stephen Hawking, Bill Gates, Martin Rees, and Elon Musk.

Experiencing a general sense of unease with the idea of creating superintelligent machines is only natural. We might call this the gorilla problem: about seven million years ago, a nowextinct primate evolved, with one branch leading to gorillas and one to humans. Today, the gorillas are not too happy about the human branch; they have essentially no control over their future. If this is the result of success in creating superhuman AI—that humans cede control over their future—then perhaps we should stop work on AI, and, as a corollary, give up the benefits it might bring. This is the essence of Turing’s warning: it is not obvious that we can control machines that are more intelligent than us.

Gorilla problem

If superhuman AI were a black box that arrived from outer space, then indeed it would be wise to exercise caution in opening the box. But it is not: we design the AI systems, so if they do end up “taking control,” as Turing suggests, it would be the result of a design failure.

To avoid such an outcome, we need to understand the source of potential failure. Norbert Wiener (1960), who was motivated to consider the long-term future of AI after seeing Arthur Samuel’s checker-playing program learn to beat its creator, had this to say:

If we use, to achieve our purposes, a mechanical agency with whose operation we cannot interfere effectively ... we had better be quite sure that the purpose put into the machine is the purpose which we really desire.

Many cultures have myths of humans who ask gods, genies, magicians, or devils for something. Invariably, in these stories, they get what they literally ask for, and then regret it. The third wish, if there is one, is to undo the first two. We will call this the King Midas problem: Midas, a legendary King in Greek mythology, asked that everything he touched should turn to gold, but then regretted it after touching his food, drink, and family members.16

16 Midas would have done better if he had followed basic principles of safety and included an “undo” button and a “pause” button in his wish.

King Midas problem

We touched on this issue in Section 1.1.5 , where we pointed out the need for a significant modification to the standard model of putting fixed objectives into the machine. The solution to Wiener’s predicament is not to have a definite “purpose put into the machine” at all. Instead, we want machines that strive to achieve human objectives but know that they don’t know for certain exactly what those objectives are.

It is perhaps unfortunate that almost all AI research to date has been carried out within the standard model, which means that almost all of the technical material in this edition reflects that intellectual framework. There are, however, some early results within the new framework. In Chapter 16 , we show that a machine has a positive incentive to allow itself to be switched off if and only if it is uncertain about the human objective. In Chapter 18 , we formulate and study assistance games, which describe mathematically the situation in which a human has an objective and a machine tries to achieve it, but is initially uncertain about what it is. In Chapter 22 , we explain the methods of inverse reinforcement learning that allow machines to learn more about human preferences from observations of the choices that humans make. In Chapter 27 , we explore two of the principal difficulties: first, that our choices depend on our preferences through a very complex cognitive architecture that is hard to invert; and, second, that we humans may not have consistent preferences in the first place—either individually or as a group—so it may not be clear what AI systems should be doing for us.

Assistance game

Inverse reinforcement learning

Summary This chapter defines AI and establishes the cultural background against which it has developed. Some of the important points are as follows:

Different people approach AI with different goals in mind. Two important questions to ask are: Are you concerned with thinking, or behavior? Do you want to model humans, or try to achieve the optimal results? According to what we have called the standard model, AI is concerned mainly with rational action. An ideal intelligent agent takes the best possible action in a situation. We study the problem of building agents that are intelligent in this sense. Two refinements to this simple idea are needed: first, the ability of any agent, human or otherwise, to choose rational actions is limited by the computational intractability of doing so; second, the concept of a machine that pursues a definite objective needs to be replaced with that of a machine pursuing objectives to benefit humans, but uncertain as to what those objectives are. Philosophers (going back to 400

BCE)

made AI conceivable by suggesting that the mind is

in some ways like a machine, that it operates on knowledge encoded in some internal language, and that thought can be used to choose what actions to take. Mathematicians provided the tools to manipulate statements of logical certainty as well as uncertain, probabilistic statements. They also set the groundwork for understanding computation and reasoning about algorithms. Economists formalized the problem of making decisions that maximize the expected utility to the decision maker. Neuroscientists discovered some facts about how the brain works and the ways in which it is similar to and different from computers. Psychologists adopted the idea that humans and animals can be considered information-processing machines. Linguists showed that language use fits into this model. Computer engineers provided the ever-more-powerful machines that make AI applications possible, and software engineers made them more usable. Control theory deals with designing devices that act optimally on the basis of feedback from the environment. Initially, the mathematical tools of control theory were quite different from those used in AI, but the fields are coming closer together.

The history of AI has had cycles of success, misplaced optimism, and resulting cutbacks in enthusiasm and funding. There have also been cycles of introducing new, creative approaches and systematically refining the best ones. AI has matured considerably compared to its early decades, both theoretically and methodologically. As the problems that AI deals with became more complex, the field moved from Boolean logic to probabilistic reasoning, and from hand-crafted knowledge to machine learning from data. This has led to improvements in the capabilities of real systems and greater integration with other disciplines. As AI systems find application in the real world, it has become necessary to consider a wide range of risks and ethical consequences. In the longer term, we face the difficult problem of controlling superintelligent AI systems that may evolve in unpredictable ways. Solving this problem seems to necessitate a change in our conception of AI.

Bibliographical and Historical Notes A comprehensive history of AI is given by Nils Nilsson (2009), one of the early pioneers of the field. Pedro Domingos (2015) and Melanie Mitchell (2019) give overviews of machine learning for a general audience, and Kai-Fu Lee (2018) describes the race for international leadership in AI. Martin Ford (2018) interviews 23 leading AI researchers.

The main professional societies for AI are the Association for the Advancement of Artificial Intelligence (AAAI), the ACM Special Interest Group in Artificial Intelligence (SIGAI, formerly SIGART), the European Association for AI, and the Society for Artificial Intelligence and Simulation of Behaviour (AISB). The Partnership on AI brings together many commercial and nonprofit organizations concerned with the ethical and social impacts of AI. AAAI’s AI Magazine contains many topical and tutorial articles, and its Web site, aaai.org, contains news, tutorials, and background information.

The most recent work appears in the proceedings of the major AI conferences: the International Joint Conference on AI (IJCAI), the annual European Conference on AI (ECAI), and the AAAI Conference. Machine learning is covered by the International Conference on Machine Learning and the Neural Information Processing Systems (NeurIPS) meeting. The major journals for general AI are Artificial Intelligence, Computational Intelligence, the IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Intelligent Systems, and the Journal of Artificial Intelligence Research. There are also many conferences and journals devoted to specific areas, which we cover in the appropriate chapters.

Chapter 2 Intelligent Agents In which we discuss the nature of agents, perfect or otherwise, the diversity of environments, and the resulting menagerie of agent types. Chapter 1  identified the concept of rational agents as central to our approach to artificial intelligence. In this chapter, we make this notion more concrete. We will see that the concept of rationality can be applied to a wide variety of agents operating in any imaginable environment. Our plan in this book is to use this concept to develop a small set of design principles for building successful agents—systems that can reasonably be called intelligent.

We begin by examining agents, environments, and the coupling between them. The observation that some agents behave better than others leads naturally to the idea of a rational agent—one that behaves as well as possible. How well an agent can behave depends on the nature of the environment; some environments are more difficult than others. We give a crude categorization of environments and show how properties of an environment influence the design of suitable agents for that environment. We describe a number of basic “skeleton” agent designs, which we flesh out in the rest of the book.

2.1 Agents and Environments An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators. This simple idea is illustrated in Figure 2.1 . A human agent has eyes, ears, and other organs for sensors and hands, legs, vocal tract, and so on for actuators. A robotic agent might have cameras and infrared range finders for sensors and various motors for actuators. A software agent receives file contents, network packets, and human input (keyboard/mouse/touchscreen/voice) as sensory inputs and acts on the environment by writing files, sending network packets, and displaying information or generating sounds. The environment could be everything—the entire universe! In practice it is just that part of the universe whose state we care about when designing this agent—the part that affects what the agent perceives and that is affected by the agent’s actions.

Figure 2.1

Agents interact with environments through sensors and actuators.

Environment

Sensor

Actuator

We use the term percept to refer to the content an agent’s sensors are perceiving. An agent’s percept sequence is the complete history of everything the agent has ever perceived. In general, an agent’s choice of action at any given instant can depend on its built-in knowledge and on the entire percept sequence observed to date, but not on anything it hasn’t perceived. By specifying the agent’s choice of action for every possible percept sequence, we have said more or less everything there is to say about the agent. Mathematically speaking, we say that an agent’s behavior is described by the agent function that maps any given percept sequence to an action.

Percept

Percept sequence

Agent function

We can imagine tabulating the agent function that describes any given agent; for most agents, this would be a very large table—infinite, in fact, unless we place a bound on the length of percept sequences we want to consider. Given an agent to experiment with, we can, in principle, construct this table by trying out all possible percept sequences and recording which actions the agent does in response.1 The table is, of course, an external characterization of the agent. Internally, the agent function for an artificial agent will be implemented by an agent program. It is important to keep these two ideas distinct. The agent function is an abstract mathematical description; the agent program is a concrete implementation, running within some physical system.

1 If the agent uses some randomization to choose its actions, then we would have to try each sequence many times to identify the probability of each action. One might imagine that acting randomly is rather silly, but we show later in this chapter that it can be very intelligent.

Agent program

To illustrate these ideas, we use a simple example—the vacuum-cleaner world, which consists of a robotic vacuum-cleaning agent in a world consisting of squares that can be either dirty or clean. Figure 2.2  shows a configuration with just two squares,

A and B. The

vacuum agent perceives which square it is in and whether there is dirt in the square. The agent starts in square

A. The available actions are to move to the right, move to the left,

suck up the dirt, or do nothing.2 One very simple agent function is the following: if the current square is dirty, then suck; otherwise, move to the other square. A partial tabulation of this agent function is shown in Figure 2.3  and an agent program that implements it appears in Figure 2.8  on page 49.

2 In a real robot, it would be unlikely to have an actions like “move right” and “move left.” Instead the actions would be “spin wheels forward” and “spin wheels backward.” We have chosen the actions to be easier to follow on the page, not for ease of implementation in an actual robot.

Figure 2.2

A vacuum-cleaner world with just two locations. Each location can be clean or dirty, and the agent can move left or right and can clean the square that it occupies. Different versions of the vacuum world allow for different rules about what the agent can perceive, whether its actions always succeed, and so on.

Figure 2.3

Partial tabulation of a simple agent function for the vacuum-cleaner world shown in Figure 2.2 . The agent cleans the current square if it is dirty, otherwise it moves to the other square. Note that the table is of unbounded size unless there is a restriction on the length of possible percept sequences.

Looking at Figure 2.3 , we see that various vacuum-world agents can be defined simply by filling in the right-hand column in various ways. The obvious question, then, is this: What is the right way to fill out the table? In other words, what makes an agent good or bad, intelligent or stupid? We answer these questions in the next section.

Before closing this section, we should emphasize that the notion of an agent is meant to be a tool for analyzing systems, not an absolute characterization that divides the world into agents and non-agents. One could view a hand-held calculator as an agent that chooses the

action of displaying “4” when given the percept sequence

“2 + 2 = ,” but such an analysis

would hardly aid our understanding of the calculator. In a sense, all areas of engineering can be seen as designing artifacts that interact with the world; AI operates at (what the authors consider to be) the most interesting end of the spectrum, where the artifacts have significant computational resources and the task environment requires nontrivial decision making.

2.2 Good Behavior: The Concept of Rationality A rational agent is one that does the right thing. Obviously, doing the right thing is better than doing the wrong thing, but what does it mean to do the right thing?

Rational agent

2.2.1 Performance measures Moral philosophy has developed several different notions of the “right thing,” but AI has generally stuck to one notion called consequentialism: we evaluate an agent’s behavior by its consequences. When an agent is plunked down in an environment, it generates a sequence of actions according to the percepts it receives. This sequence of actions causes the environment to go through a sequence of states. If the sequence is desirable, then the agent has performed well. This notion of desirability is captured by a performance measure that evaluates any given sequence of environment states.

Consequentialism

Performance measure

Humans have desires and preferences of their own, so the notion of rationality as applied to humans has to do with their success in choosing actions that produce sequences of environment states that are desirable from their point of view. Machines, on the other hand, do not have desires and preferences of their own; the performance measure is, initially at least, in the mind of the designer of the machine, or in the mind of the users the machine is

designed for. We will see that some agent designs have an explicit representation of (a version of) the performance measure, while in other designs the performance measure is entirely implicit—the agent may do the right thing, but it doesn’t know why.

Recalling Norbert Wiener’s warning to ensure that “the purpose put into the machine is the purpose which we really desire” (page 33), notice that it can be quite hard to formulate a performance measure correctly. Consider, for example, the vacuum-cleaner agent from the preceding section. We might propose to measure performance by the amount of dirt cleaned up in a single eight-hour shift. With a rational agent, of course, what you ask for is what you get. A rational agent can maximize this performance measure by cleaning up the dirt, then dumping it all on the floor, then cleaning it up again, and so on. A more suitable performance measure would reward the agent for having a clean floor. For example, one point could be awarded for each clean square at each time step (perhaps with a penalty for electricity consumed and noise generated). As a general rule, it is better to design performance measures according to what one actually wants to be achieved in the environment, rather than according to how one thinks the agent should behave.

Even when the obvious pitfalls are avoided, some knotty problems remain. For example, the notion of “clean floor” in the preceding paragraph is based on average cleanliness over time. Yet the same average cleanliness can be achieved by two different agents, one of which does a mediocre job all the time while the other cleans energetically but takes long breaks. Which is preferable might seem to be a fine point of janitorial science, but in fact it is a deep philosophical question with far-reaching implications. Which is better—a reckless life of highs and lows, or a safe but humdrum existence? Which is better—an economy where everyone lives in moderate poverty, or one in which some live in plenty while others are very poor? We leave these questions as an exercise for the diligent reader.

For most of the book, we will assume that the performance measure can be specified correctly. For the reasons given above, however, we must accept the possibility that we might put the wrong purpose into the machine—precisely the King Midas problem described on page 33. Moreover, when designing one piece of software, copies of which will belong to different users, we cannot anticipate the exact preferences of each individual user. Thus, we may need to build agents that reflect initial uncertainty about the true performance measure and learn more about it as time goes by; such agents are described in Chapters 16 , 18 , and 22 .

2.2.2 Rationality What is rational at any given time depends on four things:

The performance measure that defines the criterion of success. The agent’s prior knowledge of the environment. The actions that the agent can perform. The agent’s percept sequence to date.

This leads to a definition of a rational agent:

For each possible percept sequence, a rational agent should select an action that is expected to maximize its performance measure, given the evidence provided by the percept sequence and whatever built-in knowledge the agent has.

Definition of a rational agent

Consider the simple vacuum-cleaner agent that cleans a square if it is dirty and moves to the other square if not; this is the agent function tabulated in Figure 2.3 . Is this a rational agent? That depends! First, we need to say what the performance measure is, what is known about the environment, and what sensors and actuators the agent has. Let us assume the following:

The performance measure awards one point for each clean square at each time step, over a “lifetime” of 1000 time steps. The “geography” of the environment is known a priori (Figure 2.2 ) but the dirt distribution and the initial location of the agent are not. Clean squares stay clean and sucking cleans the current square. The Right and Left actions move the agent one square except when this would take the agent outside the environment, in which case the agent remains where it is. The only available actions are Right, Left, and Suck. The agent correctly perceives its location and whether that location contains dirt.

Under these circumstances the agent is indeed rational; its expected performance is at least as good as any other agent’s.

One can see easily that the same agent would be irrational under different circumstances. For example, once all the dirt is cleaned up, the agent will oscillate needlessly back and forth; if the performance measure includes a penalty of one point for each movement, the agent will fare poorly. A better agent for this case would do nothing once it is sure that all the squares are clean. If clean squares can become dirty again, the agent should occasionally check and re-clean them if needed. If the geography of the environment is unknown, the agent will need to explore it. Exercise 2.VACR asks you to design agents for these cases.

2.2.3 Omniscience, learning, and autonomy

Omniscience

We need to be careful to distinguish between rationality and omniscience. An omniscient agent knows the actual outcome of its actions and can act accordingly; but omniscience is impossible in reality. Consider the following example: I am walking along the Champs Elysées one day and I see an old friend across the street. There is no traffic nearby and I’m not otherwise engaged, so, being rational, I start to cross the street. Meanwhile, at 33,000 feet, a cargo door falls off a passing airliner,3 and before I make it to the other side of the street I am flattened. Was I irrational to cross the street? It is unlikely that my obituary would read “Idiot attempts to cross street.”

3 See N. Henderson, “New door latches urged for Boeing 747 jumbo jets,” Washington Post, August 24, 1989.

This example shows that rationality is not the same as perfection. Rationality maximizes expected performance, while perfection maximizes actual performance. Retreating from a requirement of perfection is not just a question of being fair to agents. The point is that if we expect an agent to do what turns out after the fact to be the best action, it will be impossible to design an agent to fulfill this specification—unless we improve the performance of crystal balls or time machines.

Our definition of rationality does not require omniscience, then, because the rational choice depends only on the percept sequence to date. We must also ensure that we haven’t inadvertently allowed the agent to engage in decidedly underintelligent activities. For example, if an agent does not look both ways before crossing a busy road, then its percept sequence will not tell it that there is a large truck approaching at high speed. Does our definition of rationality say that it’s now OK to cross the road? Far from it!

First, it would not be rational to cross the road given this uninformative percept sequence: the risk of accident from crossing without looking is too great. Second, a rational agent should choose the “looking” action before stepping into the street, because looking helps maximize the expected performance. Doing actions in order to modify future percepts— sometimes called information gathering—is an important part of rationality and is covered in depth in Chapter 16 . A second example of information gathering is provided by the exploration that must be undertaken by a vacuum-cleaning agent in an initially unknown environment.

Information gathering

Our definition requires a rational agent not only to gather information but also to learn as much as possible from what it perceives. The agent’s initial configuration could reflect some prior knowledge of the environment, but as the agent gains experience this may be modified and augmented. There are extreme cases in which the environment is completely known a priori and completely predictable. In such cases, the agent need not perceive or learn; it simply acts correctly.

Learning

Of course, such agents are fragile. Consider the lowly dung beetle. After digging its nest and laying its eggs, it fetches a ball of dung from a nearby heap to plug the entrance. If the ball

of dung is removed from its grasp en route, the beetle continues its task and pantomimes plugging the nest with the nonexistent dung ball, never noticing that it is missing. Evolution has built an assumption into the beetle’s behavior, and when it is violated, unsuccessful behavior results.

Slightly more intelligent is the sphex wasp. The female sphex will dig a burrow, go out and sting a caterpillar and drag it to the burrow, enter the burrow again to check all is well, drag the caterpillar inside, and lay its eggs. The caterpillar serves as a food source when the eggs hatch. So far so good, but if an entomologist moves the caterpillar a few inches away while the sphex is doing the check, it will revert to the “drag the caterpillar” step of its plan and will continue the plan without modification, re-checking the burrow, even after dozens of caterpillar-moving interventions. The sphex is unable to learn that its innate plan is failing, and thus will not change it.

To the extent that an agent relies on the prior knowledge of its designer rather than on its own percepts and learning processes, we say that the agent lacks autonomy. A rational agent should be autonomous—it should learn what it can to compensate for partial or incorrect prior knowledge. For example, a vacuum-cleaning agent that learns to predict where and when additional dirt will appear will do better than one that does not.

Autonomy

As a practical matter, one seldom requires complete autonomy from the start: when the agent has had little or no experience, it would have to act randomly unless the designer gave some assistance. Just as evolution provides animals with enough built-in reflexes to survive long enough to learn for themselves, it would be reasonable to provide an artificial intelligent agent with some initial knowledge as well as an ability to learn. After sufficient experience of its environment, the behavior of a rational agent can become effectively independent of its prior knowledge. Hence, the incorporation of learning allows one to design a single rational agent that will succeed in a vast variety of environments.

2.3 The Nature of Environments Now that we have a definition of rationality, we are almost ready to think about building rational agents. First, however, we must think about task environments, which are essentially the “problems” to which rational agents are the “solutions.” We begin by showing how to specify a task environment, illustrating the process with a number of examples. We then show that task environments come in a variety of flavors. The nature of the task environment directly affects the appropriate design for the agent program.

Task environment

2.3.1 Specifying the task environment In our discussion of the rationality of the simple vacuum-cleaner agent, we had to specify the performance measure, the environment, and the agent’s actuators and sensors. We group all these under the heading of the task environment. For the acronymically minded, we call this the PEAS (Performance, Environment, Actuators, Sensors) description. In designing an agent, the first step must always be to specify the task environment as fully as possible.

PEAS

The vacuum world was a simple example; let us consider a more complex problem: an automated taxi driver. Figure 2.4  summarizes the PEAS description for the taxi’s task environment. We discuss each element in more detail in the following paragraphs.

Figure 2.4

PEAS description of the task environment for an automated taxi driver.

First, what is the performance measure to which we would like our automated driver to aspire? Desirable qualities include getting to the correct destination; minimizing fuel consumption and wear and tear; minimizing the trip time or cost; minimizing violations of traffic laws and disturbances to other drivers; maximizing safety and passenger comfort; maximizing profits. Obviously, some of these goals conflict, so tradeoffs will be required.

Next, what is the driving environment that the taxi will face? Any taxi driver must deal with a variety of roads, ranging from rural lanes and urban alleys to 12-lane freeways. The roads contain other traffic, pedestrians, stray animals, road works, police cars, puddles, and potholes. The taxi must also interact with potential and actual passengers. There are also some optional choices. The taxi might need to operate in Southern California, where snow is seldom a problem, or in Alaska, where it seldom is not. It could always be driving on the right, or we might want it to be flexible enough to drive on the left when in Britain or Japan. Obviously, the more restricted the environment, the easier the design problem.

The actuators for an automated taxi include those available to a human driver: control over the engine through the accelerator and control over steering and braking. In addition, it will need output to a display screen or voice synthesizer to talk back to the passengers, and perhaps some way to communicate with other vehicles, politely or otherwise.

The basic sensors for the taxi will include one or more video cameras so that it can see, as well as lidar and ultrasound sensors to detect distances to other cars and obstacles. To avoid speeding tickets, the taxi should have a speedometer, and to control the vehicle properly, especially on curves, it should have an accelerometer. To determine the mechanical state of

the vehicle, it will need the usual array of engine, fuel, and electrical system sensors. Like many human drivers, it might want to access GPS signals so that it doesn’t get lost. Finally, it will need touchscreen or voice input for the passenger to request a destination. In Figure 2.5 , we have sketched the basic PEAS elements for a number of additional agent types. Further examples appear in Exercise 2.PEAS. The examples include physical as well as virtual environments. Note that virtual task environments can be just as complex as the “real” world: for example, a software agent (or software robot or softbot) that trades on auction and reselling Web sites deals with millions of other users and billions of objects, many with real images.

Figure 2.5

Examples of agent types and their PEAS descriptions.

Software agent

Softbot

2.3.2 Properties of task environments The range of task environments that might arise in AI is obviously vast. We can, however, identify a fairly small number of dimensions along which task environments can be categorized. These dimensions determine, to a large extent, the appropriate agent design and the applicability of each of the principal families of techniques for agent implementation. First we list the dimensions, then we analyze several task environments to illustrate the ideas. The definitions here are informal; later chapters provide more precise statements and examples of each kind of environment.

Fully observable

Partially observable

FULLY OBSERVABLE VS. PARTIALLY OBSERVABLE: If an agent’s sensors give it access to the complete state of the environment at each point in time, then we say that the task environment is fully observable. A task environment is effectively fully observable if the sensors detect all aspects that are relevant to the choice of action; relevance, in turn, depends on the performance measure. Fully observable environments are convenient because the agent need not maintain any internal state to keep track of the world. An environment might be partially observable because of noisy and inaccurate sensors or because parts of the state are simply missing from the sensor data—for example, a vacuum agent with only a local dirt sensor cannot tell whether there is dirt in other squares, and an

automated taxi cannot see what other drivers are thinking. If the agent has no sensors at all then the environment is unobservable. One might think that in such cases the agent’s plight is hopeless, but, as we discuss in Chapter 4 , the agent’s goals may still be achievable, sometimes with certainty.

Unobservable

SINGLE-AGENT VS. MULTIAGENT: The distinction between single-agent and multiagent environments may seem simple enough. For example, an agent solving a crossword puzzle by itself is clearly in a single-agent environment, whereas an agent playing chess is in a twoagent environment. However, there are some subtle issues. First, we have described how an entity may be viewed as an agent, but we have not explained which entities must be viewed as agents. Does an agent

A (the taxi driver for example) have to treat an object B (another

vehicle) as an agent, or can it be treated merely as an object behaving according to the laws of physics, analogous to waves at the beach or leaves blowing in the wind? The key

B’s behavior is best described as maximizing a performance measure whose value depends on agent A’s behavior. distinction is whether

Single-agent

Multiagent

B is trying to maximize its performance measure, which, by the rules of chess, minimizes agent A’s performance measure. Thus, chess is a For example, in chess, the opponent entity

competitive multiagent environment. On the other hand, in the taxi-driving environment, avoiding collisions maximizes the performance measure of all agents, so it is a partially

cooperative multiagent environment. It is also partially competitive because, for example, only one car can occupy a parking space.

Competitive

Cooperative

The agent-design problems in multiagent environments are often quite different from those in single-agent environments; for example, communication often emerges as a rational behavior in multiagent environments; in some competitive environments, randomized behavior is rational because it avoids the pitfalls of predictability.

Deterministic vs. nondeterministic. If the next state of the environment is completely determined by the current state and the action executed by the agent(s), then we say the environment is deterministic; otherwise, it is nondeterministic. In principle, an agent need not worry about uncertainty in a fully observable, deterministic environment. If the environment is partially observable, however, then it could appear to be nondeterministic.

Deterministic

Nondeterministic

Most real situations are so complex that it is impossible to keep track of all the unobserved aspects; for practical purposes, they must be treated as nondeterministic. Taxi driving is clearly nondeterministic in this sense, because one can never predict the behavior of traffic

exactly; moreover, one’s tires may blow out unexpectedly and one’s engine may seize up without warning. The vacuum world as we described it is deterministic, but variations can include nondeterministic elements such as randomly appearing dirt and an unreliable suction mechanism (Exercise 2.VFIN).

One final note: the word stochastic is used by some as a synonym for “nondeterministic,” but we make a distinction between the two terms; we say that a model of the environment is stochastic if it explicitly deals with probabilities (e.g., “there’s a 25% chance of rain tomorrow”) and “nondeterministic” if the possibilities are listed without being quantified (e.g., “there’s a chance of rain tomorrow”).

Stochastic

EPISODIC VS. SEQUENTIAL: In an episodic task environment, the agent’s experience is divided into atomic episodes. In each episode the agent receives a percept and then performs a single action. Crucially, the next episode does not depend on the actions taken in previous episodes. Many classification tasks are episodic. For example, an agent that has to spot defective parts on an assembly line bases each decision on the current part, regardless of previous decisions; moreover, the current decision doesn’t affect whether the next part is defective. In sequential environments, on the other hand, the current decision could affect all future decisions.4 Chess and taxi driving are sequential: in both cases, short-term actions can have long-term consequences. Episodic environments are much simpler than sequential environments because the agent does not need to think ahead.

4 The word “sequential” is also used in computer science as the antonym of “parallel.” The two meanings are largely unrelated.

Episodic

Sequential

Static

Dynamic

STATIC VS. DYNAMIC: If the environment can change while an agent is deliberating, then we say the environment is dynamic for that agent; otherwise, it is static. Static environments are easy to deal with because the agent need not keep looking at the world while it is deciding on an action, nor need it worry about the passage of time. Dynamic environments, on the other hand, are continuously asking the agent what it wants to do; if it hasn’t decided yet, that counts as deciding to do nothing. If the environment itself does not change with the passage of time but the agent’s performance score does, then we say the environment is semidynamic. Taxi driving is clearly dynamic: the other cars and the taxi itself keep moving while the driving algorithm dithers about what to do next. Chess, when played with a clock, is semidynamic. Crossword puzzles are static.

Semidynamic

DISCRETE VS. CONTINUOUS: The discrete/continuous distinction applies to the state of the environment, to the way time is handled, and to the percepts and actions of the agent. For example, the chess environment has a finite number of distinct states (excluding the clock). Chess also has a discrete set of percepts and actions. Taxi driving is a continuous-state and continuous-time problem: the speed and location of the taxi and of the other vehicles sweep through a range of continuous values and do so smoothly over time. Taxi-driving actions are also continuous (steering angles, etc.). Input from digital cameras is discrete, strictly speaking, but is typically treated as representing continuously varying intensities and locations.

Discrete

Continuous

KNOWN VS. UNKNOWN: Strictly speaking, this distinction refers not to the environment itself but to the agent’s (or designer’s) state of knowledge about the “laws of physics” of the environment. In a known environment, the outcomes (or outcome probabilities if the environment is nondeterministic) for all actions are given. Obviously, if the environment is unknown, the agent will have to learn how it works in order to make good decisions.

Known

Unknown

The distinction between known and unknown environments is not the same as the one between fully and partially observable environments. It is quite possible for a known environment to be partially observable—for example, in solitaire card games, I know the rules but am still unable to see the cards that have not yet been turned over. Conversely, an unknown environment can be fully observable—in a new video game, the screen may show the entire game state but I still don’t know what the buttons do until I try them.

As noted on page 39, the performance measure itself may be unknown, either because the designer is not sure how to write it down correctly or because the ultimate user—whose preferences matter—is not known. For example, a taxi driver usually won’t know whether a new passenger prefers a leisurely or speedy journey, a cautious or aggressive driving style. A virtual personal assistant starts out knowing nothing about the personal preferences of its

new owner. In such cases, the agent may learn more about the performance measure based on further interactions with the designer or user. This, in turn, suggests that the task environment is necessarily viewed as a multiagent environment.

The hardest case is partially observable, multiagent, nondeterministic, sequential, dynamic, continuous, and unknown. Taxi driving is hard in all these senses, except that the driver’s environment is mostly known. Driving a rented car in a new country with unfamiliar geography, different traffic laws, and nervous passengers is a lot more exciting. Figure 2.6  lists the properties of a number of familiar environments. Note that the properties are not always cut and dried. For example, we have listed the medical-diagnosis task as single-agent because the disease process in a patient is not profitably modeled as an agent; but a medical-diagnosis system might also have to deal with recalcitrant patients and skeptical staff, so the environment could have a multiagent aspect. Furthermore, medical diagnosis is episodic if one conceives of the task as selecting a diagnosis given a list of symptoms; the problem is sequential if the task can include proposing a series of tests, evaluating progress over the course of treatment, handling multiple patients, and so on.

Figure 2.6

Examples of task environments and their characteristics.

We have not included a “known/unknown” column because, as explained earlier, this is not strictly a property of the environment. For some environments, such as chess and poker, it is quite easy to supply the agent with full knowledge of the rules, but it is nonetheless

interesting to consider how an agent might learn to play these games without such knowledge.

The code repository associated with this book (aima.cs.berkeley.edu) includes multiple environment implementations, together with a general-purpose environment simulator for evaluating an agent’s performance. Experiments are often carried out not for a single environment but for many environments drawn from an environment class. For example, to evaluate a taxi driver in simulated traffic, we would want to run many simulations with different traffic, lighting, and weather conditions. We are then interested in the agent’s average performance over the environment class.

Environment class

2.4 The Structure of Agents So far we have talked about agents by describing behavior—the action that is performed after any given sequence of percepts. Now we must bite the bullet and talk about how the insides work. The job of AI is to design an agent program that implements the agent function—the mapping from percepts to actions. We assume this program will run on some sort of computing device with physical sensors and actuators—we call this the agent architecture:

agent = architecture + program .

Agent program

Agent architecture

Obviously, the program we choose has to be one that is appropriate for the architecture. If the program is going to recommend actions like Walk, the architecture had better have legs. The architecture might be just an ordinary PC, or it might be a robotic car with several onboard computers, cameras, and other sensors. In general, the architecture makes the percepts from the sensors available to the program, runs the program, and feeds the program’s action choices to the actuators as they are generated. Most of this book is about designing agent programs, although Chapters 25  and 26  deal directly with the sensors and actuators.

2.4.1 Agent programs The agent programs that we design in this book all have the same skeleton: they take the current percept as input from the sensors and return an action to the actuators.5 Notice the difference between the agent program, which takes the current percept as input, and the agent function, which may depend on the entire percept history. The agent program has no

choice but to take just the current percept as input because nothing more is available from the environment; if the agent’s actions need to depend on the entire percept sequence, the agent will have to remember the percepts.

5 There are other choices for the agent program skeleton; for example, we could have the agent programs be coroutines that run asynchronously with the environment. Each such coroutine has an input and output port and consists of a loop that reads the input port for percepts and writes actions to the output port.

We describe the agent programs in the simple pseudocode language that is defined in Appendix B . (The online code repository contains implementations in real programming languages.) For example, Figure 2.7  shows a rather trivial agent program that keeps track of the percept sequence and then uses it to index into a table of actions to decide what to do. The table—an example of which is given for the vacuum world in Figure 2.3 — represents explicitly the agent function that the agent program embodies. To build a rational agent in this way, we as designers must construct a table that contains the appropriate action for every possible percept sequence.

Figure 2.7

The TABLE-DRIVEN-AGENT program is invoked for each new percept and returns an action each time. It retains the complete percept sequence in memory.

It is instructive to consider why the table-driven approach to agent construction is doomed to failure. Let P be the set of possible percepts and let T be the lifetime of the agent (the T total number of percepts it will receive). The lookup table will contain t = 1 ∣∣P ∣∣t entries.

∑

Consider the automated taxi: the visual input from a single camera (eight cameras is typical) comes in at the rate of roughly 70 megabytes per second (30 frames per second, 1080 × 720 pixels with 24 bits of color information). This gives a lookup table with over 10600,000,000,000 entries for an hour’s driving. Even the lookup table for chess—a tiny, well-behaved fragment of the real world—has (it turns out) at least 10150 entries. In comparison, the number of atoms in the observable universe is less than 1080 . The daunting size of these tables means that (a) no physical agent in this universe will have the space to store the table; (b) the

designer would not have time to create the table; and (c) no agent could ever learn all the right table entries from its experience.

Despite all this, TABLE-DRIVEN-AGENT does do what we want, assuming the table is filled in correctly: it implements the desired agent function.

The key challenge for AI is to find out how to write programs that, to the extent possible, produce rational behavior from a smallish program rather than from a vast table.

We have many examples showing that this can be done successfully in other areas: for example, the huge tables of square roots used by engineers and schoolchildren prior to the 1970s have now been replaced by a five-line program for Newton’s method running on electronic calculators. The question is, can AI do for general intelligent behavior what Newton did for square roots? We believe the answer is yes.

In the remainder of this section, we outline four basic kinds of agent programs that embody the principles underlying almost all intelligent systems:

Simple reflex agents; Model-based reflex agents; Goal-based agents; and Utility-based agents.

Each kind of agent program combines particular components in particular ways to generate actions. Section 2.4.6  explains in general terms how to convert all these agents into learning agents that can improve the performance of their components so as to generate better actions. Finally, Section 2.4.7  describes the variety of ways in which the components themselves can be represented within the agent. This variety provides a major organizing principle for the field and for the book itself.

2.4.2 Simple reflex agents The simplest kind of agent is the simple reflex agent. These agents select actions on the basis of the current percept, ignoring the rest of the percept history. For example, the vacuum agent whose agent function is tabulated in Figure 2.3  is a simple reflex agent,

because its decision is based only on the current location and on whether that location contains dirt. An agent program for this agent is shown in Figure 2.8 .

Figure 2.8

The agent program for a simple reflex agent in the two-location vacuum environment. This program implements the agent function tabulated in Figure 2.3 .

Simple reflex agent

Notice that the vacuum agent program is very small indeed compared to the corresponding table. The most obvious reduction comes from ignoring the percept history, which cuts down the number of relevant percept sequences from 4T to just 4. A further, small reduction comes from the fact that when the current square is dirty, the action does not depend on the location. Although we have written the agent program using if-then-else statements, it is simple enough that it can also be implemented as a Boolean circuit.

Simple reflex behaviors occur even in more complex environments. Imagine yourself as the driver of the automated taxi. If the car in front brakes and its brake lights come on, then you should notice this and initiate braking. In other words, some processing is done on the visual input to establish the condition we call “The car in front is braking.” Then, this triggers some established connection in the agent program to the action “initiate braking.” We call such a connection a condition–action rule,6 written as

6 Also called situation–action rules, productions, or if–then rules.

if car-in-front-is-braking then initiate-braking.

condition–action rule

Humans also have many such connections, some of which are learned responses (as for driving) and some of which are innate reflexes (such as blinking when something approaches the eye). In the course of the book, we show several different ways in which such connections can be learned and implemented. The program in Figure 2.8  is specific to one particular vacuum environment. A more general and flexible approach is first to build a general-purpose interpreter for condition– action rules and then to create rule sets for specific task environments. Figure 2.9  gives the structure of this general program in schematic form, showing how the condition–action rules allow the agent to make the connection from percept to action. Do not worry if this seems trivial; it gets more interesting shortly.

Figure 2.9

Schematic diagram of a simple reflex agent. We use rectangles to denote the current internal state of the agent’s decision process, and ovals to represent the background information used in the process.

An agent program for Figure 2.9  is shown in Figure 2.10 . The INTERPRET-INPUT function generates an abstracted description of the current state from the percept, and the RULEMATCH function returns the first rule in the set of rules that matches the given state description. Note that the description in terms of “rules” and “matching” is purely conceptual; as noted above, actual implementations can be as simple as a collection of logic gates implementing a Boolean circuit. Alternatively, a “neural” circuit can be used, where the logic gates are replaced by the nonlinear units of artificial neural networks (see Chapter 21 ).

Figure 2.10

A simple reflex agent. It acts according to a rule whose condition matches the current state, as defined by the percept.

Simple reflex agents have the admirable property of being simple, but they are of limited intelligence. The agent in Figure 2.10  will work only if the correct decision can be made on the basis of just the current percept—that is, only if the environment is fully observable.

Even a little bit of unobservability can cause serious trouble. For example, the braking rule given earlier assumes that the condition car-in-front-is-braking can be determined from the current percept—a single frame of video. This works if the car in front has a centrally mounted (and hence uniquely identifiable) brake light. Unfortunately, older models have different configurations of taillights, brake lights, and turn-signal lights, and it is not always possible to tell from a single image whether the car is braking or simply has its taillights on. A simple reflex agent driving behind such a car would either brake continuously and unnecessarily, or, worse, never brake at all.

We can see a similar problem arising in the vacuum world. Suppose that a simple reflex vacuum agent is deprived of its location sensor and has only a dirt sensor. Such an agent has just two possible percepts: [

Dirty

]

and [

Clean . It can Suck in response to Dirty ; what ]

[

]

Clean ? Moving Left fails (forever) if it happens to start in square A, and moving Right fails (forever) if it happens to start in square B. Infinite loops should it do in response to [

]

are often unavoidable for simple reflex agents operating in partially observable environments.

Escape from infinite loops is possible if the agent can randomize its actions. For example, if the vacuum agent perceives [

Clean , it might flip a coin to choose between Right and Left. ]

It is easy to show that the agent will reach the other square in an average of two steps. Then, if that square is dirty, the agent will clean it and the task will be complete. Hence, a randomized simple reflex agent might outperform a deterministic simple reflex agent.

Randomization

We mentioned in Section 2.3  that randomized behavior of the right kind can be rational in some multiagent environments. In single-agent environments, randomization is usually not rational. It is a useful trick that helps a simple reflex agent in some situations, but in most cases we can do much better with more sophisticated deterministic agents.

2.4.3 Model-based reflex agents The most effective way to handle partial observability is for the agent to keep track of the part of the world it can’t see now. That is, the agent should maintain some sort of internal state that depends on the percept history and thereby reflects at least some of the unobserved aspects of the current state. For the braking problem, the internal state is not too extensive—just the previous frame from the camera, allowing the agent to detect when two red lights at the edge of the vehicle go on or off simultaneously. For other driving tasks such as changing lanes, the agent needs to keep track of where the other cars are if it can’t see them all at once. And for any driving to be possible at all, the agent needs to keep track of where its keys are.

Internal state

Updating this internal state information as time goes by requires two kinds of knowledge to be encoded in the agent program in some form. First, we need some information about how the world changes over time, which can be divided roughly into two parts: the effects of the agent’s actions and how the world evolves independently of the agent. For example, when the agent turns the steering wheel clockwise, the car turns to the right, and when it’s raining the car’s cameras can get wet. This knowledge about “how the world works”—whether implemented in simple Boolean circuits or in complete scientific theories—is called a transition model of the world.

Transition model

Second, we need some information about how the state of the world is reflected in the agent’s percepts. For example, when the car in front initiates braking, one or more illuminated red regions appear in the forward-facing camera image, and, when the camera gets wet, droplet-shaped objects appear in the image partially obscuring the road. This kind of knowledge is called a sensor model.

Sensor model

Together, the transition model and sensor model allow an agent to keep track of the state of the world—to the extent possible given the limitations of the agent’s sensors. An agent that uses such models is called a model-based agent.

Model-based agent

Figure 2.11  gives the structure of the model-based reflex agent with internal state, showing how the current percept is combined with the old internal state to generate the updated description of the current state, based on the agent’s model of how the world works. The agent program is shown in Figure 2.12 . The interesting part is the function UPDATE-STATE, which is responsible for creating the new internal state description. The details of how models and states are represented vary widely depending on the type of environment and the particular technology used in the agent design.

Figure 2.11

A model-based reflex agent.

Figure 2.12

A model-based reflex agent. It keeps track of the current state of the world, using an internal model. It then chooses an action in the same way as the reflex agent.

Regardless of the kind of representation used, it is seldom possible for the agent to determine the current state of a partially observable environment exactly. Instead, the box labeled “what the world is like now” (Figure 2.11 ) represents the agent’s “best guess” (or sometimes best guesses, if the agent entertains multiple possibilities). For example, an automated taxi may not be able to see around the large truck that has stopped in front of it and can only guess about what may be causing the hold-up. Thus, uncertainty about the current state may be unavoidable, but the agent still has to make a decision.

2.4.4 Goal-based agents Knowing something about the current state of the environment is not always enough to decide what to do. For example, at a road junction, the taxi can turn left, turn right, or go straight on. The correct decision depends on where the taxi is trying to get to. In other words, as well as a current state description, the agent needs some sort of goal information that describes situations that are desirable—for example, being at a particular destination. The agent program can combine this with the model (the same information as was used in the model-based reflex agent) to choose actions that achieve the goal. Figure 2.13  shows the goal-based agent’s structure.

Figure 2.13

A model-based, goal-based agent. It keeps track of the world state as well as a set of goals it is trying to achieve, and chooses an action that will (eventually) lead to the achievement of its goals.

Goal

Sometimes goal-based action selection is straightforward—for example, when goal satisfaction results immediately from a single action. Sometimes it will be more tricky—for example, when the agent has to consider long sequences of twists and turns in order to find a way to achieve the goal. Search (Chapters 3  to 5 ) and planning (Chapter 11 ) are the subfields of AI devoted to finding action sequences that achieve the agent’s goals.

Notice that decision making of this kind is fundamentally different from the condition– action rules described earlier, in that it involves consideration of the future—both “What will happen if I do such-and-such?” and “Will that make me happy?” In the reflex agent designs, this information is not explicitly represented, because the built-in rules map directly from percepts to actions. The reflex agent brakes when it sees brake lights, period. It has no idea

why. A goal-based agent brakes when it sees brake lights because that’s the only action that it predicts will achieve its goal of not hitting other cars.

Although the goal-based agent appears less efficient, it is more flexible because the knowledge that supports its decisions is represented explicitly and can be modified. For example, a goal-based agent’s behavior can easily be changed to go to a different destination, simply by specifying that destination as the goal. The reflex agent’s rules for when to turn and when to go straight will work only for a single destination; they must all be replaced to go somewhere new.

2.4.5 Utility-based agents Goals alone are not enough to generate high-quality behavior in most environments. For example, many action sequences will get the taxi to its destination (thereby achieving the goal), but some are quicker, safer, more reliable, or cheaper than others. Goals just provide a crude binary distinction between “happy” and “unhappy” states. A more general performance measure should allow a comparison of different world states according to exactly how happy they would make the agent. Because “happy” does not sound very scientific, economists and computer scientists use the term utility instead.7

7 The word “utility” here refers to “the quality of being useful,” not to the electric company or waterworks.

Utility

We have already seen that a performance measure assigns a score to any given sequence of environment states, so it can easily distinguish between more and less desirable ways of getting to the taxi’s destination. An agent’s utility function is essentially an internalization of the performance measure. Provided that the internal utility function and the external performance measure are in agreement, an agent that chooses actions to maximize its utility will be rational according to the external performance measure.

Utility function

Let us emphasize again that this is not the only way to be rational—we have already seen a rational agent program for the vacuum world (Figure 2.8 ) that has no idea what its utility function is—but, like goal-based agents, a utility-based agent has many advantages in terms of flexibility and learning. Furthermore, in two kinds of cases, goals are inadequate but a utility-based agent can still make rational decisions. First, when there are conflicting goals, only some of which can be achieved (for example, speed and safety), the utility function specifies the appropriate tradeoff. Second, when there are several goals that the agent can aim for, none of which can be achieved with certainty, utility provides a way in which the likelihood of success can be weighed against the importance of the goals.

Partial observability and nondeterminism are ubiquitous in the real world, and so, therefore, is decision making under uncertainty. Technically speaking, a rational utility-based agent chooses the action that maximizes the expected utility of the action outcomes—that is, the utility the agent expects to derive, on average, given the probabilities and utilities of each outcome. (Appendix A  defines expectation more precisely.) In Chapter 16 , we show that any rational agent must behave as if it possesses a utility function whose expected value it tries to maximize. An agent that possesses an explicit utility function can make rational decisions with a general-purpose algorithm that does not depend on the specific utility function being maximized. In this way, the “global” definition of rationality—designating as rational those agent functions that have the highest performance—is turned into a “local” constraint on rational-agent designs that can be expressed in a simple program.

Expected utility

The utility-based agent structure appears in Figure 2.14 . Utility-based agent programs appear in Chapters 16  and 17 , where we design decision-making agents that must handle the uncertainty inherent in nondeterministic or partially observable environments. Decision making in multiagent environments is also studied in the framework of utility theory, as explained in Chapter 18 .

Figure 2.14

A model-based, utility-based agent. It uses a model of the world, along with a utility function that measures its preferences among states of the world. Then it chooses the action that leads to the best expected utility, where expected utility is computed by averaging over all possible outcome states, weighted by the probability of the outcome.

At this point, the reader may be wondering, “Is it that simple? We just build agents that maximize expected utility, and we’re done?” It’s true that such agents would be intelligent, but it’s not simple. A utility-based agent has to model and keep track of its environment, tasks that have involved a great deal of research on perception, representation, reasoning, and learning. The results of this research fill many of the chapters of this book. Choosing the utility-maximizing course of action is also a difficult task, requiring ingenious algorithms that fill several more chapters. Even with these algorithms, perfect rationality is usually unachievable in practice because of computational complexity, as we noted in Chapter 1 . We also note that not all utility-based agents are model-based; we will see in Chapters 22  and 26  that a model-free agent can learn what action is best in a particular situation without ever learning exactly how that action changes the environment.

Model-free agent

Finally, all of this assumes that the designer can specify the utility function correctly; Chapters 17 , 18 , and 22  consider the issue of unknown utility functions in more depth.

2.4.6 Learning agents We have described agent programs with various methods for selecting actions. We have not, so far, explained how the agent programs come into being. In his famous early paper, Turing (1950) considers the idea of actually programming his intelligent machines by hand. He estimates how much work this might take and concludes, “Some more expeditious method seems desirable.” The method he proposes is to build learning machines and then to teach them. In many areas of AI, this is now the preferred method for creating state-of-the-art systems. Any type of agent (model-based, goal-based, utility-based, etc.) can be built as a learning agent (or not).

Learning has another advantage, as we noted earlier: it allows the agent to operate in initially unknown environments and to become more competent than its initial knowledge alone might allow. In this section, we briefly introduce the main ideas of learning agents. Throughout the book, we comment on opportunities and methods for learning in particular kinds of agents. Chapters 19 –22  go into much more depth on the learning algorithms themselves.

Learning element

Performance element

A learning agent can be divided into four conceptual components, as shown in Figure 2.15 . The most important distinction is between the learning element, which is responsible for making improvements, and the performance element, which is responsible

for selecting external actions. The performance element is what we have previously considered to be the entire agent: it takes in percepts and decides on actions. The learning element uses feedback from the critic on how the agent is doing and determines how the performance element should be modified to do better in the future.

Figure 2.15

A general learning agent. The “performance element” box represents what we have previously considered to be the whole agent program. Now, the “learning element” box gets to modify that program to improve its performance.

Critic

The design of the learning element depends very much on the design of the performance element. When trying to design an agent that learns a certain capability, the first question is not “How am I going to get it to learn this?” but “What kind of performance element will my

agent use to do this once it has learned how?” Given a design for the performance element, learning mechanisms can be constructed to improve every part of the agent.

The critic tells the learning element how well the agent is doing with respect to a fixed performance standard. The critic is necessary because the percepts themselves provide no indication of the agent’s success. For example, a chess program could receive a percept indicating that it has checkmated its opponent, but it needs a performance standard to know that this is a good thing; the percept itself does not say so. It is important that the performance standard be fixed. Conceptually, one should think of it as being outside the agent altogether because the agent must not modify it to fit its own behavior.

The last component of the learning agent is the problem generator. It is responsible for suggesting actions that will lead to new and informative experiences. If the performance element had its way, it would keep doing the actions that are best, given what it knows, but if the agent is willing to explore a little and do some perhaps suboptimal actions in the short run, it might discover much better actions for the long run. The problem generator’s job is to suggest these exploratory actions. This is what scientists do when they carry out experiments. Galileo did not think that dropping rocks from the top of a tower in Pisa was valuable in itself. He was not trying to break the rocks or to modify the brains of unfortunate pedestrians. His aim was to modify his own brain by identifying a better theory of the motion of objects.

Problem generator

The learning element can make changes to any of the “knowledge” components shown in the agent diagrams (Figures 2.9 , 2.11 , 2.13 , and 2.14 ). The simplest cases involve learning directly from the percept sequence. Observation of pairs of successive states of the environment can allow the agent to learn “What my actions do” and “How the world evolves” in response to its actions. For example, if the automated taxi exerts a certain braking pressure when driving on a wet road, then it will soon find out how much deceleration is actually achieved, and whether it skids off the road. The problem generator might identify certain parts of the model that are in need of improvement and suggest

experiments, such as trying out the brakes on different road surfaces under different conditions.

Improving the model components of a model-based agent so that they conform better with reality is almost always a good idea, regardless of the external performance standard. (In some cases, it is better from a computational point of view to have a simple but slightly inaccurate model rather than a perfect but fiendishly complex model.) Information from the external standard is needed when trying to learn a reflex component or a utility function.

For example, suppose the taxi-driving agent receives no tips from passengers who have been thoroughly shaken up during the trip. The external performance standard must inform the agent that the loss of tips is a negative contribution to its overall performance; then the agent might be able to learn that violent maneuvers do not contribute to its own utility. In a sense, the performance standard distinguishes part of the incoming percept as a reward (or penalty) that provides direct feedback on the quality of the agent’s behavior. Hard-wired performance standards such as pain and hunger in animals can be understood in this way.

Reward

Penalty

More generally, human choices can provide information about human preferences. For example, suppose the taxi does not know that people generally don’t like loud noises, and settles on the idea of blowing its horn continuously as a way of ensuring that pedestrians know it’s coming. The consequent human behavior—covering ears, using bad language, and possibly cutting the wires to the horn—would provide evidence to the agent with which to update its utility function. This issue is discussed further in Chapter 22 .

In summary, agents have a variety of components, and those components can be represented in many ways within the agent program, so there appears to be great variety

among learning methods. There is, however, a single unifying theme. Learning in intelligent agents can be summarized as a process of modification of each component of the agent to bring the components into closer agreement with the available feedback information, thereby improving the overall performance of the agent.

2.4.7 How the components of agent programs work We have described agent programs (in very high-level terms) as consisting of various components, whose function it is to answer questions such as: “What is the world like now?” “What action should I do now?” “What do my actions do?” The next question for a student of AI is, “How on Earth do these components work?” It takes about a thousand pages to begin to answer that question properly, but here we want to draw the reader’s attention to some basic distinctions among the various ways that the components can represent the environment that the agent inhabits.

Roughly speaking, we can place the representations along an axis of increasing complexity and expressive power—atomic, factored, and structured. To illustrate these ideas, it helps to consider a particular agent component, such as the one that deals with “What my actions do.” This component describes the changes that might occur in the environment as the result of taking an action, and Figure 2.16  provides schematic depictions of how those transitions might be represented.

Figure 2.16

Three ways to represent states and the transitions between them. (a) Atomic representation: a state (such as B or C) is a black box with no internal structure; (b) Factored representation: a state consists of a vector of attribute values; values can be Boolean, real-valued, or one of a fixed set of symbols. (c) Structured representation: a state includes objects, each of which may have attributes of its own as well as relationships to other objects.

In an atomic representation each state of the world is indivisible—it has no internal structure. Consider the task of finding a driving route from one end of a country to the other via some sequence of cities (we address this problem in Figure 3.1  on page 64). For the purposes of solving this problem, it may suffice to reduce the state of the world to just the name of the city we are in—a single atom of knowledge, a “black box” whose only discernible property is that of being identical to or different from another black box. The standard algorithms underlying search and game-playing (Chapters 3 –5 ), hidden Markov models (Chapter 14 ), and Markov decision processes (Chapter 17 ) all work with atomic representations.

Atomic representation

A factored representation splits up each state into a fixed set of variables or attributes, each of which can have a value. Consider a higher-fidelity description for the same driving problem, where we need to be concerned with more than just atomic location in one city or another; we might need to pay attention to how much gas is in the tank, our current GPS coordinates, whether or not the oil warning light is working, how much money we have for tolls, what station is on the radio, and so on. While two different atomic states have nothing in common—they are just different black boxes—two different factored states can share some attributes (such as being at some particular GPS location) and not others (such as having lots of gas or having no gas); this makes it much easier to work out how to turn one state into another. Many important areas of AI are based on factored representations, including constraint satisfaction algorithms (Chapter 6 ), propositional logic (Chapter 7 ), planning (Chapter 11 ), Bayesian networks (Chapters 12 –16 ), and various machine learning algorithms.

Factored representation

Variable

Attribute

Value

For many purposes, we need to understand the world as having things in it that are related to each other, not just variables with values. For example, we might notice that a large truck ahead of us is reversing into the driveway of a dairy farm, but a loose cow is blocking the truck’s path. A factored representation is unlikely to be pre-equipped with the attribute TruckAheadBackingIntoDairyFarmDrivewayBlockedByLooseCow with value true or false. Instead, we would need a structured representation, in which objects such as cows and trucks and their various and varying relationships can be described explicitly (see Figure 2.16(c) ). Structured representations underlie relational databases and first-order logic (Chapters 8 , 9 , and 10 ), first-order probability models (Chapter 15 ), and much of natural language understanding (Chapters 23  and 24 ). In fact, much of what humans express in natural language concerns objects and their relationships.

Structured representation

As we mentioned earlier, the axis along which atomic, factored, and structured representations lie is the axis of increasing expressiveness. Roughly speaking, a more expressive representation can capture, at least as concisely, everything a less expressive one can capture, plus some more. Often, the more expressive language is much more concise; for example, the rules of chess can be written in a page or two of a structured-representation language such as first-order logic but require thousands of pages when written in a factoredrepresentation language such as propositional logic and around 10

38

pages when written in

an atomic language such as that of finite-state automata. On the other hand, reasoning and

learning become more complex as the expressive power of the representation increases. To gain the benefits of expressive representations while avoiding their drawbacks, intelligent systems for the real world may need to operate at all points along the axis simultaneously.

Expressiveness

Localist representation

Another axis for representation involves the mapping of concepts to locations in physical memory, whether in a computer or in a brain. If there is a one-to-one mapping between concepts and memory locations, we call that a localist representation. On the other hand, if the representation of a concept is spread over many memory locations, and each memory location is employed as part of the representation of multiple different concepts, we call that a distributed representation. Distributed representations are more robust against noise and information loss. With a localist representation, the mapping from concept to memory location is arbitrary, and if a transmission error garbles a few bits, we might confuse Truck with the unrelated concept Truce. But with a distributed representation, you can think of each concept representing a point in multidimensional space, and if you garble a few bits you move to a nearby point in that space, which will have similar meaning.

Distributed representation

Summary This chapter has been something of a whirlwind tour of AI, which we have conceived of as the science of agent design. The major points to recall are as follows:

An agent is something that perceives and acts in an environment. The agent function for an agent specifies the action taken by the agent in response to any percept sequence. The performance measure evaluates the behavior of the agent in an environment. A rational agent acts so as to maximize the expected value of the performance measure, given the percept sequence it has seen so far. A task environment specification includes the performance measure, the external environment, the actuators, and the sensors. In designing an agent, the first step must always be to specify the task environment as fully as possible. Task environments vary along several significant dimensions. They can be fully or partially observable, single-agent or multiagent, deterministic or nondeterministic, episodic or sequential, static or dynamic, discrete or continuous, and known or unknown. In cases where the performance measure is unknown or hard to specify correctly, there is a significant risk of the agent optimizing the wrong objective. In such cases the agent design should reflect uncertainty about the true objective. The agent program implements the agent function. There exists a variety of basic agent program designs reflecting the kind of information made explicit and used in the decision process. The designs vary in efficiency, compactness, and flexibility. The appropriate design of the agent program depends on the nature of the environment. Simple reflex agents respond directly to percepts, whereas model-based reflex agents maintain internal state to track aspects of the world that are not evident in the current percept. Goal-based agents act to achieve their goals, and utility-based agents try to maximize their own expected “happiness.” All agents can improve their performance through learning.

Bibliographical and Historical Notes The central role of action in intelligence—the notion of practical reasoning—goes back at least as far as Aristotle’s Nicomachean Ethics. Practical reasoning was also the subject of McCarthy’s influential paper “Programs with Common Sense” (1958). The fields of robotics and control theory are, by their very nature, concerned principally with physical agents. The concept of a controller in control theory is identical to that of an agent in AI. Perhaps surprisingly, AI has concentrated for most of its history on isolated components of agents— question-answering systems, theorem-provers, vision systems, and so on—rather than on whole agents. The discussion of agents in the text by Genesereth and Nilsson (1987) was an influential exception. The whole-agent view is now widely accepted and is a central theme in recent texts (Padgham and Winikoff, 2004; Jones, 2007; Poole and Mackworth, 2017).

Controller

Chapter 1  traced the roots of the concept of rationality in philosophy and economics. In AI, the concept was of peripheral interest until the mid-1980s, when it began to suffuse many discussions about the proper technical foundations of the field. A paper by Jon Doyle (1983) predicted that rational agent design would come to be seen as the core mission of AI, while other popular topics would spin off to form new disciplines.

Careful attention to the properties of the environment and their consequences for rational agent design is most apparent in the control theory tradition—for example, classical control systems (Dorf and Bishop, 2004; Kirk, 2004) handle fully observable, deterministic environments; stochastic optimal control (Kumar and Varaiya, 1986; Bertsekas and Shreve, 2007) handles partially observable, stochastic environments; and hybrid control (Henzinger and Sastry, 1998; Cassandras and Lygeros, 2006) deals with environments containing both discrete and continuous elements. The distinction between fully and partially observable environments is also central in the dynamic programming literature developed in the field of operations research (Puterman, 1994), which we discuss in Chapter 17 .

Although simple reflex agents were central to behaviorist psychology (see Chapter 1 ), most AI researchers view them as too simple to provide much leverage. (Rosenschein (1985) and Brooks (1986) questioned this assumption; see Chapter 26 .) A great deal of work has gone into finding efficient algorithms for keeping track of complex environments (BarShalom et al., 2001; Choset et al., 2005; Simon, 2006), most of it in the probabilistic setting.

Goal-based agents are presupposed in everything from Aristotle’s view of practical reasoning to McCarthy’s early papers on logical AI. Shakey the Robot (Fikes and Nilsson, 1971; Nilsson, 1984) was the first robotic embodiment of a logical, goal-based agent. A full logical analysis of goal-based agents appeared in Genesereth and Nilsson (1987), and a goal-based programming methodology called agent-oriented programming was developed by Shoham (1993). The agent-based approach is now extremely popular in software engineering (Ciancarini and Wooldridge, 2001). It has also infiltrated the area of operating systems, where autonomic computing refers to computer systems and networks that monitor and control themselves with a perceive–act loop and machine learning methods (Kephart and Chess, 2003). Noting that a collection of agent programs designed to work well together in a true multiagent environment necessarily exhibits modularity—the programs share no internal state and communicate with each other only through the environment—it is common within the field of multiagent systems to design the agent program of a single agent as a collection of autonomous sub-agents. In some cases, one can even prove that the resulting system gives the same optimal solutions as a monolithic design.

Autonomic computing

The goal-based view of agents also dominates the cognitive psychology tradition in the area of problem solving, beginning with the enormously influential Human Problem Solving (Newell and Simon, 1972) and running through all of Newell’s later work (Newell, 1990). Goals, further analyzed as desires (general) and intentions (currently pursued), are central to the influential theory of agents developed by Michael Bratman (1987).

As noted in Chapter 1 , the development of utility theory as a basis for rational behavior goes back hundreds of years. In AI, early research eschewed utilities in favor of goals, with some exceptions (Feldman and Sproull, 1977). The resurgence of interest in probabilistic methods in the 1980s led to the acceptance of maximization of expected utility as the most general framework for decision making (Horvitz et al., 1988). The text by Pearl (1988) was the first in AI to cover probability and utility theory in depth; its exposition of practical methods for reasoning and decision making under uncertainty was probably the single biggest factor in the rapid shift towards utility-based agents in the 1990s (see Chapter 16 ). The formalization of reinforcement learning within a decision-theoretic framework also contributed to this shift (Sutton, 1988). Somewhat remarkably, almost all AI research until very recently has assumed that the performance measure can be exactly and correctly specified in the form of a utility function or reward function (Hadfield-Menell et al., 2017a; Russell, 2019). The general design for learning agents portrayed in Figure 2.15  is classic in the machine learning literature (Buchanan et al., 1978; Mitchell, 1997). Examples of the design, as embodied in programs, go back at least as far as Arthur Samuel’s (1959, 1967) learning program for playing checkers. Learning agents are discussed in depth in Chapters 19 –22 .

Some early papers on agent-based approaches are collected by Huhns and Singh (1998) and Wooldridge and Rao (1999). Texts on multiagent systems provide a good introduction to many aspects of agent design (Weiss, 2000a; Wooldridge, 2009). Several conference series devoted to agents began in the 1990s, including the International Workshop on Agent Theories, Architectures, and Languages (ATAL), the International Conference on Autonomous Agents (AGENTS), and the International Conference on Multi-Agent Systems (ICMAS). In 2002, these three merged to form the International Joint Conference on Autonomous Agents and Multi-Agent Systems (AAMAS). From 2000 to 2012 there were annual workshops on Agent-Oriented Software Engineering (AOSE). The journal Autonomous Agents and Multi-Agent Systems was founded in 1998. Finally, Dung Beetle Ecology (Hanski and Cambefort, 1991) provides a wealth of interesting information on the behavior of dung beetles. YouTube has inspiring video recordings of their activities.

II Problem-solving

Chapter 3 Solving Problems by Searching In which we see how an agent can look ahead to find a sequence of actions that will eventually achieve its goal.

When the correct action to take is not immediately obvious, an agent may need to to plan ahead: to consider a sequence of actions that form a path to a goal state. Such an agent is called a problem-solving agent, and the computational process it undertakes is called search.

Problem-solving agent

Search

Problem-solving agents use atomic representations, as described in Section 2.4.7 —that is, states of the world are considered as wholes, with no internal structure visible to the problem-solving algorithms. Agents that use factored or structured representations of states are called planning agents and are discussed in Chapters 7  and 11 .

We will cover several search algorithms. In this chapter, we consider only the simplest environments: episodic, single agent, fully observable, deterministic, static, discrete, and known. We distinguish between informed algorithms, in which the agent can estimate how far it is from the goal, and uninformed algorithms, where no such estimate is available. Chapter 4  relaxes the constraints on environments, and Chapter 5  considers multiple agents.

This chapter uses the concepts of asymptotic complexity (that is, unfamiliar with these concepts should consult Appendix A .

O(n) notation). Readers

3.1 Problem-Solving Agents Imagine an agent enjoying a touring vacation in Romania. The agent wants to take in the sights, improve its Romanian, enjoy the nightlife, avoid hangovers, and so on. The decision problem is a complex one. Now, suppose the agent is currently in the city of Arad and has a nonrefundable ticket to fly out of Bucharest the following day. The agent observes street signs and sees that there are three roads leading out of Arad: one toward Sibiu, one to Timisoara, and one to Zerind. None of these are the goal, so unless the agent is familiar with the geography of Romania, it will not know which road to follow.1

1 We are assuming that most readers are in the same position and can easily imagine themselves to be as clueless as our agent. We apologize to Romanian readers who are unable to take advantage of this pedagogical device.

If the agent has no additional information—that is, if the environment is unknown—then the agent can do no better than to execute one of the actions at random. This sad situation is discussed in Chapter 4 . In this chapter, we will assume our agents always have access to information about the world, such as the map in Figure 3.1 . With that information, the agent can follow this four-phase problem-solving process:

GOAL FORMULATION: The agent adopts the goal of reaching Bucharest. Goals organize behavior by limiting the objectives and hence the actions to be considered.

Goal formulation

PROBLEM FORMULATION: The agent devises a description of the states and actions necessary to reach the goal—an abstract model of the relevant part of the world. For our agent, one good model is to consider the actions of traveling from one city to an adjacent city, and therefore the only fact about the state of the world that will change due to an action is the current city.

Problem formulation

SEARCH: Before taking any action in the real world, the agent simulates sequences of actions in its model, searching until it finds a sequence of actions that reaches the goal. Such a sequence is called a solution. The agent might have to simulate multiple sequences that do not reach the goal, but eventually it will find a solution (such as going from Arad to Sibiu to Fagaras to Bucharest), or it will find that no solution is possible.

Search

Solution

EXECUTION: The agent can now execute the actions in the solution, one at a time.

Execution

Figure 3.1

A simplified road map of part of Romania, with road distances in miles.

It is an important property that in a fully observable, deterministic, known environment, the solution to any problem is a fixed sequence of actions: drive to Sibiu, then Fagaras, then Bucharest. If the model is correct, then once the agent has found a solution, it can ignore its percepts while it is executing the actions—closing its eyes, so to speak—because the solution is guaranteed to lead to the goal. Control theorists call this an open-loop system: ignoring the percepts breaks the loop between agent and environment. If there is a chance that the model is incorrect, or the environment is nondeterministic, then the agent would be safer using a closed-loop approach that monitors the percepts (see Section 4.4 ).

Open-loop

Closed-loop

In partially observable or nondeterministic environments, a solution would be a branching strategy that recommends different future actions depending on what percepts arrive. For example, the agent might plan to drive from Arad to Sibiu but might need a contingency plan in case it arrives in Zerind by accident or finds a sign saying “Drum Închis” (Road Closed).

3.1.1 Search problems and solutions A search problem can be defined formally as follows:

Problem

A set of possible states that the environment can be in. We call this the state space.

States

State space

The initial state that the agent starts in. For example: Arad.

Initial state

A set of one or more goal states. Sometimes there is one goal state (e.g., Bucharest), sometimes there is a small set of alternative goal states, and sometimes the goal is defined by a property that applies to many states (potentially an infinite number). For

example, in a vacuum-cleaner world, the goal might be to have no dirt in any location, regardless of any other facts about the state. We can account for all three of these possibilities by specifying an IS-GOAL method for a problem. In this chapter we will sometimes say “the goal” for simplicity, but what we say also applies to “any one of the possible goal states.”

Goal states

s

s

The actions available to the agent. Given a state , ACTIONS( ) returns a finite2 set of

s

s

actions that can be executed in . We say that each of these actions is applicable in . An example: 2 For problems with an infinite number of actions we would need techniques that go beyond this chapter.

ACTIONS(

Arad) = {ToSibiu, ToTimisoara, ToZerind}.

Action

Applicable

sa

A transition model, which describes what each action does. RESULT( , ) returns the state that results from doing action

a in state s. For example,

RESULT(

Transition model

Arad, ToZerind) = Zerind.

An action cost function, denoted by ACTION-COST(s, a, s′ ) when we are programming or

c(s, a, s′ ) when we are doing math, that gives the numeric cost of applying action a in state s to reach state s′ . A problem-solving agent should use a cost function that reflects its own performance measure; for example, for route-finding agents, the cost of an action might be the length in miles (as seen in Figure 3.1 ), or it might be the time it takes to complete the action.

Action cost function

A sequence of actions forms a path, and a solution is a path from the initial state to a goal state. We assume that action costs are additive; that is, the total cost of a path is the sum of the individual action costs. An optimal solution has the lowest path cost among all solutions. In this chapter, we assume that all action costs will be positive, to avoid certain complications.3

3 In any problem with a cycle of net negative cost, the cost-optimal solution is to go around that cycle an infinite number of times. The Bellman–Ford and Floyd–Warshall algorithms (not covered here) handle negative-cost actions, as long as there are no negative cycles. It is easy to accommodate zero-cost actions, as long as the number of consecutive zero-cost actions is bounded. For example, we might have a robot where there is a cost to move, but zero cost to rotate 90°; the algorithms in this chapter can handle this as long as no more than three consecutive 90° turns are allowed. There is also a complication with problems that have an infinite number of arbitrarily small action costs. Consider a version of Zeno’s paradox where there is an action to move half way to the goal, at a cost of half of the previous move. This problem has no solution with a finite number of actions, but to prevent a search from taking an unbounded number of actions without quite reaching the goal, we can require that all action costs be at least ϵ, for some small positive value ϵ.

Path

Optimal solution

The state space can be represented as a graph in which the vertices are states and the directed edges between them are actions. The map of Romania shown in Figure 3.1  is such a graph, where each road indicates two actions, one in each direction.

Graph

3.1.2 Formulating problems Our formulation of the problem of getting to Bucharest is a model—an abstract mathematical description—and not the real thing. Compare the simple atomic state description Arad to an actual cross-country trip, where the state of the world includes so many things: the traveling companions, the current radio program, the scenery out of the window, the proximity of law enforcement officers, the distance to the next rest stop, the condition of the road, the weather, the traffic, and so on. All these considerations are left out of our model because they are irrelevant to the problem of finding a route to Bucharest.

The process of removing detail from a representation is called abstraction. A good problem formulation has the right level of detail. If the actions were at the level of “move the right foot forward a centimeter” or “turn the steering wheel one degree left,” the agent would probably never find its way out of the parking lot, let alone to Bucharest.

Abstraction

Can we be more precise about the appropriate level of abstraction? Think of the abstract states and actions we have chosen as corresponding to large sets of detailed world states and detailed action sequences. Now consider a solution to the abstract problem: for example, the path from Arad to Sibiu to Rimnicu Vilcea to Pitesti to Bucharest. This abstract solution corresponds to a large number of more detailed paths. For example, we could drive with the radio on between Sibiu and Rimnicu Vilcea, and then switch it off for the rest of the trip.

Level of abstraction

The abstraction is valid if we can elaborate any abstract solution into a solution in the more detailed world; a sufficient condition is that for every detailed state that is “in Arad,” there is a detailed path to some state that is “in Sibiu,” and so on.4 The abstraction is useful if carrying out each of the actions in the solution is easier than the original problem; in our case, the action “drive from Arad to Sibiu” can be carried out without further search or planning by a driver with average skill. The choice of a good abstraction thus involves removing as much detail as possible while retaining validity and ensuring that the abstract actions are easy to carry out. Were it not for the ability to construct useful abstractions, intelligent agents would be completely swamped by the real world. 4 See Section 11.4 .

3.2 Example Problems The problem-solving approach has been applied to a vast array of task environments. We list some of the best known here, distinguishing between standardized and real-world problems. A standardized problem is intended to illustrate or exercise various problemsolving methods. It can be given a concise, exact description and hence is suitable as a benchmark for researchers to compare the performance of algorithms. A real-world problem, such as robot navigation, is one whose solutions people actually use, and whose formulation is idiosyncratic, not standardized, because, for example, each robot has different sensors that produce different data.

Standardized problem

Real-world problem

3.2.1 Standardized problems

Grid world

A grid world problem is a two-dimensional rectangular array of square cells in which agents can move from cell to cell. Typically the agent can move to any obstacle-free adjacent cell— horizontally or vertically and in some problems diagonally. Cells can contain objects, which the agent can pick up, push, or otherwise act upon; a wall or other impassible obstacle in a cell prevents an agent from moving into that cell. The vacuum world from Section 2.1  can be formulated as a grid world problem as follows:

STATES: A state of the world says which objects are in which cells. For the vacuum world, the objects are the agent and any dirt. In the simple two-cell version, the agent can be in either of the two cells, and each call can either contain dirt or not, so there are 2⋅2⋅2 = 8

states (see Figure 3.2 ). In general, a vacuum environment with n cells has

n ⋅ 2n states. Figure 3.2

The state-space graph for the two-cell vacuum world. There are 8 states and three actions for each state: L = Left, R = Right, S = Suck.

INITIAL STATE: Any state can be designated as the initial state. ACTIONS: In the two-cell world we defined three actions: Suck, move Left, and move Right. In a two-dimensional multi-cell world we need more movement actions. We could add Upward and Downward, giving us four absolute movement actions, or we could switch to egocentric actions, defined relative to the viewpoint of the agent—for example, Forward, Backward, TurnRight, and TurnLeft. TRANSITION MODEL: Suck removes any dirt from the agent’s cell; Forward moves the agent ahead one cell in the direction it is facing, unless it hits a wall, in which case the action has no effect. Backward moves the agent in the opposite direction, while TurnRight and TurnLeft change the direction it is facing by 90∘ . GOAL STATES: The states in which every cell is clean. ACTION COST: Each action costs 1.

Sokoban puzzle

Another type of grid world is the sokoban puzzle, in which the agent’s goal is to push a number of boxes, scattered about the grid, to designated storage locations. There can be at most one box per cell. When an agent moves forward into a cell containing a box and there is an empty cell on the other side of the box, then both the box and the agent move forward. The agent can’t push a box into another box or a wall. For a world with n non-obstacle cells and b boxes, there are n × n!/(b!(n − b)!) states; for example on an 8 × 8 grid with a dozen boxes, there are over 200 trillion states.

In a sliding-tile puzzle, a number of tiles (sometimes called blocks or pieces) are arranged in a grid with one or more blank spaces so that some of the tiles can slide into the blank space. One variant is the Rush Hour puzzle, in which cars and trucks slide around a 6 × 6 grid in an attempt to free a car from the traffic jam. Perhaps the best-known variant is the 8puzzle (see Figure 3.3 ), which consists of a 3 × 3 grid with eight numbered tiles and one blank space, and the 15-puzzle on a 4 × 4 grid. The object is to reach a specified goal state, such as the one shown on the right of the figure. The standard formulation of the 8 puzzle is as follows:

STATES: A state description specifies the location of each of the tiles. INITIAL STATE: Any state can be designated as the initial state. Note that a parity property partitions the state space—any given goal can be reached from exactly half of the possible initial states (see Exercise 3.PART). ACTIONS: While in the physical world it is a tile that slides, the simplest way of describing an action is to think of the blank space moving Left, Right, Up, or Down. If the blank is at an edge or corner then not all actions will be applicable. TRANSITION MODEL: Maps a state and action to a resulting state; for example, if we apply Left to the start state in Figure 3.3 , the resulting state has the 5 and the blank switched. Figure 3.3

A typical instance of the 8-puzzle.

GOAL STATE: Although any state could be the goal, we typically specify a state with the numbers in order, as in Figure 3.3 . ACTION COST: Each action costs 1.

Sliding-tile puzzle

8-puzzle

15-puzzle

Note that every problem formulation involves abstractions. The 8-puzzle actions are abstracted to their beginning and final states, ignoring the intermediate locations where the tile is sliding. We have abstracted away actions such as shaking the board when tiles get stuck and ruled out extracting the tiles with a knife and putting them back again. We are left with a description of the rules, avoiding all the details of physical manipulations.

Our final standardized problem was devised by Donald Knuth (1964) and illustrates how infinite state spaces can arise. Knuth conjectured that starting with the number 4, a

sequence of square root, floor, and factorial operations can reach any desired positive integer. For example, we can reach 5 from 4 as follows:

⌊

    ⎷⎷

√

√√ ⌋ (4!)!

= 5.

The problem definition is simple:

STATES: Positive real numbers. INITIAL STATE: 4. ACTIONS: Apply square root, floor, or factorial operation (factorial for integers only). TRANSITION MODEL: As given by the mathematical definitions of the operations. GOAL STATE: The desired positive integer. ACTION COST: Each action costs 1.

The state space for this problem is infinite: for any integer greater than 2 the factorial operator will always yield a larger integer. The problem is interesting because it explores very large numbers: the shortest path to 5 goes through (4! )! = 620,448,401,733,239,439,360,000.

Infinite state spaces arise frequently in tasks

involving the generation of mathematical expressions, circuits, proofs, programs, and other recursively defined objects.

3.2.2 Real-world problems We have already seen how the route-finding problem is defined in terms of specified locations and transitions along edges between them. Route-finding algorithms are used in a variety of applications. Some, such as Web sites and in-car systems that provide driving directions, are relatively straightforward extensions of the Romania example. (The main complications are varying costs due to traffic-dependent delays, and rerouting due to road closures.) Others, such as routing video streams in computer networks, military operations planning, and airline travel-planning systems, involve much more complex specifications. Consider the airline travel problems that must be solved by a travel-planning Web site:

STATES: Each state obviously includes a location (e.g., an airport) and the current time. Furthermore, because the cost of an action (a flight segment) may depend on previous

segments, their fare bases, and their status as domestic or international, the state must record extra information about these “historical” aspects. INITIAL STATE: The user’s home airport. ACTIONS: Take any flight from the current location, in any seat class, leaving after the current time, leaving enough time for within-airport transfer if needed. TRANSITION MODEL: The state resulting from taking a flight will have the flight’s destination as the new location and the flight’s arrival time as the new time. GOAL STATE: A destination city. Sometimes the goal can be more complex, such as “arrive at the destination on a nonstop flight.” ACTION COST: A combination of monetary cost, waiting time, flight time, customs and immigration procedures, seat quality, time of day, type of airplane, frequent-flyer reward points, and so on.

Commercial travel advice systems use a problem formulation of this kind, with many additional complications to handle the airlines’ byzantine fare structures. Any seasoned traveler knows, however, that not all air travel goes according to plan. A really good system should include contingency plans—what happens if this flight is delayed and the connection is missed?

Touring problems describe a set of locations that must be visited, rather than a single goal destination. The traveling salesperson problem (TSP) is a touring problem in which every city on a map must be visited. The aim is to find a tour with cost

< C (or in the optimization

version, to find a tour with the lowest cost possible). An enormous amount of effort has been expended to improve the capabilities of TSP algorithms. The algorithms can also be extended to handle fleets of vehicles. For example, a search and optimization algorithm for routing school buses in Boston saved $5 million, cut traffic and air pollution, and saved time for drivers and students (Bertsimas et al., 2019). In addition to planning trips, search algorithms have been used for tasks such as planning the movements of automatic circuitboard drills and of stocking machines on shop floors.

Touring problem

Traveling salesperson problem (TSP)

A VLSI layout problem requires positioning millions of components and connections on a chip to minimize area, minimize circuit delays, minimize stray capacitances, and maximize manufacturing yield. The layout problem comes after the logical design phase and is usually split into two parts: cell layout and channel routing. In cell layout, the primitive components of the circuit are grouped into cells, each of which performs some recognized function. Each cell has a fixed footprint (size and shape) and requires a certain number of connections to each of the other cells. The aim is to place the cells on the chip so that they do not overlap and so that there is room for the connecting wires to be placed between the cells. Channel routing finds a specific route for each wire through the gaps between the cells. These search problems are extremely complex, but definitely worth solving.

VLSI layout

Robot navigation is a generalization of the route-finding problem described earlier. Rather than following distinct paths (such as the roads in Romania), a robot can roam around, in effect making its own paths. For a circular robot moving on a flat surface, the space is essentially two-dimensional. When the robot has arms and legs that must also be controlled, the search space becomes many-dimensional—one dimension for each joint angle. Advanced techniques are required just to make the essentially continuous search space finite (see Chapter 26 ). In addition to the complexity of the problem, real robots must also deal with errors in their sensor readings and motor controls, with partial observability, and with other agents that might alter the environment.

Robot navigation

Automatic assembly sequencing of complex objects (such as electric motors) by a robot has been standard industry practice since the 1970s. Algorithms first find a feasible assembly sequence and then work to optimize the process. Minimizing the amount of manual human labor on the assembly line can produce significant savings in time and cost. In assembly problems, the aim is to find an order in which to assemble the parts of some object. If the wrong order is chosen, there will be no way to add some part later in the sequence without undoing some of the work already done. Checking an action in the sequence for feasibility is a difficult geometrical search problem closely related to robot navigation. Thus, the generation of legal actions is the expensive part of assembly sequencing. Any practical algorithm must avoid exploring all but a tiny fraction of the state space. One important assembly problem is protein design, in which the goal is to find a sequence of amino acids that will fold into a three-dimensional protein with the right properties to cure some disease.

Automatic assembly sequencing

Protein design

3.3 Search Algorithms A search algorithm takes a search problem as input and returns a solution, or an indication of failure. In this chapter we consider algorithms that superimpose a search tree over the state-space graph, forming various paths from the initial state, trying to find a path that reaches a goal state. Each node in the search tree corresponds to a state in the state space and the edges in the search tree correspond to actions. The root of the tree corresponds to the initial state of the problem.

Search algorithm

Node

It is important to understand the distinction between the state space and the search tree. The state space describes the (possibly infinite) set of states in the world, and the actions that allow transitions from one state to another. The search tree describes paths between these states, reaching towards the goal. The search tree may have multiple paths to (and thus multiple nodes for) any given state, but each node in the tree has a unique path back to the root (as in all trees).

Expand

Figure 3.4  shows the first few steps in finding a path from Arad to Bucharest. The root node of the search tree is at the initial state, Arad. We can expand the node, by considering the available ACTIONS for that state, using the RESULT function to see where those actions lead

to, and generating a new node (called a child node or successor node) for each of the resulting states. Each child node has Arad as its parent node.

Figure 3.4

Three partial search trees for finding a route from Arad to Bucharest. Nodes that have been expanded are lavender with bold letters; nodes on the frontier that have been generated but not yet expanded are in green; the set of states corresponding to these two types of nodes are said to have been reached. Nodes that could be generated next are shown in faint dashed lines. Notice in the bottom tree there is a cycle from Arad to Sibiu to Arad; that can’t be an optimal path, so search should not continue from there.

Generating

Child node

Successor node

Parent node

Now we must choose which of these three child nodes to consider next. This is the essence of search—following up one option now and putting the others aside for later. Suppose we choose to expand Sibiu first. Figure 3.4  (bottom) shows the result: a set of 6 unexpanded nodes (outlined in bold). We call this the frontier of the search tree. We say that any state that has had a node generated for it has been reached (whether or not that node has been expanded).5 Figure 3.5  shows the search tree superimposed on the state-space graph.

5 Some authors call the frontier the open list, which is both geographically less evocative and computationally less appropriate, because a queue is more efficient than a list here. Those authors use the term closed list to refer to the set of previously expanded nodes, which in our terminology would be the reached nodes minus the frontier.

Figure 3.5

A sequence of search trees generated by a graph search on the Romania problem of Figure 3.1 . At each stage, we have expanded every node on the frontier, extending every path with all applicable actions that don’t result in a state that has already been reached. Notice that at the third stage, the topmost city (Oradea) has two successors, both of which have already been reached by other paths, so no paths are extended from Oradea.

Frontier

Reached

Note that the frontier separates two regions of the state-space graph: an interior region where every state has been expanded, and an exterior region of states that have not yet been reached. This property is illustrated in Figure 3.6 .

Figure 3.6

The separation property of graph search, illustrated on a rectangular-grid problem. The frontier (green) separates the interior (lavender) from the exterior (faint dashed). The frontier is the set of nodes (and corresponding states) that have been reached but not yet expanded; the interior is the set of nodes (and corresponding states) that have been expanded; and the exterior is the set of states that have not been reached. In (a), just the root has been expanded. In (b), the top frontier node is expanded. In (c), the remaining successors of the root are expanded in clockwise order.

Separator

3.3.1 Best-first search How do we decide which node from the frontier to expand next? A very general approach is called best-first search, in which we choose a node, n, with minimum value of some

evaluation function, f (n). Figure 3.7  shows the algorithm. On each iteration we choose a node on the frontier with minimum f (n) value, return it if its state is a goal state, and otherwise apply EXPAND to generate child nodes. Each child node is added to the frontier if it has not been reached before, or is re-added if it is now being reached with a path that has a lower path cost than any previous path. The algorithm returns either an indication of failure, or a node that represents a path to a goal. By employing different f (n) functions, we get different specific algorithms, which this chapter will cover.

Figure 3.7

The best-first search algorithm, and the function for expanding a node. The data structures used here are described in Section 3.3.2 . See Appendix B  for yield.

Best-first search

Evaluation function

3.3.2 Search data structures Search algorithms require a data structure to keep track of the search tree. A node in the tree is represented by a data structure with four components:

node.STATE: the state to which the node corresponds; node.PARENT: the node in the tree that generated this node; node.ACTION: the action that was applied to the parent’s state to generate this node; node.PATH-COST: the total cost of the path from the initial state to this node. In mathematical formulas, we use g(node) as a synonym for PATH-COST.

Following the PARENT pointers back from a node allows us to recover the states and actions along the path to that node. Doing this from a goal node gives us the solution.

We need a data structure to store the frontier. The appropriate choice is a queue of some kind, because the operations on a frontier are:

IS-EMPTY(frontier) returns true only if there are no nodes in the frontier. POP(frontier) removes the top node from the frontier and returns it. TOP(frontier) returns (but does not remove) the top node of the frontier. ADD(node, frontier) inserts node into its proper place in the queue.

Queue

Three kinds of queues are used in search algorithms:

A priority queue first pops the node with the minimum cost according to some evaluation function, f . It is used in best-first search.

Priority queue

A FIFO queue or first-in-first-out queue first pops the node that was added to the queue first; we shall see it is used in breadth-first search.

FIFO queue

A LIFO queue or last-in-first-out queue (also known as a stack) pops first the most recently added node; we shall see it is used in depth-first search.

LIFO queue

Stack

The reached states can be stored as a lookup table (e.g. a hash table) where each key is a state and each value is the node for that state.

3.3.3 Redundant paths The search tree shown in Figure 3.4  (bottom) includes a path from Arad to Sibiu and back to Arad again. We say that Arad is a repeated state in the search tree, generated in this case by a cycle (also known as a loopy path). So even though the state space has only 20 states, the complete search tree is infinite because there is no limit to how often one can traverse a loop.

Repeated state

Cycle

Loopy path

A cycle is a special case of a redundant path. For example, we can get to Sibiu via the path Arad–Sibiu (140 miles long) or the path Arad–Zerind–Oradea–Sibiu (297 miles long). This second path is redundant—it’s just a worse way to get to the same state—and need not be considered in our quest for optimal paths.

Redundant path

Consider an agent in a 10 × 10 grid world, with the ability to move to any of 8 adjacent squares. If there are no obstacles, the agent can reach any of the 100 squares in 9 moves or 9

fewer. But the number of paths of length 9 is almost 8 (a bit less because of the edges of the grid), or more than 100 million. In other words, the average cell can be reached by over a million redundant paths of length 9, and if we eliminate redundant paths, we can complete a search roughly a million times faster. As the saying goes, algorithms that cannot remember the past are doomed to repeat it. There are three approaches to this issue.

First, we can remember all previously reached states (as best-first search does), allowing us to detect all redundant paths, and keep only the best path to each state. This is appropriate for state spaces where there are many redundant paths, and is the preferred choice when the table of reached states will fit in memory.

Graph search

Tree-like search

Second, we can not worry about repeating the past. There are some problem formulations where it is rare or impossible for two paths to reach the same state. An example would be an assembly problem where each action adds a part to an evolving assemblage, and there is an ordering of parts so that it is possible to add

A and then B

,

but not

B and then A

.

For

those problems, we could save memory space if we don’t track reached states and we don’t check for redundant paths. We call a search algorithm a graph search if it checks for redundant paths and a tree-like search6 if it does not check. The BEST-FIRST-SEARCH algorithm in Figure 3.7  is a graph search algorithm; if we remove all references to reached we get a tree-like search that uses less memory but will examine redundant paths to the same state, and thus will run slower.

6 We say “tree-like search” because the state space is still the same graph no matter how we search it; we are just choosing to treat it as if it were a tree, with only one path from each node back to the root.

Third, we can compromise and check for cycles, but not for redundant paths in general. Since each node has a chain of parent pointers, we can check for cycles with no need for additional memory by following up the chain of parents to see if the state at the end of the path has appeared earlier in the path. Some implementations follow this chain all the way up, and thus eliminate all cycles; other implementations follow only a few links (e.g., to the parent, grandparent, and great-grandparent), and thus take only a constant amount of time, while eliminating all short cycles (and relying on other mechanisms to deal with long cycles).

3.3.4 Measuring problem-solving performance Before we get into the design of various search algorithms, we will consider the criteria used to choose among them. We can evaluate an algorithm’s performance in four ways:

COMPLETENESS: Is the algorithm guaranteed to find a solution when there is one, and to correctly report failure when there is not?

Completeness

COST OPTIMALITY: Does it find a solution with the lowest path cost of all solutions?7 7 Some authors use the term “admissibility” for the property of finding the lowest-cost solution, and some use just “optimality,” but that can be confused with other types of optimality.

Cost optimality

TIME COMPLEXITY: How long does it take to find a solution? This can be measured in seconds, or more abstractly by the number of states and actions considered.

Time complexity

SPACE COMPLEXITY: How much memory is needed to perform the search?

Space complexity

To understand completeness, consider a search problem with a single goal. That goal could be anywhere in the state space; therefore a complete algorithm must be capable of systematically exploring every state that is reachable from the initial state. In finite state

spaces that is straightforward to achieve: as long as we keep track of paths and cut off ones that are cycles (e.g. Arad to Sibiu to Arad), eventually we will reach every reachable state.

In infinite state spaces, more care is necessary. For example, an algorithm that repeatedly applied the “factorial” operator in Knuth’s “4” problem would follow an infinite path from 4 to 4! to (4!)!, and so on. Similarly, on an infinite grid with no obstacles, repeatedly moving forward in a straight line also follows an infinite path of new states. In both cases the algorithm never returns to a state it has reached before, but is incomplete because wide expanses of the state space are never reached.

To be complete, a search algorithm must be systematic in the way it explores an infinite state space, making sure it can eventually reach any state that is connected to the initial state. For example, on the infinite grid, one kind of systematic search is a spiral path that covers all the cells that are

s steps from the origin before moving out to cells that are s + 1

steps away. Unfortunately, in an infinite state space with no solution, a sound algorithm needs to keep searching forever; it can’t terminate because it can’t know if the next state will be a goal.

Systematic

Time and space complexity are considered with respect to some measure of the problem difficulty. In theoretical computer science, the typical measure is the size of the state-space graph,

|V | + |E|, where |V | is the number of vertices (state nodes) of the graph and |E| is the

number of edges (distinct state/action pairs). This is appropriate when the graph is an explicit data structure, such as the map of Romania. But in many AI problems, the graph is represented only implicitly by the initial state, actions, and transition model. For an implicit

d, the depth or number of actions in an optimal solution; m, the maximum number of actions in any path; and b, the branching state space, complexity can be measured in terms of

factor or number of successors of a node that need to be considered.

Depth

Branching factor

3.4 Uninformed Search Strategies An uninformed search algorithm is given no clue about how close a state is to the goal(s). For example, consider our agent in Arad with the goal of reaching Bucharest. An uninformed agent with no knowledge of Romanian geography has no clue whether going to Zerind or Sibiu is a better first step. In contrast, an informed agent (Section 3.5 ) who knows the location of each city knows that Sibiu is much closer to Bucharest and thus more likely to be on the shortest path.

3.4.1 Breadth-first search When all actions have the same cost, an appropriate strategy is breadth-first search, in which the root node is expanded first, then all the successors of the root node are expanded next, then their successors, and so on. This is a systematic search strategy that is therefore complete even on infinite state spaces. We could implement breadth-first search as a call to BEST-FIRST-SEARCH where the evaluation function f (n) is the depth of the node—that is, the number of actions it takes to reach the node.

Breadth-first search

However, we can get additional efficiency with a couple of tricks. A first-in-first-out queue will be faster than a priority queue, and will give us the correct order of nodes: new nodes (which are always deeper than their parents) go to the back of the queue, and old nodes, which are shallower than the new nodes, get expanded first. In addition, reached can be a set of states rather than a mapping from states to nodes, because once we’ve reached a state, we can never find a better path to the state. That also means we can do an early goal test, checking whether a node is a solution as soon as it is generated, rather than the late goal test that best-first search uses, waiting until a node is popped off the queue. Figure 3.8  shows the progress of a breadth-first search on a binary tree, and Figure 3.9  shows the algorithm with the early-goal efficiency enhancements.

Figure 3.8

Breadth-first search on a simple binary tree. At each stage, the node to be expanded next is indicated by the triangular marker.

Figure 3.9

Breadth-first search and uniform-cost search algorithms.

Early goal test

Late goal test

Breadth-first search always finds a solution with a minimal number of actions, because

d

when it is generating nodes at depth , it has already generated all the nodes at depth

d − 1,

so if one of them were a solution, it would have been found. That means it is cost-optimal for problems where all actions have the same cost, but not for problems that don’t have that property. It is complete in either case. In terms of time and space, imagine searching a

b successors. The root of the search tree generates b nodes, each of which generates b more nodes, for a total of b at the second level. Each of these generates b more nodes, yielding b nodes at the third level, and so on. Now suppose that the solution is at depth d. Then the total number of nodes generated is uniform tree where every state has

2

3

1+

b+b

2

+

b

3

+⋯+

bd = O (bd )

All the nodes remain in memory, so both time and space complexity are

O(bd ). Exponential

bounds like that are scary. As a typical real-world example, consider a problem with

b = 10, processing speed 1 million nodes/second, and memory requirements of 1 Kbyte/node. A search to depth d = 10 would take less than 3 hours, but branching factor

would require 10 terabytes of memory. The memory requirements are a bigger problem for breadth-first search than the execution time. But time is still an important factor. At depth

d = 14, even with infinite memory, the search would take 3.5 years. In general, exponentialcomplexity search problems cannot be solved by uninformed search for any but the smallest instances.

3.4.2 Dijkstra’s algorithm or uniform-cost search When actions have different costs, an obvious choice is to use best-first search where the evaluation function is the cost of the path from the root to the current node. This is called Dijkstra’s algorithm by the theoretical computer science community, and uniform-cost search by the AI community. The idea is that while breadth-first search spreads out in waves of uniform depth—first depth 1, then depth 2, and so on—uniform-cost search spreads out in waves of uniform path-cost. The algorithm can be implemented as a call to BEST-FIRSTSEARCH with PATH-COST as the evaluation function, as shown in Figure 3.9 .

Uniform-cost search

Consider Figure 3.10 , where the problem is to get from Sibiu to Bucharest. The successors of Sibiu are Rimnicu Vilcea and Fagaras, with costs 80 and 99, respectively. The least-cost node, Rimnicu Vilcea, is expanded next, adding Pitesti with cost 80 + 97 = 177. The leastcost node is now Fagaras, so it is expanded, adding Bucharest with cost 99 + 211 = 310. Bucharest is the goal, but the algorithm tests for goals only when it expands a node, not when it generates a node, so it has not yet detected that this is a path to the goal.

Figure 3.10

Part of the Romania state space, selected to illustrate uniform-cost search.

The algorithm continues on, choosing Pitesti for expansion next and adding a second path to Bucharest with cost 80 + 97 + 101 = 278. It has a lower cost, so it replaces the previous path in reached and is added to the frontier. It turns out this node now has the lowest cost, so it is considered next, found to be a goal, and returned. Note that if we had checked for a goal upon generating a node rather than when expanding the lowest-cost node, then we would have returned a higher-cost path (the one through Fagaras).

The complexity of uniform-cost search is characterized in terms of

ϵ

C

∗

, the cost of the

optimal solution,8 and , a lower bound on the cost of each action, with algorithm’s worst-case time and space complexity is

b

O(b

1+⌊

ϵ > 0. Then the

C /ϵ⌋ ), which can be much greater ∗

than d . This is because uniform-cost search can explore large trees of actions with low costs before exploring paths involving a high-cost and perhaps useful action. When all action costs are equal, search.

b

1+⌊

C /ϵ⌋ is just bd+1 , and uniform-cost search is similar to breadth-first ∗

8 Here, and throughout the book, the “star” in

C

∗

means an optimal value for

C

.

Uniform-cost search is complete and is cost-optimal, because the first solution it finds will have a cost that is at least as low as the cost of any other node in the frontier. Uniform-cost search considers all paths systematically in order of increasing cost, never getting caught going down a single infinite path (assuming that all action costs are >

ϵ

> 0).

3.4.3 Depth-first search and the problem of memory

Depth-first search

Depth-first search always expands the deepest node in the frontier first. It could be implemented as a call to BEST-FIRST-SEARCH where the evaluation function

f is the negative of

the depth. However, it is usually implemented not as a graph search but as a tree-like search that does not keep a table of reached states. The progress of the search is illustrated in Figure 3.11 ; search proceeds immediately to the deepest level of the search tree, where the nodes have no successors. The search then “backs up” to the next deepest node that still has unexpanded successors. Depth-first search is not cost-optimal; it returns the first solution it finds, even if it is not cheapest.

Figure 3.11

A dozen steps (left to right, top to bottom) in the progress of a depth-first search on a binary tree from start state A to goal M. The frontier is in green, with a triangle marking the node to be expanded next. Previously expanded nodes are lavender, and potential future nodes have faint dashed lines. Expanded nodes with no descendants in the frontier (very faint lines) can be discarded.

For finite state spaces that are trees it is efficient and complete; for acyclic state spaces it may end up expanding the same state many times via different paths, but will (eventually) systematically explore the entire space.

In cyclic state spaces it can get stuck in an infinite loop; therefore some implementations of depth-first search check each new node for cycles. Finally, in infinite state spaces, depth-first search is not systematic: it can get stuck going down an infinite path, even if there are no cycles. Thus, depth-first search is incomplete.

With all this bad news, why would anyone consider using depth-first search rather than breadth-first or best-first? The answer is that for problems where a tree-like search is feasible, depth-first search has much smaller needs for memory. We don’t keep a reached

table at all, and the frontier is very small: think of the frontier in breadth-first search as the surface of an ever-expanding sphere, while the frontier in depth-first search is just a radius of the sphere. For a finite tree-shaped state-space like the one in Figure 3.11 , a depth-first tree-like search takes time proportional to the number of states, and has memory complexity of only

O(bm), where b is the branching factor and m is the maximum depth of the tree. Some problems that would require exabytes of memory with breadth-first search can be handled with only kilobytes using depth-first search. Because of its parsimonious use of memory, depth-first tree-like search has been adopted as the basic workhorse of many areas of AI, including constraint satisfaction (Chapter 6 ), propositional satisfiability (Chapter 7 ), and logic programming (Chapter 9 ).

A variant of depth-first search called backtracking search uses even less memory. (See Chapter 6  for more details.) In backtracking, only one successor is generated at a time rather than all successors; each partially expanded node remembers which successor to generate next. In addition, successors are generated by modifying the current state description directly rather than allocating memory for a brand-new state. This reduces the memory requirements to just one state description and a path of savings over

O(m) actions; a significant

O(bm) states for depth-first search. With backtracking we also have the option

of maintaining an efficient set data structure for the states on the current path, allowing us to check for a cyclic path in

O(1) time rather than O(m). For backtracking to work, we must

be able to undo each action when we backtrack. Backtracking is critical to the success of many problems with large state descriptions, such as robotic assembly.

Backtracking search

3.4.4 Depth-limited and iterative deepening search To keep depth-first search from wandering down an infinite path, we can use depth-limited search, a version of depth-first search in which we supply a depth limit, ℓ, and treat all nodes at depth ℓ as if they had no successors (see Figure 3.12 ). The time complexity is

O(b ) and the space complexity is O(bℓ). Unfortunately, if we make a poor choice for ℓ the ℓ

algorithm will fail to reach the solution, making it incomplete again.

Figure 3.12

Iterative deepening and depth-limited tree-like search. Iterative deepening repeatedly applies depthlimited search with increasing limits. It returns one of three different types of values: either a solution node; or failure, when it has exhausted all nodes and proved there is no solution at any depth; or cutoff, to mean there might be a solution at a deeper depth than ℓ. This is a tree-like search algorithm that does not keep track of reached states, and thus uses much less memory than best-first search, but runs the risk of visiting the same state multiple times on different paths. Also, if the IS-CYCLE check does not check all cycles, then the algorithm may get caught in a loop.

Depth-limited search

Since depth-first search is a tree-like search, we can’t keep it from wasting time on redundant paths in general, but we can eliminate cycles at the cost of some computation time. If we look only a few links up in the parent chain we can catch most cycles; longer cycles are handled by the depth limit.

Sometimes a good depth limit can be chosen based on knowledge of the problem. For example, on the map of Romania there are 20 cities. Therefore, ℓ = 19 is a valid limit. But if we studied the map carefully, we would discover that any city can be reached from any other city in at most 9 actions. This number, known as the diameter of the state-space graph, gives us a better depth limit, which leads to a more efficient depth-limited search. However, for most problems we will not know a good depth limit until we have solved the problem.

Diameter

Iterative deepening search solves the problem of picking a good value for ℓ by trying all values: first 0, then 1, then 2, and so on—until either a solution is found, or the depthlimited search returns the failure value rather than the cutoff value. The algorithm is shown in Figure 3.12 . Iterative deepening combines many of the benefits of depth-first and breadth-first search. Like depth-first search, its memory requirements are modest: when there is a solution, or

O(bd)

O(bm) on finite state spaces with no solution. Like breadth-first

search, iterative deepening is optimal for problems where all actions have the same cost, and is complete on finite acyclic state spaces, or on any finite state space when we check nodes for cycles all the way up the path.

Iterative deepening search

The time complexity is

O(bd ) when there is a solution, or O(bm ) when there is none. Each

iteration of iterative deepening search generates a new level, in the same way that breadthfirst search does, but breadth-first does this by storing all nodes in memory, while iterativedeepening does it by repeating the previous levels, thereby saving memory at the cost of more time. Figure 3.13  shows four iterations of iterative-deepening search on a binary search tree, where the solution is found on the fourth iteration.

Figure 3.13

M

Four iterations of iterative deepening search for goal on a binary tree, with the depth limit varying from 0 to 3. Note the interior nodes form a single path. The triangle marks the node to expand next; green nodes with dark outlines are on the frontier; the very faint nodes provably can’t be part of a solution with this depth limit.

Iterative deepening search may seem wasteful because states near the top of the search tree are re-generated multiple times. But for many state spaces, most of the nodes are in the bottom level, so it does not matter much that the upper levels are repeated. In an iterative deepening search, the nodes on the bottom level (depth d) are generated once, those on the next-to-bottom level are generated twice, and so on, up to the children of the root, which are generated d times. So the total number of nodes generated in the worst case is

N

(IDS)

=

which gives a time complexity of example, if

b

= 10

and

N N

d

= 5,

db

( )

1

+(

d

− 1)

b

2

+(

d

− 2)

b

3

…+

bd

,

O bd —asymptotically the same as breadth-first search. For (

)

the numbers are

(IDS) = 50 + 400 + 3,000 + 20,000 + 100,000 = 123,450

(BFS) = 10 + 100 + 1,000 + 10,000 + 100,000 = 111,110.

If you are really concerned about the repetition, you can use a hybrid approach that runs breadth-first search until almost all the available memory is consumed, and then runs iterative deepening from all the nodes in the frontier. In general, iterative deepening is the preferred uninformed search method when the search state space is larger than can fit in memory and the depth of the solution is not known.

3.4.5 Bidirectional search The algorithms we have covered so far start at an initial state and can reach any one of multiple possible goal states. An alternative approach called bidirectional search simultaneously searches forward from the initial state and backwards from the goal state(s), hoping that the two searches will meet. The motivation is that

bd (e.g., 50,000 times less when b

=

d

bd

/2

+

bd

/2

is much less than

= 10).

Bidirectional search

For this to work, we need to keep track of two frontiers and two tables of reached states, and we need to be able to reason backwards: if state direction, then we need to know that

s

'

is a successor of

s is a successor of s

'

s in the forward

in the backward direction. We

have a solution when the two frontiers collide.9

9 In our implementation, the reached data structure supports a query asking whether a given state is a member, and the frontier data structure (a priority queue) does not, so we check for a collision using reached; but conceptually we are asking if the two frontiers have met up. The implementation can be extended to handle multiple goal states by loading the node for each goal state into the backwards frontier and backwards reached table.

There are many different versions of bidirectional search, just as there are many different unidirectional search algorithms. In this section, we describe bidirectional best-first search. Although there are two separate frontiers, the node to be expanded next is always one with a minimum value of the evaluation function, across either frontier. When the evaluation function is the path cost, we get bidirectional uniform-cost search, and if the cost of the optimal path is

C

∗

,

then no node with cost >

C will be expanded. This can result in a ∗

2

considerable speedup. The general best-first bidirectional search algorithm is shown in Figure 3.14 . We pass in two versions of the problem and the evaluation function, one in the forward direction (subscript

F ) and one in the backward direction (subscript B). When the evaluation

function is the path cost, we know that the first solution found will be an optimal solution, but with different evaluation functions that is not necessarily true. Therefore, we keep track of the best solution found so far, and might have to update that several times before the TERMINATED test proves that there is no possible better solution remaining.

Figure 3.14

Bidirectional best-first search keeps two frontiers and two tables of reached states. When a path in one frontier reaches a state that was also reached in the other half of the search, the two paths are joined (by the function JOIN-NODES) to form a solution. The first solution we get is not guaranteed to be the best; the function TERMINATED determines when to stop looking for new solutions.

3.4.6 Comparing uninformed search algorithms Figure 3.15  compares uninformed search algorithms in terms of the four evaluation criteria set forth in Section 3.3.4 . This comparison is for tree-like search versions which don’t check for repeated states. For graph searches which do check, the main differences are that depth-first search is complete for finite state spaces, and the space and time complexities are bounded by the size of the state space (the number of vertices and edges,

Figure 3.15

|V | + |E|).

Evaluation of search algorithms. b is the branching factor; m is the maximum depth of the search tree; d is the depth of the shallowest solution, or is m when there is no solution; ℓ is the depth limit. Superscript caveats are as follows: 1 complete if b is finite, and the state space either has a solution or is finite. 2 complete if all action costs are ≥ ε > 0; 3 cost-optimal if action costs are all identical; 4 if both directions are breadth-first or uniform-cost.

3.5 Informed (Heuristic) Search Strategies This section shows how an informed search strategy—one that uses domain-specific hints about the location of goals—can find solutions more efficiently than an uninformed strategy. The hints come in the form of a heuristic function, denoted h(n):10

10 It may seem odd that the heuristic function operates on a node, when all it really needs is the node’s state. It is traditional to use

h (n) rather than h (s) to be consistent with the evaluation function f (n) and the path cost g (n) .

h(n) =  estimated cost of the cheapest path from the state at node n to a goal state.

Informed search

Heuristic function

For example, in route-finding problems, we can estimate the distance from the current state to a goal by computing the straight-line distance on the map between the two points. We study heuristics and where they come from in more detail in Section 3.6 .

3.5.1 Greedy best-first search Greedy best-first search is a form of best-first search that expands first the node with the lowest h(n) value—the node that appears to be closest to the goal—on the grounds that this is likely to lead to a solution quickly. So the evaluation function f (n)

Greedy best-first search

=

h(n).

Let us see how this works for route-finding problems in Romania; we use the straight-line

h

distance heuristic, which we will call SLD . If the goal is Bucharest, we need to know the straight-line distances to Bucharest, which are shown in Figure 3.16 . For example,

hSLD (Arad) = 366. Notice that the values of hSLD cannot be computed from the problem description itself (that is, the ACTIONS and RESULT functions). Moreover, it takes a certain

h

amount of world knowledge to know that SLD is correlated with actual road distances and is, therefore, a useful heuristic.

Figure 3.16

h

Values of SLD —straight-line distances to Bucharest.

Straight-line distance

Figure 3.17  shows the progress of a greedy best-first search using SLD to find a path from

h

Arad to Bucharest. The first node to be expanded from Arad will be Sibiu because the heuristic says it is closer to Bucharest than is either Zerind or Timisoara. The next node to be expanded will be Fagaras because it is now closest according to the heuristic. Fagaras in turn generates Bucharest, which is the goal. For this particular problem, greedy best-first

h

search using SLD finds a solution without ever expanding a node that is not on the solution path. The solution it found does not have optimal cost, however: the path via Sibiu and Fagaras to Bucharest is 32 miles longer than the path through Rimnicu Vilcea and Pitesti. This is why the algorithm is called “greedy”—on each iteration it tries to get as close to a goal as it can, but greediness can lead to worse results than being careful.

Figure 3.17

Stages in a greedy best-first tree-like search for Bucharest with the straight-line distance heuristic

h

Nodes are labeled with their -values.

hSLD .

Greedy best-first graph search is complete in finite state spaces, but not in infinite ones. The

O(|V |). With a good heuristic function, however, the complexity can be reduced substantially, on certain problems reaching O(bm). worst-case time and space complexity is

3.5.2 A* search The most common informed search algorithm is A* search (pronounced “A-star search”), a best-first search that uses the evaluation function

f ( n ) = g ( n ) + h( n )

A* search

where

gn (

)

is the path cost from the initial state to node

the shortest path from

fn (

n to a goal state, so we have

n

,

and

hn (

)

is the estimated cost of

) =  estimated cost of the best path that continues from 

n

 to a goal.

In Figure 3.18 , we show the progress of an A* search with the goal of reaching Bucharest. The values of

g are computed from the action costs in Figure 3.1 , and the values of hSLD

are given in Figure 3.16 . Notice that Bucharest first appears on the frontier at step (e), but it is not selected for expansion (and thus not detected as a solution) because at not the lowest-cost node on the frontier—that would be Pitesti, at

f

= 417.

f

= 450

it is

Another way to

say this is that there might be a solution through Pitesti whose cost is as low as 417, so the algorithm will not settle for a solution that costs 450. At step (f), a different path to Bucharest is now the lowest-cost node, at optimal solution.

Figure 3.18

f

= 418,

so it is selected and detected as the

Stages in an A* search for Bucharest. Nodes are labeled with distances to Bucharest taken from Figure 3.16 .

f = g + h. The h values are the straight-line

Admissible heuristic

A* search is complete.11 Whether A* is cost-optimal depends on certain properties of the heuristic. A key property is admissibility: an admissible heuristic is one that never overestimates the cost to reach a goal. (An admissible heuristic is therefore optimistic.) With an admissible heuristic, A* is cost-optimal, which we can show with a proof by contradiction. Suppose the optimal path has cost cost

C>C

∗

. Then there must be some node

C

∗

, but the algorithm returns a path with

n which is on the optimal path and is

unexpanded (because if all the nodes on the optimal path had been expanded, then we would have returned that optimal solution). So then, using the notation

n

cost of the optimal path from the start to , and from

n to the nearest goal, we have:

g (n) to mean the ∗

h (n) to mean the cost of the optimal path ∗

11 Again, assuming all action costs are >∈> 0, and the state space either has a solution or is finite.

f (n) f (n) f (n) f (n) f (n)

> = = ≤ ≤

C (otherwise n would have been expanded) g (n) + h (n) (by definition) g (n) + h (n) (because n is on an optimal path) g (n) + h (n) (because of admissibility, h (n) ≤ h (n)) C (by definition, C = g (n) + h (n)) ∗

∗

∗

∗

∗

∗

∗

∗

∗

The first and last lines form a contradiction, so the supposition that the algorithm could return a suboptimal path must be wrong—it must be that A* returns only cost-optimal paths.

hn

A slightly stronger property is called consistency. A heuristic ( ) is consistent if, for every node

n and every successor n

′

of

n generated by an action a, we have: h(n) ≤ c(n, a, n ) + h(n ). ′

′

Consistency

This is a form of the triangle inequality, which stipulates that a side of a triangle cannot be longer than the sum of the other two sides (see Figure 3.19 ). An example of a consistent heuristic is the straight-line distance

hSLD that we used in getting to Bucharest.

Figure 3.19

h is consistent, then the single number h(n) will be less than the sum c n a a ) of the action from n to n plus the heuristic estimate h(n ).

Triangle inequality: If the heuristic of the cost ( , ,

′

′

′

Triangle inequality

Every consistent heuristic is admissible (but not vice versa), so with a consistent heuristic, A* is cost-optimal. In addition, with a consistent heuristic, the first time we reach a state it will be on an optimal path, so we never have to re-add a state to the frontier, and never have to change an entry in reached. But with an inconsistent heuristic, we may end up with multiple paths reaching the same state, and if each new path has a lower path cost than the previous one, then we will end up with multiple nodes for that state in the frontier, costing us both time and space. Because of that, some implementations of A* take care to only enter a state into the frontier once, and if a better path to the state is found, all the successors of the state are updated (which requires that nodes have child pointers as well as parent pointers). These complications have led many implementers to avoid inconsistent heuristics, but Felner et al. (2011) argues that the worst effects rarely happen in practice, and one shouldn’t be afraid of inconsistent heuristics.

With an inadmissible heuristic, A* may or may not be cost-optimal. Here are two cases

hn

where it is: First, if there is even one cost-optimal path on which ( ) is admissible for all

n on the path, then that path will be found, no matter what the heuristic says for states off the path. Second, if the optimal solution has cost C , and the second-best has cost C , and if h(n) overestimates some costs, but never by more than C − C , then A* is nodes

∗

2

2

∗

guaranteed to return cost-optimal solutions.

3.5.3 Search contours A useful way to visualize a search is to draw contours in the state space, just like the contours in a topographic map. Figure 3.20  shows an example. Inside the contour labeled

fn

gn

hn

400, all nodes have ( ) = ( ) + ( ) ≤ 400, and so on. Then, because A* expands the

f

frontier node of lowest -cost, we can see that an A* search fans out from the start node,

f

adding nodes in concentric bands of increasing -cost.

Figure 3.20

f

f

f

Map of Romania showing contours at = 380, = 400, and = 420, with Arad as the start state. Nodes inside a given contour have = + costs less than or equal to the contour value.

f g h

Contour

g

With uniform-cost search, we also have contours, but of -cost, not

g + h. The contours with

uniform-cost search will be “circular” around the start state, spreading out equally in all directions with no preference towards the goal. With A* search using a good heuristic, the

g + h bands will stretch toward a goal state (as in Figure 3.20 ) and become more narrowly focused around an optimal path.

It should be clear that as you extend a path, the

g costs are monotonic: the path cost always

increases as you go along a path, because action costs are always positive.12 Therefore you get concentric contour lines that don’t cross each other, and if you choose to draw the lines fine enough, you can put a line between any two nodes on any path.

12 Technically, we say “strictly monotonic” for costs that always increase, and “monotonic” for costs that never decrease, but might remain the same.

Monotonic

f = g + h cost will monotonically increase. As you extend a path from n to n , the cost goes from g (n) + h (n) to g(n) + c(n, a, n ) + h(n ). Canceling out the g(n) term, we see that the path’s cost will be monotonically increasing if and only if h(n) ≤ c(n, a, n ) + h(n ); in other words if and only if the heuristic is consistent.13 But note that a path might contribute several nodes in a row with the same g(n) + h(n) score; this will happen whenever the decrease in h is exactly equal to the action cost just taken (for example, in a grid problem, when n is in the same row as the goal and you take a step towards the goal, g is increased by 1 and h is decreased by 1). If C is the cost of the optimal But it is not obvious whether the ′

′

′

′

′

∗

solution path, then we can say the following:

13 In fact, the term “monotonic heuristic” is a synonym for “consistent heuristic.” The two ideas were developed independently, and then it was proved that they are equivalent (Pearl, 1984).

A* expands all nodes that can be reached from the initial state on a path where every

fn

node on the path has ( )


C

∗

.

We say that A* with a consistent heuristic is optimally efficient in the sense that any algorithm that extends search paths from the initial state, and uses the same heuristic information, must expand all nodes that are surely expanded by A* (because any one of

fn

them could have been part of an optimal solution). Among the nodes with ( ) =

C

∗

, one

algorithm could get lucky and choose the optimal one first while another algorithm is unlucky; we don’t consider this difference in defining optimal efficiency.

Optimally efficient

A* is efficient because it prunes away search tree nodes that are not necessary for finding an optimal solution. In Figure 3.18(b)  we see that Timisoara has

f = 447 and Zerind has

f = 449. Even though they are children of the root and would be among the first nodes

expanded by uniform-cost or breadth-first search, they are never expanded by A* search because the solution with

f = 418 is found first. The concept of pruning—eliminating

possibilities from consideration without having to examine them—is important for many areas of AI.

Pruning

)

That A* search is complete, cost-optimal, and optimally efficient among all such algorithms is rather satisfying. Unfortunately, it does not mean that A* is the answer to all our searching needs. The catch is that for many problems, the number of nodes expanded can be exponential in the length of the solution. For example, consider a version of the vacuum world with a super-powerful vacuum that can clean up any one square at a cost of 1 unit, without even having to visit the square; in that scenario, squares can be cleaned in any

N initially dirty squares, there are 2N states where some subset has been cleaned; all of those states are on an optimal solution path, and hence satisfy f (n) < C order. With

∗

, so

all of them would be visited by A*.

3.5.4 Satisficing search: Inadmissible heuristics and weighted A*

Inadmissible heuristic

A* search has many good qualities, but it expands a lot of nodes. We can explore fewer nodes (taking less time and space) if we are willing to accept solutions that are suboptimal, but are “good enough”—what we call satisficing solutions. If we allow A* search to use an inadmissible heuristic—one that may overestimate—then we risk missing the optimal solution, but the heuristic can potentially be more accurate, thereby reducing the number of nodes expanded. For example, road engineers know the concept of a detour index, which is a multiplier applied to the straight-line distance to account for the typical curvature of roads. A detour index of 1.3 means that if two cities are 10 miles apart in straight-line distance, a good estimate of the best path between them is 13 miles. For most localities, the detour index ranges between 1.2 and 1.6.

Detour index

We can apply this idea to any problem, not just ones involving roads, with an approach called weighted A* search where we weight the heuristic value more heavily, giving us the

fn

gn

evaluation function ( ) = ( ) +

W × h(n), for some W > 1.

Weighted A* search

Figure 3.21  shows a search problem on a grid world. In (a), an A* search finds the optimal solution, but has to explore a large portion of the state space to find it. In (b), a weighted A* search finds a solution that is slightly costlier, but the search time is much faster. We see that the weighted search focuses the contour of reached states towards a goal. That means that fewer states are explored, but if the optimal path ever strays outside of the weighted search’s contour (as it does in this case), then the optimal path will not be found. In general,

C , a weighted A* search will find a solution that costs somewhere between C and W × C ; but in practice we usually get results much closer to C than W × C . if the optimal solution costs ∗

∗

∗

∗

∗

Figure 3.21

Two searches on the same grid: (a) an A* search and (b) a weighted A* search with weight

W = 2. The

gray bars are obstacles, the purple line is the path from the green start to red goal, and the small dots are states that were reached by each search. On this particular problem, weighted A* explores 7 times fewer states and finds a path that is 5% more costly.

We have considered searches that evaluate states by combining

g and h in various ways;

weighted A* can be seen as a generalization of the others:

gn

A* search: Uniform-cost search: Greedy best-first search: Weighted A* search:

(

) +

hn (

)

gn hn gn W hn (

) +

(

(

)

(

(

)

(

×

(

W W W

= 1) = 0) = ∞)

) (1


f

2

value; the node can come

function guarantees that we will never expand a node (from

C

∗

2

. We say the two halves of the search “meet in the middle” in

the sense that when the two frontiers touch, no node inside of either frontier has a path cost ∗ greater than the bound C2 . Figure 3.24  works through an example bidirectional search.

Figure 3.24

Bidirectional search maintains two frontiers: on the left, nodes A and B are successors of Start; on the right, node F is an inverse successor of Goal. Each node is labeled with = + values and the + ) value. (The values are the sum of the action costs as shown on each arrow; the 2 = max(2 ,

f

gg h

f g h

g

h

values are arbitrary and cannot be derived from anything in the figure.) The optimal solution, Start-A-FGoal, has cost ∗ = 4 + 2 + 4 = 10, so that means that a meet-in-the-middle bidirectional algorithm

C

should not expand any node with (each with

g

g>

C

∗

2

= 5; and indeed the next node to be expanded would be A or F

= 4), leading us to an optimal solution. If we expanded the node with lowest

then B and C would come next, and D and E would be tied with A, but they all have never expanded when

f

2

is the evaluation function.

g>

f cost first,

C and thus are 2 ∗

Front-to-end

Front-to-front

hF heuristic estimates the distance to the goal (or, when the problem has multiple goal states, the distance to the closest goal) and hB We have described an approach where the

estimates the distance to the start. This is called a front-to-end search. An alternative, called front-to-front search, attempts to estimate the distance to the other frontier. Clearly, if a frontier has millions of nodes, it would be inefficient to apply the heuristic function to every one of them and take the minimum. But it can work to sample a few nodes from the frontier. In certain specific problem domains it is possible to summarize the frontier—for example, in a grid search problem, we can incrementally compute a bounding box of the frontier, and use as a heuristic the distance to the bounding box.

Bidirectional search is sometimes more efficient than unidirectional search, sometimes not. In general, if we have a very good heuristic, then A* search produces search contours that are focused on the goal, and adding bidirectional search does not help much. With an average heuristic, bidirectional search that meets in the middle tends to expand fewer nodes and is preferred. In the worst case of a poor heuristic, the search is no longer focused on the goal, and bidirectional search has the same asymptotic complexity as A*. Bidirectional search with the optimal.

f2 evaluation function and an admissible heuristic h is complete and

3.6 Heuristic Functions In this section, we look at how the accuracy of a heuristic affects search performance, and also consider how heuristics can be invented. As our main example we’ll return to the 8puzzle. As mentioned in Section 3.2 , the object of the puzzle is to slide the tiles horizontally or vertically into the empty space until the configuration matches the goal configuration (Figure 3.25 ).

Figure 3.25

A typical instance of the 8-puzzle. The shortest solution is 26 actions long.

There are 9!/2

= 181,400

reachable states in an 8-puzzle, so a search could easily keep them

all in memory. But for the 15-puzzle, there are 16!/2 states—over 10 trillion—so to search that space we will need the help of a good admissible heuristic function. There is a long history of such heuristics for the 15-puzzle; here are two commonly used candidates:

h

1

= the number of misplaced tiles (blank not included).

are out of position, so the start state has h

1

= 8.

h

1

For Figure 3.25 , all eight tiles

is an admissible heuristic because

any tile that is out of place will require at least one move to get it to the right place.

h

2

= the sum of the distances of the tiles from their goal positions.

Because tiles cannot

move along diagonals, the distance is the sum of the horizontal and vertical distances— sometimes called the city-block distance or Manhattan distance. h is also admissible 2

because all any move can do is move one tile one step closer to the goal. Tiles 1 to 8 in the start state of Figure 3.25  give a Manhattan distance of

h

2

= 3 + 1 + 2 + 2 + 2 + 3 + 3 + 2 = 18.

Manhattan distance

As expected, neither of these overestimates the true solution cost, which is 26.

3.6.1 The effect of heuristic accuracy on performance One way to characterize the quality of a heuristic is the effective branching factor total number of nodes generated by A* for a particular problem is

d

is , then

b

∗

is the branching factor that a uniform tree of depth

order to contain

N + 1 nodes. Thus,

N +1=1+b

∗

+(

b)

∗ 2

+⋯+(

b . If the ∗

N and the solution depth

d would have to have in

b )d . ∗

Effective branching factor

For example, if A* finds a solution at depth 5 using 52 nodes, then the effective branching factor is 1.92. The effective branching factor can vary across problem instances, but usually for a specific domain (such as 8-puzzles) it is fairly constant across all nontrivial problem instances. Therefore, experimental measurements of

b

∗

on a small set of problems can

provide a good guide to the heuristic’s overall usefulness. A well-designed heuristic would have a value of

b

∗

close to 1, allowing fairly large problems to be solved at reasonable

computational cost.

Korf and Reid, (1998) argue that a better way to characterize the effect of A* pruning with a

h is that it reduces the effective depth by a constant kh compared to the true depth. This means that the total search cost is O(bd kh ) compared to O(bd ) for an uninformed search. Their experiments on Rubik’s Cube and n-puzzle problems show that this formula

given heuristic

−

gives accurate predictions for total search cost for sampled problem instances across a wide range of solution lengths—at least for solution lengths larger than kh .

Effective depth

For Figure 3.26  we generated random 8-puzzle problems and solved them with an uninformed breadth-first search and with A* search using both h1 and h2 , reporting the average number of nodes generated and the corresponding effective branching factor for each search strategy and for each solution length. The results suggest that h2 is better than

h1 , and both are better than no heuristic at all. Figure 3.26

Comparison of the search costs and effective branching factors for 8-puzzle problems using breadth-first search, A* with

h1 (misplaced tiles), and A* with

h2 (Manhattan distance). Data are averaged over 100 puzzles for each solution length

d from 6 to 28.

One might ask whether h2 is always better than h1 . The answer is “Essentially, yes.” It is easy to see from the definitions of the two heuristics that for any node n, h2 (n) ≥ h1 (n). We thus

h . Domination translates directly into efficiency: A* using h will never expand more nodes than A* using h (except in the case of breaking ties unluckily). The argument is simple. Recall the observation on page 90 that every node with f (n) < C will surely be expanded. This is the same as saying that every node with h(n) < C − g(n) is surely expanded when h is consistent. But because h is at least as big as h for all nodes, every node that is surely expanded by A* search with h is also surely expanded with h , and h might cause other nodes to be expanded as well. Hence, it is generally better to use a say that

h

2

dominates

1

2

1

∗

∗

2

1

2

1

1

heuristic function with higher values, provided it is consistent and that the computation time for the heuristic is not too long.

Domination

3.6.2 Generating heuristics from relaxed problems We have seen that both

h

1

(misplaced tiles) and

heuristics for the 8-puzzle and that

h

2

h

2

(Manhattan distance) are fairly good

is better. How might one have come up with

h ? Is it 2

possible for a computer to invent such a heuristic mechanically?

h

1

and

h

2

are estimates of the remaining path length for the 8-puzzle, but they are also

perfectly accurate path lengths for simplified versions of the puzzle. If the rules of the puzzle were changed so that a tile could move anywhere instead of just to the adjacent empty square, then

h

1

would give the exact length of the shortest solution. Similarly, if a tile could

move one square in any direction, even onto an occupied square, then

h

2

would give the

exact length of the shortest solution. A problem with fewer restrictions on the actions is called a relaxed problem. The state-space graph of the relaxed problem is a supergraph of the original state space because the removal of restrictions creates added edges in the graph.

Relaxed problem

Because the relaxed problem adds edges to the state-space graph, any optimal solution in the original problem is, by definition, also a solution in the relaxed problem; but the relaxed problem may have better solutions if the added edges provide shortcuts. Hence, the cost of an optimal solution to a relaxed problem is an admissible heuristic for the original problem. Furthermore, because the derived heuristic is an exact cost for the relaxed problem, it must obey the triangle inequality and is therefore consistent (see page 88).

If a problem definition is written down in a formal language, it is possible to construct relaxed problems automatically.14 For example, if the 8-puzzle actions are described as 14 In Chapters 8  and 11 , we describe formal languages suitable for this task; with formal descriptions that can be manipulated, the construction of relaxed problems can be automated. For now, we use English.

A tile can move from square X to square Y if X is adjacent to Y

and Y is blank,

we can generate three relaxed problems by removing one or both of the conditions:

a. A tile can move from square X to square Y if X is adjacent to Y. b. A tile can move from square X to square Y if Y is blank. c. A tile can move from square X to square Y. From (a), we can derive h2 (Manhattan distance). The reasoning is that h2 would be the proper score if we moved each tile in turn to its destination. The heuristic derived from (b) is discussed in Exercise 3.GASC. From (c), we can derive h1 (misplaced tiles) because it would be the proper score if tiles could move to their intended destination in one action. Notice that it is crucial that the relaxed problems generated by this technique can be solved essentially without search, because the relaxed rules allow the problem to be decomposed into eight independent subproblems. If the relaxed problem is hard to solve, then the values of the corresponding heuristic will be expensive to obtain.

A program called ABSOLVER can generate heuristics automatically from problem definitions, using the “relaxed problem” method and various other techniques (Prieditis, 1993). ABSOLVER generated a new heuristic for the 8-puzzle that was better than any preexisting heuristic and found the first useful heuristic for the famous Rubik’s Cube puzzle.

If a collection of admissible heuristics h1 … hm is available for a problem and none of them is clearly better than the others, which should we choose? As it turns out, we can have the best of all worlds, by defining

h(n) = max{h1 (n), … , hk (n)}. This composite heuristic picks whichever function is most accurate on the node in question. Because the hi components are admissible, h is admissible (and if hi are all consistent, h is consistent). Furthermore, h dominates all of its component heuristics. The only drawback is that h(n) takes longer to compute. If that is an issue, an alternative is to randomly select one of the heuristics at each evaluation, or use a machine learning algorithm to predict which heuristic will be best. Doing this can result in a heuristic that is inconsistent (even if every hi is consistent), but in practice it usually leads to faster problem solving.

3.6.3 Generating heuristics from subproblems: Pattern databases

Subproblem

Admissible heuristics can also be derived from the solution cost of a subproblem of a given problem. For example, Figure 3.27  shows a subproblem of the 8-puzzle instance in Figure 3.25 . The subproblem involves getting tiles 1, 2, 3, 4, and the blank into their correct positions. Clearly, the cost of the optimal solution of this subproblem is a lower bound on the cost of the complete problem. It turns out to be more accurate than Manhattan distance in some cases.

Figure 3.27

A subproblem of the 8-puzzle instance given in Figure 3.25 . The task is to get tiles 1, 2, 3, 4, and the blank into their correct positions, without worrying about what happens to the other tiles.

The idea behind pattern databases is to store these exact solution costs for every possible subproblem instance—in our example, every possible configuration of the four tiles and the blank. (There will be 9 × 8 × 7 × 6 × 5

= 15,120

patterns in the database. The identities of

the other four tiles are irrelevant for the purposes of solving the subproblem, but moves of those tiles do count toward the solution cost of the subproblem.) Then we compute an admissible heuristic

hDB for each state encountered during a search simply by looking up

the corresponding subproblem configuration in the database. The database itself is constructed by searching back from the goal and recording the cost of each new pattern encountered;15 the expense of this search is amortized over subsequent problem instances, and so makes sense if we expect to be asked to solve many problems.

15 By working backward from the goal, the exact solution cost of every instance encountered is immediately available. This is an

example of dynamic programming, which we discuss further in Chapter 17 .

Pattern database

The choice of tiles 1-2-3-4 to go with the blank is fairly arbitrary; we could also construct databases for 5-6-7-8, for 2-4-6-8, and so on. Each database yields an admissible heuristic, and these heuristics can be combined, as explained earlier, by taking the maximum value. A combined heuristic of this kind is much more accurate than the Manhattan distance; the number of nodes generated when solving random 15-puzzles can be reduced by a factor of

1000. However, with each additional database there are diminishing returns and increased memory and computation costs.

Disjoint pattern databases

One might wonder whether the heuristics obtained from the 1-2-3-4 database and the 5-6-78 could be added, since the two subproblems seem not to overlap. Would this still give an admissible heuristic? The answer is no, because the solutions of the 1-2-3-4 subproblem and the 5-6-7-8 subproblem for a given state will almost certainly share some moves—it is unlikely that 1-2-3-4 can be moved into place without touching 5-6-7-8, and vice versa. But what if we don’t count those moves—what if we don’t abstract the other tiles to stars, but rather make them disappear? That is, we record not the total cost of solving the 1-2-3-4 subproblem, but just the number of moves involving 1-2-3-4. Then the sum of the two costs is still a lower bound on the cost of solving the entire problem. This is the idea behind disjoint pattern databases. With such databases, it is possible to solve random 15-puzzles in a few milliseconds—the number of nodes generated is reduced by a factor of 10,000 compared with the use of Manhattan distance. For 24-puzzles, a speedup of roughly a factor of a million can be obtained. Disjoint pattern databases work for sliding-tile puzzles because the problem can be divided up in such a way that each move affects only one subproblem— because only one tile is moved at a time.

3.6.4 Generating heuristics with landmarks There are online services that host maps with tens of millions of vertices and find costoptimal driving directions in milliseconds (Figure 3.28 ). How can they do that, when the best search algorithms we have considered so far are about a million times slower? There are many tricks, but the most important one is precomputation of some optimal path costs. Although the precomputation can be time-consuming, it need only be done once, and then can be amortized over billions of user search requests.

Figure 3.28

A Web service providing driving directions, computed by a search algorithm.

Precomputation

We could generate a perfect heuristic by precomputing and storing the cost of the optimal path between every pair of vertices. That would take

O(∣∣V ∣∣ ) space, and O(∣∣E∣∣ ) time— 2

3

practical for graphs with 10 thousand vertices, but not 10 million.

A better approach is to choose a few (perhaps 10 or 20) landmark points16 from the

L and for each other vertex v in the graph, we compute and store C (v, L), the exact cost of the optimal path from v to L. (We also need C (L, v); on an undirected graph this is the same as C (v, L); on a directed graph—e.g., with one-way streets—we need to compute this separately.) Given the stored C tables, we can easily vertices. Then for each landmark ∗

∗

∗

∗

create an efficient (although inadmissible) heuristic: the minimum, over all landmarks, of the cost of getting from the current node to the landmark, and then to the goal:

16 Landmark points are sometimes called “pivots” or “anchors.”

hL ( n ) =

Landmark point

min

L ∈ Landmarks

C (n, L) + C (L, goal) ∗

∗

If the optimal path happens to go through a landmark, this heuristic will be exact; if not it is inadmissible—it overestimates the cost to the goal. In an A* search, if you have exact heuristics, then once you reach a node that is on an optimal path, every node you expand from then on will be on an optimal path. Think of the contour lines as following along this optimal path. The search will trace along the optimal path, on each iteration adding an

c to get to a result state whose h-value will be c less, meaning that the total g + h score will remain constant at C all along the path.

action with cost

f

=

∗

Some route-finding algorithms save even more time by adding shortcuts—artificial edges in the graph that define an optimal multi-action path. For example, if there were shortcuts predefined between all the 100 biggest cities in the U.S., and we were trying to navigate from the Berkeley campus in California to NYU in New York, we could take the shortcut between Sacramento and Manhattan and cover 90% of the path in one action.

Shortcuts

hL (n) is efficient but not admissible. But with a bit more care, we can come up with a heuristic that is both efficient and admissible:

hDH (n) = L

∈

max

Landmarks

|

C (n, L) − C (goal, L)| ∗

∗

This is called a differential heuristic (because of the subtraction). Think of this with a landmark that is somewhere out beyond the goal. If the goal happens to be on the optimal

n to the landmark, then this is saying “consider the entire path from n to L, then subtract off the last part of that path, from goal to L, giving us the exact cost of the path from n to goal.” To the extent that the goal is a bit off of the optimal path to the landmark, the path from

heuristic will be inexact, but still admissible. Landmarks that are not out beyond the goal will not be useful; a landmark that is exactly halfway between which is not helpful.

Differential heuristic

n and goal will give hDH = 0,

There are several ways to pick landmark points. Selecting points at random is fast, but we get better results if we take care to spread the landmarks out so they are not too close to each other. A greedy approach is to pick a first landmark at random, then find the point that is furthest from that, and add it to the set of landmarks, and continue, at each iteration adding the point that maximizes the distance to the nearest landmark. If you have logs of past search requests by your users, then you can pick landmarks that are frequently requested in searches. For the differential heuristic it is good if the landmarks are spread around the perimeter of the graph. Thus, a good technique is to find the centroid of the graph, arrange k pie-shaped wedges around the centroid, and in each wedge select the vertex that is farthest from the center.

Landmarks work especially well in route-finding problems because of the way roads are laid out in the world: a lot of traffic actually wants to travel between landmarks, so civil engineers build the widest and fastest roads along these routes; landmark search makes it easier to recover these routes.

3.6.5 Learning to search better

Metalevel state space

Object-level state space

We have presented several fixed search strategies—breadth-first, A*, and so on—that have been carefully designed and programmed by computer scientists. Could an agent learn how to search better? The answer is yes, and the method rests on an important concept called the metalevel state space. Each state in a metalevel state space captures the internal (computational) state of a program that is searching in an ordinary state space such as the map of Romania. (To keep the two concepts separate, we call the map of Romania an

object-level state space.) For example, the internal state of the A* algorithm consists of the current search tree. Each action in the metalevel state space is a computation step that alters the internal state; for example, each computation step in A* expands a leaf node and adds its successors to the tree. Thus, Figure 3.18 , which shows a sequence of larger and larger search trees, can be seen as depicting a path in the metalevel state space where each state on the path is an object-level search tree. Now, the path in Figure 3.18  has five steps, including one step, the expansion of Fagaras, that is not especially helpful. For harder problems, there will be many such missteps, and a metalevel learning algorithm can learn from these experiences to avoid exploring unpromising subtrees. The techniques used for this kind of learning are described in Chapter 22 . The goal of learning is to minimize the total cost of problem solving, trading off computational expense and path cost.

Metalevel learning

3.6.6 Learning heuristics from experience We have seen that one way to invent a heuristic is to devise a relaxed problem for which an optimal solution can be found easily. An alternative is to learn from experience. “Experience” here means solving lots of 8-puzzles, for instance. Each optimal solution to an 8-puzzle problem provides an example (goal, path) pair. From these examples, a learning algorithm can be used to construct a function h that can (with luck) approximate the true path cost for other states that arise during search. Most of these approaches learn an imperfect approximation to the heuristic function, and thus risk inadmissibility. This leads to an inevitable tradeoff between learning time, search run time, and solution cost. Techniques for machine learning are demonstrated in Chapter 19 . The reinforcement learning methods described in Chapter 22  are also applicable to search.

Some machine learning techniques work better when supplied with features of a state that are relevant to predicting the state’s heuristic value, rather than with just the raw state description. For example, the feature “number of misplaced tiles” might be helpful in predicting the actual distance of an 8-puzzle state from the goal. Let’s call this feature x1 (n).

We could take 100 randomly generated 8-puzzle configurations and gather statistics on their actual solution costs. We might find that when x1 (n) is 5, the average solution cost is around 14, and so on. Of course, we can use multiple features. A second feature x2 (n) might be “number of pairs of adjacent tiles that are not adjacent in the goal state.” How should x1 (n) and x2 (n) be combined to predict h(n)? A common approach is to use a linear combination: h(n) = c1 x1 (n) + c2 x2 (n).

Feature

The constants c1 and c2 are adjusted to give the best fit to the actual data across the randomly generated configurations. One expects both c1 and c2 to be positive because misplaced tiles and incorrect adjacent pairs make the problem harder to solve. Notice that this heuristic satisfies the condition h(n) = 0 for goal states, but it is not necessarily admissible or consistent.

Summary This chapter has introduced search algorithms that an agent can use to select action sequences in a wide variety of environments—as long as they are episodic, single-agent, fully observable, deterministic, static, discrete, and completely known. There are tradeoffs to be made between the amount of time the search takes, the amount of memory available, and the quality of the solution. We can be more efficient if we have domain-dependent knowledge in the form of a heuristic function that estimates how far a given state is from the goal, or if we precompute partial solutions involving patterns or landmarks.

Before an agent can start searching, a well-defined problem must be formulated. A problem consists of five parts: the initial state, a set of actions, a transition model describing the results of those actions, a set of goal states, and an action cost function. The environment of the problem is represented by a state space graph. A path through the state space (a sequence of actions) from the initial state to a goal state is a solution. Search algorithms generally treat states and actions as atomic, without any internal structure (although we introduced features of states when it came time to do learning). Search algorithms are judged on the basis of completeness, cost optimality, time complexity, and space complexity. Uninformed search methods have access only to the problem definition. Algorithms build a search tree in an attempt to find a solution. Algorithms differ based on which node they expand first: – BEST-FIRST SEARCH selects nodes for expansion using to an evaluation function. – BREADTH-FIRST SEARCH expands the shallowest nodes first; it is complete, optimal for unit action costs, but has exponential space complexity. – UNIFORM-COST SEARCH expands the node with lowest path cost, g(n), and is optimal for general action costs. – DEPTH-FIRST SEARCH expands the deepest unexpanded node first. It is neither complete nor optimal, but has linear space complexity. Depth-limited search adds a depth bound. – ITERATIVE DEEPENING SEARCH calls depth-first search with increasing depth limits until a goal is found. It is complete when full cycle checking is done, optimal for unit action costs, has time complexity comparable to breadth-first search, and has linear space complexity.

– BIDIRECTIONAL SEARCH expands two frontiers, one around the initial state and one around the goal, stopping when the two frontiers meet. Informed search methods have access to a heuristic function h(n) that estimates the cost of a solution from n. They may have access to additional information such as pattern databases with solution costs. – GREEDY BEST-FIRST SEARCH expands nodes with minimal h(n). It is not optimal but is often efficient. – A* SEARCH expands nodes with minimal f (n) = g(n) + h(n). A* is complete and optimal, provided that h(n) is admissible. The space complexity of A* is still an issue for many problems. – BIDIRECTIONAL A* SEARCH is sometimes more efficient than A* itself. – IDA* (iterative deepening A* search) is an iterative deepening version of A*, and thus adresses the space complexity issue. – RBFS (recursive best-first search) and SMA* (simplified memory-bounded A*) are robust, optimal search algorithms that use limited amounts of memory; given enough time, they can solve problems for which A* runs out of memory. – BEAM SEARCH puts a limit on the size of the frontier; that makes it incomplete and suboptimal, but it often finds reasonably good solutions and runs faster than complete searches. – WEIGHTED A* search focuses the search towards a goal, expanding fewer nodes, but sacrificing optimality. The performance of heuristic search algorithms depends on the quality of the heuristic function. One can sometimes construct good heuristics by relaxing the problem definition, by storing precomputed solution costs for subproblems in a pattern database, by defining landmarks, or by learning from experience with the problem class.

Bibliographical and Historical Notes The topic of state-space search originated in the early years of AI. Newell and Simon’s work on the Logic Theorist (1957) and GPS (1961) led to the establishment of search algorithms as the primary tool for 1960s AI researchers and to the establishment of problem solving as the canonical AI task. Work in operations research by Richard Bellman (1957) showed the importance of additive path costs in simplifying optimization algorithms. The text by Nils Nilsson (1971) established the area on a solid theoretical footing.

The 8-puzzle is a smaller cousin of the 15-puzzle, whose history is recounted at length by Slocum and Sonneveld (2006). In 1880, the 15-puzzle attracted broad attention from the public and mathematicians (Johnson and Story, 1879; Tait, 1880). The editors of the American Journal of Mathematics stated, “The ‘15’ puzzle for the last few weeks has been prominently before the American public, and may safely be said to have engaged the attention of nine out of ten persons of both sexes and all ages and conditions of the community,” while the Weekly News-Democrat of Emporia, Kansas wrote on March 12, 1880 that “It has become literally an epidemic all over the country.”

The famous American game designer Sam Loyd falsely claimed to have invented the 15 puzzle (Loyd, 1959); actually it was invented by Noyes Chapman, a postmaster in Canastota, New York, in the mid-1870s (although a generic patent covering sliding blocks was granted to Ernest Kinsey in 1878). Ratner and Warmuth (1986) showed that the general

n × n version of the 15-puzzle belongs to the class of NP-complete problems. Rubik’s Cube was of course invented in 1974 by Ernő Rubik, who also discovered an algorithm for finding good, but not optimal solutions. Korf (1997) found optimal solutions for some random problem instances using pattern databases and IDA* search. Rokicki et al., (2014) proved that any instance can be solved in 26 moves (if you consider a 180∘ twist to be two moves; 20 if it counts as one). The proof consumed 35 CPU years of computation; it does not lead immediately to an efficient algorithm. Agostinelli et al. (2019) used reinforcement learning, deep learning networks, and Monte Carlo tree search to learn a much more efficient solver for Rubik’s cube. It is not guaranteed to find a cost-optimal solution, but does so about 60% of the time, and typical solutions times are less than a second.

Each of the real-world search problems listed in the chapter has been the subject of a good deal of research effort. Methods for selecting optimal airline flights remain proprietary for the most part, but Carl de Marcken has shown by a reduction to Diophantine decision problems that airline ticket pricing and restrictions have become so convoluted that the problem of selecting an optimal flight is formally undecidable (Robinson, 2002). The traveling salesperson problem (TSP) is a standard combinatorial problem in theoretical computer science (Lawler et al., 1992). Karp (1972) proved the TSP decision problem to be NP-hard, but effective heuristic approximation methods were developed (Lin and Kernighan, 1973). Arora (1998) devised a fully polynomial approximation scheme for Euclidean TSPs. VLSI layout methods are surveyed by LaPaugh (2010), and many layout optimization papers appear in VLSI journals. Robotic navigation is discussed in Chapter 26 . Automatic assembly sequencing was first demonstrated by FREDDY (Michie, 1972); a comprehensive review is given by (Bahubalendruni and Biswal, 2016).

Uninformed search algorithms are a central topic of computer science (Cormen et al., 2009) and operations research (Dreyfus, 1969). Breadth-first search was formulated for solving mazes by Moore (1959). The method of dynamic programming (Bellman, 1957; Bellman and Dreyfus, 1962), which systematically records solutions for all subproblems of increasing lengths, can be seen as a form of breadth-first search.

Dijkstra’s algorithm in the form it is usually presented in (Dijkstra, 1959) is applicable to explicit finite graphs. Nilsson (1971) introduced a version of Dijkstra’s algorithm that he called uniform-cost search (because the algorithm “spreads out along contours of equal path cost”) that allows for implicitly defined, infinite graphs. Nilsson’s work also introduced the idea of closed and open lists, and the term “graph search.” The name BEST-FIRST-SEARCH was introduced in the Handbook of AI (Barr and Feigenbaum, 1981). The Floyd–Warshall (Floyd, 1962) and Bellman-Ford (Bellman, 1958; Ford, 1956) algorithms allow negative step costs (as long as there are no negative cycles).

A version of iterative deepening designed to make efficient use of the chess clock was first used by Slate and Atkin (1977) in the CHESS 4.5 game-playing program. Martelli’s algorithm B (1977) also includes an iterative deepening aspect. The iterative deepening technique was introduced by Bertram Raphael (1976) and came to the fore in work by Korf (1985a).

The use of heuristic information in problem solving appears in an early paper by Simon and Newell (1958), but the phrase “heuristic search” and the use of heuristic functions that estimate the distance to the goal came somewhat later (Newell and Ernst, 1965; Lin, 1965). Doran and Michie (1966) conducted extensive experimental studies of heuristic search. Although they analyzed path length and “penetrance” (the ratio of path length to the total number of nodes examined so far), they appear to have ignored the information provided by the path cost g(n). The A* algorithm, incorporating the current path cost into heuristic search, was developed by Hart, Nilsson, and Raphael (1968). Dechter and Pearl (1985) studied the conditions under which A* is optimally efficient (in number of nodes expanded).

The original A* paper (Hart et al., 1968) introduced the consistency condition on heuristic functions. The monotone condition was introduced by Pohl (1977) as a simpler replacement, but Pearl (1984) showed that the two were equivalent.

Pohl (1977) pioneered the study of the relationship between the error in heuristic functions and the time complexity of A*. Basic results were obtained for tree-like search with unit action costs and a single goal state (Pohl, 1977; Gaschnig, 1979; Huyn et al., 1980; Pearl, 1984) and with multiple goal states (Dinh et al., 2007). Korf and Reid (1998) showed how to predict the exact number of nodes expanded (not just an asymptotic approximation) on a variety of actual problem domains. The “effective branching factor” was proposed by Nilsson (1971) as an empirical measure of efficiency. For graph search, Helmert and Röger (2008) noted that several well-known problems contained exponentially many nodes on optimalcost solution paths, implying exponential time complexity for A*.

There are many variations on the A* algorithm. Pohl (1970b) introduced weighted A* search, and later a dynamic version (1973), where the weight changes over the depth of the tree. Ebendt and Drechsler (2009) synthesize the results and examine some applications. Hatem and Ruml (2014) show a simplified and improved version of weighted A* that is easier to implement. Wilt and Ruml (2014) introduce speedy search as an alternative to greedy search that focuses on minimizing search time, and Wilt and Ruml (2016) show that the best heuristics for satisficing search are different from the ones for optimal search. Burns et al. (2012) give some implementation tricks for writing fast search code, and Felner (2018) considers how the implementation changes when using an early goal test.

Pohl (1971) introduced bidirectional search. Holte et al. (2016) describe the version of bidirectional search that is guaranteed to meet in the middle, making it more widely applicable. (EckerLe et al. 2017) describe the set of surely expanded pairs of nodes, and show that no bidirectional search can be optimally efficient. The NBS algorithm (Chen et al., 2017) makes explicit use of a queue of pairs of nodes.

A combination of bidirectional A* and known landmarks was used to efficiently find driving routes for Microsoft’s online map service (Goldberg et al., 2006). After caching a set of paths between landmarks, the algorithm can find an optimal-cost path between any pair of points in a 24-million-point graph of the United States, searching less than 0.1% of the graph. Korf (1987) shows how to use subgoals, macro-operators, and abstraction to achieve remarkable speedups over previous techniques. Delling et al. (2009) describe how to use bidirectional search, landmarks, hierarchical structure, and other tricks to find driving routes. Anderson et al. (2008) describe a related technique, called coarse-to-fine search, which can be thought of as defining landmarks at various hierarchical levels of abstraction. Korf (1987) describes conditions under which coarse-to-fine search provides an exponential speedup. Knoblock (1991) provides experimental results and analysis to quantify the advantages of hierarchical search.

Coarse-to-fine search

A* and other state-space search algorithms are closely related to the branch-and-bound techniques that are widely used in operations research (Lawler and Wood, 1966; RaywardSmith et al., 1996). Kumar and Kanal (1988) attempt a “grand unification” of heuristic search, dynamic programming, and branch-and-bound techniques under the name of CDP—the “composite decision process.”

Branch-and-bound

Because most computers in the 1960s had only a few thousand words of main memory, memory-bounded heuristic search was an early research topic. The Graph Traverser (Doran and Michie, 1966), one of the earliest search programs, commits to an action after searching best-first up to the memory limit. IDA* (Korf, 1985b) was the first widely used lengthoptimal, memory-bounded heuristic search algorithm, and a large number of variants have been developed. An analysis of the efficiency of IDA* and of its difficulties with real-valued heuristics appears in Patrick et al., (1992).

Iterative expansion

The original version of RBFS (Korf, 1993) is actually somewhat more complicated than the algorithm shown in Figure 3.22 , which is actually closer to an independently developed algorithm called iterative expansion or IE (Russell, 1992). RBFS uses a lower bound as well as the upper bound; the two algorithms behave identically with admissible heuristics, but RBFS expands nodes in best-first order even with an inadmissible heuristic. The idea of keeping track of the best alternative path appeared earlier in Bratko’s (2009) elegant Prolog implementation of A* and in the DTA* algorithm (Russell and Wefald, 1991). The latter work also discusses metalevel state spaces and metalevel learning.

The MA* algorithm appeared in Chakrabarti et al. (1989). SMA*, or Simplified MA*, emerged from an attempt to implement MA* (Russell, 1992). Kaindl and Khorsand (1994 applied SMA* to produce a bidirectional search algorithm that was substantially faster than previous algorithms. Korf and Zhang (2000) describe a divide-and-conquer approach, and Zhou and Hansen, (2002) introduce memory-bounded A* graph search and a strategy for switching to breadth-first search to increase memory-efficiency (Zhou and Hansen, 2006).

The idea that admissible heuristics can be derived by problem relaxation appears in the seminal paper by Held and Karp (1970), who used the minimum-spanning-tree heuristic to solve the TSP. (See Exercise 3.MSTR.) The automation of the relaxation process was implemented successfully by Prieditis (1993). There is a growing literature on the application of machine learning to discover heuristic functions (Samadi et al., 2008; Arfaee et al., 2010; Thayer et al., 2011; Lelis et al., 2012).

The use of pattern databases to derive admissible heuristics is due to Gasser (1995) and Culberson and Schaeffer (1996, 1998); disjoint pattern databases are described by Korf and Felner, (2002); a similar method using symbolic patterns is due to Edelkamp (2009). Felner et al. (2007) show how to compress pattern databases to save space. The probabilistic interpretation of heuristics was investigated by Pearl (1984) and Hansson and Mayer (1989).

Pearl’s (1984) Heuristics and Edelkamp and Schrödl’s (2012) Heuristic Search are influential textbooks on search. Papers about new search algorithms appear at the International Symposium on Combinatorial Search (SoCS) and the International Conference on Automated Planning and Scheduling (ICAPS), as well as in general AI conferences such as AAAI and IJCAI, and journals such as Artificial Intelligence and Journal of the ACM.

Chapter 4 Search in Complex Environments In which we relax the simplifying assumptions of the previous chapter, to get closer to the real world. Chapter 3  addressed problems in fully observable, deterministic, static, known environments where the solution is a sequence of actions. In this chapter, we relax those constraints. We begin with the problem of finding a good state without worrying about the path to get there, covering both discrete (Section 4.1 ) and continuous (Section 4.2 ) states. Then we relax the assumptions of determinism (Section 4.3 ) and observability (Section 4.4 ). In a nondeterministic world, the agent will need a conditional plan and carry out different actions depending on what it observes—for example, stopping if the light is red and going if it is green. With partial observability, the agent will also need to keep track of the possible states it might be in. Finally, Section 4.5  guides the agent through an unknown space that it must learn as it goes, using online search.

4.1 Local Search and Optimization Problems In the search problems of Chapter 3  we wanted to find paths through the search space, such as a path from Arad to Bucharest. But sometimes we care only about the final state, not the path to get there. For example, in the 8-queens problem (Figure 4.3 ), we care only about finding a valid final configuration of 8 queens (because if you know the configuration, it is trivial to reconstruct the steps that created it). This is also true for many important applications such as integrated-circuit design, factory floor layout, job shop scheduling, automatic programming, telecommunications network optimization, crop planning, and portfolio management.

Local search algorithms operate by searching from a start state to neighboring states, without keeping track of the paths, nor the set of states that have been reached. That means they are not systematic—they might never explore a portion of the search space where a solution actually resides. However, they have two key advantages: (1) they use very little memory; and (2) they can often find reasonable solutions in large or infinite state spaces for which systematic algorithms are unsuitable.

Local search

Local search algorithms can also solve optimization problems, in which the aim is to find the best state according to an objective function.

Optimization problem

Objective function

State-space landscape

To understand local search, consider the states of a problem laid out in a state-space landscape, as shown in Figure 4.1 . Each point (state) in the landscape has an “elevation,” defined by the value of the objective function. If elevation corresponds to an objective function, then the aim is to find the highest peak—a global maximum—and we call the process hill climbing. If elevation corresponds to cost, then the aim is to find the lowest valley—a global minimum—and we call it gradient descent.

Figure 4.1

A one-dimensional state-space landscape in which elevation corresponds to the objective function. The aim is to find the global maximum.

Global maximum

Global minimum

4.1.1 Hill-climbing search The hill-climbing search algorithm is shown in Figure 4.2 . It keeps track of one current state and on each iteration moves to the neighboring state with highest value—that is, it heads in the direction that provides the steepest ascent. It terminates when it reaches a “peak” where no neighbor has a higher value. Hill climbing does not look ahead beyond the immediate neighbors of the current state. This resembles trying to find the top of Mount Everest in a thick fog while suffering from amnesia. Note that one way to use hill-climbing search is to use the negative of a heuristic cost function as the objective function; that will climb locally to the state with smallest heuristic distance to the goal.

Figure 4.2

The hill-climbing search algorithm, which is the most basic local search technique. At each step the current node is replaced by the best neighbor.

Hill-climbing

Steepest ascent

Complete-state formulation

To illustrate hill climbing, we will use the 8-queens problem (Figure 4.3 ). We will use a complete-state formulation, which means that every state has all the components of a

solution, but they might not all be in the right place. In this case every state has 8 queens on the board, one per column. The initial state is chosen at random, and the successors of a state are all possible states generated by moving a single queen to another square in the same column (so each state has 8 × 7 = 56 successors). The heuristic cost function h is the number of pairs of queens that are attacking each other; this will be zero only for solutions. (It counts as an attack if two pieces are in the same line, even if there is an intervening piece between them.) Figure 4.3(b)  shows a state that has h = 17. The figure also shows the h values of all its successors.

Figure 4.3

(a) The 8-queens problem: place 8 queens on a chess board so that no queen attacks another. (A queen attacks any piece in the same row, column, or diagonal.) This position is almost a solution, except for the two queens in the fourth and seventh columns that attack each other along the diagonal. (b) An 8-queens state with heuristic cost estimate h = 17. The board shows the value of h for each possible successor obtained by moving a queen within its column. There are 8 moves that are tied for best, with h = 12. The hill-climbing algorithm will pick one of these.

Greedy local search

Hill climbing is sometimes called greedy local search because it grabs a good neighbor state without thinking ahead about where to go next. Although greed is considered one of the

seven deadly sins, it turns out that greedy algorithms often perform quite well. Hill climbing can make rapid progress toward a solution because it is usually quite easy to improve a bad state. For example, from the state in Figure 4.3(b) , it takes just five steps to reach the state in Figure 4.3(a) , which has h = 1 and is very nearly a solution. Unfortunately, hill climbing can get stuck for any of the following reasons:

LOCAL MAXIMA: A local maximum is a peak that is higher than each of its neighboring states but lower than the global maximum. Hill-climbing algorithms that reach the vicinity of a local maximum will be drawn upward toward the peak but will then be stuck with nowhere else to go. Figure 4.1  illustrates the problem schematically. More concretely, the state in Figure 4.3(a)  is a local maximum (i.e., a local minimum for the cost h); every move of a single queen makes the situation worse.

Local maximum

RIDGES: A ridge is shown in Figure 4.4 . Ridges result in a sequence of local maxima that is very difficult for greedy algorithms to navigate.

Ridge

PLATEAUS: A plateau is a flat area of the state-space landscape. It can be a flat local maximum, from which no uphill exit exists, or a shoulder, from which progress is possible. (See Figure 4.1 .) A hill-climbing search can get lost wandering on the plateau.

Figure 4.4

Illustration of why ridges cause difficulties for hill climbing. The grid of states (dark circles) is superimposed on a ridge rising from left to right, creating a sequence of local maxima that are not directly connected to each other. From each local maximum, all the available actions point downhill. Topologies like this are common in low-dimensional state spaces, such as points in a two-dimensional plane. But in state spaces with hundreds or thousands of dimensions, this intuitive picture does not hold, and there are usually at least a few dimensions that make it possible to escape from ridges and plateaus.

Plateau

Shoulder

In each case, the algorithm reaches a point at which no progress is being made. Starting from a randomly generated 8-queens state, steepest-ascent hill climbing gets stuck 86% of the time, solving only 14% of problem instances. On the other hand, it works quickly, taking just 4 steps on average when it succeeds and 3 when it gets stuck—not bad for a state space with 88 ≈ 17 million states.

How could we solve more problems? One answer is to keep going when we reach a plateau —to allow a sideways move in the hope that the plateau is really a shoulder, as shown in Figure 4.1 . But if we are actually on a flat local maximum, then this approach will wander on the plateau forever. Therefore, we can limit the number of consecutive sideways moves, stopping after, say, 100 consecutive sideways moves. This raises the percentage of problem instances solved by hill climbing from 14% to 94%. Success comes at a cost: the algorithm averages roughly 21 steps for each successful instance and 64 for each failure.

Sideways move

Many variants of hill climbing have been invented. Stochastic hill climbing chooses at random from among the uphill moves; the probability of selection can vary with the steepness of the uphill move. This usually converges more slowly than steepest ascent, but in some state landscapes, it finds better solutions. First-choice hill climbing implements stochastic hill climbing by generating successors randomly until one is generated that is better than the current state. This is a good strategy when a state has many (e.g., thousands) of successors.

Stochastic hill climbing

First-choice hill climbing

Random-restart hill climbing

Another variant is random-restart hill climbing, which adopts the adage, “If at first you don’t succeed, try, try again.” It conducts a series of hill-climbing searches from randomly generated initial states, until a goal is found. It is complete with probability 1, because it will eventually generate a goal state as the initial state. If each hill-climbing search has a probability p of success, then the expected number of restarts required is 1/p. For 8-queens instances with no sideways moves allowed, p ≈ 0.14, so we need roughly 7 iterations to find a goal (6 failures and 1 success). The expected number of steps is the cost of one successful iteration plus (1 − p)/p times the cost of failure, or roughly 22 steps in all. When we allow sideways moves, 1/0.94 ≈ 1.06 iterations are needed on average and (1 × 21) + (0.06/0.94) × 64 ≈ 25 steps. For 8-queens, then, random-restart hill climbing is

very effective indeed. Even for three million queens, the approach can find solutions in seconds.1

1 Luby et al. (1993) suggest restarting after a fixed number of steps and show that this can be much more efficient than letting each search continue indefinitely.

The success of hill climbing depends very much on the shape of the state-space landscape: if there are few local maxima and plateaus, random-restart hill climbing will find a good solution very quickly. On the other hand, many real problems have a landscape that looks more like a widely scattered family of balding porcupines on a flat floor, with miniature porcupines living on the tip of each porcupine needle. NP-hard problems (see Appendix A ) typically have an exponential number of local maxima to get stuck on. Despite this, a reasonably good local maximum can often be found after a small number of restarts.

4.1.2 Simulated annealing A hill-climbing algorithm that never makes “downhill” moves toward states with lower value (or higher cost) is always vulnerable to getting stuck in a local maximum. In contrast, a purely random walk that moves to a successor state without concern for the value will eventually stumble upon the global maximum, but will be extremely inefficient. Therefore, it seems reasonable to try to combine hill climbing with a random walk in a way that yields both efficiency and completeness.

Simulated annealing is such an algorithm. In metallurgy, annealing is the process used to temper or harden metals and glass by heating them to a high temperature and then gradually cooling them, thus allowing the material to reach a low-energy crystalline state. To explain simulated annealing, we switch our point of view from hill climbing to gradient

descent (i.e., minimizing cost) and imagine the task of getting a ping-pong ball into the deepest crevice in a very bumpy surface. If we just let the ball roll, it will come to rest at a local minimum. If we shake the surface, we can bounce the ball out of the local minimum— perhaps into a deeper local minimum, where it will spend more time. The trick is to shake just hard enough to bounce the ball out of local minima but not hard enough to dislodge it from the global minimum. The simulated-annealing solution is to start by shaking hard (i.e., at a high temperature) and then gradually reduce the intensity of the shaking (i.e., lower the temperature).

Simulated annealing

The overall structure of the simulated-annealing algorithm (Figure 4.5 ) is similar to hill climbing. Instead of picking the best move, however, it picks a random move. If the move improves the situation, it is always accepted. Otherwise, the algorithm accepts the move with some probability less than 1. The probability decreases exponentially with the

E by which the evaluation is worsened. The probability also decreases as the “temperature” T goes down: “bad” moves are more likely to be allowed at the start when T is high, and they become more unlikely as T decreases. If the schedule lowers T to 0 slowly enough, then a property of the Boltzmann distribution, eΔE/T , “badness” of the move—the amount Δ

is that all the probability is concentrated on the global maxima, which the algorithm will find with probability approaching 1.

Figure 4.5

The simulated annealing algorithm, a version of stochastic hill climbing where some downhill moves are allowed. The schedule input determines the value of the “temperature” T as a function of time.

Simulated annealing was used to solve VLSI layout problems beginning in the 1980s. It has been applied widely to factory scheduling and other large-scale optimization tasks.

4.1.3 Local beam search Keeping just one node in memory might seem to be an extreme reaction to the problem of memory limitations. The local beam search algorithm keeps track of k states rather than just one. It begins with k randomly generated states. At each step, all the successors of all k states are generated. If any one is a goal, the algorithm halts. Otherwise, it selects the k best successors from the complete list and repeats.

Local beam search

At first sight, a local beam search with k states might seem to be nothing more than running

k random restarts in parallel instead of in sequence. In fact, the two algorithms are quite different. In a random-restart search, each search process runs independently of the others. In a local beam search, useful information is passed among the parallel search threads. In effect, the states that generate the best successors say to the others, “Come over here, the grass is greener!” The algorithm quickly abandons unfruitful searches and moves its resources to where the most progress is being made. Local beam search can suffer from a lack of diversity among the k states—they can become clustered in a small region of the state space, making the search little more than a k-timesslower version of hill climbing. A variant called stochastic beam search, analogous to stochastic hill climbing, helps alleviate this problem. Instead of choosing the top k successors, stochastic beam search chooses successors with probability proportional to the successor’s value, thus increasing diversity.

Stochastic beam search

4.1.4 Evolutionary algorithms

Evolutionary algorithms

recombination

Evolutionary algorithms can be seen as variants of stochastic beam search that are explicitly motivated by the metaphor of natural selection in biology: there is a population of individuals (states), in which the fittest (highest value) individuals produce offspring (successor states) that populate the next generation, a process called recombination. There are endless forms of evolutionary algorithms, varying in the following ways:

The size of the population. The representation of each individual. In genetic algorithms, each individual is a string over a finite alphabet (often a Boolean string), just as DNA is a string over the alphabet ACGT. In evolution strategies, an individual is a sequence of real numbers, and in genetic programming an individual is a computer program.

Genetic algorithm

Evolution strategies

Genetic programming

The mixing number, ρ, which is the number of parents that come together to form offspring. The most common case is ρ = 2: two parents combine their “genes” (parts of their representation) to form offspring. When ρ = 1 we have stochastic beam search (which can be seen as asexual reproduction). It is possible to have ρ > 2, which occurs only rarely in nature but is easy enough to simulate on computers. The selection process for selecting the individuals who will become the parents of the next generation: one possibility is to select from all individuals with probability proportional to their fitness score. Another possibility is to randomly select n individuals (n >

ρ), and then select the ρ most fit ones as parents.

Selection

The recombination procedure. One common approach (assuming

ρ = 2), is to randomly

select a crossover point to split each of the parent strings, and recombine the parts to form two children, one with the first part of parent 1 and the second part of parent 2; the other with the second part of parent 1 and the first part of parent 2.

Crossover point

The mutation rate, which determines how often offspring have random mutations to their representation. Once an offspring has been generated, every bit in its composition is flipped with probability equal to the mutation rate.

Mutation rate

The makeup of the next generation. This can be just the newly formed offspring, or it can include a few top-scoring parents from the previous generation (a practice called elitism, which guarantees that overall fitness will never decrease over time). The practice of culling, in which all individuals below a given threshold are discarded, can lead to a speedup (Baum et al., 1995).

Elitism

Figure 4.6(a)  shows a population of four 8-digit strings, each representing a state of the 8queens puzzle: the c-th digit represents the row number of the queen in column c. In (b), each state is rated by the fitness function. Higher fitness values are better, so for the 8queens problem we use the number of nonattacking pairs of queens, which has a value of

8 × 7/2 = 28 for a solution. The values of the four states in (b) are 24, 23, 20, and 11. The fitness scores are then normalized to probabilities, and the resulting values are shown next to the fitness values in (b).

Figure 4.6

A genetic algorithm, illustrated for digit strings representing 8-queens states. The initial population in (a) is ranked by a fitness function in (b) resulting in pairs for mating in (c). They produce offspring in (d), which are subject to mutation in (e).

In (c), two pairs of parents are selected, in accordance with the probabilities in (b). Notice that one individual is selected twice and one not at all. For each selected pair, a crossover point (dotted line) is chosen randomly. In (d), we cross over the parent strings at the crossover points, yielding new offspring. For example, the first child of the first pair gets the first three digits (327) from the first parent and the remaining digits (48552) from the second parent. The 8-queens states involved in this recombination step are shown in Figure 4.7 .

Figure 4.7

The 8-queens states corresponding to the first two parents in Figure 4.6(c)  and the first offspring in Figure 4.6(d) . The green columns are lost in the crossover step and the red columns are retained. (To

interpret the numbers in Figure 4.6 : row 1 is the bottom row, and 8 is the top row.)

Finally, in (e), each location in each string is subject to random mutation with a small independent probability. One digit was mutated in the first, third, and fourth offspring. In the 8-queens problem, this corresponds to choosing a queen at random and moving it to a random square in its column. It is often the case that the population is diverse early on in the process, so crossover frequently takes large steps in the state space early in the search process (as in simulated annealing). After many generations of selection towards higher fitness, the population becomes less diverse, and smaller steps are typical. Figure 4.8  describes an algorithm that implements all these steps.

Figure 4.8

A genetic algorithm. Within the function, population is an ordered list of individuals, weights is a list of corresponding fitness values for each individual, and fitness is a function to compute these values.

Genetic algorithms are similar to stochastic beam search, but with the addition of the crossover operation. This is advantageous if there are blocks that perform useful functions. For example, it could be that putting the first three queens in positions 2, 4, and 6 (where they do not attack each other) constitutes a useful block that can be combined with other useful blocks that appear in other individuals to construct a solution. It can be shown mathematically that, if the blocks do not serve a purpose—for example if the positions of the genetic code are randomly permuted—then crossover conveys no advantage.

The theory of genetic algorithms explains how this works using the idea of a schema, which is a substring in which some of the positions can be left unspecified. For example, the schema 246***** describes all 8-queens states in which the first three queens are in positions 2, 4, and 6, respectively. Strings that match the schema (such as 24613578) are called instances of the schema. It can be shown that if the average fitness of the instances of a schema is above the mean, then the number of instances of the schema will grow over time.

Schema

Instance

Evolution and Search The theory of evolution was developed by Charles Darwin in On the Origin of Species by Means of Natural Selection (1859) and independently by Alfred Russel Wallace (1858). The central idea is simple: variations occur in reproduction and will be preserved in successive generations approximately in proportion to their effect on reproductive fitness. Darwin’s theory was developed with no knowledge of how the traits of organisms can be inherited and modified. The probabilistic laws governing these processes were first identified by Gregor Mendel (1866), a monk who experimented with sweet peas. Much later, Watson and Crick (1953) identified the structure of the DNA molecule and its alphabet, AGTC (adenine, guanine, thymine, cytosine). In the standard model, variation occurs both by point mutations in the letter sequence and by “crossover” (in which the DNA of an offspring is generated by combining long sections of DNA from each parent). The analogy to local search algorithms has already been described; the principal difference between stochastic beam search and evolution is the use of sexual reproduction, wherein successors are generated from multiple individuals rather than just one. The actual mechanisms of evolution are, however, far richer than most genetic algorithms allow. For example, mutations can involve reversals, duplications, and movement of large chunks of DNA; some viruses borrow DNA from one organism and insert it into another; and there are transposable genes that do nothing but copy themselves many thousands of times within the genome. There are even genes that poison cells from potential mates that do not carry the gene, thereby increasing their own chances of replication. Most important is the fact that the genes themselves encode the mechanisms whereby the genome is reproduced and translated into an organism. In genetic algorithms, those

mechanisms are a separate program that is not represented within the strings being manipulated. Darwinian evolution may appear inefficient, having generated blindly some 10

43

or so organisms without improving its search heuristics one iota. But learning does play a role in evolution. Although the otherwise great French naturalist Jean Lamarck, (1809) was wrong to propose that traits acquired by adaptation during an organism’s lifetime would be passed on to its offspring, James Baldwin’s (1896) superficially similar theory is correct: learning can effectively relax the fitness landscape, leading to an acceleration in the rate of evolution. An organism that has a trait that is not quite adaptive for its environment will pass on the trait if it also has enough plasticity to learn to adapt to the environment in a way that is beneficial. Computer simulations (Hinton and Nowlan, 1987) confirm that this Baldwin effect is real, and that a consequence is that things that are hard to learn end up in the genome, but things that are easy to learn need not reside there (Morgan and Griffiths, 2015).

Clearly, this effect is unlikely to be significant if adjacent bits are totally unrelated to each other, because then there will be few contiguous blocks that provide a consistent benefit. Genetic algorithms work best when schemas correspond to meaningful components of a solution. For example, if the string is a representation of an antenna, then the schemas may represent components of the antenna, such as reflectors and deflectors. A good component is likely to be good in a variety of different designs. This suggests that successful use of genetic algorithms requires careful engineering of the representation.

In practice, genetic algorithms have their place within the broad landscape of optimization methods (Marler and Arora, 2004), particularly for complex structured problems such as circuit layout or job-shop scheduling, and more recently for evolving the architecture of deep neural networks (Miikkulainen et al., 2019). It is not clear how much of the appeal of genetic algorithms arises from their superiority on specific tasks, and how much from the appealing metaphor of evolution.

4.2 Local Search in Continuous Spaces In Chapter 2 , we explained the distinction between discrete and continuous environments, pointing out that most real-world environments are continuous. A continuous action space has an infinite branching factor, and thus can’t be handled by most of the algorithms we have covered so far (with the exception of first-choice hill climbing and simulated annealing).

This section provides a very brief introduction to some local search techniques for continuous spaces. The literature on this topic is vast; many of the basic techniques originated in the 17th century, after the development of calculus by Newton and Leibniz.2 We find uses for these techniques in several places in this book, including the chapters on learning, vision, and robotics. 2 Knowledge of vectors, matrices, and derivatives is useful for this section (see Appendix A ).

We begin with an example. Suppose we want to place three new airports anywhere in Romania, such that the sum of squared straight-line distances from each city on the map to its nearest airport is minimized. (See Figure 3.1  for the map of Romania.) The state space is then defined by the coordinates of the three airports: (

x ,y ), (x ,y ), and (x ,y ). This is a 1

1

2

2

3

3

six-dimensional space; we also say that states are defined by six variables. In general, states

n

are defined by an -dimensional vector of variables, x. Moving around in this space corresponds to moving one or more of the airports on the map. The objective function

f (x) = f (x ,y ,x ,y ,x ,y ) is relatively easy to compute for any particular state once we compute the closest cities. Let Ci be the set of cities whose closest airport (in the state x) is airport i. Then, we have 1

1

2

2

3

3

(4.1)

f ( x ) = f ( x ,y ,x ,y ,x ,y ) = 1

Variable

1

2

2

3

3

∑∑ x 3

i=1 c∈Ci

(

i − xc )

2

+(

yi − yc )

2

.

This equation is correct not only for the state x but also for states in the local neighborhood of x. However, it is not correct globally; if we stray too far from

x (by altering the location of

one or more of the airports by a large amount) then the set of closest cities for that airport changes, and we need to recompute

Ci .

One way to deal with a continuous state space is to discretize it. For example, instead of

x y

allowing the ( i , i ) locations to be any point in continuous two-dimensional space, we could limit them to fixed points on a rectangular grid with spacing of size

δ (delta). Then

instead of having an infinite number of successors, each state in the space would have only

δ

12 successors, corresponding to incrementing one of the 6 variables by ± . We can then apply any of our local search algorithms to this discrete space. Alternatively, we could make the branching factor finite by sampling successor states randomly, moving in a random

δ

direction by a small amount, . Methods that measure progress by the change in the value of the objective function between two nearby points are called empirical gradient methods. Empirical gradient search is the same as steepest-ascent hill climbing in a discretized version of the state space. Reducing the value of

δ over time can give a more accurate solution, but

does not necessarily converge to a global optimum in the limit.

Discretization

Empirical gradient

Often we have an objective function expressed in a mathematical form such that we can use calculus to solve the problem analytically rather than empirically. Many methods attempt to use the gradient of the landscape to find a maximum. The gradient of the objective function

f

is a vector ∇ that gives the magnitude and direction of the steepest slope. For our problem, we have

∇

f =(

f ∂x ∂

1

,

f ∂y ∂

1

,

f ∂x ∂

2

,

f ∂y ∂

2

,

f ∂x ∂

3

,

f ). ∂y ∂

3

Gradient

f

In some cases, we can find a maximum by solving the equation ∇ = 0. (This could be done, for example, if we were placing just one airport; the solution is the arithmetic mean of all the cities’ coordinates.) In many cases, however, this equation cannot be solved in closed form. For example, with three airports, the expression for the gradient depends on what cities are closest to each airport in the current state. This means we can compute the gradient locally (but not globally); for example,

(4.2)

f x

∂ ∂

=2

1

∑x

c ∈C

(

1

−

xc ) .

1

Given a locally correct expression for the gradient, we can perform steepest-ascent hill climbing by updating the current state according to the formula

x ← x + α ∇f ( x ) ,

α (alpha) is a small constant often called the step size. There exist a huge variety of methods for adjusting α. The basic problem is that if α is too small, too many steps are needed; if α is too large, the search could overshoot the maximum. The technique of line

where

search tries to overcome this dilemma by extending the current gradient direction—usually by repeatedly doubling

α—until f starts to decrease again. The point at which this occurs

becomes the new current state. There are several schools of thought about how the new direction should be chosen at this point.

Step size

Line search

For many problems, the most effective algorithm is the venerable Newton–Raphson method. This is a general technique for finding roots of functions—that is, solving equations

gx

of the form ( ) = 0. It works by computing a new estimate for the root

x according to

Newton’s formula

x ← x − g(x)/g (x) . ′

Newton–Raphson

f

To find a maximum or minimum of , we need to find x such that the gradient is a zero

f

gx

f

vector (i.e., ∇ (x) = 0). Thus, ( ) in Newton’s formula becomes ∇ (x), and the update equation can be written in matrix–vector form as

x ← x − H−1 f (x)∇f (x) , where Hf (x) is the Hessian matrix of second derivatives, whose elements ∂

2

Hij are given by

f /∂xi ∂xj . For our airport example, we can see from Equation (4.2)  that Hf (x) is

particularly simple: the off-diagonal elements are zero and the diagonal elements for airport

i are just twice the number of cities in Ci . A moment’s calculation shows that one step of the update moves airport i directly to the centroid of Ci , which is the minimum of the local expression for f from Equation (4.1) .3 For high-dimensional problems, however, computing the n entries of the Hessian and inverting it may be expensive, so many 2

approximate versions of the Newton–Raphson method have been developed.

3 In general, the Newton–Raphson update can be seen as fitting a quadratic surface to minimum of that surface—which is also the minimum of

f if f is quadratic.

f at x and then moving directly to the

Local search methods suffer from local maxima, ridges, and plateaus in continuous state spaces just as much as in discrete spaces. Random restarts and simulated annealing are often helpful. High-dimensional continuous spaces are, however, big places in which it is very easy to get lost.

Constrained optimization

Linear programming

A final topic is constrained optimization. An optimization problem is constrained if solutions must satisfy some hard constraints on the values of the variables. For example, in our airport-siting problem, we might constrain sites to be inside Romania and on dry land (rather than in the middle of lakes). The difficulty of constrained optimization problems depends on the nature of the constraints and the objective function. The best-known category is that of linear programming problems, in which constraints must be linear inequalities forming a convex set4 and the objective function is also linear. The time complexity of linear programming is polynomial in the number of variables. 4 A set of points S is convex if the line joining any two points in S is also contained in S . A convex function is one for which the space “above” it forms a convex set; by definition, convex functions have no local (as opposed to global) minima.

Convex set

Linear programming is probably the most widely studied and broadly useful method for optimization. It is a special case of the more general problem of convex optimization, which allows the constraint region to be any convex region and the objective to be any function that is convex within the constraint region. Under certain conditions, convex optimization problems are also polynomially solvable and may be feasible in practice with thousands of variables. Several important problems in machine learning and control theory can be formulated as convex optimization problems (see Chapter 20 ).

Convex optimization

4.3 Search with Nondeterministic Actions In Chapter 3 , we assumed a fully observable, deterministic, known environment. Therefore, an agent can observe the initial state, calculate a sequence of actions that reach the goal, and execute the actions with its “eyes closed,” never having to use its percepts.

When the environment is partially observable, however, the agent doesn’t know for sure what state it is in; and when the environment is nondeterministic, the agent doesn’t know what state it transitions to after taking an action. That means that rather than thinking “I’m in state s1 and if I do action a I’ll end up in state s2 ,” an agent will now be thinking “I’m either in state s1 or s3 , and if I do action a I’ll end up in state s2 ,s4 or s5 .” We call a set of physical states that the agent believes are possible a belief state.

Belief state

In partially observable and nondeterministic environments, the solution to a problem is no longer a sequence, but rather a conditional plan (sometimes called a contingency plan or a strategy) that specifies what to do depending on what percepts agent receives while executing the plan. We examine nondeterminism in this section and partial observability in the next.

Conditional plan

4.3.1 The erratic vacuum world The vacuum world from Chapter 2  has eight states, as shown in Figure 4.9 . There are three actions—Right, Left, and Suck—and the goal is to clean up all the dirt (states 7 and 8). If the environment is fully observable, deterministic, and completely known, then the problem

is easy to solve with any of the algorithms in Chapter 3 , and the solution is an action sequence. For example, if the initial state is 1, then the action sequence [Suck, Right, Suck] will reach a goal state, 8.

Figure 4.9

The eight possible states of the vacuum world; states 7 and 8 are goal states.

Now suppose that we introduce nondeterminism in the form of a powerful but erratic vacuum cleaner. In the erratic vacuum world, the Suck action works as follows:

When applied to a dirty square the action cleans the square and sometimes cleans up dirt in an adjacent square, too. When applied to a clean square the action sometimes deposits dirt on the carpet.5 5 We assume that most readers face similar problems and can sympathize with our agent. We apologize to owners of modern, efficient cleaning appliances who cannot take advantage of this pedagogical device.

To provide a precise formulation of this problem, we need to generalize the notion of a transition model from Chapter 3 . Instead of defining the transition model by a RESULT function that returns a single outcome state, we use a RESULTS function that returns a set of possible outcome states. For example, in the erratic vacuum world, the Suck action in state 1 cleans up either just the current location, or both locations:

RESULTS(1,

Suck) = {5,7}

If we start in state 1, no single sequence of actions solves the problem, but the following conditional plan does:

(4.3) [

Suck, if  State = 5 then [Right, Suck] else [ ]] .

Here we see that a conditional plan can contain if–then–else steps; this means that solutions are trees rather than sequences. Here the conditional in the if statement tests to see what the current state is; this is something the agent will be able to observe at runtime, but doesn’t know at planning time. Alternatively, we could have had a formulation that tests the percept rather than the state. Many problems in the real, physical world are contingency problems, because exact prediction of the future is impossible. For this reason, many people keep their eyes open while walking around.

4.3.2

AND–OR

search trees

How do we find these contingent solutions to nondeterministic problems? As in Chapter 3 , we begin by constructing search trees, but here the trees have a different character. In a deterministic environment, the only branching is introduced by the agent’s own choices in each state: I can do this action or that action. We call these nodes OR nodes. In the vacuum world, for example, at an OR node the agent chooses Left or Right or Suck. In a nondeterministic environment, branching is also introduced by the environment’s choice of outcome for each action. We call these nodes AND nodes. For example, the Suck action in state 1 results in the belief state {5,7}, so the agent would need to find a plan for state 5 and for state 7. These two kinds of nodes alternate, leading to an AND–OR tree as illustrated in Figure 4.10 .

Figure 4.10

The first two levels of the search tree for the erratic vacuum world. State nodes are OR nodes where some action must be chosen. At the AND nodes, shown as circles, every outcome must be handled, as indicated by the arc linking the outgoing branches. The solution found is shown in bold lines.

Or node

And node

And–or tree

A solution for an AND–OR search problem is a subtree of the complete search tree that (1) has a goal node at every leaf, (2) specifies one action at each of its OR nodes, and (3) includes every outcome branch at each of its AND nodes. The solution is shown in bold lines in the figure; it corresponds to the plan given in Equation (4.3) . Figure 4.11  gives a recursive, depth-first algorithm for AND–OR graph search. One key aspect of the algorithm is the way in which it deals with cycles, which often arise in nondeterministic problems (e.g., if an action sometimes has no effect or if an unintended effect can be corrected). If the current state is identical to a state on the path from the root, then it returns with failure. This doesn’t mean that there is no solution from the current state; it simply means that if there is a noncyclic solution, it must be reachable from the earlier incarnation of the current state, so the new incarnation can be discarded. With this check, we ensure that the algorithm terminates in every finite state space, because every path must reach a goal, a dead end, or a repeated state. Notice that the algorithm does not check whether the current state is a repetition of a state on some other path from the root, which is important for efficiency.

Figure 4.11

An algorithm for searching AND–OR graphs generated by nondeterministic environments. A solution is a conditional plan that considers every nondeterministic outcome and makes a plan for each one.

AND–OR

graphs can be explored either breadth-first or best-first. The concept of a heuristic

function must be modified to estimate the cost of a contingent solution rather than a sequence, but the notion of admissibility carries over and there is an analog of the A* algorithm for finding optimal solutions. (See the bibliographical notes at the end of the chapter.)

4.3.3 Try, try again Consider a slippery vacuum world, which is identical to the ordinary (non-erratic) vacuum world except that movement actions sometimes fail, leaving the agent in the same location. For example, moving Right in state 1 leads to the belief state {1,2}. Figure 4.12  shows part of the search graph; clearly, there are no longer any acyclic solutions from state 1, and ANDOR-SEARCH would return with failure. There is, however, a cyclic solution, which is to keep trying Right until it works. We can express this with a new while construct:

[

Suck,while State = 5 do Right, Suck]

Cyclic solution

or by adding a label to denote some portion of the plan and referring to that label later:

[

Suck,L

1

:  

Right, if State = 5 then L  else Suck] . 1

When is a cyclic plan a solution? A minimum condition is that every leaf is a goal state and that a leaf is reachable from every point in the plan. In addition to that, we need to consider the cause of the nondeterminism. If it is really the case that the vacuum robot’s drive mechanism works some of the time, but randomly and independently slips on other occasions, then the agent can be confident that if the action is repeated enough times, eventually it will work and the plan will succeed. But if the nondeterminism is due to some unobserved fact about the robot or environment—perhaps a drive belt has snapped and the robot will never move—then repeating the action will not help.

Figure 4.12

Part of the search graph for a slippery vacuum world, where we have shown (some) cycles explicitly. All solutions for this problem are cyclic plans because there is no way to move reliably.

One way to understand this decision is to say that the initial problem formulation (fully observable, nondeterministic) is abandoned in favor of a different formulation (partially observable, deterministic) where the failure of the cyclic plan is attributed to an unobserved property of the drive belt. In Chapter 12  we discuss how to decide which of several uncertain possibilities is more likely.

4.4 Search in Partially Observable Environments We now turn to the problem of partial observability, where the agent’s percepts are not enough to pin down the exact state. That means that some of the agent’s actions will be aimed at reducing uncertainty about the current state.

4.4.1 Searching with no observation When the agent’s percepts provide no information at all, we have what is called a sensorless problem (or a conformant problem). At first, you might think the sensorless agent has no hope of solving a problem if it has no idea what state it starts in, but sensorless solutions are surprisingly common and useful, primarily because they don’t rely on sensors working properly. In manufacturing systems, for example, many ingenious methods have been developed for orienting parts correctly from an unknown initial position by using a sequence of actions with no sensing at all. Sometimes a sensorless plan is better even when a conditional plan with sensing is available. For example, doctors often prescribe a broadspectrum antibiotic rather than using the conditional plan of doing a blood test, then waiting for the results to come back, and then prescribing a more specific antibiotic. The sensorless plan saves time and money, and avoids the risk of the infection worsening before the test results are available.

Sensorless

Conformant

Consider a sensorless version of the (deterministic) vacuum world. Assume that the agent knows the geography of its world, but not its own location or the distribution of dirt. In that case, its initial belief state is {1,2,3,4,5,6,7,8} (see Figure 4.9 ). Now, if the agent moves Right it will be in one of the states {2,4,6,8}—the agent has gained information without

perceiving anything! After [Right,Suck] the agent will always end up in one of the states {4,8}. Finally, after [Right,Suck,Left,Suck] the agent is guaranteed to reach the goal state 7, no

matter what the start state. We say that the agent can coerce the world into state 7.

Coercion

The solution to a sensorless problem is a sequence of actions, not a conditional plan (because there is no perceiving). But we search in the space of belief states rather than physical states.6 In belief-state space, the problem is fully observable because the agent always knows its own belief state. Furthermore, the solution (if any) for a sensorless problem is always a sequence of actions. This is because, as in the ordinary problems of Chapter 3 , the percepts received after each action are completely predictable—they’re always empty! So there are no contingencies to plan for. This is true even if the environment is nondeterministic. 6 In a fully observable environment, each belief state contains one physical state. Thus, we can view the algorithms in Chapter 3  as searching in a belief-state space of singleton belief states.

We could introduce new algorithms for sensorless search problems. But instead, we can use the existing algorithms from Chapter 3  if we transform the underlying physical problem into a belief-state problem, in which we search over belief states rather than physical states. The original problem,

P , has components ActionsP , ResultP etc., and the belief-state

problem has the following components:

STATES: The belief-state space contains every possible subset of the physical states. If

P has N states, then the belief-state problem has 2N belief states, although many of those may be unreachable from the initial state. INITIAL STATE: Typically the belief state consisting of all states in

P , although in some

cases the agent will have more knowledge than this. ACTIONS: This is slightly tricky. Suppose the agent is in belief state ACTIONSP (

s )≠A 1

CTIONS

b = {s ,s }, but 1

2

P (s2 ); then the agent is unsure of which actions are legal. If we

assume that illegal actions have no effect on the environment, then it is safe to take the

b

union of all the actions in any of the physical states in the current belief state :

b

s

ACTIONS( ) = ∪ ACTIONSP ( ) .

s∈b

On the other hand, if an illegal action might lead to catastrophe, it is safer to allow only the intersection, that is, the set of actions legal in all the states. For the vacuum world, every state has the same legal actions, so both methods give the same result. TRANSITION MODEL: For deterministic actions, the new belief state has one result state for each of the current possible states (although some result states may be the same):

b′ = RESULT(b, a) = {s′ : s′ = RESULTP (s,a) and s ∈ b} . (4.4) With nondeterminism, the new belief state consists of all the possible results of applying the action to any of the states in the current belief state:

b′ = RESULT(b, a)

= =

s′ : s′ ∈ RESULTSP (s,a) and s ∈ b} ∪ RESULTSP (s,a) ,

{

s∈b

The size of b′ will be the same or smaller than b for deterministic actions, but may be larger than b with nondeterministic actions (see Figure 4.13 ). Figure 4.13

(a) Predicting the next belief state for the sensorless vacuum world with the deterministic action, Right. (b) Prediction for the same belief state and action in the slippery version of the sensorless vacuum world.

GOAL TEST: The agent possibly achieves the goal if any state s in the belief state satisfies the goal test of the underlying problem, IS-GOALP (s). The agent necessarily achieves the goal if every state satisfies IS-GOALP (s). We aim to necessarily achieve the goal. ACTION COST: This is also tricky. If the same action can have different costs in different states, then the cost of taking an action in a given belief state could be one of several values. (This gives rise to a new class of problems, which we explore in Exercise 4.MVAL.) For now we assume that the cost of an action is the same in all states and so can be transferred directly from the underlying physical problem. Figure 4.14  shows the reachable belief-state space for the deterministic, sensorless vacuum 8

world. There are only 12 reachable belief states out of 2 = 256 possible belief states.

Figure 4.14

The reachable portion of the belief-state space for the deterministic, sensorless vacuum world. Each rectangular box corresponds to a single belief state. At any given point, the agent has a belief state but

does not know which physical state it is in. The initial belief state (complete ignorance) is the top center box.

The preceding definitions enable the automatic construction of the belief-state problem formulation from the definition of the underlying physical problem. Once this is done, we can solve sensorless problems with any of the ordinary search algorithms of Chapter 3 .

In ordinary graph search, newly reached states are tested to see if they were previously reached. This works for belief states, too; for example, in Figure 4.14 , the action sequence [Suck,Left,Suck] starting at the initial state reaches the same belief state as [Right,Left,Suck], namely, {5,7}. Now, consider the belief state reached by [Left], namely, {1,3,5,7}. Obviously, this is not identical to {5,7}, but it is a superset. We can discard (prune) any such superset belief state. Why? Because a solution from {1,3,5,7} must be a solution for each of the individual states 1, 3, 5, and 7, and thus it is a solution for any combination of these individual states, such as {5,7}; therefore we don’t need to try to solve {1,3,5,7}, we can concentrate on trying to solve the strictly easier belief state {5,7}.

Conversely, if {1,3,5,7} has already been generated and found to be solvable, then any subset, such as {5,7}, is guaranteed to be solvable. (If I have a solution that works when I’m very confused about what state I’m in, it will still work when I’m less confused.) This extra level of pruning may dramatically improve the efficiency of sensorless problem solving.

Even with this improvement, however, sensorless problem-solving as we have described it is seldom feasible in practice. One issue is the vastness of the belief-state space—we saw in the previous chapter that often a search space of size

N

N is too large, and now we have

search spaces of size 2 . Furthermore, each element of the search space is a set of up to elements. For large

N , we won’t be able to represent even a single belief state without

N

running out of memory space.

One solution is to represent the belief state by some more compact description. In English, we could say the agent knows “Nothing” in the initial state; after moving Left, we could say, “Not in the rightmost column,” and so on. Chapter 7  explains how to do this in a formal representation scheme.

Another approach is to avoid the standard search algorithms, which treat belief states as black boxes just like any other problem state. Instead, we can look inside the belief states

and develop incremental belief-state search algorithms that build up the solution ol_lowerromann the sensorless vacuum world, the initial belief state is {1,2,3,4,5,6,7,8}, and we have to find an action sequence that works in all 8 states. We can do this by first finding a solution that works for state 1; then we check if it works for state 2; if not, go back and find a different solution for state 1, and so on.

Incremental belief-state search

Just as an AND–OR search has to find a solution for every branch at an AND node, this algorithm has to find a solution for every state in the belief state; the difference is that AND– OR

search can find a different solution for each branch, whereas an incremental belief-state

search has to find one solution that works for all the states.

The main advantage of the incremental approach is that it is typically able to detect failure quickly—when a belief state is unsolvable, it is usually the case that a small subset of the belief state, consisting of the first few states examined, is also unsolvable. In some cases, this leads to a speedup proportional to the size of the belief states, which may themselves be as large as the physical state space itself.

4.4.2 Searching in partially observable environments Many problems cannot be solved without sensing. For example, the sensorless 8-puzzle is impossible. On the other hand, a little bit of sensing can go a long way: we can solve 8puzzles if we can see just the upper-left corner square. The solution involves moving each tile in turn into the observable square and keeping track of its location from then on .

For a partially observable problem, the problem specification will specify a PERCEPT(s) function that returns the percept received by the agent in a given state. If sensing is nondeterministic, then we can use a PERCEPTS function that returns a set of possible percepts. For fully observable problems, PERCEPT(s) = PERCEPT(s) =

null.

s

for every state s, and for sensorless problems

Consider a local-sensing vacuum world, in which the agent has a position sensor that yields the percept L in the left square, and R in the right square, and a dirt sensor that yields Dirty when the current square is dirty and Clean when it is clean. Thus, the PERCEPT in state 1 is [L,Dirty]. With partial observability, it will usually be the case that several states produce the same percept; state 3 will also produce [L,Dirty]. Hence, given this initial percept, the initial belief state will be {1,3}. We can think of the transition model between belief states for partially observable problems as occurring in three stages, as shown in Figure 4.15 : The prediction stage computes the belief state resulting from the action, RESULT(b, a), exactly as we did with sensorless problems. To emphasize that this is a prediction, we

b = RESULT(b, a), where the “hat” over the b means “estimated,” and we use the notation ˆ also use PREDICT(b, a) as a synonym for RESULT(b, a).

The possible percepts stage computes the set of percepts that could be observed in the predicted belief state (using the letter o for observation):

b

o o = PERCEPT(s) and s ∈ ˆb} .

POSSIBLE-PERCEPTS(ˆ) = { :

The update stage computes, for each possible percept, the belief state that would result from the percept. The updated belief state bo is the set of states in ˆ b that could have produced the percept:

bo = UPDATE(ˆb,o) = {s : o = PERCEPT(s) and s ∈ ˆb} . The agent needs to deal with possible percepts at planning time, because it won’t know the actual percepts until it executes the plan. Notice that nondeterminism in the physical environment can enlarge the belief state in the prediction stage, but each updated belief state bo can be no larger than the predicted belief state ˆ b; observations can only help reduce uncertainty. Moreover, for deterministic sensing, the belief states for the different possible percepts will be disjoint, forming a partition of the original predicted belief state.

Figure 4.15

Two examples of transitions in local-sensing vacuum worlds. (a) In the deterministic world, Right is applied in the initial belief state, resulting in a new predicted belief state with two possible physical states; for those states, the possible percepts are [R,Dirty] and [R,Clean], leading to two belief states, each of which is a singleton. (b) In the slippery world, Right is applied in the initial belief state, giving a new belief state with four physical states; for those states, the possible percepts are [L,Dirty], [R,Dirty], and [R,Clean], leading to three belief states as shown.

Putting these three stages together, we obtain the possible belief states resulting from a given action and the subsequent possible percepts:

(4.5)

ba

RESULTS( , ) = {

bo : bo = U o∈P

PDATE(PREDICT(

b, a),o) and  (P (b, a))} .

OSSIBLE-PERCEPTS

REDICT

4.4.3 Solving partially observable problems The preceding section showed how to derive the RESULTS function for a nondeterministic belief-state problem from an underlying physical problem, given the PERCEPT function. With this formulation, the AND–OR search algorithm of Figure 4.11  can be applied directly to derive a solution. Figure 4.16  shows part of the search tree for the local-sensing vacuum world, assuming an initial percept [A,Dirty]. The solution is the conditional plan [

Suck,Right, if Bstate = {6} then Suck else [ ]] .

Figure 4.16

The first level of the AND–OR search tree for a problem in the local-sensing vacuum world; Suck is the first action in the solution.

Notice that, because we supplied a belief-state problem to the AND–OR search algorithm, it returned a conditional plan that tests the belief state rather than the actual state. This is as it should be: in a partially observable environment the agent won’t know the actual state.

As in the case of standard search algorithms applied to sensorless problems, the AND–OR search algorithm treats belief states as black boxes, just like any other states. One can improve on this by checking for previously generated belief states that are subsets or supersets of the current state, just as for sensorless problems. One can also derive incremental search algorithms, analogous to those described for sensorless problems, that provide substantial speedups over the black-box approach.

4.4.4 An agent for partially observable environments An agent for partially observable environments formulates a problem, calls a search algorithm (such as AND-OR-SEARCH) to solve it, and executes the solution. There are two main differences between this agent and the one for fully observable deterministic environments. First, the solution will be a conditional plan rather than a sequence; to execute an if–then–else expression, the agent will need to test the condition and execute the appropriate branch of the conditional. Second, the agent will need to maintain its belief state as it performs actions and receives percepts. This process resembles the prediction– observation–update process in Equation (4.5)  but is actually simpler because the percept is given by the environment rather than calculated by the agent. Given an initial belief state b, an action a, and a percept o, the new belief state is: (4.6)

b′

= UPDATE(PREDICT(b, a),o) .

Consider a kindergarten vacuum world wherein agents sense only the state of their current square, and any square may become dirty at any time unless the agent is actively cleaning it at that moment.7 Figure 4.17  shows the belief state being maintained in this environment.

7 The usual apologies to those who are unfamiliar with the effect of small children on the environment.

Figure 4.17

Two prediction–update cycles of belief-state maintenance in the kindergarten vacuum world with local sensing.

Monitoring

Filtering

State estimation

In partially observable environments—which include the vast majority of real-world environments—maintaining one’s belief state is a core function of any intelligent system. This function goes under various names, including monitoring, filtering, and state estimation.Equation (4.6)  is called a recursive state estimator because it computes the new belief state from the previous one rather than by examining the entire percept sequence. If the agent is not to “fall behind,” the computation has to happen as fast as percepts are coming in. As the environment becomes more complex, the agent will only have time to compute an approximate belief state, perhaps focusing on the implications of the percept for the aspects of the environment that are of current interest. Most work on this problem has been done for stochastic, continuous-state environments with the tools of probability theory, as explained in Chapter 14 .

In this section we will show an example in a discrete environment with deterministic sensors and nondeterministic actions. The example concerns a robot with a particular state estimation task called localization: working out where it is, given a map of the world and a sequence of percepts and actions. Our robot is placed in the maze-like environment of Figure 4.18 . The robot is equipped with four sonar sensors that tell whether there is an obstacle—the outer wall or a dark shaded square in the figure—in each of the four compass directions. The percept is in the form of a bit vector, one bit for each of the directions north, east, south, and west in that order, so 1011 means there are obstacles to the north, south, and west, but not east.

Figure 4.18

E

Possible positions of the robot, ⊙, (a) after one observation, 1 = 1011, and (b) after moving one square and making a second observation, 2 = 1010. When sensors are noiseless and the transition model is

E

accurate, there is only one possible location for the robot consistent with this sequence of two observations.

Localization

We assume that the sensors give perfectly correct data, and that the robot has a correct map of the environment. But unfortunately, the robot’s navigational system is broken, so when it executes a Right action, it moves randomly to one of the adjacent squares. The robot’s task is to determine its current location.

Suppose the robot has just been switched on, and it does not know where it is—its initial

b consists of the set of all locations. The robot then receives the percept 1011 and does an update using the equation bo = U (1011), yielding the 4 locations shown in belief state

PDATE

Figure 4.18(a) . You can inspect the maze to see that those are the only four locations that yield the percept 1011.

Next the robot executes a Right action, but the result is nondeterministic. The new belief

ba = P (bo ,Right), contains all the locations that are one step away from the (ba ,1010) and locations in bo . When the second percept, 1010, arrives, the robot does U state,

REDICT

PDATE

finds that the belief state has collapsed down to the single location shown in Figure 4.18(b) . That’s the only location that could be the result of

b

UPDATE(PREDICT(UPDATE( ,1011),

Right),1010) .

With nondeterministic actions the PREDICT step grows the belief state, but the UPDATE step shrinks it back down—as long as the percepts provide some useful identifying information. Sometimes the percepts don’t help much for localization: If there were one or more long east-west corridors, then a robot could receive a long sequence of 1010 percepts, but never know where in the corridor(s) it was. But for environments with reasonable variation in geography, localization often converges quickly to a single point, even when actions are nondeterministic.

What happens if the sensors are faulty? If we can reason only with Boolean logic, then we have to treat every sensor bit as being either correct or incorrect, which is the same as having no perceptual information at all. But we will see that probabilistic reasoning (Chapter 12 ), allows us to extract useful information from a faulty sensor as long as it is wrong less than half the time.

4.5 Online Search Agents and Unknown Environments

Offline search

Online search

So far we have concentrated on agents that use offline search algorithms. They compute a complete solution before taking their first action. In contrast, an online search8 agent interleaves computation and action: first it takes an action, then it observes the environment and computes the next action. Online search is a good idea in dynamic or semi-dynamic environments, where there is a penalty for sitting around and computing too long. Online search is also helpful in nondeterministic domains because it allows the agent to focus its computational efforts on the contingencies that actually arise rather than those that might happen but probably won’t.

8 The term “online” here refers to algorithms that must process input as it is received rather than waiting for the entire input data set to become available. This usage of “online” is unrelated to the concept of “having an Internet connection.”

Of course, there is a tradeoff: the more an agent plans ahead, the less often it will find itself up the creek without a paddle. In unknown environments, where the agent does not know what states exist or what its actions do, the agent must use its actions as experiments in order to learn about the environment.

A canonical example of online search is the mapping problem: a robot is placed in an unknown building and must explore to build a map that can later be used for getting from to

B. Methods for escaping from labyrinths—required knowledge for aspiring heroes of

A

antiquity—are also examples of online search algorithms. Spatial exploration is not the only

form of online exploration, however. Consider a newborn baby: it has many possible actions but knows the outcomes of none of them, and it has experienced only a few of the possible states that it can reach.

Mapping problem

4.5.1 Online search problems An online search problem is solved by interleaving computation, sensing, and acting. We’ll start by assuming a deterministic and fully observable environment (Chapter 17  relaxes these assumptions) and stipulate that the agent knows only the following:

ACTIONS(s), the legal actions in state s;

c(s,a,s′ ), the cost of applying action a in state s to arrive at state s′ . Note that this cannot be used until the agent knows that s′ is the outcome. Is-GOAL(s), the goal test.

Note in particular that the agent cannot determine RESULT(s, a) except by actually being in s

and doing a. For example, in the maze problem shown in Figure 4.19 , the agent does not

know that going Up from (1,1) leads to (1,2); nor, having done that, does it know that going Down will take it back to (1,1). This degree of ignorance can be reduced in some applications—for example, a robot explorer might know how its movement actions work and be ignorant only of the locations of obstacles.

Figure 4.19

A simple maze problem. The agent starts at

S and must reach G but knows nothing of the environment.

hs

Finally, the agent might have access to an admissible heuristic function ( ) that estimates the distance from the current state to a goal state. For example, in Figure 4.19 , the agent might know the location of the goal and be able to use the Manhattan-distance heuristic (page 97).

Typically, the agent’s objective is to reach a goal state while minimizing cost. (Another possible objective is simply to explore the entire environment.) The cost is the total path cost that the agent incurs as it travels. It is common to compare this cost with the path cost the agent would incur if it knew the search space in advance—that is, the optimal path in the known environment. In the language of online algorithms, this comparison is called the competitive ratio; we would like it to be as small as possible.

Competitive ratio

Dead end

Online explorers are vulnerable to dead ends: states from which no goal state is reachable. If the agent doesn’t know what each action does, it might execute the “jump into bottomless pit” action, and thus never reach the goal. In general, no algorithm can avoid dead ends in all

state spaces. Consider the two dead-end state spaces in Figure 4.20(a) . An online search algorithm that has visited states

S and A cannot tell if it is in the top state or the bottom

one; the two look identical based on what the agent has seen. Therefore, there is no way it could know how to choose the correct action in both state spaces. This is an example of an adversary argument—we can imagine an adversary constructing the state space while the agent explores it and putting the goals and dead ends wherever it chooses, as in Figure 4.20(b) .

Figure 4.20

(a) Two state spaces that might lead an online search agent into a dead end. Any given agent will fail in at least one of these spaces. (b) A two-dimensional environment that can cause an online search agent to follow an arbitrarily inefficient route to the goal. Whichever choice the agent makes, the adversary blocks that route with another long, thin wall, so that the path followed is much longer than the best possible path.

Adversary argument

Irreversible action

Safely explorable

Dead ends are a real difficulty for robot exploration—staircases, ramps, cliffs, one-way streets, and even natural terrain all present states from which some actions are irreversible —there is no way to return to the previous state. The exploration algorithm we will present is only guaranteed to work in state spaces that are safely explorable—that is, some goal state is reachable from every reachable state. State spaces with only reversible actions, such as mazes and 8-puzzles, are clearly safely explorable (if they have any solution at all). We will cover the subject of safe exploration in more depth in Section 22.3.2 .

Even in safely explorable environments, no bounded competitive ratio can be guaranteed if there are paths of unbounded cost. This is easy to show in environments with irreversible actions, but in fact it remains true for the reversible case as well, as Figure 4.20(b)  shows. For this reason, it is common to characterize the performance of online search algorithms in terms of the size of the entire state space rather than just the depth of the shallowest goal.

4.5.2 Online search agents After each action, an online agent in an observable environment receives a percept telling it what state it has reached; from this information, it can augment its map of the environment. The updated map is then used to plan where to go next. This interleaving of planning and action means that online search algorithms are quite different from the offline search algorithms we have seen previously: offline algorithms explore their model of the state space, while online algorithms explore the real world. For example, A* can expand a node in one part of the space and then immediately expand a node in a distant part of the space, because node expansion involves simulated rather than real actions.

An online algorithm, on the other hand, can discover successors only for a state that it physically occupies. To avoid traveling all the way to a distant state to expand the next node, it seems better to expand nodes in a local order. Depth-first search has exactly this property because (except when the algorithm is backtracking) the next node expanded is a child of the previous node expanded.

An online depth-first exploration agent (for deterministic but unknown actions) is shown in Figure 4.21 . This agent stores its map in a table, result[s,a], that records the state resulting from executing action a in state s. (For nondeterministic actions, the agent could record a set of states under result[s,a].) Whenever the current state has unexplored actions, the agent tries one of those actions. The difficulty comes when the agent has tried all the actions in a state. In offline depth-first search, the state is simply dropped from the queue; in an online search, the agent has to backtrack in the physical world. In depth-first search, this means going back to the state from which the agent most recently entered the current state. To achieve that, the algorithm keeps another table that lists, for each state, the predecessor states to which the agent has not yet backtracked. If the agent has run out of states to which it can backtrack, then its search is complete.

Figure 4.21

An online search agent that uses depth-first exploration. The agent can safely explore only in state spaces in which every action can be “undone” by some other action.

We recommend that the reader trace through the progress of ONLINE-DFS-AGENT when applied to the maze given in Figure 4.19 . It is fairly easy to see that the agent will, in the worst case, end up traversing every link in the state space exactly twice. For exploration, this is optimal; for finding a goal, on the other hand, the agent’s competitive ratio could be arbitrarily bad if it goes off on a long excursion when there is a goal right next to the initial

state. An online variant of iterative deepening solves this problem; for an environment that is a uniform tree, the competitive ratio of such an agent is a small constant.

Because of its method of backtracking, ONLINE-DFS-AGENT works only in state spaces where the actions are reversible. There are slightly more complex algorithms that work in general state spaces, but no such algorithm has a bounded competitive ratio.

4.5.3 Online local search Like depth-first search, hill-climbing search has the property of locality in its node expansions. In fact, because it keeps just one current state in memory, hill-climbing search is already an online search algorithm! Unfortunately, the basic algorithm is not very good for exploration because it leaves the agent sitting at local maxima with nowhere to go. Moreover, random restarts cannot be used, because the agent cannot teleport itself to a new start state.

Random walk

Instead of random restarts, one might consider using a random walk to explore the environment. A random walk simply selects at random one of the available actions from the current state; preference can be given to actions that have not yet been tried. It is easy to prove that a random walk will eventually find a goal or complete its exploration, provided that the space is finite and safely explorable.9 On the other hand, the process can be very slow. Figure 4.22  shows an environment in which a random walk will take exponentially many steps to find the goal, because, for each state in the top row except S, backward progress is twice as likely as forward progress. The example is contrived, of course, but there are many real-world state spaces whose topology causes these kinds of “traps” for random walks.

9 Random walks are complete on infinite one-dimensional and two-dimensional grids. On a three-dimensional grid, the probability that the walk ever returns to the starting point is only about 0.3405 (Hughes, 1995).

Figure 4.22

An environment in which a random walk will take exponentially many steps to find the goal.

Augmenting hill climbing with memory rather than randomness turns out to be a more effective approach. The basic idea is to store a “current best estimate” reach the goal from each state that has been visited.

hs

H (s) of the cost to

H (s) starts out being just the heuristic

estimate ( ) and is updated as the agent gains experience in the state space. Figure 4.23  shows a simple example in a one-dimensional state space. In (a), the agent seems to be stuck in a flat local minimum at the red state. Rather than staying where it is, the agent should follow what seems to be the best path to the goal given the current cost estimates for its neighbors. The estimated cost to reach the goal through a neighbor cost to get to

s

′

csas

s

plus the estimated cost to get to a goal from there—that is, ( , , ′ ) +

′

is the

H (s ) . ′

In the example, there are two actions, with estimated costs 1 + 9 to the left and 1 + 2 to the right, so it seems best to move right.

Figure 4.23

Hs

Five iterations of LRTA* on a one-dimensional state space. Each state is labeled with ( ), the current cost estimate to reach a goal, and every link has an action cost of 1. The red state marks the location of

the agent, and the updated cost estimates at each iteration have a double circle.

In (b) it is clear that the cost estimate of 2 for the red state in (a) was overly optimistic. Since the best move cost 1 and led to a state that is at least 2 steps from a goal, the red state must be at least 3 steps from a goal, so its

H should be updated accordingly, as shown in Figure

4.23(b) . Continuing this process, the agent will move back and forth twice more, updating

H each time and “flattening out” the local minimum until it escapes to the right.

LRTA*

Optimism under uncertainty

An agent implementing this scheme, which is called learning real-time A* (LRTA*), is shown in Figure 4.24 . Like ONLINE-DFS-AGENT, it builds a map of the environment in the result table. It updates the cost estimate for the state it has just left and then chooses the “apparently best” move according to its current cost estimates. One important detail is that

s are always assumed to lead immediately to the goal with the least possible cost, namely h(s). This optimism under uncertainty actions that have not yet been tried in a state

encourages the agent to explore new, possibly promising paths.

Figure 4.24

LRTA*-AGENT selects an action according to the values of neighboring states, which are updated as the agent moves about the state space.

An LRTA* agent is guaranteed to find a goal in any finite, safely explorable environment. Unlike A*, however, it is not complete for infinite state spaces—there are cases where it can be led infinitely astray. It can explore an environment of

n states in O(n ) steps in the worst 2

case, but often does much better. The LRTA* agent is just one of a large family of online agents that one can define by specifying the action selection rule and the update rule in different ways. We discuss this family, developed originally for stochastic environments, in Chapter 22 .

4.5.4 Learning in online search The initial ignorance of online search agents provides several opportunities for learning. First, the agents learn a “map” of the environment—more precisely, the outcome of each action in each state—simply by recording each of their experiences. Second, the local search agents acquire more accurate estimates of the cost of each state by using local updating rules, as in LRTA*. In Chapter 22 , we show that these updates eventually converge to exact values for every state, provided that the agent explores the state space in the right way.

Once exact values are known, optimal decisions can be taken simply by moving to the lowest-cost successor—that is, pure hill climbing is then an optimal strategy.

If you followed our suggestion to trace the behavior of ONLINE-DFS-AGENT in the environment of Figure 4.19 , you will have noticed that the agent is not very bright. For example, after it has seen that the Up action goes from (1,1) to (1,2), the agent still has no idea that the Down action goes back to (1,1) or that the Up action also goes from (2,1) to (2,2), from (2,2) to (2,3), and so on. In general, we would like the agent to learn that Up increases the y-coordinate unless there is a wall in the way, that Down reduces it, and so on.

For this to happen, we need two things. First, we need a formal and explicitly manipulable representation for these kinds of general rules; so far, we have hidden the information inside the black box called the RESULT function. Chapters 8  to 11  are devoted to this issue. Second, we need algorithms that can construct suitable general rules from the specific observations made by the agent. These are covered in Chapter 19 .

If we anticipate that we will be called upon to solve multiple similar problems in the future then it makes sense to invest time (and memory) to make those future searches easier. There are several ways to do this, all falling under the heading of incremental search. We could keep the search tree in memory and reuse the parts of it that are unchanged in the new problem. We could keep the heuristic h values and update them as we gain new information—either because the world has changed or because we have computed a better estimate. Or we could keep the best-path g values, using them to piece together a new solution, and updating them when the world changes.

Incremental search

Summary This chapter has examined search algorithms for problems in partially observable, nondeterministic, unknown, and continuous environments.

Local search methods such as hill climbing keep only a small number of states in memory. They have been applied to optimization problems, where the idea is to find a high-scoring state, without worrying about the path to the state. Several stochastic local search algorithms have been developed, including simulated annealing, which returns optimal solutions when given an appropriate cooling schedule. Many local search methods apply also to problems in continuous spaces. Linear programming and convex optimization problems obey certain restrictions on the shape of the state space and the nature of the objective function, and admit polynomialtime algorithms that are often extremely efficient in practice. For some mathematically well-formed problems, we can find the maximum using calculus to find where the gradient is zero; for other problems we have to make do with the empirical gradient, which measures the difference in fitness between two nearby points. An evolutionary algorithm is a stochastic hill-climbing search in which a population of states is maintained. New states are generated by mutation and by crossover, which combines pairs of states. In nondeterministic environments, agents can apply AND–OR search to generate contingent plans that reach the goal regardless of which outcomes occur during execution. When the environment is partially observable, the belief state represents the set of possible states that the agent might be in. Standard search algorithms can be applied directly to belief-state space to solve sensorless problems, and belief-state AND–OR search can solve general partially observable problems. Incremental algorithms that construct solutions state by state within a belief state are often more efficient. Exploration problems arise when the agent has no idea about the states and actions of its environment. For safely explorable environments, online search agents can build a map and find a goal if one exists. Updating heuristic estimates from experience provides an effective method to escape from local minima.

Bibliographical and Historical Notes Local search techniques have a long history in mathematics and computer science. Indeed, the Newton–Raphson method (Newton, 1671; Raphson, 1690) can be seen as a very efficient local search method for continuous spaces in which gradient information is available. Brent (1973) is a classic reference for optimization algorithms that do not require such information. Beam search, which we have presented as a local search algorithm, originated as a bounded-width variant of dynamic programming for speech recognition in the HARPY system (Lowerre, 1976). A related algorithm is analyzed in depth by Pearl (1984 Ch. 5 ).

The topic of local search was reinvigorated in the early 1990s by surprisingly good results for large constraint-satisfaction problems such as n-queens (Minton et al., 1992) and Boolean satisfiability (Selman et al., 1992) and by the incorporation of randomness, multiple simultaneous searches, and other improvements. This renaissance of what Christos Papadimitriou has called “New Age” algorithms also sparked increased interest among theoretical computer scientists (Koutsoupias and Papadimitriou, 1992; Aldous and Vazirani, 1994).

In the field of operations research, a variant of hill climbing called tabu search has gained popularity (Glover and Laguna, 1997). This algorithm maintains a tabu list of k previously visited states that cannot be revisited; as well as improving efficiency when searching graphs, this list can allow the algorithm to escape from some local minima.

Tabu search

Another useful improvement on hill climbing is the STAGE algorithm (Boyan and Moore, 1998). The idea is to use the local maxima found by random-restart hill climbing to get an idea of the overall shape of the landscape. The algorithm fits a smooth quadratic surface to the set of local maxima and then calculates the global maximum of that surface analytically.

This becomes the new restart point. Gomes et al. (1998) showed that the run times of systematic backtracking algorithms often have a heavy-tailed distribution, which means that the probability of a very long run time is more than would be predicted if the run times were exponentially distributed. When the run time distribution is heavy-tailed, random restarts find a solution faster, on average, than a single run to completion. Hoos and Stützle (2004) provide a book-length coverage of the topic.

Heavy-tailed distribution

Simulated annealing was first described by Kirkpatrick et al. (1983), who borrowed directly from the Metropolis algorithm (which is used to simulate complex systems in physics (Metropolis et al., 1953) and was supposedly invented at a Los Alamos dinner party). Simulated annealing is now a field in itself, with hundreds of papers published every year.

Finding optimal solutions in continuous spaces is the subject matter of several fields, including optimization theory, optimal control theory, and the calculus of variations. The basic techniques are explained well by Bishop (1995); Press et al. (2007) cover a wide range of algorithms and provide working software.

Researchers have taken inspiration for search and optimization algorithms from a wide variety of fields of study: metallurgy (simulated annealing); biology (genetic algorithms); neuroscience (neural networks); mountaineering (hill climbing); economics (market-based algorithms (Dias et al., 2006)); physics (particle swarms (Li and Yao, 2012) and spin glasses (Mézard et al., 1987)); animal behavior (reinforcement learning, grey wolf optimizers (Mirjalili and Lewis, 2014)); ornithology (Cuckoo search (Yang and Deb, 2014)); entomology (ant colony (Dorigo et al., 2008), bee colony (Karaboga and Basturk, 2007), firefly (Yang, 2009) and glowworm (Krishnanand and Ghose, 2009) optimization); and others.

Linear programming (LP) was first studied systematically by the mathematician Leonid Kantorovich (1939). It was one of the first applications of computers; the simplex algorithm (Dantzig, 1949) is still used despite worst-case exponential complexity. Karmarkar (1984)

developed the far more efficient family of interior-point methods, which was shown to have polynomial complexity for the more general class of convex optimization problems by Nesterov and Nemirovski (1994). Excellent introductions to convex optimization are provided by Ben-Tal and Nemirovski (2001) and Boyd and Vandenberghe (2004).

Work by Sewall Wright (1931) on the concept of a fitness landscape was an important precursor to the development of genetic algorithms. In the 1950s, several statisticians, including Box (1957) and Friedman (1959), used evolutionary techniques for optimization problems, but it wasn’t until Rechenberg (1965) introduced evolution strategies to solve optimization problems for airfoils that the approach gained popularity. In the 1960s and 1970s, John Holland (1975) championed genetic algorithms, both as a useful optimization tool and as a method to expand our understanding of adaptation (Holland, 1995).

The artificial life movement (Langton, 1995) took this idea one step further, viewing the products of genetic algorithms as organisms rather than solutions to problems. The the Baldwin effect discussed in the chapter was proposed roughly simultaneously by Conwy Lloyd Morgan (1896) and James (Baldwin, 1896). Computer simulations have helped to clarify its implications (Hinton and Nowlan, 1987; Ackley and Littman, 1991; Morgan and Griffiths, 2015). Smith and Szathmáry (1999), Ridley (2004), and Carroll (2007) provide general background on evolution.

Most comparisons of genetic algorithms to other approaches (especially stochastic hill climbing) have found that the genetic algorithms are slower to converge (O’Reilly and Oppacher, 1994; Mitchell et al., 1996; Juels and Wattenberg, 1996; Baluja, 1997). Such findings are not universally popular within the GA community, but recent attempts within that community to understand population-based search as an approximate form of Bayesian learning (see Chapter 20 ) might help close the gap between the field and its critics (Pelikan et al., 1999). The theory of quadratic dynamical systems may also explain the performance of GAs (Rabani et al., 1998). There are some impressive practical applications of GAs, in areas as diverse as antenna design (Lohn et al., 2001), computer-aided design (Renner and Ekart, 2003), climate models (Stanislawska et al., 2015), medicine (Ghaheri et al., 2015), and designing deep neural networks (Miikkulainen et al., 2019).

The field of genetic programming is a subfield of genetic algorithms in which the representations are programs rather than bit strings. The programs are represented in the

form of syntax trees, either in a standard programming language or in specially designed formats to represent electronic circuits, robot controllers, and so on. Crossover involves splicing together subtrees in such a way that the offspring are guaranteed to be well-formed expressions.

Interest in genetic programming was spurred by the work of John Koza (1992, 1994), but it goes back at least to early experiments with machine code by Friedberg (1958) and with finite-state automata by Fogel et al. (1966). As with genetic algorithms, there is debate about the effectiveness of the technique. Koza et al. (1999) describe experiments in the use of genetic programming to design circuit devices.

The journals Evolutionary Computation and IEEE Transactions on Evolutionary Computation cover evolutionary algorithms; articles are also found in Complex Systems, Adaptive Behavior, and Artificial Life. The main conference is the Genetic and Evolutionary Computation Conference (GECCO). Good overview texts on genetic algorithms include those by Mitchell (1996), Fogel (2000), Langdon and Poli (2002), and Poli et al. (2008).

The unpredictability and partial observability of real environments were recognized early on in robotics projects that used planning techniques, including Shakey (Fikes et al., 1972) and FREDDY (Michie, 1972). The problems received more attention after the publication of McDermott’s (1978a) influential article Planning and Acting.

The first work to make explicit use of AND–OR trees seems to have been Slagle’s SAINT program for symbolic integration, mentioned in Chapter 1 . Amarel (1967) applied the idea to propositional theorem proving, a topic discussed in Chapter 7 , and introduced a search algorithm similar to AND-OR-GRAPH-SEARCH. The algorithm was further developed by Nilsson (1971), who also described

AO∗ —which, as its name suggests, finds optimal solutions. AO∗

was further improved by Martelli and Montanari (1973).

AO∗ is a top-down algorithm; a bottom-up generalization of A* is A*LD, for A* Lightest Derivation (Felzenszwalb and McAllester, 2007). Interest in AND–OR search underwent a revival in the early 2000s, with new algorithms for finding cyclic solutions (Jimenez and Torras, 2000; Hansen and Zilberstein, 2001) and new techniques inspired by dynamic programming (Bonet and Geffner, 2005).

The idea of transforming partially observable problems into belief-state problems originated with Astrom (1965) for the much more complex case of probabilistic uncertainty (see Chapter 17 ). Erdmann and Mason (1988) studied the problem of robotic manipulation without sensors, using a continuous form of belief-state search. They showed that it was possible to orient a part on a table from an arbitrary initial position by a well-designed sequence of tilting actions. More practical methods, based on a series of precisely oriented diagonal barriers across a conveyor belt, use the same algorithmic insights (Wiegley et al., 1996).

The belief-state approach was reinvented in the context of sensorless and partially observable search problems by Genesereth and Nourbakhsh (1993). Additional work was done on sensorless problems in the logic-based planning community (Goldman and Boddy, 1996; Smith and Weld, 1998). This work has emphasized concise representations for belief states, as explained in Chapter 11 . Bonet and Geffner (2000) introduced the first effective heuristics for belief-state search; these were refined by Bryce et al. (2006). The incremental approach to belief-state search, in which solutions are constructed incrementally for subsets of states within each belief state, was studied in the planning literature by Kurien et al. (2002); several new incremental algorithms were introduced for nondeterministic, partially observable problems by Russell and Wolfe (2005). Additional references for planning in stochastic, partially observable environments appear in Chapter 17 .

Algorithms for exploring unknown state spaces have been of interest for many centuries. Depth-first search in a reversible maze can be implemented by keeping one’s left hand on the wall; loops can be avoided by marking each junction. The more general problem of exploring Eulerian graphs (i.e., graphs in which each node has equal numbers of incoming and outgoing edges) was solved by an algorithm due to Hierholzer (1873).

Eulerian graph

The first thorough algorithmic study of the exploration problem for arbitrary graphs was carried out by Deng and Papadimitriou (1990), who developed a completely general algorithm but showed that no bounded competitive ratio is possible for exploring a general

graph. Papadimitriou and Yannakakis (1991) examined the question of finding paths to a goal in geometric path-planning environments (where all actions are reversible). They showed that a small competitive ratio is achievable with square obstacles, but with general rectangular obstacles no bounded ratio can be achieved. (See Figure 4.20 .)

In a dynamic environment, the state of the world can spontaneously change without any action by the agent. For example, the agent can plan an optimal driving route from

A to B,

but an accident or unusually bad rush hour traffic can spoil the plan. Incremental search algorithms such as Lifelong Planning A* (Koenig et al., 2004) and D* Lite (Koenig and Likhachev, 2002) deal with this situation.

The LRTA* algorithm was developed by Korf (1990) as part of an investigation into realtime search for environments in which the agent must act after searching for only a fixed amount of time (a common situation in two-player games). LRTA* is in fact a special case of reinforcement learning algorithms for stochastic environments (Barto et al., 1995). Its policy of optimism under uncertainty—always head for the closest unvisited state—can result in an exploration pattern that is less efficient in the uninformed case than simple depth-first search (Koenig, 2000). Dasgupta et al. (1994) show that online iterative deepening search is optimally efficient for finding a goal in a uniform tree with no heuristic information.

Several informed variants on the LRTA* theme have been developed with different methods for searching and updating within the known portion of the graph (Pemberton and Korf, 1992). As yet, there is no good theoretical understanding of how to find goals with optimal efficiency when using heuristic information. Sturtevant and Bulitko (2016) provide an analysis of some pitfalls that occur in practice.

Chapter 5 Adversarial Search and Games In which we explore environments where other agents are plotting against us.

In this chapter we cover competitive environments, in which two or more agents have conflicting goals, giving rise to adversarial search problems. Rather than deal with the chaos of real-world skirmishes, we will concentrate on games, such as chess, Go, and poker. For AI researchers, the simplified nature of these games is a plus: the state of a game is easy to represent, and agents are usually restricted to a small number of actions whose effects are defined by precise rules. Physical games, such as croquet and ice hockey, have more complicated descriptions, a larger range of possible actions, and rather imprecise rules defining the legality of actions. With the exception of robot soccer, these physical games have not attracted much interest in the AI community.

Adversarial search

5.1 Game Theory There are at least three stances we can take towards multi-agent environments. The first stance, appropriate when there are a very large number of agents, is to consider them in the aggregate as an economy, allowing us to do things like predict that increasing demand will cause prices to rise, without having to predict the action of any individual agent.

Economy

Second, we could consider adversarial agents as just a part of the environment—a part that makes the environment nondeterministic. But if we model the adversaries in the same way that, say, rain sometimes falls and sometimes doesn’t, we miss the idea that our adversaries are actively trying to defeat us, whereas the rain supposedly has no such intention.

The third stance is to explicitly model the adversarial agents with the techniques of adversarial game-tree search. That is what this chapter covers. We begin with a restricted class of games and define the optimal move and an algorithm for finding it: minimax search, a generalization of AND–OR search (from Figure 4.11 ). We show that pruning makes the search more efficient by ignoring portions of the search tree that make no difference to the optimal move. For nontrivial games, we will usually not have enough time to be sure of finding the optimal move (even with pruning); we will have to cut off the search at some point.

Pruning

For each state where we choose to stop searching, we ask who is winning. To answer this question we have a choice: we can apply a heuristic evaluation function to estimate who is

winning based on features of the state (Section 5.3 ), or we can average the outcomes of many fast simulations of the game from that state all the way to the end (Section 5.4 ). Section 5.5  discusses games that include an element of chance (through rolling dice or shuffling cards) and Section 5.6  covers games of imperfect information (such as poker and bridge, where not all cards are visible to all players).

Imperfect information

5.1.1 Two-player zero-sum games The games most commonly studied within AI (such as chess and Go) are what game theorists call deterministic, two-player, turn-taking, perfect information, zero-sum games. “Perfect information” is a synonym for “fully observable,”1 and “zero-sum” means that what is good for one player is just as bad for the other: there is no “win-win” outcome. For games we often use the term move as a synonym for “action” and position as a synonym for “state.”

1 Some authors make a distinction, using “imperfect information game” for one like poker where the players get private information about their own hands that the other players do not have, and “partially observable game” to mean one like StarCraft II where each player can see the nearby environment, but not the environment far away.

Perfect information

Zero-sum games

Position

Move

We will call our two players MAX and MIN, for reasons that will soon become obvious. MAX moves first, and then the players take turns moving until the game is over. At the end of the game, points are awarded to the winning player and penalties are given to the loser. A game can be formally defined with the following elements:

S0 : The initial state, which specifies how the game is set up at the start. TO-MOVE(s): The player whose turn it is to move in state s. ACTIONS(s): The set of legal moves in state s. RESULT(s, a): The transition model, which defines the state resulting from taking action a in state s.

Transition model

IS-TERMINAL(s): A terminal test, which is true when the game is over and false otherwise. States where the game has ended are called terminal states.

Terminal test

Terminal state

UTILITY(s, p): A utility function (also called an objective function or payoff function), which defines the final numeric value to player p when the game ends in terminal state s . In chess, the outcome is a win, loss, or draw, with values 1, 0, or 1/2.2  Some games

have a wider range of possible outcomes—for example, the payoffs in backgammon range from 0 to 192.

2 Chess is considered a “zero-sum” game, even though the sum of the outcomes for the two players is +1 for each game, not zero. “Constantsum” would have been a more accurate term, but zero-sum is traditional and makes sense if you imagine each player is charged an entry fee of

1/2.

Much as in Chapter 3 , the initial state, ACTIONS function, and RESULT function define the state space graph—a graph where the vertices are states, the edges are moves and a state might be reached by multiple paths. As in Chapter 3 , we can superimpose a search tree over part of that graph to determine what move to make. We define the complete game tree as a search tree that follows every sequence of moves all the way to a terminal state. The game tree may be infinite if the state space itself is unbounded or if the rules of the game allow for infinitely repeating positions.

State space graph

Search tree

Game tree

Figure 5.1  shows part of the game tree for tic-tac-toe (noughts and crosses). From the initial state, MAX has nine possible moves. Play alternates between MAX’s placing an X and

MIN’s

placing an O until we reach leaf nodes corresponding to terminal states such that one

player has three squares in a row or all the squares are filled. The number on each leaf node indicates the utility value of the terminal state from the point of view of MAX; high values are good for

MAX

and bad for MIN (which is how the players get their names).

Figure 5.1

A (partial) game tree for the game of tic-tac-toe. The top node is the initial state, and MAX moves first, placing an X in an empty square. We show part of the tree, giving alternating moves by MIN (O) and MAX (X), until we eventually reach terminal states, which can be assigned utilities according to the rules of the game.

For tic-tac-toe the game tree is relatively small—fewer than 9! = 362,880 terminal nodes (with only 5,478 distinct states). But for chess there are over 10

40

nodes, so the game tree is best

thought of as a theoretical construct that we cannot realize in the physical world.

5.2 Optimal Decisions in Games MAX

wants to find a sequence of actions leading to a win, but MIN has something to say about

it. This means that MAX’s strategy must be a conditional plan—a contingent strategy specifying a response to each of MIN’s possible moves. In games that have a binary outcome (win or lose), we could use AND–OR search (page 125) to generate the conditional plan. In fact, for such games, the definition of a winning strategy for the game is identical to the definition of a solution for a nondeterministic planning problem: in both cases the desirable outcome must be guaranteed no matter what the “other side” does. For games with multiple outcome scores, we need a slightly more general algorithm called minimax search.

Minimax search

Consider the trivial game in Figure 5.2 . The possible moves for MAX at the root node are labeled a1 , a2 , and a3 . The possible replies to a1 for MIN are b1 , b2 , b3 , and so on. This particular game ends after one move each by MAX and MIN. (NOTE: In some games, the word “move” means that both players have taken an action; therefore the word ply is used to unambiguously mean one move by one player, bringing us one level deeper in the game tree.) The utilities of the terminal states in this game range from 2 to 14.

Figure 5.2

△

▽

A two-ply game tree. The nodes are “MAX nodes,” in which it is MAX’s turn to move, and the nodes are “MIN nodes.” The terminal nodes show the utility values for MAX; the other nodes are labeled with their minimax values. MAX’s best move at the root is 1 , because it leads to the state with the highest minimax value, and MIN’s best reply is

a

b , because it leads to the state with the lowest minimax value. 1

ply

Given a game tree, the optimal strategy can be determined by working out the minimax value of each state in the tree, which we write as MINIMAX(s). The minimax value is the utility (for

MAX)

of being in that state, assuming that both players play optimally from there to the end

of the game. The minimax value of a terminal state is just its utility. In a non-terminal state, MAX prefers to move to a state of maximum value when it is MAX’s turn to move, and MIN prefers a state of minimum value (that is, minimum value for MAX and thus maximum value for MIN). So we have:

MINIMAX (s)   = ⎧ ⎪ ⎪ ⎨ ⎩ ⎪ ⎪

s

UTILITY( ,MAX)

s a (s,  a))

s

if IS-TERMINAL ( )

s ( s) =

maxa∈Actions(s) MINIMAX (RESULT ( ,   ))

if TO-MOVE ( ) =  MAX

mina∈Actions(s) MINIMAX (RESULT

if TO-MOVE

Minimax value

 MIN

Let us apply these definitions to the game tree in Figure 5.2 . The terminal nodes on the bottom level get their utility values from the game’s UTILITY function. The first MIN node, labeled

B, has three successor states with values 3, 12, and 8, so its minimax value is 3.

Similarly, the other two MIN nodes have minimax value 2. The root node is a MAX node; its successor states have minimax values 3, 2, and 2; so it has a minimax value of 3. We can also identify the minimax decision at the root: action

a1 is the optimal choice for MAX

because it leads to the state with the highest minimax value.

Minimax decision

This definition of optimal play for MAX assumes that MIN also plays optimally. What if MIN does not play optimally? Then MAX will do at least as well as against an optimal player, possibly better. However, that does not mean that it is always best to play the minimax optimal move when facing a suboptimal opponent. Consider a situation where optimal play by both sides will lead to a draw, but there is one risky move for MAX that leads to a state in which there are 10 possible response moves by MIN that all seem reasonable, but 9 of them are a loss for MIN and one is a loss for MAX. If MAX believes that MIN does not have sufficient computational power to discover the optimal move, MAX might want to try the risky move, on the grounds that a 9/10 chance of a win is better than a certain draw.

5.2.1 The minimax search algorithm Now that we can compute MINIMAX(s), we can turn that into a search algorithm that finds the best move for MAX by trying all actions and choosing the one whose resulting state has the highest MINIMAX value. Figure 5.3  shows the algorithm. It is a recursive algorithm that proceeds all the way down to the leaves of the tree and then backs up the minimax values through the tree as the recursion unwinds. For example, in Figure 5.2 , the algorithm first recurses down to the three bottom-left nodes and uses the UTILITY function on them to discover that their values are 3, 12, and 8, respectively. Then it takes the minimum of these values, 3, and returns it as the backed-up value of node backed-up values of 2 for

B. A similar process gives the

C and 2 for D. Finally, we take the maximum of 3, 2, and 2 to get

the backed-up value of 3 for the root node.

Figure 5.3

An algorithm for calculating the optimal move using minimax—the move that leads to a terminal state with maximum utility, under the assumption that the opponent plays to minimize utility. The functions MAX-VALUE and MIN-VALUE go through the whole game tree, all the way to the leaves, to determine the backed-up value of a state and the move to get there.

The minimax algorithm performs a complete depth-first exploration of the game tree. If the

m and there are b legal moves at each point, then the time complexity of the minimax algorithm is O(b m ). The space complexity is O(bm) for an algorithm that generates all actions at once, or O(m) for an algorithm that generates actions maximum depth of the tree is

one at a time (see page 80). The exponential complexity makes MINIMAX impractical for complex games; for example, chess has a branching factor of about 35 and the average game has depth of about 80 ply, and it is not feasible to search 3580 ≈ 10123 states. MINIMAX does, however, serve as a basis for the mathematical analysis of games. By approximating the minimax analysis in various ways, we can derive more practical algorithms.

5.2.2 Optimal decisions in multiplayer games Many popular games allow more than two players. Let us examine how to extend the minimax idea to multiplayer games. This is straightforward from the technical viewpoint, but raises some interesting new conceptual issues.

First, we need to replace the single value for each node with a vector of values. For example, in a three-player game with players

A, B, and C , a vector ⟨vA vB vC ⟩ is associated with each ,

,

node. For terminal states, this vector gives the utility of the state from each player’s viewpoint. (In two-player, zero-sum games, the two-element vector can be reduced to a single value because the values are always opposite.) The simplest way to implement this is to have the UTILITY function return a vector of utilities.

Now we have to consider nonterminal states. Consider the node marked shown in Figure 5.4 . In that state, player

X in the game tree

C chooses what to do. The two choices lead to terminal states with utility vectors ⟨vA vB vC ⟩ and ⟨vA vB vC ⟩. Since 6 is bigger than 3, C should choose the first move. This means that if state X is reached, subsequent play will lead to a terminal state with utilities ⟨vA vB vC ⟩. Hence, the backed-up value of X is this vector. In general, the backed-up value of a node n is the utility vector of the successor state with the highest value for the player choosing at n. = 1,

= 2,

= 6

= 4,

= 1,

= 2,

= 2,

= 3

= 6

Figure 5.4

ABC

The first three ply of a game tree with three players ( , , ). Each node is labeled with values from the viewpoint of each player. The best move is marked at the root.

Anyone who plays multiplayer games, such as Diplomacy or Settlers of Catan, quickly becomes aware that much more is going on than in two-player games. Multiplayer games

usually involve alliances, whether formal or informal, among the players. Alliances are made and broken as the game proceeds. How are we to understand such behavior? Are alliances a natural consequence of optimal strategies for each player in a multiplayer game? It turns out that they can be.

Alliance

A and B are in weak positions and C is in a stronger position. Then it is often optimal for both A and B to attack C rather than each other, lest C destroy each of For example, suppose

them individually. In this way, collaboration emerges from purely selfish behavior. Of course, as soon as either

C weakens under the joint onslaught, the alliance loses its value, and

A or B could violate the agreement.

In some cases, explicit alliances merely make concrete what would have happened anyway. In other cases, a social stigma attaches to breaking an alliance, so players must balance the immediate advantage of breaking an alliance against the long-term disadvantage of being perceived as untrustworthy. See Section 18.2  for more on these complications.

If the game is not zero-sum, then collaboration can also occur with just two players.

v

Suppose, for example, that there is a terminal state with utilities ⟨ A

= 1000,

vB

= 1000⟩

and

that 1000 is the highest possible utility for each player. Then the optimal strategy is for both players to do everything possible to reach this state—that is, the players will automatically cooperate to achieve a mutually desirable goal.

5.2.3 Alpha–Beta Pruning The number of game states is exponential in the depth of the tree. No algorithm can completely eliminate the exponent, but we can sometimes cut it in half, computing the correct minimax decision without examining every state by pruning (see page 90) large parts of the tree that make no difference to the outcome. The particular technique we examine is called alpha–beta pruning.

Alpha–beta pruning

Consider again the two-ply game tree from Figure 5.2 . Let’s go through the calculation of the optimal decision once more, this time paying careful attention to what we know at each point in the process. The steps are explained in Figure 5.5 . The outcome is that we can identify the minimax decision without ever evaluating two of the leaf nodes.

Figure 5.5

Stages in the calculation of the optimal decision for the game tree in Figure 5.2 . At each point, we show

B

B

the range of possible values for each node. (a) The first leaf below has the value 3. Hence, , which is a MIN node, has a value of at most 3. (b) The second leaf below has a value of 12; MIN would avoid this move, so the value of is still at most 3. (c) The third leaf below has a value of 8; we have seen all ’s

B

B

B

B

B

successor states, so the value of is exactly 3. Now we can infer that the value of the root is at least 3, because MAX has a choice worth 3 at the root. (d) The first leaf below has the value 2. Hence, , which

C

B

C

C

is a MIN node, has a value of at most 2. But we know that is worth 3, so MAX would never choose . Therefore, there is no point in looking at the other successor states of . This is an example of alpha–beta

D

D

C

pruning. (e) The first leaf below has the value 14, so is worth at most 14. This is still higher than best alternative (i.e., 3), so we need to keep exploring ’s successor states. Notice also that we now have bounds on all of the successors of the root, so the root’s value is also at most 14. (f) The second MAX’s

D

D

successor of is worth 5, so again we need to keep exploring. The third successor is worth 2, so now is worth exactly 2. MAX’s decision at the root is to move to , giving a value of 3.

B

D

Another way to look at this is as a simplification of the formula for MINIMAX. Let the two unevaluated successors of node

C in Figure 5.5  have values x and y. Then the value of the

root node is given by

MINIMAX(

root) = max(min(3,12,8), min(2,x,y), min(14,5,2)) = max(3, min(2,x,y),2) = max(3,z,2) where z = min(2,x,y) ≤ 2 = 3.

In other words, the value of the root and hence the minimax decision are independent of the values of the leaves

x and y, and therefore they can be pruned.

Alpha–beta pruning can be applied to trees of any depth, and it is often possible to prune

n somewhere in the tree (see Figure 5.6 ), such that Player has a choice of moving to n. If Player has a better choice either at the same level (e.g. m in Figure 5.6 ) or at any point higher up in the tree (e.g. m in Figure 5.6 ), then Player will never move to n. So once we have found out enough about n (by examining some of its descendants) to reach this entire subtrees rather than just leaves. The general principle is this: consider a node

′

conclusion, we can prune it.

Figure 5.6

The general case for alpha–beta pruning. If m or m′ is better than n for Player, we will never get to n in play.

Remember that minimax search is depth-first, so at any one time we just have to consider the nodes along a single path in the tree. Alpha–beta pruning gets its name from the two extra parameters in MAX-VALUE (state,α,β) (see Figure 5.7 ) that describe bounds on the backed-up values that appear anywhere along the path:

α = the value of the best (i.e., highest-value) choice we have found so far at any choice point along the path for MAX. Think: α = “at least.”

β = the value of the best (i.e., lowest-value) choice we have found so far at any choice point along the path for MIN. Think: β = “at most.”

Figure 5.7

The alpha–beta search algorithm. Notice that these functions are the same as the MINIMAX-SEARCH functions in Figure 5.3 , except that we maintain bounds in the variables α and β, and use them to cut off search when a value is outside the bounds.

Alpha–beta search updates the values of α and β as it goes along and prunes the remaining branches at a node (i.e., terminates the recursive call) as soon as the value of the current node is known to be worse than the current α or β value for MAX or

MIN,

respectively. The

complete algorithm is given in Figure 5.7 . Figure 5.5  traces the progress of the algorithm on a game tree.

5.2.4 Move ordering The effectiveness of alpha–beta pruning is highly dependent on the order in which the states are examined. For example, in Figure 5.5(e)  and (f) , we could not prune any

D at all because the worst successors (from the point of view of ) were generated first. If the third successor of D had been generated first, with value 2, we would successors of

MIN

have been able to prune the other two successors. This suggests that it might be worthwhile to try to first examine the successors that are likely to be best.

If this could be done perfectly, alpha–beta would need to examine only

O( b m

/2

) nodes to

O(bm ) for minimax. This means that the effective branching factor becomes √b instead of b—for chess, about 6 instead of 35. Put another way, alpha– pick the best move, instead of

beta with perfect move ordering can solve a tree roughly twice as deep as minimax in the same amount of time. With random move ordering, the total number of nodes examined will be roughly

O( b m 3

/4

b

) for moderate . Now, obviously we cannot achieve perfect move

ordering—in that case the ordering function could be used to play a perfect game! But we can often get fairly close. For chess, a fairly simple ordering function (such as trying captures first, then threats, then forward moves, and then backward moves) gets you to within about a factor of 2 of the best-case

O(bm

/2

) result.

Adding dynamic move-ordering schemes, such as trying first the moves that were found to be best in the past, brings us quite close to the theoretical limit. The past could be the previous move—often the same threats remain—or it could come from previous exploration of the current move through a process of iterative deepening (see page 80). First, search one ply deep and record the ranking of moves based on their evaluations. Then search one ply deeper, using the previous ranking to inform move ordering; and so on. The increased search time from iterative deepening can be more than made up from better move ordering. The best moves are known as killer moves, and to try them first is called the killer move heuristic.

Killer moves

In Section 3.3.3 , we noted that redundant paths to repeated states can cause an exponential increase in search cost, and that keeping a table of previously reached states can address this problem. In game tree search, repeated states can occur because of transpositions—different permutations of the move sequence that end up in the same

position, and the problem can be addressed with a transposition table that caches the heuristic value of states.

Transposition

Transposition table

For example, suppose White has a move w1 that can be answered by Black with b1 and an unrelated move w2 on the other side of the board that can be answered by b2 , and that we search the sequence of moves [w1 ,b1 ,w2 ,b2 ]; let’s call the resulting state s. After exploring a large subtree below s, we find its backed-up value, which we store in the transposition table. When we later search the sequence of moves [w2 ,b2 ,w1 ,b1 ], we end up in s again, and we can look up the value instead of repeating the search. In chess, use of transposition tables is very effective, allowing us to double the reachable search depth in the same amount of time.

Even with alpha–beta pruning and clever move ordering, minimax won’t work for games like chess and Go, because there are still too many states to explore in the time available. In the very first paper on computer game-playing, Programming a Computer for Playing Chess (Shannon, 1950), Claude Shannon recognized this problem and proposed two strategies: a Type A strategy considers all possible moves to a certain depth in the search tree, and then uses a heuristic evaluation function to estimate the utility of states at that depth. It explores a wide but shallow portion of the tree. A Type B strategy ignores moves that look bad, and follows promising lines “as far as possible.” It explores a deep but narrow portion of the tree.

Type A strategy

Type B strategy

Historically, most chess programs have been Type A (which we cover in the next section), whereas Go programs are more often Type B (covered in Section 5.4 ), because the branching factor is much higher in Go. More recently, Type B programs have shown worldchampion-level play across a variety of games, including chess (Silver et al., 2018).

5.3 Heuristic Alpha–Beta Tree Search To make use of our limited computation time, we can cut off the search early and apply a heuristic evaluation function to states, effectively treating nonterminal nodes as if they were terminal. In other words, we replace the UTILITY function with EVAL, which estimates a state’s utility. We also replace the terminal test by a cutoff test, which must return true for terminal states, but is otherwise free to decide when to cut off the search, based on the search depth and any property of the state that it chooses to consider. That gives us the formula HMINIMAX(s, d) for the heuristic minimax value of state

sd s

s at search depth d:

H-MINIMAX ( , )   = EVAL ( , MAX) ⎧ ⎪ ⎪ ⎨ ⎪ ⎩ ⎪

s a d + 1) (s,  a) , d + 1)

sd ( s) = ( s) =

if IS-CUTOFF ( , )

maxa∈Actions(s) H-MINIMAX (RESULT ( ,   ) ,

if TO-MOVE

mina∈Actions(s) H-MINIMAX (RESULT

if TO-MOVE

 MAX  MIN.

cutoff test

5.3.1 Evaluation functions

sp

A heuristic evaluation function EVAL( , ) returns an estimate of the expected utility of state

s

to player , just as the heuristic functions of Chapter 3  return an estimate of the distance to

p

sp

sp

the goal. For terminal states, it must be that EVAL( , ) = UTILITY( , ) and for nonterminal states, the evaluation must be somewhere between a loss and a win: UTILITY(

loss, p) ≤ E

VAL(

s, p ) ≤ U

TILITY(

win, p).

Beyond those requirements, what makes for a good evaluation function? First, the computation must not take too long! (The whole point is to search faster.) Second, the evaluation function should be strongly correlated with the actual chances of winning. One might well wonder about the phrase “chances of winning.” After all, chess is not a game of chance: we know the current state with certainty, and no dice are involved; if neither player makes a mistake, the outcome is predetermined. But if the search must be cut off at nonterminal states, then the algorithm will necessarily be uncertain about the final outcomes

of those states (even though that uncertainty could be resolved with infinite computing resources).

Let us make this idea more concrete. Most evaluation functions work by calculating various features of the state—for example, in chess, we would have features for the number of white pawns, black pawns, white queens, black queens, and so on. The features, taken together, define various categories or equivalence classes of states: the states in each category have the same values for all the features. For example, one category might contain all two-pawn versus one-pawn endgames. Any given category will contain some states that lead (with perfect play) to wins, some that lead to draws, and some that lead to losses.

Features

The evaluation function does not know which states are which, but it can return a single value that estimates the proportion of states with each outcome. For example, suppose our experience suggests that 82% of the states encountered in the two-pawns versus one-pawn category lead to a win (utility +1); 2% to a loss (0), and 16% to a draw (1/2). Then a reasonable evaluation for states in the category is the expected value: (0.82 × +1) + (0.02 × 0) + (0.16 × 1/2) = 0.90. In principle, the expected value can be

determined for each category of states, resulting in an evaluation function that works for any state.

Expected value

In practice, this kind of analysis requires too many categories and hence too much experience to estimate all the probabilities. Instead, most evaluation functions compute separate numerical contributions from each feature and then combine them to find the total value. For centuries, chess players have developed ways of judging the value of a position using just this idea. For example, introductory chess books give an approximate material

value for each piece: each pawn is worth 1, a knight or bishop is worth 3, a rook 5, and the queen 9. Other features such as “good pawn structure” and “king safety” might be worth half a pawn, say. These feature values are then simply added up to obtain the evaluation of the position.

Material value

Mathematically, this kind of evaluation function is called a weighted linear function because it can be expressed as

s

EVAL( ) =

w1 f1 (s) + w2 f2 (s) + ⋯ + wn fn (s) =

∑ n

w i fi ( s ) ,

i=1

Weighted linear function

where each fi is a feature of the position (such as “number of white bishops”) and each wi is a weight (saying how important that feature is). The weights should be normalized so that the sum is always within the range of a loss (0) to a win (+1). A secure advantage equivalent to a pawn gives a substantial likelihood of winning, and a secure advantage equivalent to three pawns should give almost certain victory, as illustrated in Figure 5.8(a) . We said that the evaluation function should be strongly correlated with the actual chances of winning, but it need not be linearly correlated: if state s is twice as likely to win as state s′ we don’t require that EVAL(S) be twice EVAL(S'); all we require is that EVAL(s)

Figure 5.8

> EVAL(s').

Two chess positions that differ only in the position of the rook at lower right. In (a), Black has an advantage of a knight and two pawns, which should be enough to win the game. In (b), White will capture the queen, giving it an advantage that should be strong enough to win.

Adding up the values of features seems like a reasonable thing to do, but in fact it involves a strong assumption: that the contribution of each feature is independent of the values of the other features. For this reason, current programs for chess and other games also use nonlinear combinations of features. For example, a pair of bishops might be worth more than twice the value of a single bishop, and a bishop is worth more in the endgame than earlier—when the move number feature is high or the number of remaining pieces feature is low.

Where do the features and weights come from? They’re not part of the rules of chess, but they are part of the culture of human chess-playing experience. In games where this kind of experience is not available, the weights of the evaluation function can be estimated by the machine learning techniques of Chapter 22 . Applying these techniques to chess has confirmed that a bishop is indeed worth about three pawns, and it appears that centuries of human experience can be replicated in just a few hours of machine learning.

5.3.2 Cutting off search The next step is to modify ALPHA-BETA-SEARCH so that it will call the heuristic EVAL function when it is appropriate to cut off the search. We replace the two lines in Figure 5.7  that mention IS-TERMINAL with the following line:

if game.IS-CUTOFF(state, depth) then return game.EVAL(state, player), null

We also must arrange for some bookkeeping so that the current depth is incremented on each recursive call. The most straightforward approach to controlling the amount of search is to set a fixed depth limit so that IS-CUTOFF(state, depth) returns true for all depth greater than some fixed depth d (as well as for all terminal states). The depth d is chosen so that a move is selected within the allocated time. A more robust approach is to apply iterative deepening. (See Chapter 3 .) When time runs out, the program returns the move selected by the deepest completed search. As a bonus, if in each round of iterative deepening we keep entries in the transposition table, subsequent rounds will be faster, and we can use the evaluations to improve move ordering.

These simple approaches can lead to errors due to the approximate nature of the evaluation function. Consider again the simple evaluation function for chess based on material advantage. Suppose the program searches to the depth limit, reaching the position in Figure 5.8(b) , where Black is ahead by a knight and two pawns. It would report this as the heuristic value of the state, thereby declaring that the state is a probable win by Black. But White’s next move captures Black’s queen with no compensation. Hence, the position is actually favorable for White, but this can be seen only by looking ahead.

The evaluation function should be applied only to positions that are quiescent—that is, positions in which there is no pending move (such as a capturing the queen) that would wildly swing the evaluation. For nonquiescent positions the IS-CUTOFF returns false, and the search continues until quiescent positions are reached. This extra quiescence search is sometimes restricted to consider only certain types of moves, such as capture moves, that will quickly resolve the uncertainties in the position.

Quiescence

Quiescence search

The horizon effect is more difficult to eliminate. It arises when the program is facing an opponent’s move that causes serious damage and is ultimately unavoidable, but can be temporarily avoided by the use of delaying tactics. Consider the chess position in Figure 5.9 . It is clear that there is no way for the black bishop to escape. For example, the white rook can capture it by moving to h1, then a1, then a2; a capture at depth 6 ply.

Figure 5.9

The horizon effect. With Black to move, the black bishop is surely doomed. But Black can forestall that event by checking the white king with its pawns, encouraging the king to capture the pawns. This pushes the inevitable loss of the bishop over the horizon, and thus the pawn sacrifices are seen by the search algorithm as good moves rather than bad ones.

Horizon effect

But Black does have a sequence of moves that pushes the capture of the bishop “over the horizon.” Suppose Black searches to depth 8 ply. Most moves by Black will lead to the eventual capture of the bishop, and thus will be marked as “bad” moves. But Black will also

consider the sequence of moves that starts by checking the king with a pawn, and enticing the king to capture the pawn. Black can then do the same thing with a second pawn. That takes up enough moves that the capture of the bishop would not be discovered during the remainder of Black’s search. Black thinks that the line of play has saved the bishop at the price of two pawns, when actually all it has done is waste pawns and push the inevitable capture of the bishop beyond the horizon that Black can see.

One strategy to mitigate the horizon effect is to allow singular extensions, moves that are “clearly better” than all other moves in a given position, even when the search would normally be cut off at that point. In our example, a search will have revealed that three moves of the white rook—h2 to h1, then h1 to a1, and then a1 capturing the bishop on a2— are each in turn clearly better moves, so even if a sequence of pawn moves pushes us to the horizon, these clearly better moves will be given a chance to extend the search. This makes the tree deeper, but because there are usually few singular extensions, the strategy does not add many total nodes to the tree, and has proven to be effective in practice.

Singular extension

5.3.3 Forward pruning Alpha–beta pruning prunes branches of the tree that can have no effect on the final evaluation, but forward pruning prunes moves that appear to be poor moves, but might possibly be good ones. Thus, the strategy saves computation time at the risk of making an error. In Shannon’s terms, this is a Type B strategy. Clearly, most human chess players do this, considering only a few moves from each position (at least consciously).

Forward pruning

One approach to forward pruning is beam search (see page 115): on each ply, consider only a “beam” of the n best moves (according to the evaluation function) rather than considering

all possible moves. Unfortunately, this approach is rather dangerous because there is no guarantee that the best move will not be pruned away.

The PROBCUT, or probabilistic cut, algorithm (Buro, 1995) is a forward-pruning version of alpha–beta search that uses statistics gained from prior experience to lessen the chance that the best move will be pruned. Alpha–beta search prunes any node that is provably outside the current (α, β) window. PROBCUT also prunes nodes that are probably outside the window. It computes this probability by doing a shallow search to compute the backed-up value v of a node and then using past experience to estimate how likely it is that a score of v at depth d in the tree would be outside (α, β). Buro applied this technique to his Othello program, LOGISTELLO, and found that a version of his program with PROBCUT beat the regular version 64% of the time, even when the regular version was given twice as much time.

Another technique, late move reduction, works under the assumption that move ordering has been done well, and therefore moves that appear later in the list of possible moves are less likely to be good moves. But rather than pruning them away completely, we just reduce the depth to which we search these moves, thereby saving time. If the reduced search comes back with a value above the current α value, we can re-run the search with the full depth.

Late move reduction

Combining all the techniques described here results in a program that can play creditable chess (or other games). Let us assume we have implemented an evaluation function for chess, a reasonable cutoff test with a quiescence search. Let us also assume that, after months of tedious bit-bashing, we can generate and evaluate around a million nodes per second on the latest PC. The branching factor for chess is about 35, on average, and 355 is about 50 million, so if we used minimax search, we could look ahead only five ply in about a minute of computation; the rules of competition would not give us enough time to search six ply. Though not incompetent, such a program can be defeated by an average human chess player, who can occasionally plan six or eight ply ahead.

With alpha–beta search and a large transposition table we get to about 14 ply, which results in an expert level of play. We could trade in our PC for a workstation with 8 GPUs, getting us over a billion nodes per second, but to obtain grandmaster status we would still need an extensively tuned evaluation function and a large database of endgame moves. Top chess programs like STOCKFISH have all of these, often reaching depth 30 or more in the search tree and far exceeding the ability of any human player.

5.3.4 Search versus lookup Somehow it seems like overkill for a chess program to start a game by considering a tree of a billion game states, only to conclude that it will play pawn to e4 (the most popular first move). Books describing good play in the opening and endgame in chess have been available for more than a century (Tattersall, 1911). It is not surprising, therefore, that many game-playing programs use table lookup rather than search for the opening and ending of games.

For the openings, the computer is mostly relying on the expertise of humans. The best advice of human experts on how to play each opening can be copied from books and entered into tables for the computer’s use. In addition, computers can gather statistics from a database of previously played games to see which opening sequences most often lead to a win. For the first few moves there are few possibilities, and most positions will be in the table. Usually after about 10 or 15 moves we end up in a rarely seen position, and the program must switch from table lookup to search.

Near the end of the game there are again fewer possible positions, and thus it is easier to do lookup. But here it is the computer that has the expertise: computer analysis of endgames goes far beyond human abilities. Novice humans can win a king-and-rook-versus-king (KRK) endgame by following a few simple rules. Other endings, such as king, bishop, and knight versus king (KBNK), are difficult to master and have no succinct strategy description.

A computer, on the other hand, can completely solve the endgame by producing a policy, which is a mapping from every possible state to the best move in that state. Then the computer can play perfectly by looking up the right move in this table. The table is constructed by retrograde minimax search: start by considering all ways to place the KBNK pieces on the board. Some of the positions are wins for white; mark them as such. Then reverse the rules of chess to do reverse moves rather than moves. Any move by White that,

no matter what move Black responds with, ends up in a position marked as a win, must also be a win. Continue this search until all possible positions are resolved as win, loss, or draw, and you have an infallible lookup table for all endgames with those pieces. This has been done not only for KBNK endings, but for all endings with seven or fewer pieces. The tables contain 400 trillion positions. An eight-piece table would require 40 quadrillion positions.

Retrograde

5.4 Monte Carlo Tree Search The game of Go illustrates two major weaknesses of heuristic alpha–beta tree search: First, Go has a branching factor that starts at 361, which means alpha–beta search would be limited to only 4 or 5 ply. Second, it is difficult to define a good evaluation function for Go because material value is not a strong indicator and most positions are in flux until the endgame. In response to these two challenges, modern Go programs have abandoned alpha–beta search and instead use a strategy called Monte Carlo tree search (MCTS).3

3 “Monte Carlo” algorithms are randomized algorithms named after the Casino de Monte-Carlo in Monaco.

Monte Carlo tree search (MCTS)

The basic MCTS strategy does not use a heuristic evaluation function. Instead, the value of a state is estimated as the average utility over a number of simulations of complete games starting from the state. A simulation (also called a playout or rollout) chooses moves first for one player, than for the other, repeating until a terminal position is reached. At that point the rules of the game (not fallible heuristics) determine who has won or lost, and by what score. For games in which the only outcomes are a win or a loss, “average utility” is the same as “win percentage.”

Simulation

Playout

Rollout

How do we choose what moves to make during the playout? If we just choose randomly, then after multiple simulations we get an answer to the question “what is the best move if both players play randomly?” For some simple games, that happens to be the same answer as “what is the best move if both players play well?,” but for most games it is not. To get useful information from the playout we need a playout policy that biases the moves towards good ones. For Go and other games, playout policies have been successfully learned from self-play by using neural networks. Sometimes game-specific heuristics are used, such as “consider capture moves” in chess or “take the corner square” in Othello.

Playout policy

Given a playout policy, we next need to decide two things: from what positions do we start the playouts, and how many playouts do we allocate to each position? The simplest answer, called pure Monte Carlo search, is to do

N simulations starting from the current state of the

game, and track which of the possible moves from the current position has the highest win percentage.

Pure Monte Carlo search

For some stochastic games this converges to optimal play as

N increases, but for most

games it is not sufficient—we need a selection policy that selectively focuses the computational resources on the important parts of the game tree. It balances two factors: exploration of states that have had few playouts, and exploitation of states that have done well in past playouts, to get a more accurate estimate of their value. (See Section 17.3  for more on the exploration/exploitation tradeoff.) Monte Carlo tree search does that by

maintaining a search tree and growing it on each iteration of the following four steps, as shown in Figure 5.10 :

Selection policy

Exploration

Exploitation

SELECTION: Starting at the root of the search tree, we choose a move (guided by the selection policy), leading to a successor node, and repeat that process, moving down the tree to a leaf. Figure 5.10(a)  shows a search tree with the root representing a state where white has just moved, and white has won 37 out of the 100 playouts done so far. The thick arrow shows the selection of a move by black that leads to a node where black has won 60/79 playouts. This is the best win percentage among the three moves, so selecting it is an example of exploitation. But it would also have been reasonable to select the 2/11 node for the sake of exploration—with only 11 playouts, the node still has high uncertainty in its valuation, and might end up being best if we gain more information about it. Selection continues on to the leaf node marked 27/35. EXPANSION: We grow the search tree by generating a new child of the selected node; Figure 5.10(b)  shows the new node marked with 0/0. (Some versions generate more than one child in this step.) SIMULATION: We perform a playout from the newly generated child node, choosing moves for both players according to the playout policy. These moves are not recorded in the search tree. In the figure, the simulation results in a win for black. BACK-PROPAGATION: We now use the result of the simulation to update all the search tree nodes going up to the root. Since black won the playout, black nodes are incremented in both the number of wins and the number of playouts, so 27/35 becomes

28/26 and 60/79 becomes 61/80. Since white lost, the white nodes are incremented in the number of playouts only, so 16/53 becomes 16/54 and the root 37/100 becomes 37/101.

Figure 5.10

One iteration of the process of choosing a move with Monte Carlo tree search (MCTS) using the upper confidence bounds applied to trees (UCT) selection metric, shown after 100 iterations have already been done. In (a) we select moves, all the way down the tree, ending at the leaf node marked (27/35) (for 27 wins for black out of 35 playouts). In (b) we expand the selected node and do a simulation (playout), which ends in a win for black. In (c), the results of the simulation are back-propagated up the tree.

We repeat these four steps either for a set number of iterations, or until the allotted time has expired, and then return the move with the highest number of playouts.

One very effective selection policy is called “upper confidence bounds applied to trees” or UCT. The policy ranks each possible move based on an upper confidence bound formula called UCB1. (See Section 17.3.3  for more details.) For a node , the formula is:

n

U (n) log N (P (n)) UCB1(n) = N (n) + C × √ N ( n) ARENT

UCT

UCB1

U (n) is the total utility of all playouts that went through node n, N (n) is the number U (n) of playouts through node n, and P (n) is the parent node of n in the tree. Thus is N (n) the exploitation term: the average utility of n. The term with the square root is the exploration term: it has the count N (n) in the denominator, which means the term will be where

ARENT

high for nodes that have only been explored a few times. In the numerator it has the log of

n

the number of times we have explored the parent of . This means that if we are selecting

n

some non-zero percentage of the time, the exploration term goes to zero as the counts increase, and eventually the playouts are given to the node with highest average utility.

C is a constant that balances exploitation and exploration. There is a theoretical argument that C should be √2, but in practice, game programmers try multiple values for C and choose the one that performs best. (Some programs use slightly different formulas; for example, ALPHAZERO adds in a term for move probability, which is calculated by a neural network trained from past self-play.) With highest UCB1 score, but with

C = 1.4, the 60/79 node in Figure 5.10  has the

C = 1.5, it would be the 2/11 node.

Figure 5.11  shows the complete UCT MCTS algorithm. When the iterations terminate, the move with the highest number of playouts is returned. You might think that it would be better to return the node with the highest average utility, but the idea is that a node with 65/100 wins is better than one with 2/3 wins, because the latter has a lot of uncertainty. In

any event, the UCB1 formula ensures that the node with the most playouts is almost always the node with the highest win percentage, because the selection process favors win percentage more and more as the number of playouts goes up.

Figure 5.11

The Monte Carlo tree search algorithm. A game tree, tree, is initialized, and then we repeat a cycle of SELECT / EXPAND / SIMULATE / BACK-PROPAGATE until we run out of time, and return the move that led to the node with the highest number of playouts.

The time to compute a playout is linear, not exponential, in the depth of the game tree, because only one move is taken at each choice point. That gives us plenty of time for multiple playouts. For example: consider a game with a branching factor of 32, where the average game lasts 100 ply. If we have enough computing power to consider a billion game states before we have to make a move, then minimax can search 6 ply deep, alpha–beta with perfect move ordering can search 12 ply, and Monte Carlo search can do 10 million playouts. Which approach will be better? That depends on the accuracy of the heuristic function versus the selection and playout policies.

The conventional wisdom has been that Monte Carlo search has an advantage over alpha– beta for games like Go where the branching factor is very high (and thus alpha–beta can’t search deep enough), or when it is difficult to define a good evaluation function. What alpha–beta does is choose the path to a node that has the highest achievable evaluation function score, given that the opponent will be trying to minimize the score. Thus, if the evaluation function is inaccurate, alpha–beta will be inaccurate. A miscalculation on a single node can lead alpha–beta to erroneously choose (or avoid) a path to that node. But Monte Carlo search relies on the aggregate of many playouts, and thus is not as vulnerable to a single error. It is possible to combine MCTS and evaluation functions by doing a playout for a certain number of moves, but then truncating the playout and applying an evaluation function.

It is also possible to combine aspects of alpha–beta and Monte Carlo search. For example, in games that can last many moves, we may want to use early playout termination, in which we stop a playout that is taking too many moves, and either evaluate it with a heuristic evaluation function or just declare it a draw.

Early playout termination

Monte Carlo search can be applied to brand-new games, in which there is no body of experience to draw upon to define an evaluation function. As long as we know the rules of the game, Monte Carlo search does not need any additional information. The selection and playout policies can make good use of hand-crafted expert knowledge when it is available, but good policies can be learned using neural networks trained by self-play alone.

Monte Carlo search has a disadvantage when it is likely that a single move can change the course of the game, because the stochastic nature of Monte Carlo search means it might fail to consider that move. In other words, Type B pruning in Monte Carlo search means that a vital line of play might not be explored at all. Monte Carlo search also has a disadvantage when there are game states that are “obviously” a win for one side or the other (according to human knowledge and to an evaluation function), but where it will still take many moves in a playout to verify the winner. It was long held that alpha–beta search was better suited for games like chess with low branching factor and good evaluation functions, but recently Monte Carlo approaches have demonstrated success in chess and other games.

The general idea of simulating moves into the future, observing the outcome, and using the outcome to determine which moves are good ones is one kind of reinforcement learning, which is covered in Chapter 22 .

5.5 Stochastic Games Stochastic games bring us a little closer to the unpredictability of real life by including a random element, such as the throwing of dice. Backgammon is a typical stochastic game that combines luck and skill. In the backgammon position of Figure 5.12 , White has rolled a 6–5 and has four possible moves (each of which moves one piece forward (clockwise) 5 positions, and one piece forward 6 positions).

Figure 5.12

A typical backgammon position. The goal of the game is to move all one’s pieces off the board. Black moves clockwise toward 25, and White moves counterclockwise toward 0. A piece can move to any

position unless multiple opponent pieces are there; if there is one opponent, it is captured and must start over. In the position shown, Black has rolled 6 − 5 and must choose among four legal moves: (5 − 11, 5 − 10) , (5 − 11, 19 − 24) , (5 − 10, 10 − 16) , and (5 − 11, 11 − 16) , where the notation (5 − 11, 11 − 16) means move one piece from position 5 to 11, and then move a piece from 11 to 16.

Stochastic game

At this point Black knows what moves can be made, but does not know what White is going to roll and thus does not know what White’s legal moves will be. That means Black cannot construct a standard game tree of the sort we saw in chess and tic-tac-toe. A game tree in backgammon must include chance nodes in addition to MAX and MIN nodes. Chance nodes are shown as circles in Figure 5.13 . The branches leading from each chance node denote the possible dice rolls; each branch is labeled with the roll and its probability. There are 36 ways to roll two dice, each equally likely; but because a 6–5 is the same as a 5–6, there are only 21 distinct rolls. The six doubles (1–1 through 6–6) each have a probability of 1/36, so we say P (1–1) = 1/36. The other 15 distinct rolls each have a 1/18 probability.

Figure 5.13

Schematic game tree for a backgammon position.

Chance nodes

The next step is to understand how to make correct decisions. Obviously, we still want to pick the move that leads to the best position. However, positions do not have definite minimax values. Instead, we can only calculate the expected value of a position: the average over all possible outcomes of the chance nodes.

Expected value

This leads us to the expectiminimax value for games with chance nodes, a generalization of the minimax value for deterministic games. Terminal nodes and MAX and MIN nodes work exactly the same way as before (with the caveat that the legal moves for MAX and MIN will depend on the outcome of the dice roll in the previous chance node). For chance nodes we compute the expected value, which is the sum of the value over all outcomes, weighted by the probability of each chance action:

s

EXPECTIMINIMAX( ) = ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ ⎪

s

s

UTILITY( ,MAX)

if IS-TERMINAL( )

maxa if EXPECTIMINIMAX(RESULT( , ))

if To-Move( ) = MAX

sa mina if EXPECTIMINIMAX(RESULT(s, a)) ∑ r P (r)EXPECTIMINIMAX(RESULT(s, r))

s if To-Move(s) = MIN if To-Move(s) = CHANCE

Expectiminimax value

where r represents a possible dice roll (or other chance event) and RESULT(s, r) is the same state as s, with the additional fact that the result of the dice roll is r.

5.5.1 Evaluation functions for games of chance As with minimax, the obvious approximation to make with expectiminimax is to cut the search off at some point and apply an evaluation function to each leaf. One might think that evaluation functions for games such as backgammon should be just like evaluation functions for chess—they just need to give higher values to better positions. But in fact, the presence of chance nodes means that one has to be more careful about what the values mean. Figure 5.14  shows what happens: with an evaluation function that assigns the values [1, 2, 3, 4] to the leaves, move a1 is best; with values [1, 20, 30, 400], move a2 is best. Hence, the program behaves totally differently if we make a change to some of the evaluation values, even if the preference order remains the same.

Figure 5.14

An order-preserving transformation on leaf values changes the best move.

It turns out that to avoid this problem, the evaluation function must return values that are a positive linear transformation of the probability of winning (or of the expected utility, for games that have outcomes other than win/lose). This relation to probability is an important and general property of situations in which uncertainty is involved, and we discuss it further in Chapter 16 .

If the program knew in advance all the dice rolls that would occur for the rest of the game, solving a game with dice would be just like solving a game without dice, which minimax does in

O(bm ) time, where b is the branching factor and m is the maximum depth of the

game tree. Because expectiminimax is also considering all the possible dice-roll sequences, it will take

O(bm nm ), where n is the number of distinct rolls. d

Even if the search is limited to some small depth , the extra cost compared with that of minimax makes it unrealistic to consider looking ahead very far in most games of chance. In backgammon

n is 21 and b is usually around 20, but in some situations can be as high as

4000 for dice rolls that are doubles. We could probably only manage three ply of search.

Another way to think about the problem is this: the advantage of alpha–beta is that it ignores future developments that just are not going to happen, given best play. Thus, it concentrates on likely occurrences. But in a game where a throw of two dice precedes each move, there are no likely sequences of moves; even the most likely move occurs only 1/2 of the time, because for the move to take place, the dice would first have to come out the right way to make it legal. This is a general problem whenever uncertainty enters the picture: the

possibilities are multiplied enormously, and forming detailed plans of action becomes pointless because the world probably will not play along.

It may have occurred to you that something like alpha–beta pruning could be applied to game trees with chance nodes. It turns out that it can. The analysis for MIN and MAX nodes is unchanged, but we can also prune chance nodes, using a bit of ingenuity. Consider the

C in Figure 5.13  and what happens to its value as we evaluate its children. Is it possible to find an upper bound on the value of C before we have looked at all its chance node

children? (Recall that this is what alpha–beta needs in order to prune a node and its subtree.)

At first sight, it might seem impossible because the value of

C is the average of its children’s

values, and in order to compute the average of a set of numbers, we must look at all the numbers. But if we put bounds on the possible values of the utility function, then we can arrive at bounds for the average without looking at every number. For example, say that all utility values are between

−2 and +2; then the value of leaf nodes is bounded, and in turn

we can place an upper bound on the value of a chance node without looking at all its children.

In games where the branching factor for chance nodes is high—consider a game like Yahtzee where you roll 5 dice on every turn—you may want to consider forward pruning that samples a smaller number of the possible chance branches. Or you may want to avoid using an evaluation function altogether, and opt for Monte Carlo tree search instead, where each playout includes random dice rolls.

5.6 Partially Observable Games Bobby Fischer declared that “chess is war,” but chess lacks at least one major characteristic of real wars, namely, partial observability. In the “fog of war,” the whereabouts of enemy units is often unknown until revealed by direct contact. As a result, warfare includes the use of scouts and spies to gather information and the use of concealment and bluff to confuse the enemy.

Partially observable games share these characteristics and are thus qualitatively different from the games in the preceding sections. Video games such as StarCraft are particularly challenging, being partially observable, multi-agent, nondeterministic, dynamic, and unknown.

In deterministic partially observable games, uncertainty about the state of the board arises entirely from lack of access to the choices made by the opponent. This class includes children’s games such as Battleship (where each player’s ships are placed in locations hidden from the opponent) and Stratego (where piece locations are known but piece types are hidden). We will examine the game of Kriegspiel, a partially observable variant of chess in which pieces are completely invisible to the opponent. Other games also have partially observable versions: Phantom Go, Phantom tic-tac-toe, and Screen Shogi.

Kriegspiel

5.6.1 Kriegspiel: Partially observable chess The rules of Kriegspiel are as follows: White and Black each see a board containing only their own pieces. A referee, who can see all the pieces, adjudicates the game and periodically makes announcements that are heard by both players. First, White proposes to the referee a move that would be legal if there were no black pieces. If the black pieces prevent the move, the referee announces “illegal,” and White keeps proposing moves until a legal one is found—learning more about the location of Black’s pieces in the process.

Once a legal move is proposed, the referee announces one or more of the following: “Capture on square X” if there is a capture, and “Check by D” if the black king is in check, where D is the direction of the check, and can be one of “Knight,” “Rank,” “File,” “Long diagonal,” or “Short diagonal.” If Black is checkmated or stalemated, the referee says so; otherwise, it is Black’s turn to move.

Kriegspiel may seem terrifyingly impossible, but humans manage it quite well and computer programs are beginning to catch up. It helps to recall the notion of a belief state as defined in Section 4.4  and illustrated in Figure 4.14 —the set of all logically possible board states given the complete history of percepts to date. Initially, White’s belief state is a singleton because Black’s pieces haven’t moved yet. After White makes a move and Black responds, White’s belief state contains 20 positions, because Black has 20 replies to any opening move. Keeping track of the belief state as the game progresses is exactly the problem of state estimation, for which the update step is given in Equation (4.6)  on Page 132. We can map Kriegspiel state estimation directly onto the partially observable, nondeterministic framework of Section 4.4  if we consider the opponent as the source of nondeterminism; that is, the RESULTS of White’s move are composed from the (predictable) outcome of White’s own move and the unpredictable outcome given by Black’s reply.4

4 Sometimes, the belief state will become too large to represent just as a list of board states, but we will ignore this issue for now; Chapters 7  and 8  suggest methods for compactly representing very large belief states.

Given a current belief state, White may ask, “Can I win the game?” For a partially observable game, the notion of a strategy is altered; instead of specifying a move to make for each possible move the opponent might make, we need a move for every possible percept sequence that might be received.

For Kriegspiel, a winning strategy, or guaranteed checkmate, is one that, for each possible percept sequence, leads to an actual checkmate for every possible board state in the current belief state, regardless of how the opponent moves. With this definition, the opponent’s belief state is irrelevant—the strategy has to work even if the opponent can see all the pieces. This greatly simplifies the computation. Figure 5.15  shows part of a guaranteed checkmate for the KRK (king and rook versus king) endgame. In this case, Black has just one piece (the king), so a belief state for White can be shown in a single board by marking each possible position of the Black king.

Figure 5.15

Part of a guaranteed checkmate in the KRK endgame, shown on a reduced board. In the initial belief state, Black’s king is in one of three possible locations. By a combination of probing moves, the strategy narrows this down to one. Completion of the checkmate is left as an exercise.

Guaranteed checkmate

The general AND-OR search algorithm can be applied to the belief-state space to find guaranteed checkmates, just as in Section 4.4 . The incremental belief-state algorithm

mentioned in Section 4.4.2  often finds midgame checkmates up to depth 9—well beyond the abilities of most human players.

In addition to guaranteed checkmates, Kriegspiel admits an entirely new concept that makes no sense in fully observable games: probabilistic checkmate. Such checkmates are still required to work in every board state in the belief state; they are probabilistic with respect to randomization of the winning player’s moves. To get the basic idea, consider the problem of finding a lone black king using just the white king. Simply by moving randomly, the white king will eventually bump into the black king even if the latter tries to avoid this fate, since Black cannot keep guessing the right evasive moves indefinitely. In the terminology of probability theory, detection occurs with probability 1.

Probabilistic checkmate

The KBNK endgame—king, bishop and knight versus king—is won in this sense; White presents Black with an infinite random sequence of choices, for one of which Black will guess incorrectly and reveal his position, leading to checkmate. On the other hand, the KBBK endgame is won with probability 1 − ϵ. White can force a win only by leaving one of his bishops unprotected for one move. If Black happens to be in the right place and captures the bishop (a move that would be illegal if the bishops are protected), the game is drawn. White can choose to make the risky move at some randomly chosen point in the middle of a very long sequence, thus reducing ϵ to an arbitrarily small constant, but cannot reduce ϵ to zero.

Sometimes a checkmate strategy works for some of the board states in the current belief state but not others. Trying such a strategy may succeed, leading to an accidental checkmate—accidental in the sense that White could not know that it would be checkmate— if Black’s pieces happen to be in the right places. (Most checkmates in games between humans are of this accidental nature.) This idea leads naturally to the question of how likely it is that a given strategy will win, which leads in turn to the question of how likely it is that each board state in the current belief state is the true board state.

Accidental checkmate

One’s first inclination might be to propose that all board states in the current belief state are equally likely—but this can’t be right. Consider, for example, White’s belief state after Black’s first move of the game. By definition (assuming that Black plays optimally), Black must have played an optimal move, so all board states resulting from suboptimal moves ought to be assigned zero probability.

This argument is not quite right either, because each player’s goal is not just to move pieces to the right squares but also to minimize the information that the opponent has about their location. Playing any predictable “optimal” strategy provides the opponent with information. Hence, optimal play in partially observable games requires a willingness to play somewhat randomly. (This is why restaurant hygiene inspectors do random inspection visits.) This means occasionally selecting moves that may seem “intrinsically” weak—but they gain strength from their very unpredictability, because the opponent is unlikely to have prepared any defense against them.

From these considerations, it seems that the probabilities associated with the board states in the current belief state can only be calculated given an optimal randomized strategy; in turn, computing that strategy seems to require knowing the probabilities of the various states the board might be in. This conundrum can be resolved by adopting the game-theoretic notion of an equilibrium solution, which we pursue further in Chapter 17 . An equilibrium specifies an optimal randomized strategy for each player. Computing equilibria is too expensive for Kriegspiel. At present, the design of effective algorithms for general Kriegspiel play is an open research topic. Most systems perform bounded-depth look-ahead in their own belief-state space, ignoring the opponent’s belief state. Evaluation functions resemble those for the observable game but include a component for the size of the belief state— smaller is better! We will return to partially observable games under the topic of Game Theory in Section 18.2 .

5.6.2 Card games Card games such as bridge, whist, hearts, and poker feature stochastic partial observability, where the missing information is generated by the random dealing of cards.

At first sight, it might seem that these card games are just like dice games: the cards are dealt randomly and determine the moves available to each player, but all the “dice” are rolled at the beginning! Even though this analogy turns out to be incorrect, it suggests an algorithm: treat the start of the game as a chance node with every possible deal as an outcome, and then use the EXPECTIMINIMAX formula to pick the best move. Note that in this approach the only chance node is the root node; after that the game becomes fully observable. This approach is sometimes called averaging over clairvoyance because it assumes that once the actual deal has occurred, the game becomes fully observable to both players. Despite its intuitive appeal, the strategy can lead one astray. Consider the following story:

DAY 1: Road A leads to a pot of gold; Road B leads to a fork. You can see that the left fork leads to two pots of gold, and the right fork leads to you being run over by a bus. DAY 2: Road A leads to a pot of gold; Road B leads to a fork. You can see that the right fork leads to two pots of gold, and the left fork leads to you being run over by a bus. DAY 3: Road A leads to a pot of gold; Road B leads to a fork. You are told that one fork leads to two pots of gold, and one fork leads to you being run over by a bus. Unfortunately you don’t know which fork is which.

Averaging over clairvoyance leads to the following reasoning: on Day 1, B is the right choice; on Day 2, B is the right choice; on Day 3, the situation is the same as either Day 1 or Day 2, so B must still be the right choice.

Now we can see how averaging over clairvoyance fails: it does not consider the belief state that the agent will be in after acting. A belief state of total ignorance is not desirable, especially when one possibility is certain death. Because it assumes that every future state will automatically be one of perfect knowledge, the clairvoyance approach never selects actions that gather information (like the first move in Figure 5.15 ); nor will it choose actions that hide information from the opponent or provide information to a partner, because it assumes that they already know the information; and it will never bluff in poker,5 because it assumes the opponent can see its cards. In Chapter 17 , we show how to construct algorithms that do all these things by virtue of solving the true partially observable decision problem, resulting in an optimal equilibrium strategy (see Section 18.2 ).

5 Bluffing—betting as if one’s hand is good, even when it’s not—is a core part of poker strategy.

Bluff

Despite the drawbacks, averaging over clairvoyance can be an effective strategy, with some tricks to make it work better. In most card games, the number of possible deals is rather large. For example, in bridge play, each player sees just two of the four hands; there are two unseen hands of 13 cards each, so the number of deals is (

26 13

) = 10,400,600. Solving even

one deal is quite difficult, so solving ten million is out of the question. One way to deal with this huge number is with abstraction: i.e. by treating similar hands as identical. For example, it is very important which aces and kings are in a hand, but whether the hand has a 4 or 5 is not as important, and can be abstracted away.

Another way to deal with the large number is forward pruning: consider only a small random sample of

N deals, and again calculate the EXPECTIMINIMAX score. Even for fairly small

N —say, 100 to 1,000—this method gives a good approximation. It can also be applied to

deterministic games such as Kriegspiel, where we sample over possible states of the game rather than over possible deals, as long as we have some way to estimate how likely each state is. It can also be helpful to do heuristic search with a depth cutoff rather than to search the entire game tree.

So far we have assumed that each deal is equally likely. That makes sense for games like whist and hearts. But for bridge, play is preceded by a bidding phase in which each team indicates how many tricks it expects to win. Since players bid based on the cards they hold, the other players learn something about the probability

P (s) of each deal. Taking this into

account in deciding how to play the hand is tricky, for the reasons mentioned in our description of Kriegspiel: players may bid in such a way as to minimize the information conveyed to their opponents.

Computers have reached a superhuman level of performance in poker. The poker program Libratus took on four of the top poker players in the world in a 20-day match of no-limit Texas hold ’em and decisively beat them all. Since there are so many possible states in poker, Libratus uses abstraction to reduce the state space: it might consider the two hands AAA72 and AAA64 to be equivalent (they’re both “three aces and some low cards”), and it might consider a bet of 200 dollars to be the same as 201 dollars. But Libratus also monitors

the other players, and if it detects they are exploiting an abstraction, it will do some additional computation overnight to plug that hole. Overall it used 25 million CPU hours on a supercomputer to pull off the win.

The computational costs incurred by Libratus (and similar costs by ALPHAZERO and other systems) suggests that world champion game play may not be achievable for researchers with limited budgets. To some extent that is true: just as you should not expect to be able to assemble a champion Formula One race car out of spare parts in your garage, there is an advantage to having access to supercomputers or specialty hardware such as Tensor Processing Units. That is particularly true when training a system, but training could also be done via crowdsourcing. For example the open-source LEELAZERO system is a reimplementation of ALPHAZERO that trains through self-play on the computers of volunteer participants. Once trained, the computational requirements for actual tournament play are modest. ALPHASTAR won StarCraft II games running on a commodity desktop with a single GPU, and ALPHAZERO could have been run in that mode.

5.7 Limitations of Game Search Algorithms Because calculating optimal decisions in complex games is intractable, all algorithms must make some assumptions and approximations. Alpha–beta search uses the heuristic evaluation function as an approximation, and Monte Carlo search computes an approximate average over a random selection of playouts. The choice of which algorithm to use depends in part on the features of each game: when the branching factor is high or it is difficult to define an evaluation function, Monte Carlo search is preferred. But both algorithms suffer from fundamental limitations.

One limitation of alpha–beta search is its vulnerability to errors in the heuristic function. Figure 5.16  shows a two-ply game tree for which minimax suggests taking the right-hand branch because 100 > 99. That is the correct move if the evaluations are all exactly accurate. But suppose that the evaluation of each node has an error that is independent of other nodes and is randomly distributed with a standard deviation of σ. Then the left-hand branch is actually better 71% of the time when σ = 5, and 58% of the time when σ = 2 (because one of the four right-hand leaves is likely to slip below 99 in these cases). If errors in the evaluation function are not independent, then the chance of a mistake rises. It is difficult to compensate for this because we don’t have a good model of the dependencies between the values of sibling nodes.

Figure 5.16

A two-ply game tree for which heuristic minimax may make an error.

A second limitation of both alpha–beta and Monte Carlo is that they are designed to calculate (bounds on) the values of legal moves. But sometimes there is one move that is obviously best (for example when there is only one legal move), and in that case, there is no point wasting computation time to figure out the value of the move—it is better to just make the move. A better search algorithm would use the idea of the utility of a node expansion, selecting node expansions of high utility—that is, ones that are likely to lead to the discovery of a significantly better move. If there are no node expansions whose utility is higher than their cost (in terms of time), then the algorithm should stop searching and make a move. This works not only for clear-favorite situations but also for the case of symmetrical moves, for which no amount of search will show that one move is better than another.

This kind of reasoning about what computations to do is called metareasoning (reasoning about reasoning). It applies not just to game playing but to any kind of reasoning at all. All computations are done in the service of trying to reach better decisions, all have costs, and all have some likelihood of resulting in a certain improvement in decision quality. Monte Carlo search does attempt to do metareasoning to allocate resources to the most important parts of the tree, but does not do so in an optimal way.

Metareasoning

A third limitation is that both alpha-beta and Monte Carlo do all their reasoning at the level of individual moves. Clearly, humans play games differently: they can reason at a more abstract level, considering a higher-level goal—for example, trapping the opponent’s queen —and using the goal to selectively generate plausible plans. In Chapter 11  we will study this type of planning, and in Section 11.4  we will show how to plan with a hierarchy of abstract to concrete representations.

A fourth issue is the ability to incorporate machine learning into the game search process. Early game programs relied on human expertise to hand-craft evaluation functions, opening books, search strategies, and efficiency tricks. We are just beginning to see programs like ALPHAZERO (Silver et al., 2018), which relied on machine learning from self-play rather than

game-specific human-generated expertise. We cover machine learning in depth starting with Chapter 19 .

Summary We have looked at a variety of games to understand what optimal play means, to understand how to play well in practice, and to get a feel for how an agent should act in any type of adversarial environment. The most important ideas are as follows:

A game can be defined by the initial state (how the board is set up), the legal actions in each state, the result of each action, a terminal test (which says when the game is over), and a utility function that applies to terminal states to say who won and what the final score is. In two-player, discrete, deterministic, turn-taking zero-sum games with perfect information, the minimax algorithm can select optimal moves by a depth-first enumeration of the game tree. The alpha–beta search algorithm computes the same optimal move as minimax, but achieves much greater efficiency by eliminating subtrees that are provably irrelevant. Usually, it is not feasible to consider the whole game tree (even with alpha–beta), so we need to cut the search off at some point and apply a heuristic evaluation function that estimates the utility of a state. An alternative called Monte Carlo tree search (MCTS) evaluates states not by applying a heuristic function, but by playing out the game all the way to the end and using the rules of the game to see who won. Since the moves chosen during the playout may not have been optimal moves, the process is repeated multiple times and the evaluation is an average of the results. Many game programs precompute tables of best moves in the opening and endgame so that they can look up a move rather than search. Games of chance can be handled by expectiminimax, an extension to the minimax algorithm that evaluates a chance node by taking the average utility of all its children, weighted by the probability of each child. In games of imperfect information, such as Kriegspiel and poker, optimal play requires reasoning about the current and future belief states of each player. A simple approximation can be obtained by averaging the value of an action over each possible configuration of missing information. Programs have soundly defeated champion human players at chess, checkers, Othello, Go, poker, and many other games. Humans retain the edge in a few games of imperfect

information, such as bridge and Kriegspiel. In video games such as StarCraft and Dota 2, programs are competitive with human experts, but part of their success may be due to their ability to perform many actions very quickly.

Bibliographical and Historical Notes In 1846, Charles Babbage discussed the feasibility of computer chess and checkers (Morrison and Morrison, 1961). He did not understand the exponential complexity of search trees, claiming “the combinations involved in the Analytical Engine enormously surpassed any required, even by the game of chess.” Babbage also designed, but did not build, a special-purpose machine for playing tic-tac-toe. The first game-playing machine was built around 1890 by the Spanish engineer Leonardo Torres y Quevedo. It specialized in the “KRK” (king and rook versus king) chess endgame, guaranteeing a win when the side with the rook has the move. The minimax algorithm is traced to a 1912 paper by Ernst Zermelo, the developer of modern set theory.

Game playing was one of the first tasks undertaken in AI, with early efforts by such pioneers as Konrad Zuse (1945), Norbert Wiener in his book Cybernetics (1948), and Alan Turing (1953). But it was Claude Shannon’s article Programming a Computer for Playing Chess (1950) that laid out all the major ideas: a representation for board positions, an evaluation function, quiescence search, and some ideas for selective game-tree search. Slater (1950) had the idea of an evaluation function as a linear combination of features, and stressed the mobility feature in chess.

John McCarthy conceived the idea of alpha–beta search in 1956, although the idea did not appear in print until later (Hart and Edwards, 1961). Knuth and Moore (1975) proved the correctness of alpha–beta and analysed its time complexity, while Pearl (1982b) showed alpha–beta to be asymptotically optimal among all fixed-depth game-tree search algorithms.

Berliner (1979) introduced B*, a heuristic search algorithm that maintains interval bounds on the possible value of a node in the game tree rather than giving it a single point-valued estimate. David McAllester’s (1988) conspiracy number search expands leaf nodes that, by changing their values, could cause the program to prefer a new move at the root of the tree. MGSS* (Russell and Wefald, 1989) uses the decision-theoretic techniques of Chapter 16  to estimate the value of expanding each leaf in terms of the expected improvement in decision quality at the root.

The SSS* algorithm (Stockman, 1979) can be viewed as a two-player A* that never expands more nodes than alpha–beta. The memory requirements make it impractical, but a linearspace version has been developed from the RBFS algorithm (Korf and Chickering, 1996). Baum and Smith (1997) propose a probability-based replacement for minimax, showing that it results in better choices in certain games. The expectiminimax algorithm was proposed by Donald Michie (1966). Bruce Ballard (1983) extended alpha–beta pruning to cover trees with chance nodes.

Pearl’s book Heuristics (1984) thoroughly analyzes many game-playing algorithms.

Monte Carlo simulation was pioneered by Metropolis and Ulam (1949) for calculations related to the development of the atomic bomb. Monte Carlo tree search (MCTS) was introduced by Abramson (1987). Tesauro and Galperin (1997) showed how a Monte Carlo search could be combined with an evaluation function for the game of backgammon. Early playout termination is studied by Lorentz (2015). ALPHAGO terminated playouts and applied an evaluation function (Silver et al., 2016). Kocsis and Szepesvari (2006) refined the approach with the “Upper Confidence Bounds applied to Trees” selection mechanism. Chaslot et al. (2008) show how MCTS can be applied to a variety of games and Browne et al. (2012) give a survey.

Koller and Pfeffer (1997) describe a system for completely solving partially observable games. It handles larger games than previous systems, but not the full version of complex games like poker and bridge. Frank et al. (1998) describe several variants of Monte Carlo search for partially observable games, including one where MIN has complete information but MAX does not. Schofield and Thielschers (2015) adapt a general game-playing system for partially observable games.

Ferguson hand-derived randomized strategies for winning Kriegspiel with a bishop and knight (1992) or two bishops (1995) against a king. The first Kriegspiel programs concentrated on finding endgame checkmates and performed AND–OR search in belief-state space (Sakuta and Iida, 2002; Bolognesi and Ciancarini, 2003). Incremental belief-state algorithms enabled much more complex midgame checkmates to be found (Russell and Wolfe, 2005; Wolfe and Russell, 2007), but efficient state estimation remains the primary obstacle to effective general play (Parker et al., 2005). Ciancarini and Favini (2010) apply

MCTS to Kriegspiel, and Wang et al. (2018b) describe a belief-state version of MCTS for Phantom Go.

Chess milestones have been marked by successive winners of the Fredkin Prize: BELLE (Condon and Thompson 1982), the first program to achieve master status; DEEP THOUGHT (Hsu et al., 1990), the first to reach international master status; and Deep Blue (Campbell et al., 2002; Hsu, 2004), which defeated world champion Garry Kasparov in a 1997 exhibition match. Deep Blue ran alpha–beta search at over 100 million positions per second, and could generate singular extensions to occasionally reach a depth of 40 ply.

The top chess programs today (e.g., STOCKFISH, KOMODO, HOUDINI) far exceed any human player. These programs have reduced the effective branching factor to less than 3 (compared with the actual branching factor of about 35), searching to about 20 ply at a speed of about a million nodes per second on a standard 1-core computer. They use pruning techniques such as the null move heuristic, which generates a good lower bound on the value of a position, using a shallow search in which the opponent gets to move twice at the beginning. Also important is futility pruning, which helps decide in advance which moves will cause a beta cutoff in the successor nodes. SUNFISH is a simplified chess program for teaching purposes; the core is less than 200 lines of Python.

Null move

Futility pruning

The idea of retrograde analysis for computing endgame tables is due to Bellman (1965). Using this idea, Ken Thompson (1986, 1996) and Lewis Stiller (1992, 1996) solved all chess endgames with up to five pieces. Stiller discovered one case where a forced mate existed but required 262 moves; this caused some consternation because the rules of chess require a capture or pawn move to occur within 50 moves, or else a draw is declared. In 2012 Vladimir Makhnychev and Victor Zakharov compiled the Lomonosov Endgame Tablebase,

which solved all endgame positions with up to seven pieces—some require over 500 moves without a capture. The 7-piece table consumes 140 terabytes; an 8-piece table would be 100 times larger.

In 2017, ALPHAZERO (Silver et al., 2018) defeated STOCKFISH (the 2017 TCEC computer chess champion) in a 1000-game trial, with 155 wins and 6 losses. Additional matches also resulted in decisive wins for ALPHAZERO, even when it was given only 1/10th the time allotted to STOCKFISH.

Grandmaster Larry Kaufman was surprised at the sucess of this Monte Carlo program and noted, “It may well be that the current dominance of minimax chess engines may be at an end, but it’s too soon to say so.” Garry Kasparov commented “It’s a remarkable achievement, even if we should have expected it after ALPHAGO. It approaches the Type B human-like approach to machine chess dreamt of by Claude Shannon and Alan Turing instead of brute force.” He went on to predict “Chess has been shaken to its roots by ALPHAZERO, but this is only a tiny example of what is to come. Hidebound disciplines like education and medicine will also be shaken” (Sadler and Regan, 2019).

Checkers was the first of the classic games played by a computer (Strachey, 1952). Arthur Samuel (1959, 1967) developed a checkers program that learned its own evaluation function through self-play using a form of reinforcement learning. It is quite an achievement that Samuel was able to create a program that played better than he did, on an IBM 704 computer with only 10,000 words of memory and a 0.000001 GHz processor. MENACE—the Machine Educable Noughts And Crosses Engine (Michie, 1963)—also used reinforcement learning to become competent at tic-tac-toe. Its processor was even slower: a collection of 304 matchboxes holding colored beads to represent the best learned move in each position.

In 1992, Jonathan Schaeffer’s CHINOOK checkers program challenged the legendary Marion Tinsley, who had been world champion for over 20 years. Tinsley won the match, but lost two games—the fourth and fifth losses in his entire career. After Tinsley retired for health reasons, CHINOOK took the crown. The saga was chronicled by Schaeffer (2008).

In 2007 Schaeffer and his team “solved” checkers (Schaeffer et al., 2007): the game is a draw with perfect play. Richard Bellman (1965) had predicted this: “In checkers, the number of possible moves in any given situation is so small that we can confidently expect a complete

digital computer solution to the problem of optimal play in this game.” Bellman did not anticipate the scale of the effort: the endgame table for 10 pieces has 39 trillion entries. Given this table, it took 18 CPU-years of alpha–beta search to solve the game.

I. J. Good, who was taught the Game of Go by Alan Turing, wrote (1965a) “ I think it will be even more difficult to programme a computer to play a reasonable game of Go than of chess.” He was right: through 2015, Go programs played only at an amateur level. The early literature is summarized by Bouzy and Cazenave (2001) and Müller (2002).

Visual pattern recognition was proposed as a promising technique for Go by Zobrist (1970), while Schraudolph et al. (1994) analyzed the use of reinforcement learning, Lubberts and Miikkulainen (2001) recommended neural networks, and Brügmann (1993) introduced Monte Carlo tree search to Go. ALPHAGO (Silver et al., 2016) put those four ideas together to defeat top-ranked professionals Lee Sedol (by a score of 4–1 in 2015) and Ke Jie (by 3–0 in 2016).

Ke Jie remarked “After humanity spent thousands of years improving our tactics, computers tell us that humans are completely wrong. I would go as far as to say not a single human has touched the edge of the truth of Go.” Lee Sedol retired from Go, lamenting, “Even if I became the number one, there is an entity that cannot be defeated.”

In 2018, ALPHAZERO surpassed ALPHAGO at Go, and also defeated top programs in chess and shogi, learning through self-play without any expert human knowledge and without access to any past games. (It does, of course, rely on humans to define the basic architecture as Monte Carlo tree search with deep neural networks and reinforcement learning, and to encode the rules of the game.) The success of ALPHAZERO has led to increased interest in reinforcement learning as a key component of general AI (see Chapter 22 ). Going one step further, the MUZERO system operates without even being told the rules of the game it is playing—it has to figure out the rules by making plays. MUZERO achieved state-of-the-art results in Pacman, chess, Go, and 75 Atari games (Schrittwieser et al., 2019). It learns to generalize; for example, it learns that in Pacman the “up” action moves the player up a square (unless there is a wall there), even though it has only observed the result of the “up” action in a small percentage of the locations on the board.

Othello, also called Reversi, has a smaller search space than chess, but defining an evaluation function is difficult, because material advantage is not as important as mobility. Programs have been at superhuman level since 1997 (Buro, 2002).

Backgammon, a game of chance, was analyzed mathematically by Gerolamo Cardano (1663), and taken up for computer play with the BKG program (Berliner, 1980b), which used a manually constructed evaluation function and searched only to depth 1. It was the first program to defeat a human world champion at a major game (Berliner, 1980a), although Berliner readily acknowledged that BKG was very lucky with the dice. Gerry Tesauro’s (1995) TD-GAMMON learned its evaluation function using neural networks trained by selfplay. It consistently played at world champion level and caused human analysts to change their opinion on the best opening move for several dice rolls.

Poker, like Go, has seen surprising advances in recent years. Bowling et al. (2015) used game theory (see Section 18.2 ) to determine the exact optimal strategy for a version of poker with just two players and a fixed number of raises with fixed bet sizes. In 2017, for the first time, champion poker players were beaten at heads-up (two player) no-limit Texas hold ’em in two separate matches against the programs Libratus (Brown and Sandholm, 2017) and DeepStack (Moravčík et al., 2017). In 2019, Pluribus (Brown and Sandholm, 2019) defeated top-ranked professional human players in Texas hold ’em games with six players. Multiplayer games introduce some strategic concerns that we will cover in Chapter 18 . Petosa and Balch (2019) implement a multiplayer version of ALPHAZERO.

BRIDGE: Smith et al. (1998) report on how BRIDGE BARON won the 1998 computer bridge championship, using hierarchical plans (see Chapter 11 ) and high-level actions, such as finessing and squeezing, that are familiar to bridge players. Ginsberg (2001) describes how his GIB program, based on Monte Carlo simulation (first proposed for bridge by Levy (1989)), won the following computer championship and did surprisingly well against expert human players. In the 21st century, the computer bridge championship has been dominated by two commercial programs, JACK and WBRIDGE5. Neither has been described in published articles, but both are believed to use Monte Carlo techniques. In general, bridge programs are at human champion level when actually playing the hands, but lag behind in the bidding phase, because they do not completely understand the conventions used by humans to communicate with their partners. Bridge programmers have concentrated more on

producing useful and educational programs that encourage people to take up the game, rather than on defeating human champions.

Scrabble is a game where amateur human players have difficulty coming up with highscoring words, but for a computer, it is easy to find the highest possible score for a given hand (Gordon, 1994); the hard part is planning ahead in a partially observable, stochastic game. Nevertheless, in 2006, the QUACKLE program defeated the former world champion, David Boys, 3–2. Boys took it well, stating, “It’s still better to be a human than to be a computer.” A good description of a top program, MAVEN, is given by Sheppard (2002).

Video games such as StarCraft II involve hundreds of partially observable units moving in real time with high-dimensional near-continuous6 observation and action spaces with complex rules. Oriol Vinyals, who was Spain’s StarCraft champion at age 15, described how the game can serve as a testbed and grand challenge for reinforcement learning (Vinyals et al., 2017a). In 2019, Vinyals and the team at DeepMind unveiled the ALPHASTAR program, based on deep learning and reinforcement learning, which defeated expert human players 10 games to 1, and ranks in the top 0.02% of officially ranked human players (Vinyals et al., 2019). ALPHASTAR took steps to limit the number of actions per minute it could perform in critical bursts, in response to critics who felt it had an unfair advantage.

6 To a human player, it appears that objects move continuously, but they are actually discrete at the level of a pixel on the screen.

Computers have defeated top humans in other popular video games such as Super Smash Bros. (Firoiu et al., 2017), Quake III (Jaderberg et al., 2019), and Dota 2 (Fernandez and Mahlmann, 2018), all using deep learning techniques.

Physical games such as robotic soccer (Visser et al., 2008; Barrett and Stone, 2015), billiards (Lam and Greenspan, 2008; Archibald et al., 2009), and ping-pong (Silva et al., 2015) have attracted some attention in AI. They combine all the complications of video games with the messiness of the real world.

Computer game competitions occur annually, including the Computer Olympiads since 1989. The General Game Competition (Love et al., 2006) tests programs that must learn to play an unknown game given only a logical description of the rules of the game. The International Computer Games Association (ICGA) publishes the ICGA Journal and runs two alternating biennial conferences, The International Conference on Computers and Games

(ICCG or CG) and the International Conference on Advances in Computer Games (ACG). The IEEE publishes IEEE Transactions on Games and runs an annual Conference on Computational Intelligence and Games.

Chapter 6 Constraint Satisfaction Problems In which we see how treating states as more than just little black boxes leads to new search methods and a deeper understanding of problem structure. Chapters 3  and 4  explored the idea that problems can be solved by searching the state space: a graph where the nodes are states and the edges between them are actions. We saw that domain-specific heuristics could estimate the cost of reaching the goal from a given state, but that from the point of view of the search algorithm, each state is atomic, or indivisible—a black box with no internal structure. For each problem we need domainspecific code to describe the transitions between states.

In this chapter we break open the black box by using a factored representation for each state: a set of variables, each of which has a value. A problem is solved when each variable has a value that satisfies all the constraints on the variable. A problem described this way is called a constraint satisfaction problem, or CSP.

Constraint satisfaction problem

CSP search algorithms take advantage of the structure of states and use general rather than domain-specific heuristics to enable the solution of complex problems. The main idea is to eliminate large portions of the search space all at once by identifying variable/value combinations that violate the constraints. CSPs have the additional advantage that the actions and transition model can be deduced from the problem description.

6.1 Defining Constraint Satisfaction Problems A constraint satisfaction problem consists of three components,

X,D, and C :

X is a set of variables, {X , … ,Xn }. 1

D is a set of domains, {D , … ,Dn }, one for each variable. 1

C is a set of constraints that specify allowable combinations of values. Di , consists of a set of allowable values, {v , … ,vk }, for variable Xi . For example, a Boolean variable would have the domain {true,false}. Different variables can have different domains of different sizes. Each constraint Cj consists of a pair ⟨scope,rel⟩, where scope is a tuple of variables that participate in the constraint and rel is a relation that defines A domain,

1

the values that those variables can take on. A relation can be represented as an explicit set of all tuples of values that satisfy the constraint, or as a function that can compute whether a

X both have the domain {1,2,3}, then the constraint saying that X must be greater than X can be written as ⟨(X ,X ),{(3,1),(3,2),(2,1)}⟩ or as ⟨(X ,X ),X > X ⟩. tuple is a member of the relation. For example, if

X

1

and

1

1

2

2

2

1

2

1

2

Relation

X

v X

v

CSPs deal with assignments of values to variables, { i = i , j = j , …}. An assignment that does not violate any constraints is called a consistent or legal assignment. A complete assignment is one in which every variable is assigned a value, and a solution to a CSP is a consistent, complete assignment. A partial assignment is one that leaves some variables unassigned, and a partial solution is a partial assignment that is consistent. Solving a CSP is an NP-complete problem in general, although there are important subclasses of CSPs that can be solved very efficiently.

Assignments

Consistent

Complete assignment

Solution

Partial assignment

Partial solution

6.1.1 Example problem: Map coloring Suppose that, having tired of Romania, we are looking at a map of Australia showing each of its states and territories (Figure 6.1(a) ). We are given the task of coloring each region either red, green, or blue in such a way that no two neighboring regions have the same color. To formulate this as a CSP, we define the variables to be the regions:

X Figure 6.1

WA NT Q NSW V SA T

= {

,

,

,

,

,

,

}.

(a) The principal states and territories of Australia. Coloring this map can be viewed as a constraint satisfaction problem (CSP). The goal is to assign colors to each region so that no neighboring regions have the same color. (b) The map-coloring problem represented as a constraint graph.

The domain of every variable is the set

Di = {red,green,blue}. The constraints require

neighboring regions to have distinct colors. Since there are nine places where regions border, there are nine constraints:

C

=

SA ≠ WA,SA ≠ NT ,SA ≠ Q,SA ≠ NSW ,SA ≠ V , WA ≠ NT ,NT ≠ Q,Q ≠ NSW ,NSW ≠ V }.

{

Here we are using abbreviations;

SA ≠ WA is a shortcut for ⟨(SA,WA),SA ≠ WA⟩, where

SA ≠ WA can be fully enumerated in turn as {(

red,green),(red,blue),(green,red),(green,blue),(blue,red),(blue,green)}.

There are many possible solutions to this problem, such as

WA = red,NT = green,Q = red,NSW = green,V

{

=

red,SA = blue,T = red }.

It can be helpful to visualize a CSP as a constraint graph, as shown in Figure 6.1(b) . The nodes of the graph correspond to variables of the problem, and an edge connects any two variables that participate in a constraint.

Constraint graph

Why formulate a problem as a CSP? One reason is that the CSPs yield a natural representation for a wide variety of problems; it is often easy to formulate a problem as a CSP. Another is that years of development work have gone into making CSP solvers fast and efficient. A third is that a CSP solver can quickly prune large swathes of the search space that an atomic state-space searcher cannot. For example, once we have chosen {

SA blue =

}

in the Australia problem, we can conclude that none of the five neighboring variables can take on the value

blue. A search procedure that does not use constraints would have to

consider 3

assignments for the five neighboring variables; with constraints we have

only

2

5

5

= 243

= 32

assignments to consider, a reduction of 87%.

In atomic state-space search we can only ask: is this specific state a goal? No? What about this one? With CSPs, once we find out that a partial assignment violates a constraint, we can immediately discard further refinements of the partial assignment. Furthermore, we can see why the assignment is not a solution—we see which variables violate a constraint—so we can focus attention on the variables that matter. As a result, many problems that are intractable for atomic state-space search can be solved quickly when formulated as a CSP.

6.1.2 Example problem: Job-shop scheduling Factories have the problem of scheduling a day’s worth of jobs, subject to various constraints. In practice, many of these problems are solved with CSP techniques. Consider the problem of scheduling the assembly of a car. The whole job is composed of tasks, and we can model each task as a variable, where the value of each variable is the time that the task starts, expressed as an integer number of minutes. Constraints can assert that one task must occur before another—for example, a wheel must be installed before the hubcap is put on—and that only so many tasks can go on at once. Constraints can also specify that a task takes a certain amount of time to complete.

We consider a small part of the car assembly, consisting of 15 tasks: install axles (front and back), affix all four wheels (right and left, front and back), tighten nuts for each wheel, affix hubcaps, and inspect the final assembly. We can represent the tasks with 15 variables:

X

=

AxleF AxleB WheelRF WheelLF WheelRB WheelLB NutsRF NutsLF NutsRB NutsLB CapRF CapLF CapRB CapLB Inspect

{

,

,

,

,

,

,

,

,

,

,

,

,

,

,

}.

Next, we represent precedence constraints between individual tasks. Whenever a task must occur before task

T , and task T 2

1

takes duration

d

1

T

1

to complete, we add an arithmetic

constraint of the form

T

1

+

d

1

≤

T. 2

Precedence constraint

In our example, the axles have to be in place before the wheels are put on, and it takes 10 minutes to install an axle, so we write

AxleF + 10 ≤ WheelRF ; AxleF + 10 ≤ WheelLF ; AxleB + 10 ≤ WheelRB ; AxleB + 10 ≤ WheelLB . Next we say that for each wheel, we must affix the wheel (which takes 1 minute), then tighten the nuts (2 minutes), and finally attach the hubcap (1 minute, but not represented yet):

WheelRF + 1 ≤ NutsRF ; WheelLF + 1 ≤ NutsLF ; WheelRB + 1 ≤ NutsRB ; WheelLB + 1 ≤ NutsLB ;

NutsRF + 2 ≤ CapRF ; NutsLF + 2 ≤ CapLF ; NutsRB + 2 ≤ CapRB ; NutsLB + 2 ≤ CapLB .

Suppose we have four workers to install wheels, but they have to share one tool that helps put the axle in place. We need a disjunctive constraint to say that

AxleF and AxleB must

not overlap in time; either one comes first or the other does:

(

AxleF + 10 ≤ AxleB )

Disjunctive constraint

or

(

AxleB + 10 ≤ AxleF )

This looks like a more complicated constraint, combining arithmetic and logic. But it still reduces to a set of pairs of values that

AxleF and AxleB can take on.

We also need to assert that the inspection comes last and takes 3 minutes. For every variable except

Inspect we add a constraint of the form X + dX ≤ Inspect. Finally, suppose there is a

requirement to get the whole assembly done in 30 minutes. We can achieve that by limiting the domain of all variables:

Di = {0,1,2,3, … ,30}. This particular problem is trivial to solve, but CSPs have been successfully applied to jobshop scheduling problems like this with thousands of variables.

6.1.3 Variations on the CSP formalism The simplest kind of CSP involves variables that have discrete, finite domains. Mapcoloring problems and scheduling with time limits are both of this kind. The 8-queens problem (Figure 4.3 ) can also be viewed as a finite-domain CSP, where the variables

Q1 , … ,Q8 correspond to the queens in columns 1 to 8, and the domain of each variable specifies the possible row numbers for the queen in that column, Di = {1,2,3,4,5,6,7,8}. The constraints say that no two queens can be in the same row or diagonal.

Discrete domain

Finite domain

A discrete domain can be infinite, such as the set of integers or strings. (If we didn’t put a deadline on the job-scheduling problem, there would be an infinite number of start times for each variable.) With infinite domains, we must use implicit constraints like

T1 + d 1 ≤ T2

rather than explicit tuples of values. Special solution algorithms (which we do not discuss here) exist for linear constraints on integer variables—that is, constraints, such as the one

just given, in which each variable appears only in linear form. It can be shown that no algorithm exists for solving general nonlinear constraints on integer variables—the problem is undecidable.

Infinite

Linear constraints

Nonlinear constraints

Constraint satisfaction problems with continuous domains are common in the real world and are widely studied in the field of operations research. For example, the scheduling of experiments on the Hubble Space Telescope requires very precise timing of observations; the start and finish of each observation and maneuver are continuous-valued variables that must obey a variety of astronomical, precedence, and power constraints. The best-known category of continuous-domain CSPs is that of linear programming problems, where constraints must be linear equalities or inequalities. Linear programming problems can be solved in time polynomial in the number of variables. Problems with different types of constraints and objective functions have also been studied—quadratic programming, second-order conic programming, and so on. These problems constitute an important area of applied mathematics.

Continuous domains

In addition to examining the types of variables that can appear in CSPs, it is useful to look at the types of constraints. The simplest type is the unary constraint, which restricts the value of a single variable. For example, in the map-coloring problem it could be the case that South Australians won’t tolerate the color green; we can express that with the unary constraint ⟨(

SA),SA ≠ green⟩. (The initial specification of the domain of a variable can also

be seen as a unary constraint.)

Unary constraint

A binary constraint relates two variables. For example,

SA ≠ NSW is a binary constraint. A

binary CSP is one with only unary and binary constraints; it can be represented as a constraint graph, as in Figure 6.1(b) .

Binary constraint

Binary CSP

We can also define higher-order constraints. The ternary constraint

XY Z X Y

>

Z ⟩.

Between(X,Y ,Z), for

A constraint involving an arbitrary number of variables is called a global constraint. (The name is traditional but confusing because a global constraint need not involve all the variables in a problem). One of the most common global constraints is

Alldiff , which says

that all of the variables involved in the constraint must have different values. In Sudoku problems (see Section 6.2.6 ), all variables in a row, column, or 3 × 3 box must satisfy an

Alldiff constraint.

Global constraint

Another example is provided by cryptarithmetic puzzles (Figure 6.2(a) ). Each letter in a cryptarithmetic puzzle represents a different digit. For the case in Figure 6.2(a) , this would

Alldiff (F ,T ,U ,W ,R,O). The addition constraints on the four columns of the puzzle can be written as the following n-ary constraints: be represented as the global constraint

O + O = R + 10 ⋅ C C +W  +W  =U + 10 ⋅ C C +T  +T  = O+10 ⋅ C C  = F  , 1

1

2

2

3

3

Figure 6.2

(a) A cryptarithmetic problem. Each letter stands for a distinct digit; the aim is to find a substitution of digits for letters such that the resulting sum is arithmetically correct, with the added restriction that no leading zeroes are allowed. (b) The constraint hypergraph for the cryptarithmetic problem, showing the constraint (square box at the top) as well as the column addition constraints (four square boxes

Alldiff

in the middle). The variables left.

Cryptarithmetic

C , C , and C 1

2

3

represent the carry digits for the three columns from right to

where

C , C , and C 1

2

3

are auxiliary variables representing the digit carried over into the tens,

hundreds, or thousands column. These constraints can be represented in a constraint hypergraph, such as the one shown in Figure 6.2(b) . A hypergraph consists of ordinary

n

nodes (the circles in the figure) and hypernodes (the squares), which represent -ary constraints—constraints involving

n variables.

Constraint hypergraph

Alternatively, as Exercise 6.NARY asks you to prove, every finite-domain constraint can be reduced to a set of binary constraints if enough auxiliary variables are introduced. This means that we could transform any CSP into one with only binary constraints—which

n

certainly makes the life of the algorithm designer simpler. Another way to convert an -ary CSP to a binary one is the dual graph transformation: create a new graph in which there will be one variable for each constraint in the original graph, and one binary constraint for each pair of constraints in the original graph that share variables.

Dual graph

X = {X,Y ,Z}, each with the domain {1,2,3,4,5}, and with the two constraints C : ⟨(X ,Y ,Z ),X + Y = Z ⟩ and C : ⟨(X,Y ),X + 1 = Y ⟩. Then the dual graph would have the variables X = {C ,C }, where the domain of the C variable in the dual graph is the set of {(xi ,yj ,zk )} tuples from the C constraint in the original problem, and similarly the domain of C is the set of {(xi ,yj )} tuples. The dual graph has the binary constraint ⟨(C ,C ),R ⟩, where R is a new relation that defines the constraint between C and C ; in this case it would be R = {((1,2,3),(1,2)),((2,3,5),(2,3))}. For example, consider a CSP with the variables 1

2

1

2

1

1

2

1

1

2

1

1

2

1

There are however two reasons why we might prefer a global constraint such as

Alldiff

rather than a set of binary constraints. First, it is easier and less error-prone to write the

problem description using

Alldiff . Second, it is possible to design special-purpose inference

algorithms for global constraints that are more efficient than operating with primitive constraints. We describe these inference algorithms in Section 6.2.5 .

The constraints we have described so far have all been absolute constraints, violation of which rules out a potential solution. Many real-world CSPs include preference constraints indicating which solutions are preferred. For example, in a university class-scheduling problem there are absolute constraints that no professor can teach two classes at the same time. But we also may allow preference constraints: Prof. R might prefer teaching in the morning, whereas Prof. N prefers teaching in the afternoon. A schedule that has Prof. R teaching at 2 p.m. would still be an allowable solution (unless Prof. R happens to be the department chair) but would not be an optimal one.

Preference constraints

Preference constraints can often be encoded as costs on individual variable assignments—for example, assigning an afternoon slot for Prof. R costs 2 points against the overall objective function, whereas a morning slot costs 1. With this formulation, CSPs with preferences can be solved with optimization search methods, either path-based or local. We call such a problem a constrained optimization problem, or COP. Linear programs are one class of COPs.

Constrained optimization problem

6.2 Constraint Propagation: Inference in CSPs

Constraint propagation

An atomic state-space search algorithm makes progress in only one way: by expanding a node to visit the successors. A CSP algorithm has choices. It can generate successors by choosing a new variable assignment, or it can do a specific type of inference called constraint propagation: using the constraints to reduce the number of legal values for a variable, which in turn can reduce the legal values for another variable, and so on. The idea is that this will leave fewer choices to consider when we make the next choice of a variable assignment. Constraint propagation may be intertwined with search, or it may be done as a preprocessing step, before search starts. Sometimes this preprocessing can solve the whole problem, so no search is required at all.

The key idea is local consistency. If we treat each variable as a node in a graph (see Figure 6.1(b) ) and each binary constraint as an edge, then the process of enforcing local consistency in each part of the graph causes inconsistent values to be eliminated throughout the graph. There are different types of local consistency, which we now cover in turn.

Local consistency

6.2.1 Node consistency A single variable (corresponding to a node in the CSP graph) is node-consistent if all the values in the variable’s domain satisfy the variable’s unary constraints. For example, in the variant of the Australia map-coloring problem (Figure 6.1 ) where South Australians dislike green, the variable

SA starts with domain red green blue {

,

,

},

and we can make it node

consistent by eliminating

green, leaving SA with the reduced domain {red,blue}. We say

that a graph is node-consistent if every variable in the graph is node-consistent.

Node consistency

It is easy to eliminate all the unary constraints in a CSP by reducing the domain of variables with unary constraints at the start of the solving process. As mentioned earlier, it is also

n

possible to transform all -ary constraints into binary ones. Because of this, some CSP solvers work with only binary constraints, expecting the user to eliminate the other constraints ahead of time. We make that assumption for the rest of this chapter, except where noted.

6.2.2 Arc consistency A variable in a CSP is arc-consistent1 if every value in its domain satisfies the variable’s

Xi is arc-consistent with respect to another variable Xj if for every value in the current domain Di there is some value in the domain Dj that satisfies the binary constraint on the arc (Xi ,Xj ). A graph is arc-consistent if every variable is arcconsistent with every other variable. For example, consider the constraint Y = X where the domain of both X and Y is the set of decimal digits. We can write this constraint explicitly binary constraints. More formally,

2

as

1 We have been using the term “edge” rather than “arc,” so it would make more sense to call this “edge-consistent,” but the name “arcconsistent” is historical.

⟨(

X,Y ),{(0,0),(1,1),(2,4),(3,9)}⟩.

Arc consistency

X arc-consistent with respect to Y , we reduce X’s domain to {0,1,2,3}. If we also make Y arc-consistent with respect to X, then Y ’s domain becomes {0,1,4,9}, and the whole To make

CSP is arc-consistent. On the other hand, arc consistency can do nothing for the Australia map-coloring problem. Consider the following inequality constraint on ( {(

SA,WA):

red,green),(red,blue),(green,red),(green,blue),(blue,red),(blue,green)}.

No matter what value you choose for

SA (or for WA), there is a valid value for the other

variable. So applying arc consistency has no effect on the domains of either variable. The most popular algorithm for enforcing arc consistency is called AC-3 (see Figure 6.3 ). To make every variable arc-consistent, the AC-3 algorithm maintains a queue of arcs to consider. Initially, the queue contains all the arcs in the CSP. (Each binary constraint

XX

becomes two arcs, one in each direction.) AC-3 then pops off an arbitrary arc ( i , j ) from

Xi arc-consistent with respect to Xj . If this leaves Di unchanged, the algorithm just moves on to the next arc. But if this revises Di (makes the domain smaller), then we add to the queue all arcs (Xk ,Xi ) where Xk is a neighbor of Xi . We need to do that because the change in Di might enable further reductions in Dk , even if we have previously considered Xk . If Di is revised down to nothing, then we know the whole CSP has no the queue and makes

consistent solution, and AC-3 can immediately return failure. Otherwise, we keep checking, trying to remove values from the domains of variables until no more arcs are in the queue. At that point, we are left with a CSP that is equivalent to the original CSP—they both have the same solutions—but the arc-consistent CSP will be faster to search because its variables have smaller domains. In some cases, it solves the problem completely (by reducing every domain to size 1) and in others it proves that no solution exists (by reducing some domain to size 0).

Figure 6.3

The arc-consistency algorithm AC-3. After applying AC-3, either every arc is arc-consistent, or some variable has an empty domain, indicating that the CSP cannot be solved. The name “AC-3” was used by the algorithm’s inventor (Mackworth, 1977) because it was the third version developed in the paper.

n variables, each with domain size at most d, and with c binary constraints (arcs). Each arc (Xk ,Xi ) can be inserted in the queue only d times because Xi has at most d values to delete. Checking consistency of an arc can be done in O(d ) time, so we get O(cd ) total worst-case time. The complexity of AC-3 can be analyzed as follows. Assume a CSP with

2

3

6.2.3 Path consistency Suppose we are to color the map of Australia with just two colors, red and blue. Arc consistency does nothing because every constraint can be satisfied individually with red at one end and blue at the other. But clearly there is no solution to the problem: because Western Australia, Northern Territory, and South Australia all touch each other, we need at least three colors for them alone.

Arc consistency tightens down the domains (unary constraints) using the arcs (binary constraints). To make progress on problems like map coloring, we need a stronger notion of consistency. Path consistency tightens the binary constraints by using implicit constraints that are inferred by looking at triples of variables.

Path consistency

XX Xm if, for every assignment {Xi = a,Xj = b} consistent with the constraints (if any) on {Xi ,Xj }, there is an assignment to Xm that satisfies the constraints on {Xi ,Xm } and {Xm ,Xj }. The name refers to the overall consistency of the path from Xi to Xj with Xm in the middle. A two-variable set { i , j } is path-consistent with respect to a third variable

Let’s see how path consistency fares in coloring the Australia map with two colors. We will

WA,SA} path-consistent with respect to NT . We start by enumerating the consistent assignments to the set. In this case, there are only two: {WA = red,SA = blue} and {WA = blue,SA = red}. We can see that with both of these assignments NT can be neither red nor blue (because it would conflict with either WA or SA). Because there is no valid choice for NT , we eliminate both assignments, and we end up with no valid assignments for {WA,SA}. Therefore, we know that there can be no solution to this make the set {

problem.

6.2.4

K-consistency k

k

Stronger forms of propagation can be defined with the notion of -consistency. A CSP is -

k − 1 variables and for any consistent assignment to those variables, a consistent value can always be assigned to any kth variable. 1-consistency says consistent if, for any set of

that, given the empty set, we can make any set of one variable consistent: this is what we called node consistency. 2-consistency is the same as arc consistency. For binary constraint graphs, 3-consistency is the same as path consistency.

K-consistency

k

k

k

k

A CSP is strongly -consistent if it is -consistent and is also ( − 1)-consistent, ( − 2)consistent, … all the way down to 1-consistent. Now suppose we have a CSP with

n

k

n nodes

k = n). We can then solve the problem as follows: First, we choose a consistent value for X . We are then guaranteed and make it strongly -consistent (i.e., strongly -consistent for

1

to be able to choose a value for

X

2

because the graph is 2-consistent, for

X

3

because it is 3-

Xi , we need only search through the d values in the domain to find a value consistent with X , … ,Xi . The total run time is only O(n d). consistent, and so on. For each variable

1

2

−1

Strongly k-consistent

Of course, there is no free lunch: constraint satisfaction is NP-complete in general, and any

n

algorithm for establishing -consistency must take time exponential in

n

n

n in the worst case.

Worse, -consistency also requires space that is exponential in . In practice, determining the appropriate level of consistency checking is mostly an empirical science. Computing 2consistency is common, and 3-consistency less common.

6.2.5 Global constraints Remember that a global constraint is one involving an arbitrary number of variables (but not necessarily all variables). Global constraints occur frequently in real problems and can be handled by special-purpose algorithms that are more efficient than the general-purpose methods described so far. For example, the

Alldiff constraint says that all the variables

involved must have distinct values (as in the cryptarithmetic problem above and Sudoku

Alldiff constraints works as follows: if m variables are involved in the constraint, and if they have n possible distinct values altogether, and m > n, then the constraint cannot be satisfied. puzzles below). One simple form of inconsistency detection for

This leads to the following simple algorithm: First, remove any variable in the constraint that has a singleton domain, and delete that variable’s value from the domains of the remaining variables. Repeat as long as there are singleton variables. If at any point an empty domain is produced or there are more variables than domain values left, then an inconsistency has been detected.

WA = red, NSW = red} for

This method can detect the inconsistency in the assignment { Figure 6.1 . Notice that the variables

SA, NT , and Q are effectively connected by an

Alldiff constraint because each pair must have two different colors. After applying AC-3 with the partial assignment, the domains of SA, NT , and Q are all reduced to {green,blue}.

That is, we have three variables and only two colors, so the

Alldiff constraint is violated.

Thus, a simple consistency procedure for a higher-order constraint is sometimes more effective than applying arc consistency to an equivalent set of binary constraints.

Another important higher-order constraint is the resource constraint, sometimes called the

Atmost constraint. For example, in a scheduling problem, let P , … ,P 1

4

denote the numbers

of personnel assigned to each of four tasks. The constraint that no more than 10 personnel are assigned in total is written as

Atmost(10,P ,P ,P ,P ). We can detect an inconsistency 1

2

3

4

simply by checking the sum of the minimum values of the current domains; for example, if each variable has the domain {3,4,5,6}, the

Atmost constraint cannot be satisfied. We can

also enforce consistency by deleting the maximum value of any domain if it is not consistent with the minimum values of the other domains. Thus, if each variable in our example has the domain {2,3,4,5,6}, the values 5 and 6 can be deleted from each domain.

Resource constraint

For large resource-limited problems with integer values—such as logistical problems involving moving thousands of people in hundreds of vehicles—it is usually not possible to represent the domain of each variable as a large set of integers and gradually reduce that set by consistency-checking methods. Instead, domains are represented by upper and lower bounds and are managed by bounds propagation. For example, in an airline-scheduling

F , for which the planes have capacities 165 and 385, respectively. The initial domains for the numbers of passengers on flights F and F are then problem, let’s suppose there are two flights,

F

1

and

2

1

2

D

Bounds propagation

1

= [0,165]

and

D

2

= [0,385].

Now suppose we have the additional constraint that the two flights together must carry 420 people:

F1 + F2 = 420. Propagating bounds constraints, we reduce the domains to D1 = [35,165] and D2 = [255,385].

X, and for both the lowerbound and upper-bound values of X, there exists some value of Y that satisfies the constraint between X and Y for every variable Y . This kind of bounds propagation is widely

We say that a CSP is bounds-consistent if for every variable

used in practical constraint problems.

Bounds-consistent

6.2.6 Sudoku The popular Sudoku puzzle has introduced millions of people to constraint satisfaction problems, although they may not realize it. A Sudoku board consists of 81 squares, some of which are initially filled with digits from 1 to 9. The puzzle is to fill in all the remaining squares such that no digit appears twice in any row, column, or A row, column, or box is called a unit.

Figure 6.4

3 × 3 box (see Figure 6.4 ).

(a) A Sudoku puzzle and (b) its solution.

Sudoku

The Sudoku puzzles that appear in newspapers and puzzle books have the property that there is exactly one solution. Although some can be tricky to solve by hand, taking tens of minutes, a CSP solver can handle thousands of puzzles per second.

A Sudoku puzzle can be considered a CSP with 81 variables, one for each square. We use the variable names

A1 through A9 for the top row (left to right), down to I 1 through I 9 for

the bottom row. The empty squares have the domain {1,2,3,4,5,6,7,8,9} and the pre-filled squares have a domain consisting of a single value. In addition, there are 27 different

Alldiff constraints, one for each unit (row, column, and box of 9 squares): Alldiff (A1,A2,A3,A4,A5,A6,A7,A8,A9) Alldiff (B1,B2,B3,B4,B5,B6,B7,B8,B9) ⋯

Alldiff (A1,B1,C 1,D1,E1,F 1,G1,H 1,I 1) Alldiff (A2,B2,C 2,D2,E2,F 2,G2,H 2,I 2) ⋯

Alldiff (A1,A2,A3,B1,B2,B3,C 1,C 2,C 3) Alldiff (A4,A5,A6,B4,B5,B6,C 4,C 5,C 6) ⋯

Let us see how far arc consistency can take us. Assume that the been expanded into binary constraints (such as algorithm directly. Consider variable

Alldiff constraints have

A1 ≠ A2) so that we can apply the AC-3

E6 from Figure 6.4(a) —the empty square between

the 2 and the 8 in the middle box. From the constraints in the box, we can remove 1, 2, 7,

E6 ’s domain. From the constraints in its column, we can eliminate 5, 6, 2, 8, 9, and 3 (although 2 and 8 were already removed). That leaves E6 with a domain of {4}; in other words, we know the answer for E6 . Now consider variable I6—the square in the and 8 from

bottom middle box surrounded by 1, 3, and 3. Applying arc consistency in its column, we

E6 must be 4), 8, 9, and 3. We eliminate 1 by arc consistency with I 5, and we are left with only the value 7 in the domain of I6. Now there are eliminate 5, 6, 2, 4 (since we now know

8 known values in column 6, so arc consistency can infer that

A6 must be 1. Inference

continues along these lines, and eventually, AC-3 can solve the entire puzzle—all the variables have their domains reduced to a single value, as shown in Figure 6.4(b) .

Of course, Sudoku would soon lose its appeal if every puzzle could be solved by a mechanical application of AC-3, and indeed AC-3 works only for the easiest Sudoku puzzles. Slightly harder ones can be solved by PC-2, but at a greater computational cost: there are 255,960 different path constraints to consider in a Sudoku puzzle. To solve the hardest puzzles and to make efficient progress, we will have to be more clever.

Indeed, the appeal of Sudoku puzzles for the human solver is the need to be resourceful in applying more complex inference strategies. Aficionados give them colorful names, such as “naked triples.” That strategy works as follows: in any unit (row, column or box), find three squares that each have a domain that contains the same three numbers or a subset of those numbers. For example, the three domains might be {1,8}, {3,8}, and {1,3,8}. From that we don’t know which square contains 1, 3, or 8, but we do know that the three numbers must be distributed among the three squares. Therefore we can remove 1, 3, and 8 from the domains of every other square in the unit.

It is interesting to note how far we can go without saying much that is specific to Sudoku. We do of course have to say that there are 81 variables, that their domains are the digits 1 to 9, and that there are 27

Alldiff constraints. But beyond that, all the strategies—arc

consistency, path consistency, and so on—apply generally to all CSPs, not just to Sudoku problems. Even naked triples is really a strategy for enforcing consistency of

Alldiff

constraints and is not specific to Sudoku per se. This is the power of the CSP formalism: for each new problem area, we only need to define the problem in terms of constraints; then the general constraint-solving mechanisms can take over.

6.3 Backtracking Search for CSPs Sometimes we can finish the constraint propagation process and still have variables with multiple possible values. In that case we have to search for a solution. In this section we cover backtracking search algorithms that work on partial assignments; in the next section we look at local search algorithms over complete assignments. Consider how a standard depth-limited search (from Chapter 3 ) could solve CSPs. A state would be a partial assignment, and an action would extend the assignment, adding, say,

NSW red or SA blue for the Australia map-coloring problem. For a CSP with n variables of domain size d we would end up with a search tree where all the complete assignments (and thus all the solutions) are leaf nodes at depth n. But notice that the branching factor at the top level would be nd because any of d values can be assigned to any of n variables. At the next level, the branching factor is n d, and so on for n levels. So n n the tree has n d leaves, even though there are only d possible complete assignments! =

=

(

− 1)

!⋅

Commutativity

We can get back that factor of

n

!

by recognizing a crucial property of CSPs: commutativity.

A problem is commutative if the order of application of any given set of actions does not matter. In CSPs, it makes no difference if we first assign

NSW red and then SA blue, or =

=

the other way around. Therefore, we need only consider a single variable at each node in the

SA red, SA green, and SA blue, but we would never choose between NSW red and SA blue. With this restriction, the number of leaves is d n , as we would hope. At each level of the tree we do search tree. At the root we might make a choice between =

=

=

=

=

have to choose which variable we will deal with, but we never have to backtrack over that choice. Figure 6.5  shows a backtracking search procedure for CSPs. It repeatedly chooses an unassigned variable, and then tries all values in the domain of that variable in turn, trying to

extend each one into a solution via a recursive call. If the call succeeds, the solution is returned, and if it fails, the assignment is restored to the previous state, and we try the next value. If no value works then we return failure. Part of the search tree for the Australia problem is shown in Figure 6.6 , where we have assigned variables in the order

WA NT Q ,

,

, ….

Figure 6.5

A simple backtracking algorithm for constraint satisfaction problems. The algorithm is modeled on the recursive depth-first search of Chapter 3 . The functions SELECT-UNASSIGNED-VARIABLE and ORDER-DOMAIN-

VALUES, implement the general-purpose heuristics discussed in Section 6.3.1 . The INFERENCE function can optionally impose arc-, path-, or -consistency, as desired. If a value choice leads to failure (noticed either by INFERENCE or by BACKTRACK), then value assignments (including those made by INFERENCE) are retracted

k

and a new value is tried.

Figure 6.6

Part of the search tree for the map-coloring problem in Figure 6.1 .

Notice that BACKTRACKING-SEARCH keeps only a single representation of a state (assignment) and alters that representation rather than creating new ones (see page 80). Whereas the uninformed search algorithms of Chapter 3  could be improved only by supplying them with domain-specific heuristics, it turns out that backtracking search can be improved using domain-independent heuristics that take advantage of the factored representation of CSPs. In the following four sections we show how this is done: (6.3.1 ) Which variable should be assigned next (SELECT-UNASSIGNED-VARIABLE), and in what order should its values be tried (ORDER-DOMAIN-VALUES)? (6.3.2 ) What inferences should be performed at each step in the search (INFERENCE)? (6.3.3 ) Can we BACKTRACK more than one step when appropriate? (6.3.4 ) Can we save and reuse partial results from the search?

6.3.1 Variable and value ordering The backtracking algorithm contains the line

var

←S

ELECT-UNASSIGNED-VARIABLE(csp,

assignment).

The simplest strategy for SELECT-UNASSIGNED-VARIABLE is static ordering: choose the variables in order, {

X X 1,

2,

…}.

The next simplest is to choose randomly. Neither strategy is optimal.

WA red and NT green in Figure 6.6 , there is blue next rather than only one possible value for SA, so it makes sense to assign SA assigning Q. In fact, after SA is assigned, the choices for Q, NSW , and V are all forced.

For example, after the assignments for

=

=

=

This intuitive idea—choosing the variable with the fewest “legal” values—is called the minimum-remaining-values (MRV) heuristic. It also has been called the “most constrained variable” or “fail-first” heuristic, the latter because it picks a variable that is most likely to cause a failure soon, thereby pruning the search tree. If some variable left, the MRV heuristic will select

X has no legal values

X and failure will be detected immediately—avoiding

pointless searches through other variables. The MRV heuristic usually performs better than a random or static ordering, sometimes by orders of magnitude, although the results vary depending on the problem.

Minimum-remaining-values

The MRV heuristic doesn’t help at all in choosing the first region to color in Australia, because initially every region has three legal colors. In this case, the degree heuristic comes in handy. It attempts to reduce the branching factor on future choices by selecting the variable that is involved in the largest number of constraints on other unassigned variables. In Figure 6.1 ,

SA is the variable with highest degree, 5; the other variables have degree 2 or 3, except for T , which has degree 0. In fact, once SA is chosen, applying the degree heuristic solves the problem without any false steps—you can choose any consistent color at each choice point and still arrive at a solution with no backtracking. The minimumremaining-values heuristic is usually a more powerful guide, but the degree heuristic can be useful as a tie-breaker.

Degree heuristic

Least-constraining-value

Once a variable has been selected, the algorithm must decide on the order in which to examine its values. The least-constraining-value heuristic is effective for this. It prefers the value that rules out the fewest choices for the neighboring variables in the constraint graph. For example, suppose that in Figure 6.1  we have generated the partial assignment with

WA = red and NT = green and that our next choice is for Q. Blue would be a bad choice because it eliminates the last legal value left for Q’s neighbor, SA. The least-constrainingvalue heuristic therefore prefers red to blue. In general, the heuristic is trying to leave the maximum flexibility for subsequent variable assignments.

Why should variable selection be fail-first, but value selection be fail-last? Every variable has to be assigned eventually, so by choosing the ones that are likely to fail first, we will on average have fewer successful assignments to backtrack over. For value ordering, the trick is that we only need one solution; therefore it makes sense to look for the most likely values first. If we wanted to enumerate all solutions rather than just find one, then value ordering would be irrelevant.

6.3.2 Interleaving search and inference We saw how AC-3 can reduce the domains of variables before we begin the search. But inference can be even more powerful during the course of a search: every time we make a choice of a value for a variable, we have a brand-new opportunity to infer new domain reductions on the neighboring variables.

One of the simplest forms of inference is called forward checking. Whenever a variable

X is

assigned, the forward-checking process establishes arc consistency for it: for each

Y that is connected to X by a constraint, delete from Y ’s domain any value that is inconsistent with the value chosen for X. unassigned variable

Forward checking

Figure 6.7  shows the progress of backtracking search on the Australia CSP with forward checking. There are two important points to notice about this example. First, notice that after

WA red and Q green are assigned, the domains of NT and SA are reduced to a =

=

single value; we have eliminated branching on these variables altogether by propagating information from

WA and Q. A second point to notice is that after V

=

blue, the domain of

SA is empty. Hence, forward checking has detected that the partial assignment WA red Q green V blue is inconsistent with the constraints of the problem, and the {

=

,

=

,

=

}

algorithm backtracks immediately.

Figure 6.7

WA red is assigned first; then forward NT and SA. After Q green is assigned, green is deleted from the domains of NT , SA, and NSW . After V blue is assigned, blue is deleted from the domains of NSW and SA, leaving SA with no legal values. The progress of a map-coloring search with forward checking. = checking deletes from the domains of the neighboring variables

red

=

=

For many problems the search will be more effective if we combine the MRV heuristic with forward checking. Consider Figure 6.7  after assigning {

WA red . Intuitively, it seems that that assignment constrains its neighbors, NT and SA, so we should handle those =

}

variables next, and then all the other variables will fall into place. That’s exactly what

NT and SA each have two values, so one of them is chosen first, then the other, then Q, NSW , and V in order. Finally T still has three values, and any one of happens with MRV:

them works. We can view forward checking as an efficient way to incrementally compute the information that the MRV heuristic needs to do its job.

Although forward checking detects many inconsistencies, it does not detect all of them. The

Q green row of Figure 6.7 . We’ve made WA and Q arc-consistent, but we’ve left both NT and SA with problem is that it doesn’t look ahead far enough. For example, consider the

=

blue as their only possible value, which is an inconsistency, since they are neighbors.

Maintaining Arc Consistency

The algorithm called MAC (for Maintaining Arc Consistency) detects inconsistencies like

Xi is assigned a value, the INFERENCE procedure calls AC-3, but instead of a queue of all arcs in the CSP, we start with only the arcs (Xj ,Xi ) for all Xj that are unassigned variables that are neighbors of Xi . From there, AC-3 does constraint

this. After a variable

propagation in the usual way, and if any variable has its domain reduced to the empty set, the call to AC-3 fails and we know to backtrack immediately. We can see that MAC is strictly more powerful than forward checking because forward checking does the same thing as MAC on the initial arcs in MAC’s queue; but unlike MAC, forward checking does not recursively propagate constraints when changes are made to the domains of variables.

6.3.3 Intelligent backtracking: Looking backward The BACKTRACKING-SEARCH algorithm in Figure 6.5  has a very simple policy for what to do when a branch of the search fails: back up to the preceding variable and try a different value for it. This is called chronological backtracking because the most recent decision point is revisited. In this subsection, we consider better possibilities.

Chronological backtracking

Consider what happens when we apply simple backtracking in Figure 6.1  with a fixed

Q, NSW , V , T , SA, WA, NT . Suppose we have generated the partial assignment {Q = red,NSW = green,V = blue,T = red}. When we try the next variable, SA, we see that every value violates a constraint. We back up to T and try a new color for variable ordering

Tasmania! Obviously this is silly—recoloring Tasmania cannot possibly help in resolving the problem with South Australia.

A more intelligent approach is to backtrack to a variable that might fix the problem—a

SA impossible. To do this, we will keep track of a set of assignments that are in conflict with some value for SA. variable that was responsible for making one of the possible values of

The set (in this case {

Q red NSW green V =

,

=

,

=

blue

}),

is called the conflict set for

SA.

The backjumping method backtracks to the most recent assignment in the conflict set; in this case, backjumping would jump over Tasmania and try a new value for

V . This method is

easily implemented by a modification to BACKTRACK such that it accumulates the conflict set while checking for a legal value to assign. If no legal value is found, the algorithm should return the most recent element of the conflict set along with the failure indicator.

Conflict set

Backjumping

The sharp-eyed reader may have noticed that forward checking can supply the conflict set with no extra work: whenever forward checking based on an assignment

X x deletes a =

Y ’s domain, it should add X x to Y ’s conflict set. If the last value is deleted from Y ’s domain, then the assignments in the conflict set of Y are added to the conflict set of X. That is, we now know that X x leads to a contradiction (in Y ), and thus a different assignment should be tried for X. value from

=

=

The eagle-eyed reader may have noticed something odd: backjumping occurs when every value in a domain is in conflict with the current assignment; but forward checking detects this event and prevents the search from ever reaching such a node! In fact, it can be shown that every branch pruned by backjumping is also pruned by forward checking. Hence, simple backjumping is redundant in a forward-checking search or, indeed, in a search that uses stronger consistency checking, such as MAC—you need only do one or the other.

Despite the observations of the preceding paragraph, the idea behind backjumping remains a good one: to backtrack based on the reasons for failure. Backjumping notices failure when a variable’s domain becomes empty, but in many cases a branch is doomed long before this

WA red NSW red (which, from our earlier discussion, is inconsistent). Suppose we try T red next and then assign NT , Q, V , occurs. Consider again the partial assignment {

=

=

,

=

}

SA. We know that no assignment can work for these last four variables, so eventually we run out of values to try at NT . Now, the question is, where to backtrack? Backjumping cannot work, because NT does have values consistent with the preceding assigned variables —NT doesn’t have a complete conflict set of preceding variables that caused it to fail. We know, however, that the four variables NT , Q, V , and SA, taken together, failed because of a set of preceding variables, which must be those variables that directly conflict with the four.

This leads to a different–and deeper–notion of the conflict set for a variable such as

NT : it is

NT , together with any subsequent variables, to have no consistent solution. In this case, the set is WA and NSW , so the algorithm should backtrack to NSW and skip over Tasmania. A backjumping algorithm that uses conflict sets

that set of preceding variables that caused

defined in this way is called conflict-directed backjumping.

Conflict-directed backjumping

We must now explain how these new conflict sets are computed. The method is in fact quite simple. The “terminal” failure of a branch of the search always occurs because a variable’s domain becomes empty; that variable has a standard conflict set. In our example,

SA fails,

WA,NT ,Q}. We backjump to Q, and Q absorbs the conflict set from SA (minus Q itself, of course) into its own direct conflict set, which is {NT ,NSW }; the new conflict set is {WA,NT ,NSW }. That is, there is no solution from Q onward, given the preceding assignment to {WA,NT ,NSW }. Therefore, we backtrack to NT , the most recent of these. NT absorbs {WA,NT ,NSW } − {NT } into its own direct conflict set {WA}, giving {WA,NSW } (as stated in the previous paragraph). Now the algorithm backjumps to NSW , as we would hope. To summarize: let Xj be the current variable, and let conf (Xj ) be its conflict set. If every possible value for Xj fails, backjump to the most recent variable Xi in conf (Xj ) and recompute the conflict set for Xi as follows: and its conflict set is (say) {

conf (Xi ) ← conf (Xi ) ∪ conf (Xj ) − {Xi }. 6.3.4 Constraint learning

When we reach a contradiction, backjumping can tell us how far to back up, so we don’t waste time changing variables that won’t fix the problem. But we would also like to avoid running into the same problem again. When the search arrives at a contradiction, we know that some subset of the conflict set is responsible for the problem. Constraint learning is the idea of finding a minimum set of variables from the conflict set that causes the problem. This set of variables, along with their corresponding values, is called a no-good. We then record the no-good, either by adding a new constraint to the CSP to forbid this combination of assignments or by keeping a separate cache of no-goods.

Constraint learning

No-good

WA red NT green Q blue

For example, consider the state {

=

,

=

,

=

}

in the bottom row of

Figure 6.6 . Forward checking can tell us this state is a no-good because there is no valid assignment to

SA. In this particular case, recording the no-good would not help, because

once we prune this branch from the search tree, we will never encounter this combination again. But suppose that the search tree in Figure 6.6  were actually part of a larger search tree that started by first assigning values for record

WA red NT green Q blue

{

=

,

=

,

=

}

V

and

T . Then it would be worthwhile to

as a no-good because we are going to run into the

same problem again for each possible set of assignments to

V

and

T.

No-goods can be effectively used by forward checking or by backjumping. Constraint learning is one of the most important techniques used by modern CSP solvers to achieve efficiency on complex problems.

6.4 Local Search for CSPs Local search algorithms (see Section 4.1 ) turn out to be very effective in solving many CSPs. They use a complete-state formulation (as introduced in Section 4.1.1 ) where each state assigns a value to every variable, and the search changes the value of one variable at a time. As an example, we’ll use the 8-queens problem, as defined as a CSP on page 183. In Figure 6.8  we start on the left with a complete assignment to the 8 variables; typically this will violate several constraints. We then randomly choose a conflicted variable, which turns out to be

Q8 , the rightmost column. We’d like to change the value to something that brings

us closer to a solution; the most obvious approach is to select the value that results in the minimum number of conflicts with other variables—the min-conflicts heuristic.

Figure 6.8

A two-step solution using min-conflicts for an 8-queens problem. At each stage, a queen is chosen for reassignment in its column. The number of conflicts (in this case, the number of attacking queens) is shown in each square. The algorithm moves the queen to the min-conflicts square, breaking ties randomly.

Min-conflicts

In the figure we see there are two rows that only violate one constraint; we pick

Q8 = 3 (that

is, we move the queen to the 8th column, 3rd row). On the next iteration, in the middle board of the figure, we select

Q6 as the variable to change, and note that moving the queen

to the 8th row results in no conflicts. At this point there are no more conflicted variables, so we have a solution. The algorithm is shown in Figure 6.9 .2

2 Local search can easily be extended to constrained optimization problems (COPs). In that case, all the techniques for hill climbing and simulated annealing can be applied to optimize the objective function.

Figure 6.9

The MIN-CONFLICTS local search algorithm for CSPs. The initial state may be chosen randomly or by a greedy assignment process that chooses a minimal-conflict value for each variable in turn. The CONFLICTS function counts the number of constraints violated by a particular value, given the rest of the current assignment.

Min-conflicts is surprisingly effective for many CSPs. Amazingly, on the n-queens problem, if you don’t count the initial placement of queens, the run time of min-conflicts is roughly independent of problem size. It solves even the million-queens problem in an average of 50 steps (after the initial assignment). This remarkable observation was the stimulus leading to a great deal of research in the 1990s on local search and the distinction between easy and hard problems, which we take up in Section 7.6.3 . Roughly speaking, n-queens is easy for local search because solutions are densely distributed throughout the state space. Minconflicts also works well for hard problems. For example, it has been used to schedule observations for the Hubble Space Telescope, reducing the time taken to schedule a week of observations from three weeks (!) to around 10 minutes. All the local search techniques from Section 4.1  are candidates for application to CSPs, and some of those have proved especially effective. The landscape of a CSP under the minconflicts heuristic usually has a series of plateaus. There may be millions of variable assignments that are only one conflict away from a solution. Plateau search—allowing sideways moves to another state with the same score—can help local search find its way off

this plateau. This wandering on the plateau can be directed with a technique called tabu search: keeping a small list of recently visited states and forbidding the algorithm to return to those states. Simulated annealing can also be used to escape from plateaus.

Another technique called constraint weighting aims to concentrate the search on the important constraints. Each constraint is given a numeric weight, initially all 1. At each step of the search, the algorithm chooses a variable/value pair to change that will result in the lowest total weight of all violated constraints. The weights are then adjusted by incrementing the weight of each constraint that is violated by the current assignment. This has two benefits: it adds topography to plateaus, making sure that it is possible to improve from the current state, and it also adds learning: over time the difficult constraints are assigned higher weights.

Constraint weighting

Another advantage of local search is that it can be used in an online setting (see Section 4.5 ) when the problem changes. Consider a scheduling problem for an airline’s weekly flights. The schedule may involve thousands of flights and tens of thousands of personnel assignments, but bad weather at one airport can render the schedule infeasible. We would like to repair the schedule with a minimum number of changes. This can be easily done with a local search algorithm starting from the current schedule. A backtracking search with the new set of constraints usually requires much more time and might find a solution with many changes from the current schedule.

6.5 The Structure of Problems In this section, we examine ways in which the structure of the problem, as represented by the constraint graph, can be used to find solutions quickly. Most of the approaches here also apply to other problems besides CSPs, such as probabilistic reasoning.

The only way we can possibly hope to deal with the vast real world is to decompose it into subproblems. Looking again at the constraint graph for Australia (Figure 6.1(b) , repeated as Figure 6.12(a) ), one fact stands out: Tasmania is not connected to the mainland.3 Intuitively, it is obvious that coloring Tasmania and coloring the mainland are independent subproblems—any solution for the mainland combined with any solution for Tasmania yields a solution for the whole map.

3 A careful cartographer or patriotic Tasmanian might object that Tasmania should not be colored the same as its nearest mainland neighbor, to avoid the impression that it might be part of that state.

Independent subproblems

Independence can be ascertained simply by finding connected components of the constraint graph. Each component corresponds to a subproblem

CSP i . If assignment Si is a

CSP i , then ⋃i Si is a solution of ⋃i CSP i . Why is this important? Suppose each CSP i has c variables from the total of n variables, where c is a constant. Then there are n/c subproblems, each of which takes at most d c work to solve, where d is the size of the domain. Hence, the total work is O(d c n/c), which is linear in n; without the decomposition, the total work is O(d n ), which is exponential in n. Let’s make this more concrete: dividing a solution of

Boolean CSP with 100 variables into four subproblems reduces the worst-case solution time from the lifetime of the universe down to less than a second.

Connected component

Completely independent subproblems are delicious, then, but rare. Fortunately, some other graph structures are also easy to solve. For example, a constraint graph is a tree when any two variables are connected by only one path. We will show that any tree-structured CSP can be solved in time linear in the number of variables.4 The key is a new notion of consistency, called directional arc consistency or DAC. A CSP is defined to be directional arc-consistent under an ordering of variables each

Xj for j > i.

X ,X , … ,Xn if and only if every Xi is arc-consistent with 1

2

4 Sadly, very few regions of the world have tree-structured maps, although Sulawesi comes close.

Directional arc consistency

Topological sort

To solve a tree-structured CSP, first pick any variable to be the root of the tree, and choose an ordering of the variables such that each variable appears after its parent in the tree. Such an ordering is called a topological sort. Figure 6.10(a)  shows a sample tree and (b) shows

n nodes has n − 1 edges, so we can make this graph directed arc-consistent in O(n) steps, each of which must compare up to d possible domain values for two variables, for a total time of O(nd ). Once we have a directed arc-consistent one possible ordering. Any tree with

2

graph, we can just march down the list of variables and choose any remaining value. Since each edge from a parent to its child is arc-consistent, we know that for any value we choose for the parent, there will be a valid value left to choose for the child. That means we won’t have to backtrack; we can move linearly through the variables. The complete algorithm is shown in Figure 6.11 .

Figure 6.10

(a) The constraint graph of a tree-structured CSP. (b) A linear ordering of the variables consistent with the tree with as the root. This is known as a topological sort of the variables.

A

Figure 6.11

The TREE-CSP-SOLVER algorithm for solving tree-structured CSPs. If the CSP has a solution, we will find it in linear time; if not, we will detect a contradiction.

Now that we have an efficient algorithm for trees, we can consider whether more general constraint graphs can be reduced to trees somehow. There are two ways to do this: by removing nodes (Section 6.5.1 ) or by collapsing nodes together (Section 6.5.2 ).

6.5.1 Cutset conditioning The first way to reduce a constraint graph to a tree involves assigning values to some variables so that the remaining variables form a tree. Consider the constraint graph for Australia, shown again in Figure 6.12(a) . Without South Australia, the graph would become a tree, as in (b). Fortunately, we can delete South Australia (in the graph, not the

SA and deleting from the domains of the other variables any values that are inconsistent with the value chosen for SA. country) by fixing a value for

Figure 6.12

(a) The original constraint graph from Figure 6.1 . (b) After the removal of becomes a forest of two trees.

Now, any solution for the CSP after with the value chosen for

SA, the constraint graph

SA and its constraints are removed will be consistent

SA. (This works for binary CSPs; the situation is more

complicated with higher-order constraints.) Therefore, we can solve the remaining tree with the algorithm given above and thus solve the whole problem. Of course, in the general case (as opposed to map coloring), the value chosen for

SA could be the wrong one, so we

would need to try each possible value. The general algorithm is as follows:

S of the CSP’s variables such that the constraint graph becomes a tree after removal of S . S is called a cycle cutset.

1. Choose a subset

Cycle cutset

2. For each possible assignment to the variables in

S that satisfies all constraints on S,

a. remove from the domains of the remaining variables any values that are inconsistent with the assignment for

S, and

b. if the remaining CSP has a solution, return it together with the assignment for

S. c

If the cycle cutset has size , then the total run time is

O(d c ⋅ (n − c)d ): we have to try each 2

d c combinations of values for the variables in S, and for each combination we must solve a tree problem of size n − c. If the graph is “nearly a tree,” then c will be small and the of the

savings over straight backtracking will be huge—for our 100-Boolean-variable example, if

c = 20, this would get us down from the lifetime of the Universe to a few minutes. In the worst case, however, c can be as large as (n − 2). Finding we could find a cutset of size

the smallest cycle cutset is NP-hard, but several efficient approximation algorithms are known. The overall algorithmic approach is called cutset conditioning; it comes up again in Chapter 13 , where it is used for reasoning about probabilities.

Cutset conditioning

6.5.2 Tree decomposition

Tree decomposition

The second way to reduce a constraint graph to a tree is based on constructing a tree decomposition of the constraint graph: a transformation of the original graph into a tree where each node in the tree consists of a set of variables, as in Figure 6.13 . A tree decomposition must satisfy these three requirements:

Every variable in the original problem appears in at least one of the tree nodes. If two variables are connected by a constraint in the original problem, they must appear together (along with the constraint) in at least one of the tree nodes. If a variable appears in two nodes in the tree, it must appear in every node along the path connecting those nodes.

Figure 6.13

A tree decomposition of the constraint graph in Figure 6.12(a) .

The first two conditions ensure that all the variables and constraints are represented in the tree decomposition. The third condition seems rather technical, but allows us to say that any variable from the original problem must have the same value wherever it appears: the constraints in the tree say that a variable in one node of the tree must have the same value as the corresponding variable in the adjacent node in the tree. For example,

SA appears in

all four of the connected nodes in Figure 6.13 , so each edge in the tree decomposition therefore includes the constraint that the value of value of

SA in one node must be the same as the

SA in the next. You can verify from Figure 6.12  that this decomposition makes

sense.

Once we have a tree-structured graph, we can apply TREE-CSP-SOLVER to get a solution in

O(nd

2

) time, where

n is the number of tree nodes and d is the size of the largest domain.

But note that in the tree, a domain is a set of tuples of values, not just individual values. For example, the top left node in Figure 6.13  represents, at the level of the original

WA,NT ,SA}, domain {red,green,blue}, and constraints WA ≠ NT ,SA ≠ NT ,WA ≠ SA. At the level of the tree, the node represents a single variable, which we can call SANTWA, whose value must be a three-tuple of colors, such as (red,green,blue), but not (red,red,blue), because that would violate the constraint problem, a subproblem with variables {

SA ≠ NT from the original problem. We can then move from that node to the adjacent one, with the variable we can call SANTQ, and find that there is only one tuple, (red,green,blue), that is consistent with the choice for SANTWA. The exact same process is repeated for the next two nodes, and independently we can make any choice for T . We can solve any tree decomposition problem in

O(nd

2

) time with TREE-CSP-SOLVER, which

d remains small. Going back to our example with 100 Boolean variables, if each node has 10 variables, then d = 2 and we should be able to solve the will be efficient as long as

10

problem in seconds. But if there is a node with 30 variables, it would take centuries.

Tree width

A given graph admits many tree decompositions; in choosing a decomposition, the aim is to make the subproblems as small as possible. (Putting all the variables into one node is technically a tree, but is not helpful.) The tree width of a tree decomposition of a graph is one less than the size of the largest node; the tree width of the graph itself is defined to be the minimum width among all its tree decompositions. If a graph has tree width problem can be solved in

O(nd w

+1

w then the

) time given the corresponding tree decomposition.

Hence, CSPs with constraint graphs of bounded tree width are solvable in polynomial time.

Unfortunately, finding the decomposition with minimal tree width is NP-hard, but there are heuristic methods that work well in practice. Which is better: the cutset decomposition with

O(d c ⋅ (n − c)d ), or the tree decomposition with time O(nd w )? Whenever you have a cycle-cutset of size c, there is also a tree width of size w < c + 1, and it may be far smaller in time

2

+1

some cases. So time consideration favors tree decomposition, but the advantage of the cycle-cutset approach is that it can be executed in linear memory, while tree decomposition requires memory exponential in

w.

6.5.3 Value symmetry So far, we have looked at the structure of the constraint graph. There can also be important structure in the values of variables, or in the structure of the constraint relations themselves. Consider the map-coloring problem with

d colors. For every consistent solution, there is

actually a set of

d

!

solutions formed by permuting the color names. For example, on the

Australia map we know that 3! = 6

WA, NT , and SA must all have different colors, but there are

ways to assign three colors to three regions. This is called value symmetry. We would

like to reduce the search space by a factor of

d

!

by breaking the symmetry in assignments.

We do this by introducing a symmetry-breaking constraint. For our example, we might

NT SA WA, that requires the three values to be in alphabetical order. This constraint ensures that only one of the d solutions is possible: NT blue SA green WA red . impose an arbitrary ordering constraint,

60 minutes. A set of 12 examples, taken from the experience of one of us (SR), is shown in Figure 19.2 . 6

2

2

Note how skimpy these data are: there are 2 × 3 × 4 = 9, 216 possible combinations of values for the input attributes, but we are given the correct output for only 12 of them; each of the other 9,204 could be either true or false; we don’t know. This is the essence of induction: we need to make our best guess at these missing 9,204 output values, given only the evidence of the 12 examples.

Figure 19.2

Examples for the restaurant domain.

19.3 Learning Decision Trees A decision tree is a representation of a function that maps a vector of attribute values to a single output value—a “decision.” A decision tree reaches its decision by performing a sequence of tests, starting at the root and following the appropriate branch until a leaf is reached. Each internal node in the tree corresponds to a test of the value of one of the input attributes, the branches from the node are labeled with the possible values of the attribute, and the leaf nodes specify what value is to be returned by the function.

Decision tree

In general, the input and output values can be discrete or continuous, but for now we will consider only inputs consisting of discrete values and outputs that are either true (a positive

j to index the examples (xj is the input vector for the jth example and yj is the output), and xj i for the ith attribute of the jth example. example) or false (a negative example). We call this Boolean classification. We will use

,

Positive

Negative

The tree representing the decision function that SR uses for the restaurant problem is shown in Figure 19.3 . Following the branches, we see that an example with

WaitEstimate

= 0–10

Patrons Full and =

will be classified as positive (i.e., yes, we will wait for a table).

Figure 19.3

A decision tree for deciding whether to wait for a table.

19.3.1 Expressiveness of decision trees A Boolean decision tree is equivalent to a logical statement of the form:

Output where each

⇔

(

Path

1

∨

Path

2

∨ ⋯) ,

Pathi is a conjunction of the form (Am = vx ∧ An = vy ∧ ⋯) of attribute-value

tests corresponding to a path from the root to a true leaf. Thus, the whole expression is in disjunctive normal form, which means that any function in propositional logic can be expressed as a decision tree.

For many problems, the decision tree format yields a nice, concise, understandable result. Indeed, many “How To” manuals (e.g., for car repair) are written as decision trees. But some functions cannot be represented concisely. For example, the majority function, which returns true if and only if more than half of the inputs are true, requires an exponentially large decision tree, as does the parity function, which returns true if and only if an even number of input attributes are true. With real-valued attributes, the function

y>A

1

+

A

2

is

hard to represent with a decision tree because the decision boundary is a diagonal line, and all decision tree tests divide the space up into rectangular, axis-aligned boxes. We would have to stack a lot of boxes to closely approximate the diagonal line. In other words, decision trees are good for some kinds of functions and bad for others.

Is there any kind of representation that is efficient for all kinds of functions? Unfortunately, the answer is no—there are just too many functions to be able to represent them all with a small number of bits. Even just considering Boolean functions with n Boolean attributes, the n truth table will have 2n rows, and each row can output true or false, so there are 22 different functions. With 20 attributes there are 21,048,576

≈ 10

300,000

functions, so if we limit ourselves

to a million-bit representation, we can’t represent all these functions.

19.3.2 Learning decision trees from examples We want to find a tree that is consistent with the examples in Figure 19.2  and is as small as possible. Unfortunately, it is intractable to find a guaranteed smallest consistent tree. But with some simple heuristics, we can efficiently find one that is close to the smallest. The LEARN-DECISION-TREE algorithm adopts a greedy divide-and-conquer strategy: always test the most important attribute first, then recursively solve the smaller subproblems that are defined by the possible results of the test. By “most important attribute,” we mean the one that makes the most difference to the classification of an example. That way, we hope to get to the correct classification with a small number of tests, meaning that all paths in the tree will be short and the tree as a whole will be shallow. Figure 19.4(a)  shows that Type is a poor attribute, because it leaves us with four possible outcomes, each of which has the same number of positive as negative examples. On the other hand, in (b) we see that Patrons is a fairly important attribute, because if the value is None or Some, then we are left with example sets for which we can answer definitively (No and Yes, respectively). If the value is Full, we are left with a mixed set of examples. There are four cases to consider for these recursive subproblems:

1. If the remaining examples are all positive (or all negative), then we are done: we can answer Yes or No. Figure 19.4(b)  shows examples of this happening in the None and Some branches. 2. If there are some positive and some negative examples, then choose the best attribute to split them. Figure 19.4(b)  shows Hungry being used to split the

remaining examples. 3. If there are no examples left, it means that no example has been observed for this combination of attribute values, and we return the most common output value from the set of examples that were used in constructing the node’s parent. 4. If there are no attributes left, but both positive and negative examples, it means that these examples have exactly the same description, but different classifications. This can happen because there is an error or noise in the data; because the domain is nondeterministic; or because we can’t observe an attribute that would distinguish the examples. The best we can do is return the most common output value of the remaining examples.

Noise

Figure 19.4

Splitting the examples by testing on attributes. At each node we show the positive (light boxes) and negative (dark boxes) examples remaining. (a) Splitting on Type brings us no nearer to distinguishing between positive and negative examples. (b) Splitting on Patrons does a good job of separating positive and negative examples. After splitting on Patrons, Hungry is a fairly good second test.

The LEARN-DECISION-TREE algorithm is shown in Figure 19.5 . Note that the set of examples is an input to the algorithm, but nowhere do the examples appear in the tree returned by the algorithm. A tree consists of tests on attributes in the interior nodes, values of attributes on

the branches, and output values on the leaf nodes. The details of the IMPORTANCE function are given in Section 19.3.3 . The output of the learning algorithm on our sample training set is shown in Figure 19.6 . The tree is clearly different from the original tree shown in Figure 19.3 . One might conclude that the learning algorithm is not doing a very good job of learning the correct function. This would be the wrong conclusion to draw, however. The learning algorithm looks at the examples, not at the correct function, and in fact, its hypothesis (see Figure 19.6 ) not only is consistent with all the examples, but is considerably simpler than the original tree! With slightly different examples the tree might be very different, but the function it represents would be similar.

Figure 19.5

The decision tree learning algorithm. The function IMPORTANCE is described in Section 19.3.3 . The function PLURALITY-VALUE selects the most common output value among a set of examples, breaking ties randomly.

Figure 19.6

The decision tree induced from the 12-example training set.

The learning algorithm has no reason to include tests for Raining and Reservation, because it can classify all the examples without them. It has also detected an interesting and previously unsuspected pattern: SR will wait for Thai food on weekends. It is also bound to make some mistakes for cases where it has seen no examples. For example, it has never seen a case where the wait is 0–10 minutes but the restaurant is full. In that case it says not to wait when Hungry is false, but SR would certainly wait. With more training examples the learning program could correct this mistake.

We can evaluate the performance of a learning algorithm with a learning curve, as shown in Figure 19.7 . For this figure we have 100 examples at our disposal, which we split randomly into a training set and a test set. We learn a hypothesis h with the training set and measure its accuracy with the test set. We can do this starting with a training set of size 1 and increasing one at a time up to size 99. For each size, we actually repeat the process of randomly splitting into training and test sets 20 times, and average the results of the 20 trials. The curve shows that as the training set size grows, the accuracy increases. (For this reason, learning curves are also called happy graphs.) In this graph we reach 95% accuracy, and it looks as if the curve might continue to increase if we had more data.

Figure 19.7

A learning curve for the decision tree learning algorithm on 100 randomly generated examples in the restaurant domain. Each data point is the average of 20 trials.

Learning curve

Happy graphs

19.3.3 Choosing attribute tests The decision tree learning algorithm chooses the attribute with the highest IMPORTANCE. We will now show how to measure importance, using the notion of information gain, which is defined in terms of entropy, which is the fundamental quantity in information theory (Shannon and Weaver, 1949).

Entropy

Entropy is a measure of the uncertainty of a random variable; the more information, the less entropy. A random variable with only one possible value—a coin that always comes up heads—has no uncertainty and thus its entropy is defined as zero. A fair coin is equally likely to come up heads or tails when flipped, and we will soon show that this counts as “1 bit” of entropy. The roll of a fair four-sided die has 2 bits of entropy, because there are 22 equally probable choices. Now consider an unfair coin that comes up heads 99% of the time. Intuitively, this coin has less uncertainty than the fair coin—if we guess heads we’ll be wrong only 1% of the time—so we would like it to have an entropy measure that is close to zero, but positive. In general, the entropy of a random variable probability

P (vk ) is defined as Entropy:

H (V ) =

∑P v k

(

k ) log2

1

P (vk )

V with values vk having

=−

∑P v k

(

k ) log2 P (vk ).

We can check that the entropy of a fair coin flip is indeed 1 bit:

H (Fair) = −(0.5 log

2

0.5 + 0.5 log2 0.5) = 1 .

And of a four-sided die is 2 bits:

H (Die4) = −(0.25 log

2

0.25 + 0.25 log2 0.25 + 0.25 log2 0.25 + 0.25 log2 0.25) = 2

For the loaded coin with 99% heads, we get

H (Loaded) = −(0.99 log It will help to define

q

2

0.99 + 0.01 log2 0.01) ≈ 0.08 bits.

B(q) as the entropy of a Boolean random variable that is true with

probability :

B(q) = −(q log q + (1 − q) log (1 − q)). 2

2

H (Loaded) = B(0.99) ≈ 0.08. Now let’s get back to decision tree learning. If a training set contains p positive examples and n negative examples, then the entropy of the output Thus,

variable on the whole set is

H (Output) = B( p +p n ).

The restaurant training set in Figure 19.2  has

p = n = 6, so the corresponding entropy is B(0.5) or exactly 1 bit. The result of a test on an attribute A will give us some information, thus reducing the overall entropy by some amount. We can measure this reduction by looking at the entropy remaining after the attribute test.

A with d distinct values divides the training set E into subsets E , … , Ed . Each subset Ek has pk positive examples and nk negative examples, so if we go along that branch, we will need an additional B(pk / (pk + nk )) bits of information to answer the question. A randomly chosen example from the training set has the kth value for the attribute (i.e., is in Ek with probability (pk + nk )/(p + n)), so the expected entropy remaining after testing attribute A is An attribute

1

d

n k B ( p k ). Remainder(A) = ∑ ppk + p k + nk +n k =1

The information gain from the attribute test on

A is the expected reduction in entropy:

Gain(A) = B( p +p n ) − Remainder(A).

Information gain

In fact

Gain(A) is just what we need to implement the IMPORTANCE function. Returning to the

attributes considered in Figure 19.4 , we have

Gain(Patrons) = 1 − [ B ( ) + B ( ) + B ( )] ≈ 0.541 bits, Gain(Type) = 1 − [ B ( ) + B ( ) + B ( ) + B ( )] = 0 bits, 2

0

4

4

6

12

2

12

4

12

2 6

2

1

2

1

4

2

4

2

12

2

12

2

12

4

12

4

confirming our intuition that Patrons is a better attribute to split on first. In fact, Patrons has the maximum information gain of any of the attributes and thus would be chosen by the decision tree learning algorithm as the root.

19.3.4 Generalization and overfitting

We want our learning algorithms to find a hypothesis that fits the training data, but more importantly, we want it to generalize well for previously unseen data. In Figure 19.1  we saw that a high-degree polynomial can fit all the data, but has wild swings that are not warranted by the data: it fits but can overfit. Overfitting becomes more likely as the number of attributes grows, and less likely as we increase the number of training examples. Larger hypothesis spaces (e.g., decision trees with more nodes or polynomials with high degree) have more capacity both to fit and to overfit; some model classes are more prone to overfitting than others.

For decision trees, a technique called decision tree pruning combats overfitting. Pruning works by eliminating nodes that are not clearly relevant. We start with a full tree, as generated by LEARN-DECISION-TREE. We then look at a test node that has only leaf nodes as descendants. If the test appears to be irrelevant—detecting only noise in the data—then we eliminate the test, replacing it with a leaf node. We repeat this process, considering each test with only leaf descendants, until each one has either been pruned or accepted as is.

Decision tree pruning

The question is, how do we detect that a node is testing an irrelevant attribute? Suppose we are at a node consisting of p positive and n negative examples. If the attribute is irrelevant, we would expect that it would split the examples into subsets such that each subset has roughly the same proportion of positive examples as the whole set, p/(p + n), and so the information gain will be close to zero.3 Thus, a low information gain is a good clue that the attribute is irrelevant. Now the question is, how large a gain should we require in order to split on a particular attribute?

3 The gain will be strictly positive except for the unlikely case where all the proportions are exactly the same. (See Exercise 19.NNGA.)

We can answer this question by using a statistical significance test. Such a test begins by assuming that there is no underlying pattern (the so-called null hypothesis). Then the actual data are analyzed to calculate the extent to which they deviate from a perfect absence of pattern. If the degree of deviation is statistically unlikely (usually taken to mean a 5% probability or less), then that is considered to be good evidence for the presence of a

significant pattern in the data. The probabilities are calculated from standard distributions of the amount of deviation one would expect to see in random sampling.

Significance test

Null hypothesis

In this case, the null hypothesis is that the attribute is irrelevant and, hence, that the information gain for an infinitely large sample would be zero. We need to calculate the probability that, under the null hypothesis, a sample of size v = n + p would exhibit the observed deviation from the expected distribution of positive and negative examples. We can measure the deviation by comparing the actual numbers of positive and negative ˆ k , assuming true examples in each subset, pk and nk , with the expected numbers, pˆk and n

irrelevance:

pˆk

=

p×

p k + nk p+n

ˆk n

=

n×

p k + nk . p+n

A convenient measure of the total deviation is given by

∑ d

Δ=

k=1

ˆk ) 2 (pk − p

pˆk

+

ˆ k )2 (nk − n ˆk n

.

Under the null hypothesis, the value of Δ is distributed according to the χ2 (chi-squared) distribution with d − 1 degrees of freedom. We can use a χ2 statistics function to see if a particular Δ value confirms or rejects the null hypothesis. For example, consider the restaurant Type attribute, with four values and thus three degrees of freedom. A value of Δ = 7.82 or more would reject the null hypothesis at the 5% level (and a value of Δ = 11.35

or more would reject at the 1% level). Values below that lead to accepting the hypothesis that the attribute is irrelevant, and thus the associated branch of the tree should be pruned away. This is known as χ2 pruning.

χ2 pruning

With pruning, noise in the examples can be tolerated. Errors in the example’s label (e.g., an

Y es) that should be (x, No)) give a linear increase in prediction error, whereas errors in the descriptions of examples (e.g., Price = $ when it was actually Price = $$) have example (x,

an asymptotic effect that gets worse as the tree shrinks down to smaller sets. Pruned trees perform significantly better than unpruned trees when the data contain a large amount of noise. Also, the pruned trees are often much smaller and hence easier to understand and more efficient to execute.

One final warning: You might think that

χ2 pruning and information gain look similar, so

why not combine them using an approach called early stopping—have the decision tree algorithm stop generating nodes when there is no good attribute to split on, rather than going to all the trouble of generating nodes and then pruning them away. The problem with early stopping is that it stops us from recognizing situations where there is no one good attribute, but there are combinations of attributes that are informative. For example, consider the XOR function of two binary attributes. If there are roughly equal numbers of examples for all four combinations of input values, then neither attribute will be informative, yet the correct thing to do is to split on one of the attributes (it doesn’t matter which one), and then at the second level we will get splits that are very informative. Early stopping would miss this, but generate-and-then-prune handles it correctly.

Early stopping

19.3.5 Broadening the applicability of decision trees Decision trees can be made more widely useful by handling the following complications:

MISSING DATA: In many domains, not all the attribute values will be known for every example. The values might have gone unrecorded, or they might be too expensive to obtain. This gives rise to two problems: First, given a complete decision tree, how

should one classify an example that is missing one of the test attributes? Second, how should one modify the information-gain formula when some examples have unknown values for the attribute? These questions are addressed in Exercise 19.MISS. CONTINUOUS AND MULTIVALUED INPUT ATTRIBUTES: For continuous attributes like Height, Weight, or Time, it may be that every example has a different attribute value. The information gain measure would give its highest score to such an attribute, giving us a shallow tree with this attribute at the root, and single-example subtrees for each possible value below it. But that doesn’t help when we get a new example to classify with an attribute value that we haven’t seen before.

Split point

A better way to deal with continuous values is a split point test—an inequality test on the value of an attribute. For example, at a given node in the tree, it might be the case that testing on

Weight

> 160

gives the most information. Efficient methods exist for

finding good split points: start by sorting the values of the attribute, and then consider only split points that are between two examples in sorted order that have different classifications, while keeping track of the running totals of positive and negative examples on each side of the split point. Splitting is the most expensive part of realworld decision tree learning applications. For attributes that are not continuous and do not have a meaningful ordering, but have a large number of possible values (e.g., Zipcode or CreditCardNumber), a measure called the information gain ratio (see Exercise 19.GAIN) can be used to avoid splitting into lots of single-example subtrees. Another useful approach is to allow an equality test of the form

A vk =

.

For example, the test

Zipcode

= 10002

could be used to pick out a large

group of people in this zip code in New York City, and to lump everyone else into the “other” subtree. CONTINUOUS-VALUED OUTPUT ATTRIBUTE: If we are trying to predict a numerical output value, such as the price of an apartment, then we need a regression tree rather than a classification tree. A regression tree has at each leaf a linear function of some subset of numerical attributes, rather than a single output value. For example, the branch for two-bedroom apartments might end with a linear function of square

footage and number of bathrooms. The learning algorithm must decide when to stop splitting and begin applying linear regression (see Section 19.6 ) over the attributes. The name CART, standing for Classification And Regression Trees, is used to cover both classes.

Regression tree

CART

A decision tree learning system for real-world applications must be able to handle all of these problems. Handling continuous-valued variables is especially important, because both physical and financial processes provide numerical data. Several commercial packages have been built that meet these criteria, and they have been used to develop thousands of fielded systems. In many areas of industry and commerce, decision trees are the first method tried when a classification method is to be extracted from a data set.

Decision trees have a lot going for them: ease of understanding, scalability to large data sets, and versatility in handling discrete and continuous inputs as well as classification and regression. However, they can have suboptimal accuracy (largely due to the greedy search), and if trees are very deep, then getting a prediction for a new example can be expensive in run time. Decision trees are also unstable in that adding just one new example can change the test at the root, which changes the entire tree. In Section 19.8.2  we will see that the random forest model can fix some of these issues.

Unstable

19.4 Model Selection and Optimization Our goal in machine learning is to select a hypothesis that will optimally fit future examples. To make that precise we need to define “future example” and “optimal fit.”

First we will make the assumption that the future examples will be like the past. We call this the stationarity assumption; without it, all bets are off. We assume that each example

Ej has

the same prior probability distribution:

P (Ej ) = P (Ej

+1

)=

P (Ej

+2

) = ⋯,

Stationarity

and is independent of the previous examples:

P (Ej ) = P (Ej ∣Ej , Ej −1

−2 ,

…) .

Examples that satisfy these equations are independent and identically distributed or i.i.d..

I.i.d.

The next step is to define “optimal fit.” For now, we will say that the optimal fit is the

hx

hypothesis that minimizes the error rate: the proportion of times that ( ) ≠

y for an (x, y)

example. (Later we will expand on this to allow different errors to have different costs, in effect giving partial credit for answers that are “almost” correct.) We can estimate the error rate of a hypothesis by giving it a test: measure its performance on a test set of examples. It would be cheating for a hypothesis (or a student) to peek at the test answers before taking

the test. The simplest way to ensure this doesn’t happen is to split the examples you have into two sets: a training set to create the hypothesis, and a test set to evaluate it.

Error rate

If we are only going to create one hypothesis, then this approach is sufficient. But often we will end up creating multiple hypotheses: we might want to compare two completely different machine learning models, or we might want to adjust the various “knobs” within one model. For example, we could try different thresholds for χ2 pruning of decision trees, or different degrees for polynomials. We call these “knobs” hyperparameters—parameters of the model class, not of the individual model.

Hyperparameters

Suppose a researcher generates a hypotheses for one setting of the χ2 pruning hyperparameter, measures the error rates on the test set, and then tries different hyperparameters. No individual hypothesis has peeked at the test set data, but the overall process did, through the researcher.

The way to avoid this is to really hold out the test set—lock it away until you are completely done with training, experimenting, hyperparameter-tuning, re-training, etc. That means you need three data sets:

1. A training set to train candidate models. 2. A validation set, also known as a development set or dev set, to evaluate the candidate models and choose the best one.

Validation set

3. A test set to do a final unbiased evaluation of the best model.

What if we don’t have enough data to make all three of these data sets? We can squeeze more out of the data using a technique called k-fold cross-validation. The idea is that each example serves double duty—as training data and validation data—but not at the same time. First we split the data into k equal subsets. We then perform k rounds of learning; on each round

1/k of the data are held out as a validation set and the remaining examples are used

as the training set. The average test set score of the k rounds should then be a better estimate than a single score. Popular values for k are 5 and 10—enough to give an estimate that is statistically likely to be accurate, at a cost of 5 to 10 times longer computation time. The extreme is k

= n, also known as leave-one-out cross-validation or LOOCV. Even with

cross-validation, we still need a separate test set.

K-fold cross-validation

LOOCV

In Figure 19.1  (page 654) we saw a linear function underfit the data set, and a high-degree polynomial overfit the data. We can think of the task of finding a good hypothesis as two subtasks: model selection4 chooses a good hypothesis space, and optimization (also called training) finds the best hypothesis within that space.

4 Although the name “model selection” is in common use, a better name would have been “model class selection” or “hypothesis space selection.” The word “model” has been used in the literature to refer to three different levels of specificity: a broad hypothesis space (like “polynomials”), a hypothesis space with hyperparameters filled in (like “degree-2 polynomials”), and a specific hypothesis with all

5 + 3x − 2).

parameters filled in (like x2

Model selection

Optimization

Part of model selection is qualitative and subjective: we might select polynomials rather than decision trees based on something that we know about the problem. And part is quantitative and empirical: within the class of polynomials, we might select

Degree

= 2,

because that value performs best on the validation data set.

19.4.1 Model selection Figure 19.8  describes a simple MODEL-SELECTION algorithm. It takes as argument a learning algorithm, Learner (for example, it could be LEARN-DECISION-TREE). Learner takes one hyperparameter, which is named size in the figure. For decision trees it could be the number of nodes in the tree; for polynomials size would be Degree. MODEL-SELECTION starts with the smallest value of size, yielding a simple model (which will probably underfit the data) and iterates through larger values of size, considering more complex models. In the end MODELSELECTION selects the model that has the lowest average error rate on the held-out validation data.

Figure 19.8

An algorithm to select the model that has the lowest validation error. It builds models of increasing complexity, and choosing the one with best empirical error rate, err, on the validation data set. Learner(size,examples) returns a hypothesis whose complexity is set by the parameter size, and which is trained on examples. In CROSS-VALIDATION, each iteration of the for loop selects a different slice of the examples as the validation set, and keeps the other examples as the training set. It then returns the average validation set error over all the folds. Once we have determined which value of the size parameter is best, MODEL-SELECTION returns the model (i.e., learner/hypothesis) of that size, trained on all the training examples, along with its error rate on the held-out test examples.

In Figure 19.9  we see two typical patterns that occur in model selection. In both (a) and (b) the training set error decreases monotonically (with slight random fluctuation) as we increase the complexity of the model. Complexity is measured by the number of decision tree nodes in (a) and by the number of neural network parameters (wi ) in (b). For many model classes, the training set error reaches zero as the complexity increases.

Figure 19.9

Error rates on training data (lower, green line) and validation data (upper, orange line) for models of different complexity on two different problems. MODEL-SELECTION picks the hyperparameter value with the lowest validation-set error. In (a) the model class is decision trees and the hyperparameter is the number of nodes. The data is from a version of the restaurant problem. The optimal size is 7. In (b) the model

class is convolutional neural networks (see Section 21.3 ) and the hyperparameter is the number of regular parameters in the network. The data is the MNIST data set of images of digits; the task is to identify each digit. The optimal number of parameters is 1,000,000 (note the log scale).

The two cases differ markedly in validation set error. In (a) we see a U-shaped validationerror curve: error decreases for a while as model complexity increases, but then we reach a point where the model begins to overfit, and validation error rises. MODEL-SELECTION picks the value at the bottom of the U-shaped validation-error curve: in this case a tree with size 7. This is the spot that best balances underfitting and overfitting. In (b) we see an initial Ushaped curve just as in (a) but then the validation error starts to decrease again; the lowest validation error rate is the final point in the plot, with 1,000,000 parameters.

Why are some validation-error curves like (a) and some like (b)? It comes down to how the different model classes make use of excess capacity, and how well that matches up with the problem at hand. As we add capacity to a model class, we often reach the point where all the training examples can be represented perfectly within the model. For example, given a training set with n distinct examples, there is always a decision tree with n leaf nodes that can represent all the examples.

Interpolated

We say that a model that exactly fits all the training data has interpolated the data.5 Model classes typically start to overfit as the capacity approaches the point of interpolation. That seems to be because most of the model’s capacity is concentrated on the training examples, and the capacity that remains is allocated rather randomly in a way that is not representative of the patterns in the validation data set. Some model classes never recover from this overfitting, as with the decision trees in (a). But for other model classes, adding capacity means that there are more candidate functions, and some of them are naturally well-suited to the patterns of data that are in the true function f (x). The higher the capacity, the more of these suitable representations there are, and the more likely that the optimization mechanism will be able to land on one.

5 Some authors say the model has “memorized” the data.

Deep neural networks (Chapter 21 ), kernel machines (Section 19.7.5 ), random forests (Section 19.8.2 ), and boosted ensembles (Section 19.8.4 ) all have the property that their validation set error tends to decrease as capacity increases, as in Figure 19.9(b) .

We could extend the model selection algorithm in various ways: we could compare disparate model classes, by calling MODEL-SELECTION with DECISION-TREE-LEARNER as an argument and then with POLYNOMIAL-LEARNER, and seeing which does better. We could allow multiple hyperparameters, which means we would need a more complex optimization algorithm, such as a grid search (see Section 19.9.3 ) rather than a linear search.

19.4.2 From error rates to loss So far, we have been trying to minimize error rate. This is clearly better than maximizing error rate, but it is not the full story. Consider the problem of classifying email messages as spam or non-spam. It is worse to classify non-spam as spam (and thus potentially miss an important message) than to classify spam as non-spam (and thus suffer a few seconds of annoyance). So a classifier with a 1% error rate, where almost all the errors were classifying spam as non-spam, would be better than a classifier with only a 0.5% error rate, if most of those errors were classifying non-spam as spam. We saw in Chapter 16  that decision makers should maximize expected utility, and utility is what learners should maximize as well. However, in machine learning it is traditional to express this as a negative: to minimize a loss function rather than maximize a utility function. The loss function L(x, y, yˆ)

is defined as the amount of utility lost by predicting h(x)

fx (

) =

y

y when the correct answer is

= ˆ

:

Lxyy (

,

, ˆ) = −

Utility Utility

y y

(result of using   given an input 

x x

)

(result of using  ˆ given an input 

)

Loss function

This is the most general formulation of the loss function. Often a simplified version is used,

Lyy

( , ˆ),

that is independent of x. We will use the simplified version for the rest of this

chapter, which means we can’t say that it is worse to misclassify a letter from Mom than it is to misclassify a letter from our annoying cousin, but we can say that it is 10 times worse to classify non-spam as spam than vice versa:

L spam nospam (

,

L nospam spam

) = 1,

(

,

) = 10.

Note that L(y, y) is always zero; by definition there is no loss when you guess exactly right. For functions with discrete outputs, we can enumerate a loss value for each possible misclassification, but we can’t enumerate all the possibilities for real-valued data. If f (x) is 137.035999, we would be fairly happy with

hx (

) = 137.036,

but just how happy should we

be? In general, small errors are better than large ones; two functions that implement that idea are the absolute value of the difference (called the L loss), and the square of the 1

difference (called the L loss; think “2” for square). For discrete-valued outputs, if we are 2

content with the idea of minimizing error rate, we can use the L

0/1

loss function, which has

a loss of 1 for an incorrect answer:

Absolute-value loss: Squared-error loss: 0/1 loss:

L yy L yy L yy

y y y y y y

1(

, ˆ) = |

2(

, ˆ) = (

0/1 (

− ˆ|

− ˆ)

, ˆ) = 0 if 

2

= ˆ,  else 1

Theoretically, the learning agent maximizes its expected utility by choosing the hypothesis that minimizes expected loss over all input–output pairs it will see. To compute this expectation we need to define a prior probability distribution

PXY (

,

)

over examples. Let ε

be the set of all possible input–output examples. Then the expected generalization loss for a hypothesis

h (with respect to loss function L) is GenLossL (h) =

∑ Lyhx

xy ε

( , ( ))

P ( x, y ) ,

( , )∈

Generalization loss

and the best hypothesis,

h , is the one with the minimum expected generalization loss: ∗

h

∗

= argmin

h ∈H

GenLossL (h).

P (x, y) is not known in most cases, the learning agent can only estimate generalization loss with empirical loss on a set of examples E of size N : Because

EmpLossL E (h) = ,

∑ Lyhx

xy E

( , ( ))

( , )∈

1

N.

Empirical loss

h

∗ The estimated best hypothesis ˆ is then the one with minimum empirical loss:

hˆ h

∗

= argmin

∗

h ∈H

EmpLossL E (h). ,

f

There are four reasons why ˆ may differ from the true function, : unrealizability, variance, noise, and computational complexity.

H actually contains the true function f . If H is the set of linear functions, and the true function f is a quadratic function, then no amount of data will recover the true f . Second, variance means that a First, we say that a learning problem is realizable if the hypothesis space

learning algorithm will in general return different hypotheses for different sets of examples. If the problem is realizable, then variance decreases towards zero as the number of training

f may be nondeterministic or noisy—it may return different values of f (x) for the same x. By definition, noise cannot be predicted (it can only be characterized). And finally, when H is a complicated function in a large hypothesis space, it

examples increases. Third,

can be computationally intractable to systematically search all possibilities; in that case, a search can explore part of the space and return a reasonably good hypothesis, but can’t always guarantee the best one.

Realizable

Noise

Traditional methods in statistics and the early years of machine learning concentrated on small-scale learning, where the number of training examples ranged from dozens to the low thousands. Here the generalization loss mostly comes from the approximation error of not having the true

f in the hypothesis space, and from the estimation error of not having

enough training examples to limit variance.

Small-scale learning

In recent years there has been more emphasis on large-scale learning, with millions of examples. Here the generalization loss may be dominated by limits of computation: there are enough data and a rich enough model that we could find an

f

h that is very close to the

true , but the computation to find it is complex, so we settle for an approximation.

Large-scale learning

19.4.3 Regularization In Section 19.4.1 , we saw how to do model selection with cross-validation. An alternative approach is to search for a hypothesis that directly minimizes the weighted sum of empirical loss and the complexity of the hypothesis, which we will call the total cost:

Cost(h) = EmpLoss(h) + λ Complexity(h) hˆ = argmin Cost(h). ∗

h ∈H

λ is a hyperparameter, a positive number that serves as a conversion rate between loss and hypothesis complexity. If λ is chosen well, it nicely balances the empirical loss of a Here

simple function against a complicated function’s tendency to overfit.

This process of explicitly penalizing complex hypotheses is called regularization: we’re looking for functions that are more regular. Note that we are now making two choices: the

L

loss function (

1

or

L ), and the complexity measure, which is called a regularization 2

function. The choice of regularization function depends on the hypothesis space. For example, for polynomials, a good regularization function is the sum of the squares of the coefficients—keeping the sum small would guide us away from the wiggly degree-12 polynomial in Figure 19.1 . We will show an example of this type of regularization in Section 19.6.3 .

Regularization

Regularization function

Another way to simplify models is to reduce the dimensions that the models work with. A process of feature selection can be performed to discard attributes that appear to be irrelevant. χ2 pruning is a kind of feature selection.

Feature selection

It is in fact possible to have the empirical loss and the complexity measured on the same scale, without the conversion factor λ: they can both be measured in bits. First encode the hypothesis as a Turing machine program, and count the number of bits. Then count the number of bits required to encode the data, where a correctly predicted example costs zero bits and the cost of an incorrectly predicted example depends on how large the error is. The minimum description length or MDL hypothesis minimizes the total number of bits required. This works well in the limit, but for smaller problems the choice of encoding for the program—how best to encode a decision tree as a bit string—affects the outcome. In Chapter 20  (page 724), we describe a probabilistic interpretation of the MDL approach.

Minimum description length

19.4.4 Hyperparameter tuning In Section 19.4.1  we showed how to select the best value of the hyperparameter size by applying cross-validation to each possible value until the validation error rate increases. That is a good approach when there is a single hyperparameter with a small number of possible values. But when there are multiple hyperparameters, or when they have continuous values, it is more difficult to choose good values.

The simplest approach to hyperparameter tuning is hand-tuning: guess some parameter values based on past experience, train a model, measure its performance on the validation data, analyze the results, and use your intuition to suggest new parameter values. Repeat

until you have satisfactory performance (or you run out of time, computing budget, or patience).

Hand-tuning

If there are only a few hyperparameters, each with a small number of possible values, then a more systematic approach called grid search is appropriate: try all combinations of values and see which performs best on the validation data. Different combinations can be run in parallel on different machines, so if you have sufficient computing resources, this need not be slow, although in some cases model selection has been known to suck up resources on thousand-computer clusters for days at a time.

Grid search

The search strategies from Chapters 3  and 4  can also come into play. For example, if two hyperparameters are independent of each other, they can be optimized separately.

If there are too many combinations of possible values, then random search samples uniformly from the set of all possible hyperparameter settings, repeating for as long as you are willing to spend the time and computational resources. Random sampling is also a good way to handle continuous values.

Random search

When each training run takes a long time, it can be helpful to get useful information out of each one. Bayesian optimization treats the task of choosing good hyperparameter values as

a machine learning problem in itself. That is, think of the vector of hyperparameter values x as an input, and the total loss on the validation set for the model built with those hyperparameters as an output, y; then we are trying to find the function y = f (x) that minimizes the loss y. Each time we do a training run we get a new (y, f (x)) pair, which we can use to update our belief about the shape of the function f .

Bayesian optimization

The idea is to trade off exploitation (choosing parameter values that are near to a previous good result) with exploration (trying novel parameter values). This is the same tradeoff we saw in Monte Carlo tree search (Section 5.4 ), and in fact the idea of upper confidence bounds is used here as well to minimize regret. If we make the assumption that f can be approximated by a Gaussian process, then the math of updating our belief about f works out nicely. Snoek et al. (2013) explain the math and give a practical guide to the approach, showing that it can outperform hand-tuning of parameters, even by experts.

Population-based training (PBT)

An alternative to Bayesian optimization is population-based training (PBT). PBT starts by using random search to train (in parallel) a population of models, each with different hyperparameter values. Then a second generation of models are trained, but they can choose hyperparameter values based on the successful values from the previous generation, as well as by random mutation, as in genetic algorithms (Section 4.1.4 ). Thus, populationbased training shares the advantage of random search that many runs can be done in parallel, and it shares the advantage of Bayesian optimization (or of hand-tuning by a clever human) that we can gain information from earlier runs to inform later ones.

19.5 The Theory of Learning How can we be sure that our learned hypothesis will predict well for previously unseen inputs? That is, how do we know that the hypothesis h is close to the target function f if we don’t know what f is? These questions have been pondered for centuries, by Ockham, Hume, and others. In recent decades, other questions have emerged: how many examples do we need to get a good h? What hypothesis space should we use? If the hypothesis space is very complex, can we even find the best h, or do we have to settle for a local maximum? How complex should h be? How do we avoid overfitting? This section examines these questions.

We’ll start with the question of how many examples are needed for learning. We saw from the learning curve for decision tree learning on the restaurant problem (Figure 19.7  on page 661 ) that accuracy improves with more training data. Learning curves are useful, but they are specific to a particular learning algorithm on a particular problem. Are there some more general principles governing the number of examples needed?

Questions like this are addressed by computational learning theory, which lies at the intersection of AI, statistics, and theoretical computer science. The underlying principle is that any hypothesis that is seriously wrong will almost certainly be “found out” with high probability after a small number of examples, because it will make an incorrect prediction. Thus, any hypothesis that is consistent with a sufficiently large set of training examples is unlikely to be seriously wrong: that is, it must be probably approximately correct (PAC).

Computational learning theory

Probably approximately correct (PAC)

Any learning algorithm that returns hypotheses that are probably approximately correct is called a PAC learning algorithm; we can use this approach to provide bounds on the performance of various learning algorithms.

PAC learning

PAC-learning theorems, like all theorems, are logical consequences of axioms. When a theorem (as opposed to, say, a political pundit) states something about the future based on the past, the axioms have to provide the “juice” to make that connection. For PAC learning, the juice is provided by the stationarity assumption introduced on page 665 , which says that future examples are going to be drawn from the same fixed distribution

E

P( ) = P(

X, Y ) as past examples. (Note that we do not have to know what distribution that

is, just that it doesn’t change.) In addition, to keep things simple, we will assume that the true function

f is deterministic and is a member of the hypothesis space H that is being

considered.

The simplest PAC theorems deal with Boolean functions, for which the 0/1 loss is

h

appropriate. The error rate of a hypothesis , defined informally earlier, is defined formally here as the expected generalization error for examples drawn from the stationary distribution:

h

error( ) =

h

GenLossL (h) = 0/1

In other words, error( ) is the probability that

∑L x, y

0/1 (

y, h(x)) P (x, y).

h misclassifies a new example. This is the

same quantity being measured experimentally by the learning curves shown earlier.

h is called approximately correct if error(h) ≤ ϵ, where ϵ is a small constant. We will show that we can find an N such that, after training on N examples, with high A hypothesis

probability, all consistent hypotheses will be approximately correct. One can think of an approximately correct hypothesis as being “close” to the true function in hypothesis space: it

ϵ

f

lies inside what is called the -ball around the true function . The hypothesis space outside this ball is called

H

 bad .

ϵ

− ball

We can derive a bound on the probability that a “seriously wrong” hypothesis

N examples as follows. We know that ϵ probability that it agrees with a given example is at most independent, the bound for N examples is: consistent with the first

1 −

P hb (

The probability that

H

 bad

 agrees with 

N

error( .

 examples) ≤ (1 −

hb

) >

ϵ

.

hb H ∈

 bad

is

Thus, the

Since the examples are

ϵN )

.

contains at least one consistent hypothesis is bounded by the sum

of the individual probabilities:

PH (

 bad  contains

a consistent hypothesis) ≤ ∣ ∣

where we have used the fact that

H

 bad

is a subset of

H

∣

 bad ∣(1

−

ϵN )

H and thus H |

≤ ∣ ∣

 bad |

like to reduce the probability of this event below some small number

PH (

Given that 1 −

ϵ e ≤

 bad  contains

−

a consistent hypothesis) ≤ |

H

H

|(1 −

δ

∣(1 − ∣

≤ |

H

)

,

We would

:

ϵN δ )

|.

ϵN

≤

.

ϵ , we can achieve this if we allow the algorithm to see

(19.1)

N

ϵ( δ

1 ≥

1

ln

examples. Thus, with probability at least 1 −

δ

,

+ ln |

H) |

after seeing this many examples, the learning

algorithm will return a hypothesis that has error at most

ϵ

.

In other words, it is probably

approximately correct. The number of required examples, as a function of the sample complexity of the learning algorithm.

Sample complexity

ϵ and δ

,

is called

As we saw earlier, if

H is the set of all Boolean functions on n attributes, then ∣∣H ∣∣ = 22n .

Thus, the sample complexity of the space grows as examples is also

2n . Because the number of possible

2n , this suggests that PAC-learning in the class of all Boolean functions

requires seeing all, or nearly all, of the possible examples. A moment’s thought reveals the

H contains enough hypotheses to classify any given set of examples in all possible ways. In particular, for any set of N examples, the set of hypotheses consistent with those examples contains equal numbers of hypotheses that predict xN +1 to be positive and hypotheses that predict xN +1 to be negative. reason for this:

To obtain real generalization to unseen examples, then, it seems we need to restrict the hypothesis space

H in some way; but of course, if we do restrict the space, we might

eliminate the true function altogether. There are three ways to escape this dilemma.

The first is to bring prior knowledge to bear on the problem. The second, which we introduced in Section 19.4.3 , is to insist that the algorithm return not just any consistent hypothesis, but preferably a simple one (as is done in decision tree learning). In cases where finding simple consistent hypotheses is tractable, the sample complexity results are generally better than for analyses based only on consistency.

The third, which we pursue next, is to focus on learnable subsets of the entire hypothesis space of Boolean functions. This approach relies on the assumption that the restricted hypothesis space contains a hypothesis

h that is close enough to the true function f ; the

benefits are that the restricted hypothesis space allows for effective generalization and is typically easier to search. We now examine one such restricted hypothesis space in more detail.

19.5.1 PAC learning example: Learning decision lists We now show how to apply PAC learning to a new hypothesis space: decision lists. A decision list consists of a series of tests, each of which is a conjunction of literals. If a test succeeds when applied to an example description, the decision list specifies the value to be returned. If the test fails, processing continues with the next test in the list. Decision lists resemble decision trees, but their overall structure is simpler: they branch only in one direction. In contrast, the individual tests are more complex. Figure 19.10  shows a decision list that represents the following hypothesis:

WillWait

⇔

(

Patrons Some =

)∨(

Patrons Full Fri Sat =

∧

/

).

Figure 19.10

A decision list for the restaurant problem.

Decision lists

If we allow tests of arbitrary size, then decision lists can represent any Boolean function (Exercise 19.DLEX). On the other hand, if we restrict the size of each test to at most

k literals,

then it is possible for the learning algorithm to generalize successfully from a small number

k conjunctions. The includes example in Figure 19.10  is in 2-DL. It is easy to show (Exercise 19.DLEX) that k as a subset k DT the set of all decision trees of depth at most k We will use the notation k n to denote a k using n Boolean attributes. of examples. We use the notation

k

 -DL

for a decision list with up to

 -DL

−

 -DL(

)

,

.

 -DL

K-DT

The first task is to show that

k

 -DL

is learnable—that is, that any function in

k

 -DL

can be

approximated accurately after training on a reasonable number of examples. To do this, we need to calculate the number of possible hypotheses. Let the set of conjunctions of at most literals using

n attributes be Conj n k (

,

).

k

Because a decision list is constructed from tests, and

because each test can be attached to either a Yes or a No outcome or can be absent from the decision list, there are at most 3|Conj(n,k)| distinct sets of component tests. Each of these sets of tests can be in any order, so

|

k

 -DL(

n

)| ≤ 3

c c! where c = |Conj(n, k)|.

k literals from n attributes is given by

The number of conjunctions of at most

∣ ∣ ∣ ∣

Conj n k (

,

∑ ( in ) k

∣ )∣ = ∣ ∣

i

2

=

O nk (

).

=0

Hence, after some work, we obtain

∣ ∣

k

 -DL(

n

∣ )∣ = 2

O nk (

log ( 2

nk

))

.

We can plug this into Equation (19.1)  to show that the number of examples needed for PAC-learning a

k

 -DL(

n

)

function is polynomial in

N

k ϵ( δ O n

:

1

1 ≥

n

ln

+

(

log2 (

nk ) ))

.

Therefore, any algorithm that returns a consistent decision list will PAC-learn a function in a reasonable number of examples, for small

k

k

 -DL

.

The next task is to find an efficient algorithm that returns a consistent decision list. We will use a greedy algorithm called DECISION-LIST-LEARNING that repeatedly finds a test that agrees exactly with some subset of the training set. Once it finds such a test, it adds it to the decision list under construction and removes the corresponding examples. It then constructs the remainder of the decision list, using just the remaining examples. This is repeated until there are no examples left. The algorithm is shown in Figure 19.11 .

Figure 19.11

An algorithm for learning decision lists.

This algorithm does not specify the method for selecting the next test to add to the decision list. Although the formal results given earlier do not depend on the selection method, it would seem reasonable to prefer small tests that match large sets of uniformly classified examples, so that the overall decision list will be as compact as possible. The simplest strategy is to find the smallest test t that matches any uniformly classified subset, regardless of the size of the subset. Even this approach works quite well, as Figure 19.12  suggests. For this problem, the decision tree learns a bit faster than the decision list, but has more variation. Both methods are over 90% accurate after 100 trials.

Figure 19.12

Learning curve for DECISION-LIST-LEARNING algorithm on the restaurant data. The curve for LEARN-DECISIONTREE is shown for comparison; decision trees do slightly better on this particular problem.

19.6 Linear Regression and Classification Now it is time to move on from decision trees and lists to a different hypothesis space, one that has been used for hundreds of years: the class of linear functions of continuous-valued inputs. We’ll start with the simplest case: regression with a univariate linear function, otherwise known as “fitting a straight line.” Section 19.6.3  covers the multivariable case. Sections 19.6.4  and 19.6.5  show how to turn linear functions into classifiers by applying hard and soft thresholds.

Linear function

19.6.1 Univariate linear regression A univariate linear function (a straight line) with input x and output y has the form

y = w1 x + w0 , where w0 and w1 are real-valued coefficients to be learned. We use the letter w because we think of the coefficients as weights; the value of y is changed by changing the relative weight of one term or another. We’ll define

w to be the vector ⟨w , w ⟩, and define 0

1

the linear function with those weights as

h w ( x) = w 1 x + w 0 .

Weight

Figure 19.13(a)  shows an example of a training set of n points in the x, y plane, each point representing the size in square feet and the price of a house offered for sale. The task of finding the hw that best fits these data is called linear regression. To fit a line to the data, all we have to do is find the values of the weights ⟨w0 , w1 ⟩ that minimize the empirical loss. It is

traditional (going back to Gauss6) to use the squared-error loss function, L2 , summed over all the training examples: 6 Gauss showed that if the yj values have normally distributed noise, then the most likely values of w1 and w0 are obtained by using

L2 loss, minimizing the sum of the squares of the errors. (If the values have noise that follows a Laplace (double exponential)

distribution, then L1 loss is appropriate.)

Loss(hw ) =

∑L N

j=1

2 (yj , hw (xj )) =

∑y N

j=1

( j − hw (xj )) = 2

∑y N

j=1

( j − (w1 xj + w0 )) . 2

Linear regression

Figure 19.13

(a) Data points of price versus floor space of houses for sale in Berkeley, CA, in July 2009, along with the linear function hypothesis that minimizes squared-error loss: y = 0.232x + 246. (b) Plot of the loss function ∑j (yj − w1 xj + w0 )2 for various values of w0 , w1 . Note that the loss function is convex, with a single global minimum.

w

Loss(hw ). The sum ∑Nj=1 (yj − (w1 xj + w0 ))2 is minimized when its partial derivatives with respect to w0 and w1 are zero: We would like to find

∗

= argminw

(19.2)

∑y w N

∂ ∂

0

j=1

(

2 j − (w1 xj + w0 )) = 0 and 

∑y w N

∂ ∂

1

j=1

(

2 j − (w1 xj + w0 )) = 0.

These equations have a unique solution:

(19.3)

w

N (∑ xj yj ) − (∑ xj ) (∑ yj ) 1

=

N (∑ xj ) − (∑ xj )

2

2

;

  w

0

For the example in Figure 19.13(a) , the solution is w1

=

(∑ yj − w (∑ xj )) /N .

= 0.232,

1

w

0

= 246,

and the line with

those weights is shown as a dashed line in the figure.

Many forms of learning involve adjusting weights to minimize a loss, so it helps to have a mental picture of what’s going on in weight space—the space defined by all possible settings of the weights. For univariate linear regression, the weight space defined by w0 and w1 is two-dimensional, so we can graph the loss as a function of w0 and w1 in a 3D plot (see

Figure 19.13(b) ). We see that the loss function is convex, as defined on page 122; this is true for every linear regression problem with an L2 loss function, and implies that there are no local minima. In some sense that’s the end of the story for linear models; if we need to fit lines to data, we apply Equation (19.3) .7 7 With some caveats: the L2 loss function is appropriate when there is normally distributed noise that is independent of x; all results rely on the stationarity assumption; etc.

Weight space

19.6.2 Gradient descent The univariate linear model has the nice property that it is easy to find an optimal solution where the partial derivatives are zero. But that won’t always be the case, so we introduce here a method for minimizing loss that does not depend on solving to find zeroes of the derivatives, and can be applied to any loss function, no matter how complex.

Gradient descent

As discussed in Section 4.2  (page 119) we can search through a continuous weight space by incrementally modifying the parameters. There we called the algorithm hill climbing, but here we are minimizing loss, not maximizing gain, so we will use the term gradient descent. We choose any starting point in weight space—here, a point in the (w0 ,

w1 ) plane—

and then compute an estimate of the gradient and move a small amount in the steepest downhill direction, repeating until we converge on a point in weight space with (local) minimum loss. The algorithm is as follows:

(19.4)

w ← any point in the parameter space while not converged do    for each wi in w do      wi ← wi − α ∂ Loss (w) ∂ wi

Gradient descent

The parameter α, which we called the step size in Section 4.2 , is usually called the learning rate when we are trying to minimize loss in a learning problem. It can be a fixed constant, or it can decay over time as the learning process proceeds.

Learning rate

For univariate regression, the loss is quadratic, so the partial derivative will be linear. (The only calculus you need to know is the chain rule: ∂ g(f (x))/∂ x =

g ′ (f (x)) ∂f (x)/∂x, plus the

facts that ∂∂x x2 = 2x and ∂∂x x = 1.) Let’s first work out the partial derivatives—the slopes—in the simplified case of only one training example, (x, y):

(19.5) ∂ ∂

wi

Loss(w)

∂

∂ 2 (y − hw (x))   =  2(y − hw (x)) × (y − hw (x)) wi ∂wi ∂ 2(y − hw (x)) × (y − (w1 x + w0 )). ∂wi

=

∂

=

Chain rule

Applying this to both w0 and w1 we get: ∂

∂

w0

Loss(w) = −2 (y − hw (x)) ;

∂

∂

w1

Loss (w) = −2 (y − hw (x)) × x.

Plugging this into Equation (19.4) , and folding the 2 into the unspecified learning rate α, we get the following learning rule for the weights:

w0 ← w0 + α  (y − hw (x))  ; w1 ← w1 + α  (y − hw (x)  ) ×  x. These updates make intuitive sense: if hw (x) >

y (i.e., the output is too large), reduce w0 a bit, and reduce w1 if x was a positive input but increase w1 if x was a negative input. The preceding equations cover one training example. For N training examples, we want to minimize the sum of the individual losses for each example. The derivative of a sum is the sum of the derivatives, so we have:

w 0 ← w0 + α

∑y j

(

j − hw (xj )) ;

w1 ← w 1 + α

∑y j

(

j − hw (xj ))  ×  xj .

These updates constitute the batch gradient descent learning rule for univariate linear regression (also called deterministic gradient descent). The loss surface is convex, which means that there are no local minima to get stuck in, and convergence to the global minimum is guaranteed (as long as we don’t pick an α that is so large that it overshoots),

but may be very slow: we have to sum over all N training examples for every step, and there may be many steps. The problem is compounded if N is larger than the processor’s memory size. A step that covers all the training examples is called an epoch.

Batch gradient descent

A faster variant is called stochastic gradient descent or SGD: it randomly selects a small number of training examples at each step, and updates according to Equation (19.5) . The original version of SGD selected only one training example for each step, but it is now more

m out of the N examples. Suppose we have N examples and choose a minibatch of size m Then on each step we have reduced the

common to select a minibatch of

= 10, 000

= 100.

amount of computation by a factor of 100; but because the standard error of the estimated mean gradient is proportional to the square root of the number of examples, the standard error increases by only a factor of 10. So even if we have to take 10 times more steps before convergence, minibatch SGD is still 10 times faster than full batch SGD in this case.

Epoch

Stochastic gradient descent

SGD

Minibatch

With some CPU or GPU architectures, we can choose

m to take advantage of parallel vector

m examples almost as fast as a step with only a single example. Within these constraints, we would treat m as a hyperparameter that should be operations, making a step with

tuned for each learning problem.

Convergence of minibatch SGD is not strictly guaranteed; it can oscillate around the minimum without settling down. We will see on page 684 how a schedule of decreasing the learning rate, α, (as in simulated annealing) does guarantee convergence.

SGD can be helpful in an online setting, where new data are coming in one at a time, and the stationarity assumption may not hold. (In fact, SGD is also known as online gradient descent.) With a good choice for α a model will slowly evolve, remembering some of what it learned in the past, but also adapting to the changes represented by the new data.

Online gradient descent

SGD is widely applied to models other than linear regression, in particular neural networks. Even when the loss surface is not convex, the approach has proven effective in finding good local minima that are close to the global minimum.

19.6.3 Multivariable linear regression We can easily extend to multivariable linear regression problems, in which each example

xj is an n-element vector.8 Our hypothesis space is the set of functions of the form

8 The reader may wish to consult Appendix A  for a brief summary of linear algebra. Also, note that we use the term “multivariable regression” to mean that the input is a vector of multiple values, but the output is a single variable. We will use the term “multivariate regression” for the case where the output is also a vector of multiple variables. However, other authors use the two terms interchangeably.

hw (xj ) = w0 + w1 xj,1 + ⋯ + wn xj,n = w0 +

∑ i

w i x j, i .

Multivariable linear regression

The w0 term, the intercept, stands out as different from the others. We can fix that by inventing a dummy input attribute, xj,0 , which is defined as always equal to 1. Then h is

simply the dot product of the weights and the input vector (or equivalently, the matrix product of the transpose of the weights and the input vector):

h w ( x j ) = w ⋅ x j = w⊤ xj = The best vector of weights,

∑w x i

i j, i .

w , minimizes squared-error loss over the examples: ∗

w

∗

= argmin

w

∑L y w x 2(

j

j,

⋅

j ).

Multivariable linear regression is actually not much more complicated than the univariate case we just covered. Gradient descent will reach the (unique) minimum of the loss function; the update equation for each weight wi is (19.6)

wi

←

wi + α

∑y j

(

x

j − hw ( j )) × xj,i .

With the tools of linear algebra and vector calculus, it is also possible to solve analytically for the

w that minimizes loss. Let y be the vector of outputs for the training examples, and

X be the data matrix—that is, the matrix of inputs with one n-dimensional example per ˆ = Xw and the squared-error loss over all the row. Then the vector of predicted outputs is y training data is

L(w) =∥ yˆ − y ∥2 =∥ Xw − y∥∥ . 2

Data matrix

We set the gradient to zero:

L w) = 2X⊤ (Xw − y) = 0.

∇w (

Rearranging, we find that the minimum-loss weight vector is given by

(19.7)

w We call the expression (

X X) X ⊤

−1

⊤

∗

=(

X X) X y . ⊤

−1

⊤

the pseudoinverse of the data matrix, and Equation

(19.7)  is called the normal equation.

Pseudoinverse

Normal equation

With univariate linear regression we didn’t have to worry about overfitting. But with multivariable linear regression in high-dimensional spaces it is possible that some dimension that is actually irrelevant appears by chance to be useful, resulting in overfitting.

Thus, it is common to use regularization on multivariable linear functions to avoid overfitting. Recall that with regularization we minimize the total cost of a hypothesis, counting both the empirical loss and the complexity of the hypothesis:

Cost(h) = EmpLoss(h) + λ Complexity(h). For linear functions the complexity can be specified as a function of the weights. We can consider a family of regularization functions:

Complexity(hw ) = Lq (w) =

∑w i

|

i|

q.

q = 1 we have L regularization9, which minimizes the sum of the absolute values; with q = 2, L regularization minimizes the sum of squares. Which regularization function should you pick? That depends on the specific problem, but L As with loss functions, with

1

2

1

regularization has an important advantage: it tends to produce a sparse model. That is, it often sets many weights to zero, effectively declaring the corresponding attributes to be

completely irrelevant—just as LEARN-DECISION-TREE does (although by a different mechanism). Hypotheses that discard attributes can be easier for a human to understand, and may be less likely to overfit.

L and L is used for both loss functions and regularization functions. They need not be used in pairs: you could use L loss with L regularization, or vice versa.

9 It is perhaps confusing that the notation 2

1

2

1

Sparse model

Figure 19.14  gives an intuitive explanation of why

L

1

regularization leads to weights of

Loss(w) + λComplexity(w) is equivalent to minimizing Loss(w) subject to the constraint that Complexity(w) ≤ c, for some constant c that is related to λ. Now, in Figure 19.14(a)  the diamond-shaped box represents the set of points w in two-dimensional weight space that have L complexity less than c; our solution will have to be somewhere inside this box. The concentric ovals zero, while

L

2

regularization does not. Note that minimizing

1

represent contours of the loss function, with the minimum loss at the center. We want to find the point in the box that is closest to the minimum; you can see from the diagram that, for an arbitrary position of the minimum and its contours, it will be common for the corner of the box to find its way closest to the minimum, just because the corners are pointy. And of course the corners are the points that have a value of zero in some dimension.

Figure 19.14

Why L1 regularization tends to produce a sparse model. Left: With L1 regularization (box), the minimal

achievable loss (concentric contours) often occurs on an axis, meaning a weight of zero. Right: With L2 regularization (circle), the minimal loss is likely to occur anywhere on the circle, giving no preference to zero weights.

In Figure 19.14(b) , we’ve done the same for the L2 complexity measure, which represents a circle rather than a diamond. Here you can see that, in general, there is no reason for the intersection to appear on one of the axes; thus L2 regularization does not tend to produce

zero weights. The result is that the number of examples required to find a good h is linear in the number of irrelevant features for L2 regularization, but only logarithmic with L1 regularization. Empirical evidence on many problems supports this analysis. Another way to look at it is that L1 regularization takes the dimensional axes seriously,

L2 treats them as arbitrary. The L2 function is spherical, which makes it rotationally invariant: Imagine a set of points in a plane, measured by their x and y coordinates. Now imagine rotating the axes by 45o . You’d get a different set of (x′ , y ′ ) values representing the same points. If you apply L2 regularization before and after rotating, you get exactly the same point as the answer (although the point would be described with the new (x′ , y ′ ) while

coordinates). That is appropriate when the choice of axes really is arbitrary—when it doesn’t matter whether your two dimensions are distances north and east; or distances northeast and southeast. With L1 regularization you’d get a different answer, because the L1 function is not rotationally invariant. That is appropriate when the axes are not interchangeable; it doesn’t make sense to rotate “number of bathrooms” 45o towards “lot size.”

19.6.4 Linear classifiers with a hard threshold Linear functions can be used to do classification as well as regression. For example, Figure 19.15(a)  shows data points of two classes: earthquakes (which are of interest to seismologists) and underground explosions (which are of interest to arms control experts). Each point is defined by two input values, x1 and x2 , that refer to body and surface wave magnitudes computed from the seismic signal. Given these training data, the task of classification is to learn a hypothesis h that will take new (x1 , x2 ) points and return either 0 for earthquakes or 1 for explosions.

Figure 19.15

(a) Plot of two seismic data parameters, body wave magnitude x1 and surface wave magnitude x2 , for earthquakes (open orange circles) and nuclear explosions (green circles) occurring between 1982 and 1990 in Asia and the Middle East (Kebeasy et al., 1998). Also shown is a decision boundary between the classes. (b) The same domain with more data points. The earthquakes and explosions are no longer linearly separable.

A decision boundary is a line (or a surface, in higher dimensions) that separates the two classes. In Figure 19.15(a) , the decision boundary is a straight line. A linear decision boundary is called a linear separator and data that admit such a separator are called linearly separable. The linear separator in this case is defined by

x2 = 1.7x1 − 4.9

Decision boundary

or

− 4.9 + 1.7x1 − x2 = 0.

Linear separator

Linear separability

The explosions, which we want to classify with value 1, are below and to the right of this line; they are points for which −4.9 + 1.7x1 − x2 > 0, while earthquakes have −4.9 + 1.7x1 − x2 < 0.. We can make the equation easier to deal with by changing it into the

vector dot product form—with x0 = 1 we have −4.9x0 + 1.7x1 − x2 = 0 ,

and we can define the vector of weights,

w = ⟨−4.9, 1.7, −1⟩, and write the classification hypothesis

hw (x) = 1 if w ⋅ x ≥ 0 and 0 otherwise. Alternatively, we can think of h as the result of passing the linear function

w ⋅ x through a

threshold function:

hw (x) = T hreshold(w ⋅ x) where T hreshold(z) = 1 if z ≥ 0 and 0 otherwise.

Threshold function

The threshold function is shown in Figure 19.17(a) .

x

Now that the hypothesis hw ( ) has a well-defined mathematical form, we can think about choosing the weights

w to minimize the loss. In Sections 19.6.1  and 19.6.3 , we did this

both in closed form (by setting the gradient to zero and solving for the weights) and by gradient descent in weight space. Here we cannot do either of those things because the gradient is zero almost everywhere in weight space except at those points where

w ⋅ x = 0,

and at those points the gradient is undefined.

There is, however, a simple weight update rule that converges to a solution—that is, to a linear separator that classifies the data perfectly—provided the data are linearly separable.

x

For a single example ( , y), we have (19.8)

wi

←

wi + α (y − hw (x))

×

xi

which is essentially identical to Equation (19.6) , the update rule for linear regression! This rule is called the perceptron learning rule, for reasons that will become clear in Chapter 21 . Because we are considering a 0/1 classification problem, however, the behavior is

x

somewhat different. Both the true value y and the hypothesis output hw ( ) are either 0 or 1, so there are three possibilities: If the output is correct (i.e., y =

x

hw (x)) then the weights are not changed.

If y is 1 but hw ( ) is 0, then wi is increased when the corresponding input xi is positive and decreased when xi is negative. This makes sense, because we want to make

x

w⋅x

bigger so that hw ( ) outputs a 1.

x

If y is 0 but hw ( ) is 1, then wi is decreased when the corresponding input xi is positive and increased when xi is negative. This makes sense, because we want to make

x

w⋅x

smaller so that hw ( ) outputs a 0.

Perceptron learning rule

Typically the learning rule is applied one example at a time, choosing examples at random (as in stochastic gradient descent). Figure 19.16(a)  shows a training curve for this learning

rule applied to the earthquake/explosion data shown in Figure 19.15(a) . A training curve measures the classifier performance on a fixed training set as the learning process proceeds one update at a time on that training set. The curve shows the update rule converging to a zero-error linear separator. The “convergence” process isn’t exactly pretty, but it always works. This particular run takes 657 steps to converge, for a data set with 63 examples, so each example is presented roughly 10 times on average. Typically, the variation across runs is large.

Figure 19.16

(a) Plot of total training-set accuracy vs. number of iterations through the training set for the perceptron

learning rule, given the earthquake/explosion data in Figure 19.15(a) . (b) The same plot for the noisy, nonseparable data in Figure 19.15(b) ; note the change in scale of the -axis. (c) The same plot as in (b), with a learning rate schedule

α(t) = 1000/(1000 + t).

x

Training curve

We have said that the perceptron learning rule converges to a perfect linear separator when the data points are linearly separable; but what if they are not? This situation is all too common in the real world. For example, Figure 19.15(b)  adds back in the data points left out by Kebeasy et al., (1998) when they plotted the data shown in Figure 19.15(a) . In Figure 19.16(b) , we show the perceptron learning rule failing to converge even after 10,000 steps: even though it hits the minimum-error solution (three errors) many times, the algorithm keeps changing the weights. In general, the perceptron rule may not converge to a stable solution for fixed learning rate

α, but if α decays as O(1/t) where t is the iteration

number, then the rule can be shown to converge to a minimum-error solution when examples are presented in a random sequence.10 It can also be shown that finding the

minimum-error solution is NP-hard, so one expects that many presentations of the examples will be required for convergence to be achieved. Figure 19.16(c)  shows the training process with a learning rate schedule α(t) = 1000/(1000 + t): convergence is not perfect after 100,000 iterations, but it is much better than the fixed-α case.

2 α(t) = ∞ and ∑∞ t=1 α (t) < ∞. The learning rate α(t) = O(1/t) satisfies these conditions. Often we use c/(c + t) for some fairly large constant c.

10 Technically, we require that

∑t

∞ =1

19.6.5 Linear classification with logistic regression We have seen that passing the output of a linear function through the threshold function creates a linear classifier; yet the hard nature of the threshold causes some problems: the

x

hypothesis hw ( ) is not differentiable and is in fact a discontinuous function of its inputs and its weights. This makes learning with the perceptron rule a very unpredictable adventure. Furthermore, the linear classifier always announces a completely confident prediction of 1 or 0, even for examples that are very close to the boundary; it would be better if it could classify some examples as a clear 0 or 1, and others as unclear borderline cases.

All of these issues can be resolved to a large extent by softening the threshold function— approximating the hard threshold with a continuous, differentiable function. In Chapter 13  (page 424), we saw two functions that look like soft thresholds: the integral of the standard normal distribution (used for the probit model) and the logistic function (used for the logit model). Although the two functions are very similar in shape, the logistic function

Logistic(z) =

1 1+

e−z

has more convenient mathematical properties. The function is shown in Figure 19.17(b) . With the logistic function replacing the threshold function, we now have

hw (x) = Logistic(w ⋅ x) = Figure 19.17

1

1+

e − w⋅x .

(a) The hard threshold function Threshold(z) with 0/1 output. Note that the function is nondifferentiable at z = 0. (b) The logistic function, Logistic(z) =

x

logistic regression hypothesis hw ( ) =

1

1+

e−z , also known as the sigmoid function. (c) Plot of a

Logistic(w ⋅ x) for the data shown in Figure 19.15(b) .

An example of such a hypothesis for the two-input earthquake/explosion problem is shown in Figure 19.17(c) . Notice that the output, being a number between 0 and 1, can be interpreted as a probability of belonging to the class labeled 1. The hypothesis forms a soft boundary in the input space and gives a probability of 0.5 for any input at the center of the boundary region, and approaches 0 or 1 as we move away from the boundary.

The process of fitting the weights of this model to minimize loss on a data set is called logistic regression. There is no easy closed-form solution to find the optimal value of

w

with this model, but the gradient descent computation is straightforward. Because our hypotheses no longer output just 0 or 1, we will use the L2 loss function; also, to keep the formulas readable, we’ll use g to stand for the logistic function, with g ′ its derivative.

Logistic regression

x

For a single example ( , y), the derivation of the gradient is the same as for linear regression (Equation (19.5) ) up to the point where the actual form of h is inserted. (For this derivation, we again need the chain rule.) We have

∂ ∂

wi

Loss(w) =

∂ ∂

wi

y hw (x))2

( −

y hw (x)) ×

= 2( −

∂ ∂

wi

y hw (x))

( −

y hw (x)) × g ′ (w ⋅ x) × w⋅x ∂ wi = −2(y − hw (x)) × g ′ (w ⋅ x) × xi .

= −2( −

∂

The derivative g ′ of the logistic function satisfies g ′ (z) = g(z)(1 − g(z)), so we have

g ′ (w ⋅ x) = g(w ⋅ x)(1 − g(w ⋅ x)) = hw (x)(1 − hw (x)) so the weight update for minimizing the loss takes a step in the direction of the difference

x

between input and prediction, (y − hw ( )), and the length of that step depends on the constant

α and g ′ :

(19.9)

wi

←

wi + α (y − hw (x)) × hw (x)(1 − hw (x)) × xi .

Repeating the experiments of Figure 19.16  with logistic regression instead of the linear threshold classifier, we obtain the results shown in Figure 19.18 . In (a), the linearly separable case, logistic regression is somewhat slower to converge, but behaves much more predictably. In (b) and (c), where the data are noisy and nonseparable, logistic regression converges far more quickly and reliably. These advantages tend to carry over into real-world applications, and logistic regression has become one of the most popular classification techniques for problems in medicine, marketing, survey analysis, credit scoring, public health, and other applications.

Figure 19.18

Repeat of the experiments in Figure 19.16  using logistic regression. The plot in (a) covers 5000 iterations rather than 700, while the plots in (b) and (c) use the same scale as before.

19.7 Nonparametric Models Linear regression uses the training data to estimate a fixed set of parameters our hypothesis

w. That defines

hw (x), and at that point we can throw away the training data, because they

are all summarized by

w. A learning model that summarizes data with a set of parameters of

fixed size (independent of the number of training examples) is called a parametric model.

Parametric model

When data sets are small, it makes sense to have a strong restriction on the allowable hypotheses, to avoid overfitting. But when there are millions or billions of examples to learn from, it seems like a better idea to let the data speak for themselves rather than forcing them to speak through a tiny vector of parameters. If the data say that the correct answer is a very wiggly function, we shouldn’t restrict ourselves to linear or slightly wiggly functions.

Nonparametric model

Instance-based learning

A nonparametric model is one that cannot be characterized by a bounded set of parameters. For example, the piecewise linear function from Figure 19.1  retains all the data points as part of the model. Learning methods that do this have also been described as instance-based learning or memory-based learning. The simplest instance-based learning method is table lookup: take all the training examples, put them in a lookup table, and then

x

when asked for h( ), see if

x is in the table; if it is, return the corresponding y.

Table lookup

The problem with this method is that it does not generalize well: when x is not in the table we have no information about a plausible value.

19.7.1 Nearest-neighbor models We can improve on table lookup with a slight variation: given a query xq , instead of finding

k examples that are nearest to xq . This is called knearest-neighbors lookup. We’ll use the notation NN (k, xq ) to denote the set of k

an example that is equal to xq , find the neighbors nearest to xq .

Nearest neighbors

NN (k, xq ) and take the most common output value—for example, if k = 3 and the output values are ⟨Y es, No, Y es⟩, then the classification will be Yes. To avoid ties on binary classification, k is usually chosen to be an odd number. To do classification, find the set of neighbors

To do regression, we can take the mean or median of the

k neighbors, or we can solve a

linear regression problem on the neighbors. The piecewise linear function from Figure 19.1  solves a (trivial) linear regression problem with the two data points to the right and left of xq . (When the

xi data points are equally spaced, these will be the two nearest

neighbors.)

k

In Figure 19.19 , we show the decision boundary of -nearest-neighbors classification for

k = 1 and 5 on the earthquake data set from Figure 19.15 . Nonparametric methods are still subject to underfitting and overfitting, just like parametric methods. In this case 1-nearestneighbors is overfitting; it reacts too much to the black outlier in the upper right and the white outlier at (5.4, 3.7). The 5-nearest-neighbors decision boundary is good; higher

k

would underfit. As usual, cross-validation can be used to select the best value of .

k

Figure 19.19

(a) A k-nearest-neighbors model showing the extent of the explosion class for the data in Figure 19.15 , with k = 1. Overfitting is apparent. (b) With k = 5, the overfitting problem goes away for this data set.

The very word “nearest” implies a distance metric. How do we measure the distance from a query point xq to an example point xj ? Typically, distances are measured with a Minkowski distance or

Lp norm, defined as

Lp ( x

j , xq ) =

(

∑x ∣

i

∣ ∣ ∣

∣

∣p

j i − xq i ∣ ,

,

∣

)

1/

p .

Minkowski distance

With

p

= 2

this is Euclidean distance and with p

= 1

it is Manhattan distance. With Boolean

attribute values, the number of attributes on which the two points differ is called the Hamming distance. Often Euclidean distance is used if the dimensions are measuring similar properties, such as the width, height and depth of parts, and Manhattan distance is used if they are dissimilar, such as age, weight, and gender of a patient. Note that if we use the raw numbers from each dimension then the total distance will be affected by a change in units in any dimension. That is, if we change the height dimension from meters to miles while keeping the width and depth dimensions the same, we’ll get different nearest neighbors. And how do we compare a difference in age to a difference in weight? A

common approach is to applynormalization to the measurements in each dimension. We

μi and standard deviation σi of the values in each dimension, and rescale them so that xj i becomes (xj i − μi )/σi . A more complex metric known as the can compute the mean

,

,

Mahalanobis distance takes into account the covariance between dimensions.

Hamming distance

Normalization

Mahalanobis distances

In low-dimensional spaces with plenty of data, nearest neighbors works very well: we are likely to have enough nearby data points to get a good answer. But as the number of dimensions rises we encounter a problem: the nearest neighbors in high-dimensional spaces

k

N points uniformly distributed throughout the interior of an n-dimensional unit hypercube. We’ll define the kneighborhood of a point as the smallest hypercube that contains the k nearest neighbors. are usually not very near! Consider -nearest-neighbors on a data set of

Let ℓ be the average side length of a neighborhood. Then the volume of the neighborhood

k points) is ℓn and the volume of the full cube (which contains N points) is 1. So, on average, ℓn = k/N . Taking nth roots of both sides we get ℓ = (k/N ) n . (which contains

1/

To be concrete, let

k = 10 and N = 1, 000, 000. In two dimensions (n = 2; a unit square), the

average neighborhood has ℓ = 0.003, a small fraction of the unit square, and in 3 dimensions ℓ is just 2% of the edge length of the unit cube. But by the time we get to 17 dimensions, ℓ is

half the edge length of the unit hypercube, and in 200 dimensions it is 94%. This problem has been called the curse of dimensionality.

Curse of dimensionality

Another way to look at it: consider the points that fall within a thin shell making up the outer 1% of the unit hypercube. These are outliers; in general it will be hard to find a good value for them because we will be extrapolating rather than interpolating. In one dimension, these outliers are only 2% of the points on the unit line (those points where

x < .01 or

x > .99), but in 200 dimensions, over 98% of the points fall within this thin shell—almost all

the points are outliers. You can see an example of a poor nearest-neighbors fit on outliers if you look ahead to Figure 19.20(b) .

The

NN (k, xq ) function is conceptually trivial: given a set of N examples and a query xq ,

iterate through the examples, measure the distance to xq from each one, and keep the best

k. If we are satisfied with an implementation that takes O(N ) execution time, then that is the

end of the story. But instance-based methods are designed for large data sets, so we would like something faster. The next two subsections show how trees and hash tables can be used to speed up the computation.

k

19.7.2 Finding nearest neighbors with -d trees A balanced binary tree over data with an arbitrary number of dimensions is called a k-d

k

k

tree, for -dimensional tree. The construction of a -d tree is similar to the construction of a balanced binary tree. We start with a set of examples and at the root node we split them

i xi ≤ m, where m is the median of the examples along the ith dimension; thus half the examples will be in the left branch of the tree and half along the th dimension by testing whether

in the right. We then recursively make a tree for the left and right sets of examples, stopping when there are fewer than two examples left. To choose a dimension to split on at each

i

n

i

node of the tree, one can simply select dimension  mod  at level of the tree. (Note that we may need to split on any given dimension several times as we proceed down the tree.) Another strategy is to split on the dimension that has the widest spread of values.

K-d tree

k

Exact lookup from a -d tree is just like lookup from a binary tree (with the slight complication that you need to pay attention to which dimension you are testing at each node). But nearest-neighbor lookup is more complicated. As we go down the branches, splitting the examples in half, in some cases we can ignore half of the examples. But not always. Sometimes the point we are querying for falls very close to the dividing boundary. The query point itself might be on the left hand side of the boundary, but one or more of the

k nearest neighbors might actually be on the right-hand side.

We have to test for this possibility by computing the distance of the query point to the dividing boundary, and then searching both sides if we can’t find

k

k examples on the left that

are closer than this distance. Because of this problem, -d trees are appropriate only when

k

there are many more examples than dimensions, preferably at least 2n examples. Thus, -d trees are a good choice for up to about 10 dimensions when there are thousands of examples or up to 20 dimensions with millions of examples.

19.7.3 Locality-sensitive hashing Hash tables have the potential to provide even faster lookup than binary trees. But how can we find nearest neighbors using a hash table, when hash codes rely on an exact match? Hash codes randomly distribute values among the bins, but we want to have near points grouped together in the same bin; we want a locality-sensitive hash (LSH).

Locality-sensitive hash

Approximate near-neighbors

We can’t use hashes to solve

NN (k, xq ) exactly, but with a clever use of randomized

algorithms, we can find an approximate solution. First we define the approximate nearneighbors problem: given a data set of example points and a query point xq , find, with high probability, an example point (or points) that is near xq . To be more precise, we require that if there is a point xj that is within a radius

r of xq , then with high probability the algorithm

c r of xq . If there is no point within radius r then the algorithm is allowed to report failure. The values of c and “high probability” are will find a point xj′ that is within distance hyperparameters of the algorithm.

g

To solve approximate near neighbors, we will need a hash function (x) that has the property that, for any two points xj and xj′ , the probability that they have the same hash code is small if their distance is more than

c r, and is high if their distance is less than r. For

simplicity we will treat each point as a bit string. (Any features that are not Boolean can be encoded into a set of Boolean features.)

n

We rely on the intuition that if two points are close together in an -dimensional space, then they will necessarily be close when projected down onto a one-dimensional space (a line). In fact, we can discretize the line into bins—hash buckets—so that, with high probability, near points project down to the same bin. Points that are far away from each other will tend to project down into different bins, but there will always be a few projections that coincidentally project far-apart points into the same bin. Thus, the bin for point xq contains many (but not all) points that are near xq , and it might contain some points that are far away.

The trick of LSH is to create multiple random projections and combine them. A random projection is just a random subset of the bit-string representation. We choose ℓ different random projections and create ℓ hash tables,

g (x), … , g (x). We then enter all the 1

ℓ

examples into each hash table. Then when given a query point xq , we fetch the set of points

g points, C . Then we compute the actual distance to xq for each of the points in C and return the k closest points. With high probability, each of the points that are near to xq will show in bin i (xq ) of each hash table, and union these ℓ sets together into a set of candidate

up in at least one of the bins, and although some far-away points will show up as well, we can ignore those. With large real-world problems, such as finding the near neighbors in a data set of 13 million Web images using 512 dimensions (Torralba et al., 2008), localitysensitive hashing needs to examine only a few thousand images out of 13 million to find

k

nearest neighbors—a thousand-fold speedup over exhaustive or -d tree approaches.

19.7.4 Nonparametric regression Now we’ll look at nonparametric approaches to regression rather than classification. Figure 19.20  shows an example of some different models. In (a), we have perhaps the simplest

method of all, known informally as “connect-the-dots,” and superciliously as “piecewiselinear nonparametric regression.” This model creates a function h(x) that, when given a query xq , considers the training examples immediately to the left and right of xq , and interpolates between them. When noise is low, this trivial method is actually not too bad, which is why it is a standard feature of charting software in spreadsheets. But when the data are noisy, the resulting function is spiky and does not generalize well.

Figure 19.20

Nonparametric regression models: (a) connect the dots, (b) 3-nearest neighbors average, (c) 3-nearestneighbors linear regression, (d) locally weighted regression with a quadratic kernel of width 10.

k-nearest-neighbors regression improves on connect-the-dots. Instead of using just the two examples to the left and right of a query point xq , we use the k nearest neighbors. (Here we are using k = 3.). A larger value of k tends to smooth out the magnitude of the spikes,

although the resulting function has discontinuities. Figure 19.20  shows two versions of knearest-neighbors regression. In (b), we have the k-nearest-neighbors average: h(x) is the

mean value of the

k points, ∑ yj /k. Notice that at the outlying points, near x = 0 and

x = 14, the estimates are poor because all the evidence comes from one side (the interior), and ignores the trend. In (c), we have k-nearest-neighbor linear regression, which finds the best line through the k examples. This does a better job of capturing trends at the outliers, but is still discontinuous. In both (b) and (c), we’re left with the question of how to choose a

k

good value for . The answer, as usual, is cross-validation.

Nearest-neighbors regression

Locally weighted regression (Figure 19.20(d) ) gives us the advantages of nearest

hx

neighbors, without the discontinuities. To avoid discontinuities in ( ), we need to avoid

hx

discontinuities in the set of examples we use to estimate ( ). The idea of locally weighted regression is that at each query point

xq , the examples that are close to xq are weighted

heavily, and the examples that are farther away are weighted less heavily, and the farthest not at all. The decrease in weight over distance is typically gradual, not sudden.

Locally weighted regression

We decide how much to weight each example with a function known as a kernel, whose input is a distance between the query point and the example. A kernel function decreasing function of distance with a maximum at 0, so that

K is a

K(Distance(xj , xq )) gives

higher weight to examples xj that are closer to the query point xq for which we are trying to predict the function value. The integral of the kernel value over the entire input space for x must be finite—and if we choose to make the integral 1, certain calculations are easier.

Kernel

Figure 19.20(d)  was generated with a quadratic kernel, kernel width

K(d) = max(0, 1 − (2∣d∣/w) ), with 2

w = 10. Other shapes, such as Gaussians, are also used. Typically, the width

matters more than the exact shape: this is a hyperparameter of the model that is best chosen by cross-validation. If the kernels are too wide we’ll get underfitting and if they are too narrow we’ll get overfitting. In Figure 19.20(d) , a kernel width of 10 gives a smooth curve that looks just about right.

Doing locally weighted regression with kernels is now straightforward. For a given query point

xq we solve the following weighted regression problem: w

∗

= argmin

w

∑j K Distance xq xj (

(

,

)) (

yj − w ⋅ xj )

2

,

where Distance is any of the distance metrics discussed for nearest neighbors. Then the

h x q ) = w ⋅ xq .

answer is (

∗

Note that we need to solve a new regression problem for every query point—that’s what it means to be local. (In ordinary linear regression, we solved the regression problem once, globally, and then used the same

hw for any query point.) Mitigating against this extra work

is the fact that each regression problem will be easier to solve, because it involves only the examples with nonzero weight—the examples that are within the kernel width of the query. When kernel widths are small, this may be just a few points.

Most nonparametric models have the advantage that it is easy to do leave-one-out cross-

k

validation without having to recompute everything. With a -nearest-neighbors model, for

xy k nearest neighbors once, compute the per-example loss L(y, h(x)) from them, and record that as the leave-one-out result for every example that is not one of the neighbors. Then we retrieve the k + 1 nearest neighbors and record distinct results for leaving out each of the k neighbors. With N examples the whole process is O(k), not O(kN ). instance, when given a test example ( , ) we retrieve the

19.7.5 Support vector machines In the early 2000s, the support vector machine (SVM) model class was the most popular approach for “off-the-shelf” supervised learning, for when you don’t have any specialized

prior knowledge about a domain. That position has now been taken over by deep learning networks and random forests, but SVMs retain three attractive properties:

1. SVMs construct a maximum margin separator—a decision boundary with the largest possible distance to example points. This helps them generalize well. 2. SVMs create a linear separating hyperplane, but they have the ability to embed the data into a higher-dimensional space, using the so-called kernel trick. Often, data that are not linearly separable in the original input space are easily separable in the higher-dimensional space. 3. SVMs are nonparametric—the separating hyperplane is defined by a set of example points, not by a collection of parameter values. But while nearest-neighbor models need to retain all the examples, an SVM model keeps only the examples that are closest to the separating plane—usually only a small constant times the number of dimensions. Thus SVMs combine the advantages of nonparametric and parametric models: they have the flexibility to represent complex functions, but they are resistant to overfitting.

Support vector machine (SVM)

We see in Figure 19.21(a)  a binary classification problem with three candidate decision boundaries, each a linear separator. Each of them is consistent with all the examples, so from the point of view of 0/1 loss, each would be equally good. Logistic regression would find some separating line; the exact location of the line depends on all the example points. The key insight of SVMs is that some examples are more important than others, and that paying attention to them can lead to better generalization.

Figure 19.21

Support vector machine classification: (a) Two classes of points (orange open and green filled circles) and three candidate linear separators. (b) The maximum margin separator (heavy line), is at the midpoint of the margin (area between dashed lines). The support vectors (points with large black circles) are the examples closest to the separator; here there are three.

Consider the lowest of the three separating lines in (a). It comes very close to five of the black examples. Although it classifies all the examples correctly, and thus minimizes loss, it should make you nervous that so many examples are close to the line; it may be that other black examples will turn out to fall on the wrong side of the line.

SVMs address this issue: Instead of minimizing expected empirical loss on the training data, SVMs attempt to minimize expected generalization loss. We don’t know where the as-yetunseen points may fall, but under the probabilistic assumption that they are drawn from the same distribution as the previously seen examples, there are some arguments from computational learning theory (Section 19.5 ) suggesting that we minimize generalization loss by choosing the separator that is farthest away from the examples we have seen so far. We call this separator, shown in Figure 19.21(b)  the maximum margin separator. The margin is the width of the area bounded by dashed lines in the figure—twice the distance from the separator to the nearest example point.

Maximum margin separator

Margin

Now, how do we find this separator? Before showing the equations, some notation: Traditionally SVMs use the convention that class labels are +1 and -1, instead of the +1 and 0 we have been using so far. Also, whereas we previously put the intercept into the weight vector

w (and a corresponding dummy 1 value into x

j,0 ),

SVMs do not do that; they keep

the intercept as a separate parameter, b.

With that in mind, the separator is defined as the set of points { search the space of

x : w ⋅ x + b = 0}. We could

w and b with gradient descent to find the parameters that maximize the

margin while correctly classifying all the examples.

However, it turns out there is another approach to solving this problem. We won’t show the details, but will just say that there is an alternative representation called the dual representation, in which the optimal solution is found by solving

(19.10) argmax α

∑

αj −

j

1 2

∑

αj αk yj yk (

j, k

x ⋅x ) j

k

Quadratic programming

subject to the constraints αj ≥ 0 and

∑ αy j

j j

= 0. This is a quadratic programming

optimization problem, for which there are good software packages. Once we have found the vector α we can get back to

w with the equation w = ∑

j

αj yj

x , or we can stay in the dual j

representation. There are three important properties of Equation (19.10) . First, the

expression is convex; it has a single global maximum that can be found efficiently. Second, the data enter the expression only in the form of dot products of pairs of points. This second property is also true of the equation for the separator itself; once the optimal αj have been calculated, the equation is11

11 The function sign(x) returns +1 for a positive x, −1 for a negative x.

(19.11)

⎛ ⎝

h(x) = sign ⎜

∑ j

⎞ ⎠

αj yj (x ⋅ xj ) − b⎟ .

A final important property is that the weights αj associated with each data point are zero except for the support vectors—the points closest to the separator. (They are called “support” vectors because they “hold up” the separating plane.) Because there are usually many fewer support vectors than examples, SVMs gain some of the advantages of parametric models.

Support vector

What if the examples are not linearly separable? Figure 19.22(a)  shows an input space defined by attributes x = (x1 , x2 ), with positive examples (y = +1) inside a circular region and negative examples (y = −1) outside. Clearly, there is no linear separator for this problem. Now, suppose we re-express the input data—that is, we map each input vector x to a new vector of feature values, F (x). In particular, let us use the three features (19.12)

f1 = x21 , Figure 19.22

f2 = x22 ,

f 3 = √ 2 x1 x 2 .

(a) A two-dimensional training set with positive examples as green filled circles and negative examples as orange open circles. The true decision boundary, x21 + x22 ≤ 1, is also shown. (b) The same data after

mapping into a three-dimensional input space (x21 , x22 , √2x1 x2 ). The circular decision boundary in (a) becomes a linear decision boundary in three dimensions. Figure 19.21(b)  gives a closeup of the separator in (b).

We will see shortly where these came from, but for now, just look at what happens. Figure 19.22(b)  shows the data in the new, three-dimensional space defined by the three features; the data are linearly separable in this space! This phenomenon is actually fairly general: if data are mapped into a space of sufficiently high dimension, then they will almost always be linearly separable—if you look at a set of points from enough directions, you’ll find a way to make them line up. Here, we used only three dimensions;12 Exercise 19.SVME asks you to show that four dimensions suffice for linearly separating a circle anywhere in the plane (not just at the origin), and five dimensions suffice to linearly separate any ellipse. In general (with some special cases excepted) if we have N data points then they will always be separable in spaces of N − 1 dimensions or more (Exercise 19.EMBE). 12 The reader may notice that we could have used just f1 and f2 , but the 3D mapping illustrates the idea better.

Now, we would not usually expect to find a linear separator in the input space x, but we can find linear separators in the high-dimensional feature space F (x) simply by replacing xj ⋅ xk in Equation (19.10)  with F (xj ) ⋅ F (xk ). This by itself is not remarkable—replacing x by

F (x) in any learning algorithm has the required effect—but the dot product has some special properties. It turns out that F (xj ) ⋅ F (xk ) can often be computed without first computing F

for each point. In our three-dimensional feature space defined by Equation (19.12) , a little bit of algebra shows that

F ( xj ) ⋅ F ( xk ) = ( xj ⋅ xk ) 2 .

x x

(That’s why the √2 is in f3 .) The expression ( j ⋅ k )2 is called a kernel function,13 and is

x x

usually written as K ( j , k ). The kernel function can be applied to pairs of input data to evaluate dot products in some corresponding feature space. So, we can find linear

x

x x

separators in the higher-dimensional feature space F ( ) simply by replacing j ⋅ k in

x x

Equation (19.10)  with a kernel function K ( j , k ). Thus, we can learn in the higher-

dimensional space, but we compute only kernel functions rather than the full list of features for each data point.

13 This usage of “kernel function” is slightly different from the kernels in locally weighted regression. Some SVM kernels are distance metrics, but not all are.

Kernel function

x x

x x

The next step is to see that there’s nothing special about the kernel K ( j , k ) = ( j ⋅ k )2 . It corresponds to a particular higher-dimensional feature space, but other kernel functions correspond to other feature spaces. A venerable result in mathematics, Mercer’s theorem (1909), tells us that any “reasonable”14 kernel function corresponds to some feature space. These feature spaces can be very large, even for innocuous-looking kernels. For example,

x x

x x

the polynomial kernel, K ( j , k ) = (1 + j ⋅ k )d , corresponds to a feature space whose ∣ ∣2 dimension is exponential in d. A common kernel is the Gaussian: K ( j , k ) = e−γ∣xj −xk ∣ . 14 Here, “reasonable” means that the matrix

Mercer’s theorem

Kjk = K(xj, xk) is positive definite.

x x

Polynomial kernel

19.7.6 The kernel trick This then is the clever kernel trick: Plugging these kernels into Equation (19.10) , optimal linear separators can be found efficiently in feature spaces with billions of (or even infinitely many) dimensions. The resulting linear separators, when mapped back to the original input space, can correspond to arbitrarily wiggly, nonlinear decision boundaries between the positive and negative examples.

Kernel trick

In the case of inherently noisy data, we may not want a linear separator in some highdimensional space. Rather, we’d like a decision surface in a lower-dimensional space that does not cleanly separate the classes, but reflects the reality of the noisy data. That is possible with the soft margin classifier, which allows examples to fall on the wrong side of the decision boundary, but assigns them a penalty proportional to the distance required to move them back to the correct side.

Soft margin

The kernel method can be applied not only with learning algorithms that find optimal linear separators, but also with any other algorithm that can be reformulated to work only with dot products of pairs of data points, as in Equations (19.10)  and (19.11) . Once this is done, the dot product is replaced by a kernel function and we have a kernelized version of the algorithm.

Kernelization

19.8 Ensemble Learning So far we have looked at learning methods in which a single hypothesis is used to make predictions. The idea of ensemble learning is to select a collection, or ensemble, of hypotheses, h1 , h2 , … , hn , and combine their predictions by averaging, voting, or by another level of machine learning. We call the individual hypotheses base models and their combination an ensemble model.

Ensemble learning

Base model

Ensemble model

There are two reasons to do this. The first is to reduce bias. The hypothesis space of a base model may be too restrictive, imposing a strong bias (such as the bias of a linear decision boundary in logistic regression). An ensemble can be more expressive, and thus have less bias, than the base models. Figure 19.23  shows that an ensemble of three linear classifiers can represent a triangular region that could not be represented by a single linear classifier. An ensemble of n linear classifiers allows more functions to be realizable, at a cost of only n times more computation; this is often better than allowing a completely general hypothesis space that might require exponentially more computation.

Figure 19.23

Illustration of the increased expressive power obtained by ensemble learning. We take three linear threshold hypotheses, each of which classifies positively on the unshaded side, and classify as positive any example classified positively by all three. The resulting triangular region is a hypothesis not expressible in the original hypothesis space.

The second reason is to reduce variance. Consider an ensemble of

K = 5 binary classifiers

that we combine using majority voting. For the ensemble to misclassify a new example, at least three of the five classifiers have to misclassify it. The hope is that this is less likely than a single misclassification by a single classifier. To quantify that, suppose you have trained a single classifier that is correct in 80% of cases. Now create an ensemble of 5 classifiers, each trained on a different subset of the data so that they are independent. Let’s assume this leads to some reduction in quality, and each individual classifier is correct in only 75% of cases. But together, the majority vote of the ensemble will be correct in 89% of cases (and 99% with 17 classifiers), assuming true independence.

In practice the independence assumption is unreasonable—individual classifiers share some of the same data and assumptions, and thus are not completely independent, and will share some of the same errors. But if the component classifiers are at least somewhat uncorrelated then ensemble learning will make fewer misclassifications. We will now consider four ways of creating ensembles: bagging, random forests, stacking, and boosting.

19.8.1 Bagging

K distinct training sets by sampling with replacement from the original training set. That is, we randomly pick N examples from the training set, but each

In bagging,15 we generate

of those picks might be an example we picked before. We then run our machine learning

algorithm on the

N examples to get a hypothesis. We repeat this process K times, getting K

different hypotheses. Then, when asked to predict the value of a new input, we aggregate the predictions from all

K hypotheses. For classification problems, that means taking the

plurality vote (the majority vote for binary classification). For regression problems, the final output is the average:

15 Note on terminology: In statistics, a sample with replacement is called a bootstrap, and “bagging” is short for “bootstrap aggregating.”

h(x) = K1

∑iK hi x

( )

=1

Bagging

Bagging tends to reduce variance and is a standard approach when there is limited data or when the base model is seen to be overfitting. Bagging can be applied to any class of model, but is most commonly used with decision trees. It is appropriate because decision trees are unstable: a slightly different set of examples can lead to a wildly different tree. Bagging smoothes out this variance. If you have access to multiple computers then bagging is efficient, because the hypotheses can be computed in parallel.

19.8.2 Random forests Unfortunately, bagging decision trees often ends up giving us

K trees that are highly

correlated. If there is one attribute with a very high information gain, it is likely to be the root of most of the trees. The random forest model is a form of decision tree bagging in which we take extra steps to make the ensemble of

K trees more diverse, to reduce

variance. Random forests can be used for classification or regression.

Random forests

The key idea is to randomly vary the attribute choices (rather than the training examples). At each split point in constructing the tree, we select a random sampling of attributes, and then compute which of those gives the highest information gain. If there are

n

n attributes, a

common default choice is that each split randomly picks √ attributes to consider for classification problems, or

n

/3

for regression problems.

A further improvement is to use randomness in selecting the split point value: for each selected attribute, we randomly sample several candidate values from a uniform distribution over the attribute’s range. Then we select the value that has the highest information gain. That makes it more likely that every tree in the forest will be different. Trees constructed in this fashion are called extremely randomized trees (ExtraTrees).

Extremely randomized trees (ExtraTrees)

Random forests are efficient to create. You might think that it would take create an ensemble of

K times longer to

K trees, but it is not that bad, for three reasons: (a) each split point

runs faster because we are considering fewer attributes, (b) we can skip the pruning step for each individual tree, because the ensemble as a whole decreases overfitting, and (c) if we happen to have

K computers available, we can build all the trees in parallel. For example,

Adele Cutler reports that for a 100-attribute problem, if we have just three CPUs we can grow a forest of

K

= 100

trees in about the same time as it takes to create a single decision

tree on a single CPU.

All the hyperparameters of random forests can be trained by cross-validation: the number of trees

K

,

the number of examples used by each tree

N (often expressed as a percentage of

the complete data set), the number of attributes used at each split point (often expressed as

n

a function of the total number of attributes, such as √ ), and the number of random split points tried if we are using ExtraTrees. In place of the regular cross-validation strategy, we could measure the out-of-bag error: the mean error on each example, using only the trees whose example set didn’t include that particular example.

Out-of-bag error

We have been warned that more complex models can be prone to overfitting, and observed that to be true for decision trees, where we found that pruning was an answer to prevent overfitting. Random forests are complex, unpruned models. Yet they are resistant to overfitting. As you increase capacity by adding more trees to the forest they tend to improve on validation-set error rate. The curve typically looks like Figure 19.9(b) , not (a) .

Breiman (2001) gives a mathematical proof that (in almost all cases) as you add more trees to the forest, the error converges; it does not grow. One way to think of it is that the random selection of attributes yields a variety of trees, thus reducing variance, but because we don’t need to prune the trees, they can cover the full input space at higher resolution. Some number of trees can cover unique cases that appear only a few times in the data, and their votes can prove decisive, but they can be outvoted when they do not apply. That said, random forests are not totally immune to overfitting. Although the error can’t increase in the limit, that does not mean that the error will go to zero.

Random forests have been very successful across a wide variety of application problems. In Kaggle data science competitions they were the most popular approach of winning teams from 2011 through 2014, and remain a common approach to this day (although deep learning and gradient boosting have become even more common among recent winners). The randomForest package in R has been a particular favorite. In finance, random forests have been used for credit card default prediction, household income prediction, and option pricing. Mechanical applications include machine fault diagnosis and remote sensing. Bioinformatic and medical applications include diabetic retinopathy, microarray gene expression, mass spectrum protein expression analysis, biomarker discovery, and protein– protein interaction prediction.

19.8.3 Stacking Whereas bagging combines multiple base models of the same model class trained on different data, the technique of stacked generalization (or stacking for short) combines multiple base models from different model classes trained on the same data. For example, suppose we are given the restaurant data set, the first row of which is shown here:

x1 =Yes, No, No, Yes, Some, $$$, No, Yes, French, 0

− 10; y1 = Yes

Stacked generalization

We separate the data into training, validation, and test sets and use the training set to train, say, three separate base models—an SVM model, a logistic regression model, and a decision tree model.

In the next step we take the validation data set and augment each row with the predictions made from the three base models, giving us rows that look like this (where the predictions are shown in bold):

x2

= Yes, No, No, Yes, Full, $, No, No, Thai, 30 − 60,  Yes,  No,  No; y2 = No

We use this validation set to train a new ensemble model, let’s say a logistic regression model (but it need not be one of the base model classes). The ensemble model can use the predictions and the original data as it sees fit. It might learn a weighted average of the base models, for example that the predictions should be weighted in a ratio of 50%:30%:20%. Or it might learn nonlinear interactions between the data and the predictions, perhaps trusting the SVM model more when the wait time is long, for example. We used the same training data to train each of the base models, and then used the held-out validation data (plus predictions) to train the ensemble model. It is also possible to use cross-validation if desired.

The method is called “stacking” because it can be thought of as a layer of base models with an ensemble model stacked above it, operating on the output of the base models. In fact, it is possible to stack multiple layers, each one operating on the output of the previous layer. Stacking reduces bias, and usually leads to performance that is better than any of the individual base models. Stacking is frequently used by winning teams in data science competitions (such as Kaggle and the KDD Cup), because individuals can work independently, each refining their own base model, and then come together to build the final stacked ensemble model.

19.8.4 Boosting The most popular ensemble method is called boosting. To understand how it works, we need first to introduce the idea of a weighted training set, in which each example has an associated weight

wj

≥ 0

that describes how much the example should count during

training. For example, if one example had a weight of 3 and the other examples all had a weight of 1, that would be equivalent to having 3 copies of the one example in the training set.

Boosting

Weighted training set

Boosting starts with equal weights generates the first hypothesis,

h

1.

wj

= 1

for all the examples. From this training set, it

In general,

h

1

will classify some of the training examples

correctly and some incorrectly. We would like the next hypothesis to do better on the misclassified examples, so we increase their weights while decreasing the weights of the correctly classified examples.

From this new weighted training set, we generate hypothesis this way until we have generated

h

2.

The process continues in

K hypotheses, where K is an input to the boosting

algorithm. Examples that are difficult to classify will get increasingly larger weights until the algorithm is forced to create a hypothesis that classifies them correctly. Note that this is a greedy algorithm in the sense that it does not backtrack; once it has chosen a hypothesis

hi

it will never undo that choice; rather it will add new hypotheses. It is also a sequential algorithm, so we can’t compute all the hypotheses in parallel as we could with bagging.

The final ensemble lets each hypothesis vote, as in bagging, except that each hypothesis gets a weighted number of votes—the hypotheses that did better on their respective weighted training sets are given more voting weight. For regression or binary classification we have

h(x) =

∑iK zihi x

( )

=1

where

zi is the weight of the ith hypothesis. (This weighting of hypotheses is distinct from

the weighting of examples.) Figure 19.24  shows how the algorithm works conceptually. There are many variants of the basic boosting idea, with different ways of adjusting the example weights and combining the hypotheses. The variants all share the general idea that difficult examples get more weight as we move from one hypothesis to the next. Like the Bayesian learning methods we will see in Chapter 20 , they also give more weight to more accurate hypotheses.

Figure 19.24

How the boosting algorithm works. Each shaded rectangle corresponds to an example; the height of the rectangle corresponds to the weight. The checks and crosses indicate whether the example was classified correctly by the current hypothesis. The size of the decision tree indicates the weight of that hypothesis in the final ensemble.

One specific algorithm, called ADABOOST, is shown in Figure 19.25 . It is usually applied with decision trees as the component hypotheses; often the trees are limited in size. ADABOOST has a very important property: if the input learning algorithm

L is a weak

L always returns a hypothesis with accuracy on the training set that is slightly better than random guessing (that is, 50% +ϵ for Boolean learning algorithm—which means that

classification)—then ADABOOST will return a hypothesis that classifies the training data perfectly for large enough

K

.

Thus, the algorithm boosts the accuracy of the original learning

algorithm on the training data.

Figure 19.25

The ADABOOST variant of the boosting method for ensemble learning. The algorithm generates hypotheses by successively reweighting the training examples. The function WEIGHTED-MAJORITY generates a hypothesis that returns the output value with the highest vote from the hypotheses in h, with votes weighted by z. For regression problems, or for binary classification with two classes

∑k h k z k

−1

and 1, this is

[ ] [ ].

Weak learning

In other words, boosting can overcome any amount of bias in the base model, as long as the base model is

ϵ better than random guessing. (In our pseudocode we stop generating

hypotheses if we get one that is worse than random.) This result holds no matter how inexpressive the original hypothesis space and no matter how complex the function being learned. The exact formulas for weights in Figure 19.25  (with

error/(1 − error, etc.) are

chosen to make the proof of this property easy (see (Freund and Schapire, 1996)). Of course, this property does not guarantee accuracy on previously unseen examples.

Let us see how well boosting does on the restaurant data. We will choose as our original hypothesis space the class of decision stumps, which are decision trees with just one test, at the root. The lower curve in Figure 19.26(a)  shows that unboosted decision stumps are not very effective for this data set, reaching a prediction performance of only 81% on 100 training examples. When boosting is applied (with

K = 5), the performance is better,

reaching 93% after 100 examples.

Figure 19.26

K

(a) Graph showing the performance of boosted decision stumps with = 5 versus unboosted decision stumps on the restaurant data. (b) The proportion correct on the training set and the test set as a function

K

of , the number of hypotheses in the ensemble. Notice that the test set accuracy improves slightly even after the training accuracy reaches 1, i.e., after the ensemble fits the data exactly.

Decision stump

K increases. Figure 19.26(b)  shows the training set performance (on 100 examples) as a function of K . Notice that the error reaches An interesting thing happens as the ensemble size

zero when

K is 20; that is, a weighted-majority combination of 20 decision stumps suffices

to fit the 100 examples exactly—this is the interpolation pont. As more stumps are added to the ensemble, the error remains at zero. The graph also shows that the test set performance continues to increase long after the training set error has reached zero. At

K

= 20,

the test

performance is 0.95 (or 0.05 error), and the performance increases to 0.98 as late as

K

= 137,

before gradually dropping to 0.95.

This finding, which is quite robust across data sets and hypothesis spaces, came as quite a surprise when it was first noticed. Ockham’s razor tells us not to make hypotheses more complex than necessary, but the graph tells us that the predictions improve as the ensemble hypothesis gets more complex! Various explanations have been proposed for this. One view is that boosting approximates Bayesian learning (see Chapter 20 ), which can be shown to be an optimal learning algorithm, and the approximation improves as more hypotheses are added. Another possible explanation is that the addition of further hypotheses enables the ensemble to be more confident in its distinction between positive and negative examples, which helps it when it comes to classifying new examples.

19.8.5 Gradient boosting For regression and classification of factored tabular data, gradient boosting, sometimes called gradient boosting machines (GBM) or gradient boosted regression trees (GBRT), has become a very popular method. As the name implies, gradient boosting is a form of boosting using gradient descent. Recall that in ADABOOST, we start with one hypothesis h1 , and boost it with a sequence of hypotheses that pay special attention to the examples that the previous ones got wrong. In gradient boosting we also add new boosting hypotheses, which pay attention not to specific examples, but to the gradient between the right answers and the answers given by the previous hypotheses.

Gradient boosting

As in the other algorithms that used gradient descent, we start with a differentiable loss function; we might use squared error for regression, or logarithmic loss for classification. As in ADABOOST, we then build a decision tree. In Section 19.6.2 , we used gradient descent to

minimize the parameters of a model—we calculate the loss, and update the parameters in the direction of less loss. With gradient boosting, we are not updating parameters of the existing model, we are updating the parameters of the next tree—but we must do that in a way that reduces the loss by moving in the right direction along the gradient. As in the models we saw in Section 19.4.3 , regularization can help prevent overfitting. That can come in the form of limiting the number of trees or their size (in terms of their

,

depth or number of nodes). It can come from the learning rate, α which says how far to move along the direction of the gradient; values in the range 0.1 to 0.3 are common, and the smaller the learning rate, the more trees we will need in the ensemble.

Gradient boosting is implemented in the popular XGBOOST (eXtreme Gradient Boosting) package, which is routinely used for both large-scale applications in industry (for problems with billions of examples), and by the winners of data science competitions (in 2015, it was used by every team in the top 10 of the KDDCup). XGBOOST does gradient boosting with pruning and regularization, and takes care to be efficient, carefully organizing memory to avoid cache misses, and allowing for parallel computation on multiple machines.

19.8.6 Online learning So far, everything we have done in this chapter has relied on the assumption that the data are i.i.d. (independent and identically distributed). On the one hand, that is a sensible assumption: if the future bears no resemblance to the past, then how can we predict anything? On the other hand, it is too strong an assumption: we know that there are correlations between the past and the future, and in complex scenarios it is unlikely that we will capture all the data that would make the future independent of the past given the data.

In this section we examine what to do when the data are not i.i.d.—when they can change over time. In this case, it matters when we make a prediction, so we will adopt the perspective called online learning: an agent receives an input xj from nature, predicts the

,

corresponding yj and then is told the correct answer. Then the process repeats with xj+1

,

and so on. One might think this task is hopeless—if nature is adversarial, all the predictions may be wrong. It turns out that there are some guarantees we can make.

Online learning

Let us consider the situation where our input consists of predictions from a panel of experts. For example, each day

K pundits predict whether the stock market will go up or down, and

our task is to pool those predictions and make our own. One way to do this is to keep track of how well each expert performs, and choose to believe them in proportion to their past performance. This is called the randomized weighted majority algorithm. We can describe it more formally:

Initialize a set of weights {

w , … , wK } all to 1. 1

for each problem to be solved do

y

y

1. Receive the predictions { ˆ1 , … , ˆK } from the experts. 2. Randomly choose an expert

y

k

∗

in proportion to its weight:

P ( k) = w k .

3. yield ˆk∗ as the answer to this problem.

y 5. For each expert k such that yˆk ≠ y, update wk ← βwk 6. Normalize the weights so that ∑k wk = 1. 4. Receive the correct answer .

Randomized weighted majority algorithm

Here

β is a number, 0 < β < 1, that tells how much to penalize an expert for each mistake.

We measure the success of this algorithm in terms of regret, which is defined as the number of additional mistakes we make compared to the expert who, in hindsight, had the best

M be the number of mistakes made by the best expert. Then the number of mistakes, M , made by the random weighted majority algorithm, is bounded by16 prediction record. Let

∗

16 Blum (1996) gives an elegant proof.

M< M

∗

β

ln(1/ ) + ln 1−

β

K.

Regret

This bound holds for any sequence of examples, even ones chosen by adversaries trying to

K = 10 experts, if we choose β = 1/2 then our number of mistakes is bounded by 1.39M + 4.6, and if β = 3/4 by 1.15M + 9.2. In general, if β is close to 1 then we are responsive to change over the long run; if the best expert do their worst. To be specific, when there are ∗

∗

changes, we will pick up on it before too long. However, we pay a penalty at the beginning, when we start with all experts trusted equally; we may accept the advice of the bad experts

β is closer to 0, these two factors are reversed. Note that we can choose β so that M gets asymptotically close to M in the long run; this is called no-regret learning for too long. When

∗

(because the average amount of regret per trial tends to 0 as the number of trials increases).

No-regret learning

Online learning is helpful when the data may be changing rapidly over time. It is also useful for applications that involve a large collection of data that is constantly growing, even if changes are gradual. For example, with a data set of millions of Web images, you wouldn’t want to retrain from scratch every time a single new image is added. It would be more practical to have an online algorithm that allows images to be added incrementally. For most learning algorithms based on minimizing loss, there is an online version based on minimizing regret. Many of these online algorithms come with guaranteed bounds on regret.

It may seem surprising that there are such tight bounds on how well we can do compared to a panel of experts. What is even more surprising is that when such panels convene to prognosticate about political contests or sporting events, the viewing public is so willing to listen to their predictions and so uninterested in knowing their error rates.

19.9 Developing Machine Learning Systems In this chapter we have concentrated on explaining the theory of machine learning. The practice of using machine learning to solve practical problems is a separate discipline. Over the last 50 years, the software industry has evolved a software development methodology that makes it more likely that a (traditional) software project will be a success. But we are still in the early stages of defining a methodology for machine learning projects; the tools and techniques are not as well-developed. Here is a breakdown of typical steps in the process.

19.9.1 Problem formulation The first step is to figure out what problem you want to solve. There are two parts to this. First ask, “what problem do I want to solve for my users?” An answer such as “make it easier for users to organize and access their photos” is too vague; “help a user find all photos that match a specific term, such as Paris” is better. Then ask, “what part(s) of the problem can be solved by machine learning?” perhaps settling on “learn a function that maps a photo to a set of labels; then, when given a label as a query, retrieve all photos with that label.”

To make this concrete, you need to specify a loss function for your machine learning component, perhaps measuring the system’s accuracy at predicting a correct label. This objective should be correlated with your true goals, but usually will be distinct—the true goal might be to maximize the number of users you gain and keep on your system, and the revenue that they produce. Those are metrics you should track, but not necessarily ones that you can directly build a machine learning model for.

When you have decomposed your problem into parts, you may find that there are multiple components that can be handled by old-fashioned software engineering, not machine learning. For example, for a user who asks for “best photos,” you could implement a simple procedure that sorts photos by the number of likes and views. Once you have developed your overall system to the point where it is viable, you can then go back and optimize, replacing the simple components with more sophisticated machine learning models.

Part of problem formulation is deciding whether you are dealing with supervised, unsupervised, or reinforcement learning. The distinctions are not always so crisp. In semisupervised learning we are given a few labeled examples and use them to mine more information from a large collection of unlabeled examples. This has become a common approach, with companies emerging whose missions are to quickly label some examples, in order to help machine learning systems make better use of the remaining unlabeled examples.

Semisupervised learning

Sometimes you have a choice of which approach to use. Consider a system to recommend songs or movies to customers. We could approach this as a supervised learning problem, where the inputs include a representation of the customer and the labeled output is whether or not they liked the recommendation, or we could approach it as a reinforcement learning problem, where the system makes a series of recommendation actions, and occasionally gets a reward from the customer for making a good suggestion.

The labels themselves may not be the oracular truths that we hope for. Imagine that you are trying to build a system to guess a person’s age from a photo. You gather some labeled examples by having people upload photos and state their age. That’s supervised learning. But in reality some of the people lied about their age. It’s not just that there is random noise in the data; rather the inaccuracies are systematic, and to uncover them is an unsupervised learning problem involving images, self-reported ages, and true (unknown) ages. Thus, both noise and lack of labels create a continuum between supervised and unsupervised learning. The field of weakly supervised learning focuses on using labels that are noisy, imprecise, or supplied by non-experts.

Weakly supervised learning

19.9.2 Data collection, assessment, and management Every machine learning project needs data; in the case of our photo identification project there are freely available image data sets, such as ImageNet, which has over 14 million photos with about 20,000 different labels. Sometimes we may have to manufacture our own data, which can be done by our own labor, or by crowdsourcing to paid workers or unpaid volunteers operating over an Internet service. Sometimes data come from your users. For example, the Waze navigation service encourages users to upload data about traffic jams, and uses that to provide up-to-date navigation directions for all users. Transfer learning (see Section 21.7.2 ) can be used when you don’t have enough of your own data: start with a publicly available general-purpose data set (or a model that has been pretrained on this data), and then add specific data from your users and retrain.

ImageNet

If you deploy a system to users, your users will provide feedback—perhaps by clicking on one item and ignoring the others. You will need a strategy for dealing with this data. That involves a review with privacy experts (see Section 27.3.2 ) to make sure that you get the proper permission for the data you collect, and that you have processes for insuring the integrity of the user’s data, and that they understand what you will do with it. You also need to ensure that your processes are fair and unbiased (see Section 27.3.3 ). If there is data that you feel is too sensitive to collect but that would be useful for a machine learning model, consider a federated learning approach where the data stays on the user’s device, but model parameters are shared in a way that does not reveal private data.

It is good practice to maintain data provenance for all your data. For each column in your data set, you should know the exact definition, where the data come from, what the possible values are, and who has worked on it. Were there periods of time in which a data feed was interrupted? Did the definition of some data source evolve over time? You’ll need to know this if you want to compare results across time periods.

Data provenance

This is particularly true if you are relying on data that are produced by someone else—their needs and yours might diverge, and they might end up changing the way the data are produced, or might stop updating it all together. You need to monitor your data feeds to catch this. Having a reliable, flexible, secure, data-handling pipeline is more critical to success than the exact details of the machine learning algorithm. Provenance is also important for legal reasons, such as compliance with privacy law.

For any task there will be questions about the data: Is this the right data for my task? Does it capture enough of the right inputs to give us a chance of learning a model? Does it contain the outputs I want to predict? If not, can I build an unsupervised model? Or can I label a portion of the data and then do semisupervised learning? Is it relevant data? It is great to have 14 million photos, but if all your users are specialists interested in a specific topic, then a general database won’t help—you’ll need to collect photos on the specific topic. How much training data is enough? (Do I need to collect more data? Can I discard some data to make computation faster?) The best way to answer this is to reason by analogy to a similar project with known training set size. Once you get started you can draw a learning curve (see Figure 19.7 ) to see if more data will help, or if learning has already plateaued. There are endless ad hoc, unjustified rules of thumb for the number of training examples you’ll need: millions for hard problems; thousands for average problems; hundreds or thousands for each class in a classification problem; 10 times more examples than parameters of the model; 10 times more examples than input features;

O(d log d) examples for d input features; more examples for nonlinear

models than for linear models; more examples if greater accuracy is required; fewer examples if you use regularization; enough examples to achieve the statistical power necessary to reject the null hypothesis in classification. All these rules come with caveats—as does the sensible rule that suggests trying what has worked in the past for similar problems.

You should think defensively about your data. Could there be data entry errors? What can be done with missing data fields? If you collect data from your customers (or other people) could some of the people be adversaries out to game the system? Are there spelling errors or

inconsistent terminology in text data? (For example, do “Apple,” “AAPL,” and “Apple Inc.” all refer to the same company?) You will need a process to catch and correct all these potential sources of data error.

When data are limited, data augmentation can help. For example, with a data set of images, you can create multiple versions of each image by rotating, translating, cropping, or scaling each image, or by changing the brightness or color balance or adding noise. As long as these are small changes, the image label should remain the same, and a model trained on such augmented data will be more robust.

Data augmentation

Sometimes data are plentiful but are classified into unbalanced classes. For example, a training set of credit card transactions might consist of 10,000,000 valid transactions and 1,000 fraudulent ones. A classifier that says “valid” regardless of the input will achieve 99.99% accuracy on this data set. To go beyond that, a classifier will have to pay more attention to the fraudulent examples. To help it do that, you can undersample the majority class (i.e., ignore some of the “valid” class examples) or over-sample the minority class (i.e., duplicate some of the “fraudulent” class examples). You can use a weighted loss function that gives a larger penalty to missing a fraudulent case.

Unbalanced classes

Undersampling

Over-sample

Boosting can also help you focus on the minority class. If you are using an ensemble method, you can change the rules by which the ensemble votes and give “fraudulent” as the response even if only a minority of the ensemble votes for “fraudulent.” You can help balance unbalanced classes by generating synthetic data with techniques such as SMOTE (Chawla et al., 2002) or ADASYN (He et al., 2008).

You should carefully consider outliers in your data. An outlier is a data point that is far from other points. For example, in the restaurant problem, if price were a numeric value rather than a categorical one, and if one example had a price of $316 while all the others were $30 or less, that example would be an outlier. Methods such as linear regression are susceptible to outliers because they must form a single global linear model that takes all inputs into account—they can’t treat the outlier differently from other example points, and thus a single outlier can have a large effect on all the parameters of the model.

Outlier

With attributes like price that are positive numbers, we can diminish the effect of outliers by transforming the data, taking the logarithm of each value, so $20, $25, and $316 become 1.3, 1.4, and 2.5. This makes sense from a practical point of view because the high value now has less influence on the model, and from a theoretical point of view because, as we saw in Section 16.3.2 , the utility of money is logarithmic.

Methods such as decision trees that are built from multiple local models can treat outliers individually: it doesn’t matter if the biggest value is $300 or $31; either way it can be treated in its own local node after a test of the form cost ≤ 30. That makes decision trees (and thus random forests and gradient boosting) more robust to outliers.

Feature engineering After correcting overt errors, you may also want to preprocess your data to make it easier to digest. We have already seen the process of quantization: forcing a continuous valued input, such as the wait time, into fixed bins (0 − 10 minutes, 10 − 30, 30 − 60, or > 60) . Domain knowledge can tell you what thresholds are important, such as comparing age ≥ 18 when

studying voting patterns. We also saw (page 688 ) that nearest-neighbor algorithms perform better when data are normalized to have a standard deviation of 1. With categorical attributes such as sunny/cloudy/rainy, it is often helpful to transform the data into three separate Boolean attributes, exactly one of which is true (we call this a one-hot encoding). This is particularly useful when the machine learning model is a neural network.

One-hot encoding

You can also introduce new attributes based on your domain knowledge. For example, given a data set of customer purchases where each entry has a date attribute, you might want to augment the data with new attributes saying whether the date is a weekend or holiday.

As another example, consider the task of estimating the true value of houses that are for sale. In Figure 19.13  we showed a toy version of this problem, doing linear regression of house size to asking price. But we really want to estimate the selling price of a house, not the asking price. To solve this task we’ll need data on actual sales. But that doesn’t mean we should throw away the data about asking price—we can use it as one of the input features. Besides the size of the house, we’ll need more information: the number of rooms, bedrooms, and bathrooms; whether the kitchen and bathrooms have been recently remodeled; the age of the house and perhaps its state of repair; whether it has central heating and air conditioning; the size of the yard and the state of the landscaping.

We’ll also need information about the lot and the neighborhood. But how do we define neighborhood? By zip code? What if a zip code straddles a desirable and an undesirable neighborhood? What about the school district? Should the name of the school district be a feature, or the average test scores? The ability to do a good job of feature engineering is critical to success. As Pedro Domingos (2012) says, “At the end of the day, some machine learning projects succeed and some fail. What makes the difference? Easily the most important factor is the features used.”

Exploratory data analysis and visualization

John Tukey (1977) coined the term exploratory data analysis (EDA) for the process of exploring data in order to gain an understanding of it, not to make predictions or test hypotheses. This is done mostly with visualizations, but also with summary statistics. Looking at a few histograms or scatter plots can often help determine if data are missing or erroneous; whether your data are normally distributed or heavy-tailed; and what learning model might be appropriate.

It can be helpful to cluster your data and then visualize a prototype data point at the center of each cluster. For example, in the data set of images, I can identify that here is a cluster of cat faces; nearby is a cluster of sleeping cats; other clusters depict other objects. Expect to iterate several times between visualizing and modeling—to create clusters you need a distance function to tell you which items are near each other, but to choose a good distance function you need some feel for the data.

It is also helpful to detect outliers that are far from the prototypes; these can be considered critics of the prototype model, and can give you a feel for what type of errors your system might make. An example would be a cat wearing a lion costume.

Our computer display devices (screens or paper) are two-dimensional, which means that it is easy to visualize two-dimensional data. And our eyes are experienced at understanding three-dimensional data that has been projected down to two dimensions. But many data sets have dozens or even millions of dimensions. In order to visualize them we can do dimensionality reduction, projecting the data down to a map in two dimensions (or sometimes to three dimensions, which can then be explored interactively).17

17 Geoffrey Hinton provides the helpful advice “To deal with a 14-dimensional space, visualize a 3D space and say ‘fourteen’ to yourself very loudly.”

The map can’t maintain all relationships between data points, but should have the property that similar points in the original data set are close together in the map. A technique called t-distributed stochastic neighbor embedding (t-SNE) does just that. Figure 19.27  shows a t-SNE map of the MNIST digit recognition data set. Data analysis and visualization packages such as Pandas, Bokeh, and Tableau can make it easier to work with your data.

Figure 19.27

A two-dimensional t-SNE map of the MNIST data set, a collection of 60,000 images of handwritten digits, each 28 × 28 pixels and thus 784 dimensions. You can clearly see clusters for the ten digits, with a few confusions in each cluster; for example the top cluster is for the digit 0, but within the bounds of the cluster are a few data points representing the digits 3 and 6. The t-SNE algorithm finds a representation that accentuates the differences between clusters.

T-distributed stochastic neighbor embedding (t-SNE)

19.9.3 Model selection and training With cleaned data in hand and an intuitive feel for it, it is time to build a model. That means choosing a model class (random forests? deep neural networks? an ensemble?), training your model with the training data, tuning any hyperparameters of the class (number of trees? number of layers?) with the validation data, debugging the process, and finally evaluating the model on the test data.

There is no guaranteed way to pick the best model class, but there are some rough guidelines. Random forests are good when there are a lot of categorical features and you believe that many of them may be irrelevant. Nonparametric methods are good when you have a lot of data and no prior knowledge, and when you don’t want to worry too much about choosing just the right features (as long as there are fewer than 20 or so). However, nonparametric methods usually give you a function h that is more expensive to run.

Logistic regression does well when the data are linearly separable, or can be converted to be so with clever feature engineering. Support vector machines are a good method to try when the data set is not too large; they perform similarly to logistic regression on separable data and can be better for high-dimensional data. Problems dealing with pattern recognition, such as image or speech processing, are most often approached with deep neural networks (see Chapter 21 ).

Choosing hyperparameters can be done with a combination of experience—do what worked well in similar past problems—and search: run experiments with multiple possible values for hyperparameters. As you run more experiments you will get ideas for different models to try. However, if you measure performance on the validation data, get a new idea, and run more experiments, then you run the risk of overfitting on the validation data. If you have enough data, you may want to have several separate validation data sets to avoid this problem. This is especially true if you inspect the validation data by eye, rather than just run evaluations on it.

Suppose you are building a classifier—for example a system to classify spam email. Labeling a legitimate piece of mail as spam is called a false positive. There will be a tradeoff between false positives and false negatives (labeling a piece of spam as legitimate); if you want to keep more legitimate mail out of the spam folder, you will necessarily end up sending more spam to the inbox. But what is the best way to make the tradeoff? You can try different values of hyperparameters and get different rates for the two types of errors—different points on this tradeoff. A chart called the receiver operating characteristic (ROC) curve plots false positives versus true positives for each value of the hyperparameter, helping you visualize values that would be good choices for the tradeoff. A metric called the “area under the ROC curve” or AUC provides a single-number summary of the ROC curve, which is useful if you want to deploy a system and let each user choose their tradeoff point.

False positive

Receiver operating characteristic (ROC) curve

AUC

Another helpful visualization tool for classification problems is a confusion matrix: a twodimensional table of counts of how often each category is classified or misclassified as each other category.

Confusion matrix

There can be tradeoffs in factors other than the loss function. If you can train a stock market prediction model that makes you $10 on every trade, that’s great—but not if it costs you $20 in computation cost for each prediction. A machine translation program that runs on your phone and allows you to read signs in a foreign city is helpful—but not if it runs down the battery after an hour of use. Keep track of all the factors that lead to acceptance or rejection of your system, and design a process where you can quickly iterate the process of getting a new idea, running an experiment, and evaluating the results of the experiment to see if you have made progress. Making this iteration process fast is one of the most important factors for success in machine learning.

19.9.4 Trust, interpretability, and explainability We have described a machine learning methodology where you develop your model with training data, choose hyperparameters with validation data, and get a final metric with test data. Doing well on that metric is a necessary but not sufficient condition for you to trust

your model. And it is not just you—other stakeholders including regulators, lawmakers, the press, and your users are also interested in the trustworthiness of your system (as well as in related attributes such as reliability, accountability, and safety).

A machine learning system is still a piece of software, and you can build trust with all the typical tools for verifying and validating any software system:

SOURCE CONTROL: Systems for version control, build, and bug/issue tracking. TESTING: Unit tests for all the components covering simple canonical cases as well as tricky adversarial cases, fuzz tests (where random inputs are generated), regression tests, load tests, and system integration tests: these are all important for any software system. For machine learning, we also have tests on the training, validation, and test data sets. REVIEW: Code walk-throughs and reviews, privacy reviews, fairness reviews (see Section 27.3.3 ), and other legal compliance reviews. MONITORING: Dashboards and alerts to make sure that the system is up and running and is continuing to performing at a high level of accuracy. ACCOUNTABILITY: What happens when the system is wrong? What is the process for complaining about or appealing the system’s decision? How can we track who was responsible for the error? Society expects (but doesn’t always get) accountability for important decisions made by banks, politicians, and the law, and they should expect accountability from software systems including machine learning systems.

In addition, there are some factors that are especially important for machine learning systems, as we shall detail below.

INTERPRETABILITY: We say that a machine learning model is interpretable if you can inspect the actual model and understand why it got a particular answer for a given input, and how the answer would change when the input changes.18 Decision tree models are considered to be highly interpretable; we can understand that following the path

Patrons Full and WaitEstimate =

= 0–10

in a decision tree leads to a decision to

wait

.

A

decision tree is interpretable for two reasons. First, we humans have experience in understanding IF/THEN rules. (In contrast, it is very difficult for humans to get an intuitive understanding of the result of a matrix multiply followed by an activation function, as is done in some neural network models.) Second, the decision tree was in a sense constructed

to be interpretable—the root of the tree was chosen to be the attribute with the highest information gain.

18 This terminology is not universally accepted; some authors use “interpretable” and “explainable” as synonyms, both referring to reaching some kind of understanding of a model.

Interpretability

Linear regression models are also considered to be interpretable; we can examine a model for predicting the rent on an apartment and see that for each bedroom added, the rent increases by $500, according to the model. This idea of “If I change

X, how will the output

change?” is at the core of interpretability. Of course, correlation is not causation, so interpretable models are answering what is the case, but not necessarily why it is the case.

Explainability

EXPLAINABILITY: An explainable model is one that can help you understand “why was this output produced for this input?” In our terminology, interpretability derives from inspecting the actual model, whereas explainability can be provided by a separate process. That is, the model itself can be a hard-to-understand black box, but an explanation module can summarize what the model does. For a neural network image-recognition system that classifies a picture as dog, if we tried to interpret the model directly, the best we could come away with would be something like “after processing the convolutional layers, the activation for the dog output in the softmax layer was higher than any other class.” That’s not a very compelling argument. But a separate explanation module might be able to examine the neural network model and come up with the explanation “it has four legs, fur, a tail, floppy ears, and a long snout; it is smaller than a wolf, and it is lying on a dog bed, so I think it is a dog.” Explanations are one way to build trust, and some regulations such as the European GDPR (General Data Protection Regulation) require systems to provide explanations.

As an example of a separate explanation module, the local interpretable model-agnostic explanations (LIME) system works like this: no matter what model class you use, LIME builds an interpretable model—often a decision tree or linear model—that is an approximation of your model, and then interprets the linear model to create explanations that say how important each feature is. LIME accomplishes this by treating the machinelearned model as a black box, and probing it with different random input values to create a data set from which the interpretable model can be built. This approach is appropriate for structured data, but not for things like images, where each pixel is a feature, and no one pixel is “important” by itself.

Sometimes we choose a model class because of its explainability—we might choose decision trees over neural networks not because they have higher accuracy but because the explainability gives us more trust in them.

However, a simple explanation can lead to a false sense of security. After all, we typically choose to use a machine learning model (rather than a hand-written traditional program) because the problem we are trying to solve is inherently complex, and we don’t know how to write a traditional program. In that case, we shouldn’t expect that there will necessarily be a simple explanation for every prediction.

If you are building a machine learning model primarily for the purpose of understanding the domain, then interpretability and explainability will help you arrive at that understanding. But if you just want the best-performing piece of software then testing may give you more confidence and trust than explanations. Which would you trust: an experimental aircraft that has never flown before but has a detailed explanation of why it is safe, or an aircraft that safely completed 100 previous flights and has been carefully maintained, but comes with no guaranted explanation?

19.9.5 Operation, monitoring, and maintenance Once you are happy with your model’s performance, you can deploy it to your users. You’ll face additional challenges. First, there is the problem of the long tail of user inputs. You may have tested your system on a large test set, but if your system is popular, you will soon see inputs that were never tested before. You need to know whether your model generalizes well for them, which means you need to monitor your performance on live data—tracking statistics, displaying a dashboard, and sending alerts when key metrics fall below a

threshold. In addition to automatically updating statistics on user interactions, you may need to hire and train human raters to look at your system and grade how well it is doing.

Long tail

Monitoring

Second, there is the problem of nonstationarity—the world changes over time. Suppose your system classifies email as spam or non-spam. As soon as you successfully classify a batch of spam messages, the spammers will see what you have done and change their tactics, sending a new type of message you haven’t seen before. Non-spam also evolves, as users change the mix of email versus messaging or desktop versus mobile services that they use.

Nonstationarity

You will continually face the question of what is better: a model that has been well tested but was built from older data, versus a model that is built from the latest data but has not been tested in actual use. Different systems have different requirements for freshness: some problems benefit from a new model every day, or even every hour, while other problems can keep the same model for months. If you are deploying a new model every hour, it will be impractical to run a heavy test suite and a manual review process for each update. You will need to automate the testing and release process so that small changes can be automatically approved, but larger changes trigger appropriate review. You can consider the tradeoff between an online model where new data incrementally modifies the existing model, versus an offline model where each new release requires building a new model from scratch.

It it is not just that the data will be changing—for example, new words will be used in spam email messages. It is also that the entire data schema may change—you might start out classifying spam email, and need to adapt to classify spam text messages, spam voice messages, spam videos, etc. Figure 19.28  gives a general rubric to guide the practitioner in choosing the appropriate level of testing and monitoring.

Figure 19.28

A set of criteria to see how well you are doing at deploying your machine learning model with sufficient tests. Abridged from Breck et al. (2016), who also provide a scoring metric.

Summary This chapter introduced machine learning, and focused on supervised learning from examples. The main points were:

Learning takes many forms, depending on the nature of the agent, the component to be improved, and the available feedback. If the available feedback provides the correct answer for example inputs, then the learning problem is called supervised learning. The task is to learn a function y = h(x). Learning a function whose output is a continuous or ordered value (like weight) is called regression; learning a function with a small number of possible output categories is called classification; We want to learn a function that not only agrees with the data but also is likely to agree with future data. We need to balance agreement with the data against simplicity of the hypothesis. Decision trees can represent all Boolean functions. The information-gain heuristic provides an efficient method for finding a simple, consistent decision tree. The performance of a learning algorithm can be visualized by a learning curve, which shows the prediction accuracy on the test set as a function of the training set size. When there are multiple models to choose from, model selection can pick good values of hyperparameters, as confirmed by cross-validation on validation data. Once the hyperparameter values are chosen, we build our best model using all the training data. Sometimes not all errors are equal. A loss function tells us how bad each error is; the goal is then to minimize loss over a validation set. Computational learning theory analyzes the sample complexity and computational complexity of inductive learning. There is a tradeoff between the expressiveness of the hypothesis space and the ease of learning. Linear regression is a widely used model. The optimal parameters of a linear regression model can be calculated exactly, or can be found by gradient descent search, which is a technique that can be applied to models that do not have a closed-form solution. A linear classifier with a hard threshold—also known as a perceptron—can be trained by a simple weight update rule to fit data that are linearly separable. In other cases, the rule fails to converge.

Logistic regression replaces the perceptron’s hard threshold with a soft threshold defined by a logistic function. Gradient descent works well even for noisy data that are not linearly separable. Nonparametric models use all the data to make each prediction, rather than trying to summarize the data with a few parameters. Examples include nearest neighbors and locally weighted regression. Support vector machines find linear separators with maximum margin to improve the generalization performance of the classifier. Kernel methods implicitly transform the input data into a high-dimensional space where a linear separator may exist, even if the original data are nonseparable. Ensemble methods such as bagging and boosting often perform better than individual methods. In online learning we can aggregate the opinions of experts to come arbitrarily close to the best expert’s performance, even when the distribution of the data are constantly shifting. Building a good machine learning model requires experience in the complete development process, from managing data to model selection and optimization, to continued maintenance.

Bibliographical and Historical Notes Chapter 1  covered the history of philosophical investigations into the topic of inductive learning. William of Ockham (1280–1349), the most influential philosopher of his century and a major contributor to medieval epistemology, logic, and metaphysics, is credited with a statement called “Ockham’s Razor”—in Latin, Entia non sunt multiplicanda praeter necessitatem, and in English, “Entities are not to be multiplied beyond necessity.” Unfortunately, this laudable piece of advice is nowhere to be found in his writings in precisely these words (although he did say “Pluralitas non est ponenda sine necessitate,” or “Plurality shouldn’t be posited without necessity”). A similar sentiment was expressed by Aristotle in 350 BCE in Physics book I, chapter VI: “For the more limited, if adequate, is always preferable.”

David Hume (1711–1776) formulated the problem of induction, recognizing that generalizing from examples admits the possibility of errors, in a way that logical deduction does not. He saw that there was no way to have a guaranteed correct solution to the problem, but proposed the principle of uniformity of nature, which we have called stationarity. What Ockham and Hume were getting at is that when we do induction, we are choosing from the multitude of consistent models one that is more likely—because it is simpler and matches our expectations. In modern day, the no free lunch theorem (Wolpert and Macready, 1997; Wolpert, 2013) says that if a learning algorithm performs well on a certain set of problems, it is only because it will perform poorly on a different set: if our decision tree correctly predicts SR’s restaurant waiting behavior, it must perform poorly for some other hypothetical person who has the opposite waiting behavior on the unobserved inputs.

Machine learning was one of the key ideas at the birth of computer science. Alan Turing (1947) anticipated it, saying “Let us suppose we have set up a machine with certain initial instruction tables, so constructed that these tables might on occasion, if good reason arose, modify those tables.” Arthur Samuel (1959) defined machine learning as the “field of study that gives computers the ability to learn without being explicitly programmed” while creating his learning checkers program.

The first notable use of decision trees was in EPAM, the “Elementary Perceiver And Memorizer” (Feigenbaum, 1961), which was a simulation of human concept learning. ID3

(Quinlan, 1979) added the crucial idea of choosing the attribute with maximum entropy. The concepts of entropy and information theory were developed by Claude Shannon to aid in the study of communication (Shannon and Weaver, 1949). (Shannon also contributed one of the earliest examples of machine learning, a mechanical mouse named Theseus that learned to navigate through a maze by trial and error.) The χ2 method of tree pruning was described by Quinlan (1986). A description of C4.5, an industrial-strength decision tree package, can be found in Quinlan (1993). An alternative industrial-strength software package, CART (for Classification and Regression Trees) was developed by the statistician Leo Breiman and his colleagues (Breiman et al., 1984).

Hyafil and Rivest, (1976) proved that finding an optimal decision tree (rather than finding a good tree through locally greedy selections) is NP-complete. But Bertsimas and Dunn, (2017) point out that in the last 25 years, advances in hardware design and in algorithms for mixed-integer programming have resulted in an 800 billion-fold speedup, which means that it is now feasible to solve this NP-hard problem at least for problems with not more than a few thousand examples and a few dozen features.

Cross-validation was first introduced by Larson, (1931), and in a form close to what we show by Stone, (1974) and Golub et al., (1979). The regularization procedure is due to Tikhonov, (1963).

On the question of overfitting, John von Neumann was quoted (Dyson, 2004) as boasting, “With four parameters I can fit an elephant, and with five I can make him wiggle his trunk,” meaning that a high-degree polynomial can be made to fit almost any data, but at the cost of potentially overfitting. Mayer et al., (2010) proved him right by demonstrating a fourparameter elephant and five-parameter wiggle, and Boué, (2019) went even further, demonstrating an elephant and other animals with a one-parameter chaotic function.

Zhang et al., (2016) analyze under what conditions a model can memorize the training data. They perform experiments using random data—surely an algorithm that gets zero error on a training set with random labels must be memorizing the data set. However, they conclude that the field has yet to discover a precise measure of what it means for a model to be “simple” in the sense of Ockham’s razor. Arpit et al., (2017) show that the conditions under which memorization can occur depend on details of both the model and the data set.

Belkin et al., (2019) discuss the bias–variance tradeoff in machine learning and why some model classes continue to improve after reaching the interpolation point, while other model classes exhibit the U-shaped curve. Berrada et al., (2019) develop a new learning algorithm based on gradient descent that exploits the ability of models to memorize to set good values for the learning rate hyperparameter.

Theoretical analysis of learning algorithms began with the work of Gold, (1967) on identification in the limit. This approach was motivated in part by models of scientific discovery from the philosophy of science (Popper, 1962), but has been applied mainly to the problem of learning grammars from example sentences (Osherson et al., 1986).

Whereas the identification-in-the-limit approach concentrates on eventual convergence, the study of Kolmogorov complexity or algorithmic complexity, developed independently by Solomonoff (1964, 2009) and Kolmogorov (1965), attempts to provide a formal definition for the notion of simplicity used in Ockham’s razor. To escape the problem that simplicity depends on the way in which information is represented, it is proposed that simplicity be measured by the length of the shortest program for a universal Turing machine that correctly reproduces the observed data. Although there are many possible universal Turing machines, and hence many possible “shortest” programs, these programs differ in length by at most a constant that is independent of the amount of data. This beautiful insight, which essentially shows that any initial representation bias will eventually be overcome by the data, is marred only by the undecidability of computing the length of the shortest program. Approximate measures such as the minimum description length, or MDL (Rissanen, 1984; Rissanen, 2007) can be used instead and have produced excellent results in practice. The text by Li and Vitanyi (2008) is the best source for Kolmogorov complexity.

Kolmogorov complexity

The theory of PAC learning was inaugurated by Leslie Valiant (1984), stressing the importance of computational and sample complexity. With Michael Kearns (1990), Valiant showed that several concept classes cannot be PAC-learned tractably, even though sufficient

information is available in the examples. Some positive results were obtained for classes such as decision lists (Rivest, 1987).

An independent tradition of sample-complexity analysis has existed in statistics, beginning with the work on uniform convergence theory (Vapnik and Chervonenkis, 1971). The socalled VC dimension provides a measure roughly analogous to, but more general than, the

H

ln | | measure obtained from PAC analysis. The VC dimension can be applied to continuous function classes, to which standard PAC analysis does not apply. PAC-learning theory and VC theory were first connected by the “four Germans” (none of whom actually is German): Blumer, Ehrenfeucht, Haussler, and Warmuth (1989).

VC dimension

Linear regression with squared error loss goes back to Legendre, (1805) and Gauss, (1809), who were both working on predicting orbits around the sun. (Gauss claimed to be using the technique since 1795, but delayed in publishing it.) The modern use of multivariable regression for machine learning is covered in texts such as Bishop, (2007). The differences between

L1 and L2 regularization are analyzed by Ng, (2004) and Moore and DeNero,

(2011).

The term logistic function comes from Pierre-François Verhulst (1804–1849), a statistician who used the curve to model population growth with limited resources, a more realistic model than the unconstrained geometric growth proposed by Thomas Malthus. Verhulst called it the courbe logistique, because of its relation to the logarithmic curve. The term curse of dimensionality comes from Richard Bellman, (1961).

Logistic regression can be solved with gradient descent or with the Newton–Raphson method (Newton, 1671; Raphson, 1690). A variant of the Newton method called L-BFGS is often used for large-dimensional problems; the L stands for “limited memory,” meaning that it avoids creating the full matrices all at once, and instead creates parts of them on the fly. BFGS are the authors’ initials (Byrd et al., 1995). The idea of gradient descent goes back to Cauchy, (1847); stochastic gradient descent (SGD) was introduced in the statistical

optimization community by Robbins and Monro, (1951), rediscovered for neural networks by Rosenblatt, (1960), and popularized for large-scale machine learning by Bottou and Bousquet, (2008). Bottou et al., (2018) reconsider the topic of large-scale learning with a decade of additional experience.

Nearest-neighbors models date back at least to Fix and Hodges, (1951) and have been a standard tool in statistics and pattern recognition ever since. Within AI, they were popularized by Stanfill and Waltz, (1986), who investigated methods for adapting the distance metric to the data. Hastie and Tibshirani, (1996) developed a way to localize the metric to each point in the space, depending on the distribution of data around that point. Gionis et al., (1999) introduced locality-sensitive hashing (LSH), which revolutionized the retrieval of similar objects in high-dimensional spaces. Andoni and Indyk, (2006) provide a survey of LSH and related methods, and Samet, (2006) covers properties of highdimensional spaces. The technique is particularly useful for genomic data, where each record has millions of attributes (Berlin et al., 2015).

The ideas behind kernel machines come from Aizerman et al., (1964) (who also introduced the kernel trick), but the full development of the theory is due to Vapnik and his colleagues (Boser et al., 1992). SVMs were made practical with the introduction of the soft-margin classifier for handling noisy data in a paper that won the 2008 ACM Theory and Practice Award (Cortes and Vapnik, 1995), and of the Sequential Minimal Optimization (SMO) algorithm for efficiently solving SVM problems using quadratic programming (Platt, 1999). SVMs have proven to be very effective for tasks such as text categorization (Joachims, 2001), computational genomics (Cristianini and Hahn, 2007), and handwritten digit recognition of DeCoste and Schölkopf, (2002).

As part of this process, many new kernels have been designed that work with strings, trees, and other nonnumerical data types. A related technique that also uses the kernel trick to implicitly represent an exponential feature space is the voted perceptron (Freund and Schapire, 1999; Collins and Duffy, 2002). Textbooks on SVMs include Cristianini and Shawe-Taylor, (2000) and Schölkopf and Smola, (2002). A friendlier exposition appears in the AI Magazine article by Cristianini and Schölkopf, (2002). Bengio and LeCun, (2007) show some of the limitations of SVMs and other local, nonparametric methods for learning functions that have a global structure but do not have local smoothness.

The first mathematical proof of the value of an ensemble was Condorcet’s jury theorem (1785), which proved that if jurors are independent and an individual juror has at least a 50% chance of deciding a case correctly, then the more jurors you add, the better the chance of deciding the case correctly. More recently, ensemble learning has become an increasingly popular technique for improving the performance of learning algorithms.

The first random forest algorithm, using random attribution selection, is by Ho, (1995); an independent version was introduced by Amit and Geman, (1997). Breiman, (2001) added the ideas of bagging and “out-of-bag error.” Friedman, (2001) introduced the terminology Gradient Boosting Machine (GBM), expanding the approach to allow for multiclass classification, regression, and ranking problems.

Michel Kearns, (1988) defined the Hypothesis Boosting Problem: given a learner that predicts only slightly better than random guessing, is it possible to derive a learner that performs arbitrarily well? The problem was answered in the affirmative in a theoretical paper by Schapire, (1990) that led to the ADABOOST algorithm Freund and Schapire, (1996) and to further theoretical work Schapire, (2003). Friedman et al., (2000) explain boosting from a statistician’s viewpoint. Chen and Guestrin, (2016) describe the XGBOOST system, which has been used with great success in many large-scale applications.

Online learning is covered in a survey by Blum, (1996) and a book by Cesa-Bianchi and Lugosi, (2006). Dredze et al., (2008) introduce the idea of confidence-weighted online learning for classification: in addition to keeping a weight for each parameter, they also maintain a measure of confidence, so that a new example can have a large effect on features that were rarely seen before (and thus had low confidence) and a small effect on common features that have already been well estimated. Yu et al., (2011) describe how a team of students work together to build an ensemble classifier in the KDD competition. One exciting possibility is to create an “outrageously large” mixture-of-experts ensemble that uses a sparse subset of experts for each incoming example (Shazeer et al., 2017). Seni and Elder, (2010) survey ensemble methods.

In terms of practical advice for building machine learning systems, Pedro Domingos describes a few things to know (2012). Andrew Ng gives hints for developing and debugging a product using machine learning (Ng, 2019). O’Neil and Schutt, (2013) describe the process of doing data science. Tukey, (1977) introduced exploratory data analysis, and

Gelman, (2004) gives an updated view of the process. Bien et al., (2011) describe the process of choosing prototypes for interpretability, and Kim et al., (2017) show how to find critics that are maximally distant from the prototypes using a metric called maximum mean discrepancy. Wattenberg et al., (2016) describe how to use t-SNE. To get a comprehensive view of how well your deployed machine learning system is doing, Breck et al., (2016) offer a checklist of 28 tests that you can apply to get an overall ML test score. Riley, (2019) describes three common pitfalls of ML development.

Banko and Brill, (2001), Halevy et al., (2009), and Gandomi and Haider, (2015) discuss the advantages of using the large amounts of data that are now available. Lyman and Varian, (2003) estimated that about 5 exabytes (5 × 1018 bytes) of data was produced in 2002, and that the rate of production is doubling every 3 years; Hilbert and Lopez, (2011) estimated

2 × 1021 bytes for 2007, indicating an acceleration. Guyon and Elisseeff, (2003) discuss the problem of feature selection with large data sets.

Doshi-Velez and Kim, (2017) propose a framework for interpretable machine learning or explainable AI (XAI). Miller et al., (2017) point out that there are two kinds of explanations, one for the designers of an AI system and one for the users, and we need to be clear what we are aiming for. The LIME system (Ribeiro et al., 2016) builds interpretable linear models that approximate whatever machine learning system you have. A similar system, SHAP (Lundberg and Lee, 2018) (Shapley Additive exPlanations), uses the notion of a Shapley value (page 628) to determine the contribution of each feature.

The idea that we could apply machine learning to the task of solving machine learning problems is a tantalizing one. Thrun and Pratt, (2012) give an early overview of the field in an edited collection titled Learning to Learn. Recently the field has adopted the name automated machine learning (AutoML); Hutter et al., (2019) give an overview.

Automated machine learning (AutoML)

Kanter and Veeramachaneni, (2015) describe a system for doing automated feature selection. Bergstra and Bengio, (2012) describe a system for searching the space of

hyperparameters, as do Thornton et al., (2013) and Bermúdez-Chacón et al., (2015). Wong et al., (2019) show how transfer learning can speed up AutoML for deep learning models. Competitions have been organized to see which systems are best at AutoML tasks (Guyon et al., 2015). (Steinruecken et al., 2019) describe a system called the Automatic Statistician: you give it some data and it writes a report, mixing text, charts, and calculations. The major cloud computing providers have included AutoML as part of their offerings. Some researchers prefer the term metalearning: for example, the MAML (Model-Agnostic MetaLearning) system (Finn et al., 2017) works with any model that can be trained by gradient descent; it trains a core model so that it will be easy to fine-tune the model with new data on new tasks.

Despite all this work, we still don’t have a complete system for automatically solving machine learning problems. To do that with supervised machine learning we would need to start with a data set of (xj , yj ) examples. Here the input xj is a specification of the problem, in the form that a problem is initially encountered: a vague description of the goals, and some data to work with, perhaps with a vague plan for how to acquire more data. The output yi would be a complete running machine learning program, along with a methodology for maintaining the program: gathering more data, cleaning it, testing and monitoring the system, etc. One would expect we would need a data set of thousands of such examples. But no such data set exists, so existing AutoML systems are limited in what they can accomplish.

There is a dizzying array of books that introduce data science and machine learning in conjunction with software packages such as Python (Segaran, 2007; Raschka, 2015; Nielsen, 2015), Scikit-Learn (Pedregosa et al., 2011), R (Conway and White, 2012), Pandas (McKinney, 2012), NumPy (Marsland, 2014), PyTorch (Howard and Gugger, 2020), TensorFlow (Ramsundar and Zadeh, 2018), and Keras (Chollet, 2017; Géron, 2019).

There are a number of valuable textbooks in machine learning (Bishop, 2007; Murphy, 2012) and in the closely allied and overlapping fields of pattern recognition (Ripley, 1996; Duda et al., 2001), statistics (Wasserman, 2004; Hastie et al., 2009; James et al., 2013), data science (Blum et al., 2020), data mining (Han et al., 2011; Witten and Frank, 2016; Tan et al., 2019), computational learning theory (Kearns and Vazirani, 1994; Vapnik, 1998), and information theory (Shannon and Weaver, 1949; MacKay, 2002; Cover and Thomas, 2006). Burkov, (2019) attempts the shortest possible introduction to machine learning, and

Domingos, (2015) offers a nontechnical overview of the field. Current research in machine learning is published in the annual proceedings of the International Conference on Machine Learning (ICML), the International Conference on Learning Representations (ICLR), and the conference on Neural Information Processing Systems (NeurIPS); and in Machine Learning and the Journal of Machine Learning Research.

Chapter 20 Learning Probabilistic Models In which we view learning as a form of uncertain reasoning from observations, and devise models to represent the uncertain world. Chapter 12  pointed out the prevalence of uncertainty in real environments. Agents can handle uncertainty by using the methods of probability and decision theory, but first they must learn their probabilistic theories of the world from experience. This chapter explains how they can do that, by formulating the learning task itself as a process of probabilistic inference (Section 20.1 ). We will see that a Bayesian view of learning is extremely powerful, providing general solutions to the problems of noise, overfitting, and optimal prediction. It also takes into account the fact that a less-than-omniscient agent can never be certain about which theory of the world is correct, yet must still make decisions by using some theory of the world.

We describe methods for learning probability models—primarily Bayesian networks—in Sections 20.2  and 20.3 . Some of the material in this chapter is fairly mathematical, although the general lessons can be understood without plunging into the details. It may benefit the reader to review Chapters 12  and 13  and peek at Appendix A .

20.1 Statistical Learning The key concepts in this chapter, just as in Chapter 19 , are data and hypotheses. Here, the data are evidence—that is, instantiations of some or all of the random variables describing the domain. The hypotheses in this chapter are probabilistic theories of how the domain works, including logical theories as a special case.

Consider a simple example. Our favorite surprise candy comes in two flavors: cherry (yum) and lime (ugh). The manufacturer has a peculiar sense of humor and wraps each piece of candy in the same opaque wrapper, regardless of flavor. The candy is sold in very large bags, of which there are known to be five kinds—again, indistinguishable from the outside:

h1 : 100% cherry, h2 : 75% cherry + 25% lime, h3 : 50% cherry + 50% lime, h4 : 25% cherry + 75% lime, h5 : 100% lime. Given a new bag of candy, the random variable

H (for hypothesis) denotes the type of the

h1 through h5 . H is not directly observable, of course. As the pieces of candy are opened and inspected, data are revealed—D1 , D2 ,  … , DN , where each Di is a random variable with possible values cherry and lime. The basic task faced by the bag, with possible values

agent is to predict the flavor of the next piece of candy.1 Despite its apparent triviality, this scenario serves to introduce many of the major issues. The agent really does need to infer a theory of its world, albeit a very simple one.

1 Statistically sophisticated readers will recognize this scenario as a variant of the urn-and-ball setup. We find urns and balls less compelling than candy.

Bayesian learning simply calculates the probability of each hypothesis, given the data, and makes predictions on that basis. That is, the predictions are made by using all the hypotheses, weighted by their probabilities, rather than by using just a single “best” hypothesis. In this way, learning is reduced to probabilistic inference.

Bayesian learning

Let

D represent all the data, with observed value d. The key quantities in the Bayesian

approach are the hypothesis prior, P (hi ), and the likelihood of the data under each hypothesis, P (

d hi . The probability of each hypothesis is obtained by Bayes’ rule: |

)

(20.1)

P hi d (

|

) =

αP

(

d hi P hi |

)

(

).

Hypothesis prior

Likelihood

Now, suppose we want to make a prediction about an unknown quantity X. Then we have (20.2)

P( X d ) ∣ ∣ ∣

∑ P( X h ) P ( h d ) ∣

∣

=

i

∣ ∣ ∣

∣

i

i ∣∣

,

∣

where each hypothesis determines a probability distribution over X. This equation shows that predictions are weighted averages over the predictions of the individual hypotheses, where the weight P (hi |

d

)

is proportional to the prior probability of hi and its degree of fit,

according to Equation (20.1) . The hypotheses themselves are essentially “intermediaries” between the raw data and the predictions.

For our candy example, we will assume for the time being that the prior distribution over

h

1,

…,

h

5

is given by ⟨0.1,0.2,0.4,0.2,0.1⟩, as advertised by the manufacturer. The likelihood

of the data is calculated under the assumption that the observations are i.i.d. (see page 665), so that

(20.3)

P ( d hi ) ∣ ∣ ∣ ∣

∏j P (dj hi) ∣

=

∣ ∣ ∣

h ) and the first 10 candies are all , because half the candies in an h bag are lime.2 Figure 20.1(a) 

For example, suppose the bag is really an all-lime bag ( lime; then

P dh (

|

3)

is 0.5

10

.

5

3

shows how the posterior probabilities of the five hypotheses change as the sequence of 10 lime candies is observed. Notice that the probabilities start out at their prior values, so

h

3

is

initially the most likely choice and remains so after 1 lime candy is unwrapped. After 2 lime candies are unwrapped,

h

4

is most likely; after 3 or more,

h

5

(the dreaded all-lime bag) is

the most likely. After 10 in a row, we are fairly certain of our fate. Figure 20.1(b)  shows the predicted probability that the next candy is lime, based on Equation (20.2) . As we would expect, it increases monotonically toward 1.

2 We stated earlier that the bags of candy are very large; otherwise, the i.i.d. assumption fails to hold. Technically, it is more correct (but less hygienic) to rewrap each candy after inspection and return it to the bag.

Figure 20.1

(a) Posterior probabilities

P hi d (

|

1,

…,

dN

)

from Equation (20.1) . The number of observations

from 1 to 10, and each observation is of a lime candy. (b) Bayesian prediction from Equation (20.2) .

P DN (

+1

=

lime d |

N ranges dN

1,

…,

)

The example shows that the Bayesian prediction eventually agrees with the true hypothesis. This is characteristic of Bayesian learning. For any fixed prior that does not rule out the true hypothesis, the posterior probability of any false hypothesis will, under certain technical conditions, eventually vanish. This happens simply because the probability of generating “uncharacteristic” data indefinitely is vanishingly small. (This point is analogous to one made in the discussion of PAC learning in Chapter 19 .) More important, the Bayesian prediction is optimal, whether the data set is small or large. Given the hypothesis prior, any other prediction is expected to be correct less often.

The optimality of Bayesian learning comes at a price, of course. For real learning problems, the hypothesis space is usually very large or infinite, as we saw in Chapter 19 . In some cases, the summation in Equation (20.2)  (or integration, in the continuous case) can be carried out tractably, but in most cases we must resort to approximate or simplified methods.

A very common approximation—one that is usually adopted in science—is to make

d

predictions based on a single most probable hypothesis—that is, an hi that maximizes P (hi | ) . This is often called a maximum a posteriori or MAP (pronounced “em-ay-pee”) hypothesis. Predictions made according to an MAP hypothesis hMAP are approximately Bayesian to the extent that

P ( X |d ) ≈ P ( X |h

MAP )

. In our candy example, hMAP =

h5 after

three lime candies in a row, so the MAP learner then predicts that the fourth candy is lime with probability 1.0—a much more dangerous prediction than the Bayesian prediction of 0.8 shown in Figure 20.1(b) . As more data arrive, the MAP and Bayesian predictions become closer, because the competitors to the MAP hypothesis become less and less probable.

Maximum a posteriori

Although this example doesn’t show it, finding MAP hypotheses is often much easier than Bayesian learning, because it requires solving an optimization problem instead of a large summation (or integration) problem.

In both Bayesian learning and MAP learning, the hypothesis prior

P (hi ) plays an important

role. We saw in Chapter 19  that overfitting can occur when the hypothesis space is too

expressive, that is, when it contains many hypotheses that fit the data set well. Bayesian and MAP learning methods use the prior to penalize complexity. Typically, more complex hypotheses have a lower prior probability—in part because there so many of them. On the other hand, more complex hypotheses have a greater capacity to fit the data. (In the extreme case, a lookup table can reproduce the data exactly.) Hence, the hypothesis prior embodies a tradeoff between the complexity of a hypothesis and its degree of fit to the data.

We can see the effect of this tradeoff most clearly in the logical case, where deterministic hypotheses (such as

H contains only

h , which says that every candy is cherry). In that case, 1

P (d|hi ) is 1 if hi is consistent and 0 otherwise. Looking at Equation (20.1) , we see that h will then be the simplest logical theory that is consistent with the data. Therefore, MAP

maximum a posteriori learning provides a natural embodiment of Ockham’s razor.

Another insight into the tradeoff between complexity and degree of fit is obtained by taking the logarithm of Equation (20.1) . Choosing

h

MAP

to maximize

P (d|hi )P (hi ) is equivalent

to minimizing − log2

P (d|hi ) − log P (hi ) . 2

Using the connection between information encoding and probability that we introduced in

P (hi ) term equals the number of bits required to specify the hypothesis hi . Furthermore, − log P (d|hi ) is the additional number of bits

Section 19.3.3 , we see that the − log2

2

required to specify the data, given the hypothesis. (To see this, consider that no bits are required if the hypothesis predicts the data exactly—as with

h

5

and the string of lime candies

—and log2 1 = 0.) Hence, MAP learning is choosing the hypothesis that provides maximum compression of the data. The same task is addressed more directly by the minimum description length, or MDL, learning method. Whereas MAP learning expresses simplicity by assigning higher probabilities to simpler hypotheses, MDL expresses it directly by counting the bits in a binary encoding of the hypotheses and data.

A final simplification is provided by assuming a uniform prior over the space of hypotheses. In that case, MAP learning reduces to choosing an maximum-likelihood hypothesis,

h

ML

hi that maximizes P (d|hi ). This is called a

. Maximum-likelihood learning is very common in

statistics, a discipline in which many researchers distrust the subjective nature of hypothesis

priors. It is a reasonable approach when there is no reason to prefer one hypothesis over another a priori—for example, when all hypotheses are equally complex.

Maximum-likelihood

When the data set is large, the prior distribution over hypotheses is less important—the evidence from the data is strong enough to swamp the prior distribution over hypotheses. That means maximum likelihood learning is a good approximation to Bayesian and MAP learning with large data sets, but it has problems (as we shall see) with small data sets.

20.2 Learning with Complete Data The general task of learning a probability model, given data that are assumed to be generated from that model, is called density estimation. (The term applied originally to probability density functions for continuous variables, but it is used now for discrete distributions too.) Density estimation is a form of unsupervised learning. This section covers the simplest case, where we have complete data. Data are complete when each data point contains values for every variable in the probability model being learned. We focus on parameter learning—finding the numerical parameters for a probability model whose structure is fixed. For example, we might be interested in learning the conditional probabilities in a Bayesian network with a given structure. We will also look briefly at the problem of learning structure and at nonparametric density estimation.

Density estimation

Complete data

Parameter learning

20.2.1 Maximum-likelihood parameter learning: Discrete models Suppose we buy a bag of lime and cherry candy from a new manufacturer whose flavor proportions are completely unknown; the fraction of cherry could be anywhere between 0 and 1. In that case, we have a continuum of hypotheses. The parameter in this case, which we call θ, is the proportion of cherry candies, and the hypothesis is hθ . (The proportion of lime candies is just 1 − θ.) If we assume that all proportions are equally likely a priori, then a

maximum-likelihood approach is reasonable. If we model the situation with a Bayesian network, we need just one random variable, Flavor (the flavor of a randomly chosen candy from the bag). It has values cherry and lime, where the probability of cherry is θ (see Figure 20.2(a) ). Now suppose we unwrap N candies, of which c are cherry and ℓ

=

N

−

lime. According to Equation (20.3) , the likelihood of this particular data set is

P (d hθ ) ∣ ∣ ∣

∏ P (d h ) N

∣

=

j

∣

j ∣∣ θ

=

θc ⋅

(

1 −

θ)

c are

ℓ

.

∣

=1

The maximum-likelihood hypothesis is given by the value of θ that maximizes this expression. Because the log function is monotonic, the same value is obtained by maximizing the log likelihood instead:

L( d h θ ) ∣ ∣ ∣

= log

P ( d hθ ) ∣ ∣ ∣

∑ N

∣

∣

=

j

∣

log

=1

P ( dj h θ ) ∣ ∣ ∣

=

c

log

θ

+ ℓ log

(

1 −

θ)

.

Log likelihood

(By taking logarithms, we reduce the product to a sum over the data, which is usually easier to maximize.) To find the maximum-likelihood value of θ, we differentiate L with respect to

θ and set the resulting expression to zero: dL d hθ dθ (

|

) =

c θ

ℓ − 1 −

θ

= 0

θ

⇒

In English, then, the maximum-likelihood hypothesis h

ML

=

c

c + ℓ

=

c N

.

asserts that the actual proportion

of cherry candies in the bag is equal to the observed proportion in the candies unwrapped so far!

It appears that we have done a lot of work to discover the obvious. In fact, though, we have laid out one standard method for maximum-likelihood parameter learning, a method with broad applicability:

1. Write down an expression for the likelihood of the data as a function of the parameter(s). 2. Write down the derivative of the log likelihood with respect to each parameter. 3. Find the parameter values such that the derivatives are zero.

The trickiest step is usually the last. In our example, it was trivial, but we will see that in many cases we need to resort to iterative solution algorithms or other numerical optimization techniques, as described in Section 4.2 . (We will need to verify that the Hessian matrix is negative-definite.) The example also illustrates a significant problem with maximum-likelihood learning in general: when the data set is small enough that some events have not yet been observed—for instance, no cherry candies—the maximum-likelihood hypothesis assigns zero probability to those events. Various tricks are used to avoid this problem, such as initializing the counts for each event to 1 instead of 0.

Let us look at another example. Suppose this new candy manufacturer wants to give a little hint to the consumer and uses candy wrappers colored red and green. The Wrapper for each candy is selected probabilistically, according to some unknown conditional distribution, depending on the flavor. The corresponding probability model is shown in Figure 20.2(b) . Notice that it has three parameters: θ, θ1 , and θ2 . With these parameters, the likelihood of seeing, say, a cherry candy in a green wrapper can be obtained from the standard semantics for Bayesian networks (page 415):

P (Flavor = cherry, Wrapper = green∣hθ,θ ,θ ) = P (Flavor = cherry∣hθ,θ ,θ ) P (Wrapper = green∣Flavor = cherry, hθ,θ ,θ ) = θ ⋅ (1 − θ1 ) . 1

1

Figure 20.2

2

2

1

2

(a) Bayesian network model for the case of candies with an unknown proportion of cherry and lime. (b) Model for the case where the wrapper color depends (probabilistically) on the candy flavor.

Now we unwrap N candies, of which c are cherry and ℓ are lime. The wrapper counts are as follows: rc of the cherry candies have red wrappers and gc have green, while rℓ of the lime candies have red and gℓ have green. The likelihood of the data is given by ℓ

gc

P (d∣∣∣hθ,θ ,θ ) = θc (1 − θ) ⋅ θr1c (1 − θ1 ) ⋅ θr2 (1 − θ2 ) ℓ

1

2

gℓ .

This looks pretty horrible, but taking logarithms helps:

L = [c log θ + ℓ log(1 − θ)] + [rc log θ1 + gc log(1 − θ1 )] + [rℓ log θ2 + gℓ log(1 − θ2 )] . The benefit of taking logs is clear: the log likelihood is the sum of three terms, each of which contains a single parameter. When we take derivatives with respect to each parameter and set them to zero, we get three independent equations, each containing just one parameter:

L θ L θ L θ

c θ rc θ r θ

∂

=

∂ ∂ ∂

=

∂ ∂

The solution for

1 −

−

1 −

2

2

= 0

⇒

= 0

⇒

= 0

⇒

θ

1

θ

ℓ

ℓ

=

θ gc θ g θ

1 −

−

1

1

θ

ℓ −

2

=

1

=

2

=

c

c + ℓ

rc

r c gc r r g + ℓ

+

ℓ

.

ℓ

θ is the same as before. The solution for θ , the probability that a cherry 1

candy has a red wrapper, is the observed fraction of cherry candies with red wrappers, and similarly for

θ. 2

These results are very comforting, and it is easy to see that they can be extended to any Bayesian network whose conditional probabilities are represented as tables. The most important point is that with complete data, the maximum-likelihood parameter learning problem for a Bayesian network decomposes into separate learning problems, one for each parameter. (See Exercise 20.NORX for the nontabulated case, where each parameter affects several conditional probabilities.) The second point is that the parameter values for a variable, given its parents, are just the observed frequencies of the variable values for each setting of the parent values. As before, we must be careful to avoid zeroes when the data set is small.

20.2.2 Naive Bayes models Probably the most common Bayesian network model used in machine learning is the naive Bayes model first introduced on page 402. In this model, the “class” variable be predicted) is the root and the “attribute” variables

C (which is to

Xi are the leaves. The model is “naive”

because it assumes that the attributes are conditionally independent of each other, given the class. (The model in Figure 20.2(b)  is a naive Bayes model with class

Flavor and just one

attribute, Wrapper.) In the case of Boolean variables, the parameters are

θ P C true θi =

(

=

),

1

=

P Xi true C true θi (

=

|

=

),

2

=

P Xi true C false (

=

|

=

).

The maximum-likelihood parameter values are found in exactly the same way as in Figure 20.2(b) . Once the model has been trained in this way, it can be used to classify new examples for which the class variable

x

1,

…,

C is unobserved. With observed attribute values

xn , the probability of each class is given by

P(C x ∣ ∣ ∣ ∣

1,

…,

xn ) α P ( C ) =

∏ P(x C ) ∣

i

i ∣∣ ∣

.

A deterministic prediction can be obtained by choosing the most likely class. Figure 20.3  shows the learning curve for this method when it is applied to the restaurant problem from Chapter 19 . The method learns fairly well but not as well as decision tree learning; this is presumably because the true hypothesis—which is a decision tree—is not representable exactly using a naive Bayes model. Naive Bayes learning turns out to do surprisingly well in a wide range of applications; the boosted version (Exercise 20.BNBX) is one of the most effective general-purpose learning algorithms. Naive Bayes learning scales well to very large problems: with

n Boolean attributes, there are just 2n + 1 parameters, and no search is

required to find hML , the maximum-likelihood naive Bayes hypothesis. Finally, naive Bayes learning systems deal well with noisy or missing data and can give probabilistic predictions when appropriate. Their primary drawback is the fact that the conditional independence assumption is seldom accurate; as noted on page 403, the assumption leads to overconfident probabilities that are often very close to 0 or 1, especially with large numbers of attributes.

Figure 20.3

The learning curve for naive Bayes learning applied to the restaurant problem from Chapter 19 ; the learning curve for decision tree learning is shown for comparison.

20.2.3 Generative and discriminative models We can distinguish two kinds of machine learning models used for classifiers: generative and discriminative. A generative model models the probability distribution of each class. For example, the naive Bayes text classifier from Section 12.6.1  creates a separate model for each possible category of text—one for sports, one for weather, and so on. Each model

P (Category = weather)—as well as the conditional probability P(Inputs|Category = weather). From these we can compute the joint probability P(Inputs, Category = weather)) and we can generate a random includes the prior probability of the category—for example

selection of words that is representative of texts in the weather category.

Generative model

A discriminative model directly learns the decision boundary between classes. That is, it learns

P(Category|Inputs). Given example inputs, a discriminative model will come up with

an output category, but you cannot use a discriminative model to, say, generate random words that are representative of a category. Logistic regression, decision trees, and support vector machines are all discriminative models.

Discriminative model

Since discriminative models put all their emphasis on defining the decision boundary—that is, actually doing the classification task they were asked to do—they tend to perform better in the limit, with an arbitrary amount of training data. However, with limited data, in some cases a generative model performs better. (Ng and Jordan, 2002) compare the generative naive Bayes classifier to the discriminative logistic regression classifier on 15 (small) data sets, and find that with the maximum amount of data, the discriminative model does better on 9 out of 15 data sets, but with only a small amount of data, the generative model does better on 14 out of 15 data sets.

20.2.4 Maximum-likelihood parameter learning: Continuous models Continuous probability models such as the linear–Gaussian model were shown on page 422. Because continuous variables are ubiquitous in real-world applications, it is important

to know how to learn the parameters of continuous models from data. The principles for maximum-likelihood learning are identical in the continuous and discrete cases.

Let us begin with a very simple case: learning the parameters of a Gaussian density function on a single variable. That is, we assume the data are generated as follows:

⎛

⎞

⎜ ⎜

⎟ ⎟

⎝

⎠

P x

=

1

σ √2 π

e

( x−μ ) −

σ

2 2

2

.

The parameters of this model are the mean μ and the standard deviation σ. (Notice that the normalizing “constant” depends on σ, so we cannot ignore it.) Let the observed values be

x1 , … , xN . Then the log likelihood is N

1

j=1

σ √2 π

L = ∑ log

e−

( xj −μ ) σ

2 2

2

=

N

⎛ ⎜− ⎜

log √2

π − log σ

⎟ ⎟ ⎠

⎝

N

⎞

−∑

(x j − μ) 2

j=1

σ2

2

.

Setting the derivatives to zero as usual, we obtain

(20.4)

L ∂μ

∂

∂

L σ

∂

=

=

−

−

1

σ2 N σ

∑Nj

=1

+

1

σ3

( xj − μ ) = 0

∑Nj

=1

( xj − μ )

⇒

2

=0

⇒

∑j xj μ= N σ=

   ⎷

∑j(xj

−

N

μ)

2

.

That is, the maximum-likelihood value of the mean is the sample average and the maximum-likelihood value of the standard deviation is the square root of the sample variance. Again, these are comforting results that confirm “commonsense” practice. Now consider a linear–Gaussian model with one continuous parent X and a continuous

Y . As explained on page 422, Y has a Gaussian distribution whose mean depends linearly on the value of X and whose standard deviation is fixed. To learn the conditional distribution P (Y |X), we can maximize the conditional likelihood child

(20.5)

⎛

∣

P yx ∣ ⎜ ∣ ⎜ ∣ ⎝ ∣

(y (θ x θ )) −

⎞

1

⎟ = ⎟

σ π √2

⎠

e

+

1

− 2

σ

2

2

2

.

Here, the parameters are θ , θ , and σ. The data are a collection of (xj , yj ) pairs, as 1

2

illustrated in Figure 20.4 . Using the usual methods (Exercise 20.LINR), we can find the maximum-likelihood values of the parameters. The point here is different. If we consider just the parameters θ and θ that define the linear relationship between x and y, it becomes 1

2

clear that maximizing the log likelihood with respect to these parameters is the same as minimizing the numerator (y − (θ

1

x θ )) +

2

2

in the exponent of Equation (20.5) . This is

the L loss, the squared error between the actual value y and the prediction θ 2

1

x θ. +

2

Figure 20.4

(a) A linear–Gaussian model described as y

=

θx θ 1

+

2

plus Gaussian noise with fixed variance. (b) A set

of 50 data points generated from this model and the best-fit line.

This is the quantity minimized by the standard linear regression procedure described in Section 19.6 . Now we can understand why: minimizing the sum of squared errors gives the maximum-likelihood straight-line model, provided that the data are generated with Gaussian noise of fixed variance.

20.2.5 Bayesian parameter learning Maximum-likelihood learning gives rise to simple procedures, but it has serious deficiencies with small data sets. For example, after seeing one cherry candy, the maximum-likelihood hypothesis is that the bag is 100% cherry (i.e., θ

= 1.0

). Unless one’s hypothesis prior is that

bags must be either all cherry or all lime, this is not a reasonable conclusion. It is more likely that the bag is a mixture of lime and cherry. The Bayesian approach to parameter learning starts with a hypothesis prior and updates the distribution as data arrive. The candy example in Figure 20.2(a)  has one parameter, : the probability that a randomly

θ

selected piece of candy is cherry-flavored. In the Bayesian view,

θ is the (unknown) value of

a random variable Θ that defines the hypothesis space; the hypothesis prior is the prior distribution over

P(Θ). Thus, P (Θ = θ) is the prior probability that the bag has a fraction θ

of cherry candies.

If the parameter

θ can be any value between 0 and 1, then P(Θ) is a continuous probability

density function (see Section A.3 ). If we don’t know anything about the possible values of

θ we can use the uniform density function P (θ) = Uniform(θ; 0, 1), which says all values are equally likely.

A more flexible family of probability density functions is known as the beta distributions. Each beta distribution is defined by two hyperparameters3 

a and b such that

(20.6)

Beta(θ; a, b) = α θa

−1

(1 − θ )

b−1

,

3 They are called hyperparameters because they parameterize a

θ

distribution over , which is itself a parameter.

Beta distribution

Hyperparameter

θ in the range [0, 1]. The normalization constant α, which makes the distribution integrate to 1, depends on a and b. Figure 20.5  shows what the distribution looks like for various values of a and b. The mean value of the beta distribution is a/(a + b), so larger values of a suggest a belief that Θ is closer to 1 than to 0. Larger values of a + b make the for

distribution more peaked, suggesting greater certainty about the value of Θ. It turns out that the uniform density function is the same as

Beta(1, 1): the mean is 1/2, and the distribution

is flat.

Figure 20.5

Examples of the

Beta(a, b) distribution for different values of (a, b).

Besides its flexibility, the beta family has another wonderful property: if Θ has a prior

Beta(a, b), then, after a data point is observed, the posterior distribution for Θ is also a beta distribution. In other words, Beta is closed under update. The beta family is called the conjugate prior for the family of distributions for a Boolean variable.4 Let’s see how this works. Suppose we observe a cherry candy; then we have

4 Other conjugate priors include the Dirichlet family for the parameters of a discrete multivalued distribution and the Normal– Wishart family for the parameters of a Gaussian distribution. See Bernardo and Smith (1994).

P ( θ|D

1

=

cherry) = α P (D = cherry|θ)P (θ) = α θ ⋅ Beta(θ; a, b) = α θ ⋅ θa 1

′

=

α θ a (1 − θ ) ′

′

b−1

=

−1

(1 − θ )

b−1

α  Beta(θ; a + 1, b) . ′

Conjugate prior

a parameter to get the posterior; similarly, after seeing a lime candy, we increment the b parameter. Thus, we can view the a and b hyperparameters as virtual counts, in the sense that a prior Beta(a, b) behaves exactly as if we had started out with a uniform prior Beta(1, 1) and seen a − 1 actual cherry candies and b − 1 actual lime candies. Thus, after seeing a cherry candy, we simply increment the

Virtual count

By examining a sequence of beta distributions for increasing values of

a and b, keeping the

proportions fixed, we can see vividly how the posterior distribution over the parameter Θ changes as data arrive. For example, suppose the actual bag of candy is 75% cherry. Figure 20.5(b)  shows the sequence

Beta(3, 1), Beta(6, 2), Beta(30,10). Clearly, the distribution is

converging to a narrow peak around the true value of Θ. For large data sets, then, Bayesian learning (at least in this case) converges to the same answer as maximum-likelihood learning. Now let us consider a more complicated case. The network in Figure 20.2(b)  has three

θ θ , and θ , where θ

parameters, ,

θ

2

1

2

1

is the probability of a red wrapper on a cherry candy and

is the probability of a red wrapper on a lime candy. The Bayesian hypothesis prior must

cover all three parameters—that is, we need to specify

P(Θ, Θ , Θ ). Usually, we assume 1

parameter independence:

P(Θ, Θ , Θ ) = P(Θ)P(Θ )P(Θ ) . 1

Parameter independence

2

1

2

2

With this assumption, each parameter can have its own beta distribution that is updated separately as data arrive. Figure 20.6  shows how we can incorporate the hypothesis prior and any data into a Bayesian network, in which we have a node for each parameter variable.

Figure 20.6

A Bayesian network that corresponds to a Bayesian learning process. Posterior distributions for the parameter variables Θ, Θ1 , and Θ2 can be inferred from their prior distributions and the evidence in

Flavori and Wrapperi .

The nodes Θ, Θ1 , Θ2 have no parents. For the ith observation of a wrapper and corresponding flavor of a piece of candy, we add nodes Wrapperi and Flavori . Flavori is dependent on the flavor parameter Θ:

P (Flavori = cherry|Θ = θ) = θ . Wrapperi is dependent on Θ1 and Θ2 :

P (Wrapperi = red|Flavori = cherry, Θ1 = θ1 ) = θ1 P (Wrapperi = red|Flavori = lime, Θ2 = θ2 ) = θ2 . Now, the entire Bayesian learning process for the original Bayes net in Figure 20.2(b)  can be formulated as an inference problem in the derived Bayes net shown in Figure 20.6 , where the data and parameters become nodes. Once we have added all the new evidence nodes, we can then query the parameter variables (in this case, Θ, Θ1 , Θ2 ). Under this formulation there is just one learning algorithm—the inference algorithm for Bayesian networks. Of course, the nature of these networks is somewhat different from those of Chapter 13  because of the potentially huge number of evidence variables representing the training set and the prevalence of continuous-valued parameter variables. Exact inference may be impossible except in very simple cases such as the naive Bayes model. Practitioners typically use approximate inference methods such as MCMC (Section 13.4.2 ); many statistical software packages incorporate efficient implementations of MCMC for this purpose.

20.2.6 Bayesian linear regression Here we illustrate how to apply a Bayesian approach to a standard statistical task: linear regression. The conventional approach was described in Section 19.6  as minimizing the sum of squared errors and reinterpreted in Section 20.2.4  as maximizing likelihood assuming a Gaussian error model. These produce a single best hypothesis: a straight line with specific values for the slope and intercept and a fixed variance for the prediction error at any given point. There is no measure of how confident one should be in the slope and intercept values.

Furthermore, if one is predicting a value for an unseen data point far from the observed data points, it seems to make no sense to assume a prediction error that is the same as the prediction error for a data point right next to an observed data point. It would seem more sensible for the prediction error to be larger, the farther the data point is from the observed data, because a small change in the slope will cause a large change in the predicted value for a distant point.

The Bayesian approach fixes both of these problems. The general idea, as in the preceding section, is to place a prior on the model parameters—here, the coefficients of the linear

model and the noise variance—and then to compute the parameter posterior given the data. For multivariate data and unknown noise model, this leads to rather a lot of linear algebra, so we focus on a simple case: univariable data, a model that is constrained to go through the origin, and known noise: a normal distribution with variance parameter

θ and the model is

σ . Then we have just one 2

(20.7) ⎛

∣

⎞

∣

P yx θ ⎜ ∣ ⎜ ⎜ ∣ ⎜ ⎜ ∣ ∣ ⎝ ∣

,

⎛

⎟ ⎟ ⎟ = ⎟ ⎟

⎛

⎞

1

N y θx σy ⎜ ⎜ ⎜ ; ⎜ ⎜

⎠

,

⎟ 2⎟ ⎟ = ⎟ ⎟

⎝

1

σ π √2

e

− 2

(y θx)

2

−

⎜ ⎜

σ

⎝

⎞ ⎟ ⎟

2

⎠

.

⎠

θ

As the log likelihood is quadratic in , the appropriate form for a conjugate prior on a Gaussian. This ensures that the posterior for

θ

0

and variance

σ

2 0

θ is also

θ will also be Gaussian. We’ll assume a mean

for the prior, so that

(20.8) ⎛ ⎞

P θ ⎜ ⎜ ⎜ ⎜ ⎜

⎟ ⎟ ⎟ = ⎟ ⎟

⎝ ⎠

⎛

N θθ σ ⎜ ⎜ ⎜ ; ⎜ ⎜ ⎝

⎛

⎞

0,

1

⎟

2⎟

0⎟

⎟ ⎟

=

σ

1

0

√2

π

e

− 2

⎜ ⎜ ⎝

(θ θ ) −

0

σ

2

0

2

⎞ ⎟ ⎟ ⎠

.

⎠

Uninformative prior

θ to expect, or one might be completely agnostic. In the latter case, it makes sense to choose θ to be 0 and σ to be large—a so-called uninformative prior. Finally, we can assume a prior P x for the x-value of each data point, but this is completely immaterial to the analysis because it doesn’t depend on θ. Depending on the data being modeled, one might have some idea of what sort of slope

0

2

0

(

)

θ using Equation (20.1) : x y xN yN , so the

Now the setup is complete, so we can compute the posterior for

P θd ( |

) ∝

P d θ P θ . The observed data points are d (

| )

( )

= (

1,

1 ),

…,(

,

)

likelihood for the data is obtained from Equation (20.7)  as follows:

P dθ (

| )

=

(

∏i P xi ∏i P yi xi θ (

))

αe ∑i( 1

−

=

2

(

(

yi θxi σ −

)

2

2

)

|

,

α∏e

)) =

1

− 2

(

(

yi θxi σ −

2

)

2

)

i

,

x

where we have absorbed the -value priors and the normalizing constants for the Gaussians into a constant

N

α that is independent of θ. Now we combine this and the

parameter prior from Equation (20.8)  to obtain the posterior:

⎛

∣ ∣

P θd ⎜ ∣ ⎜ ⎜ ∣ ⎜ ⎜ ∣ ∣ ⎝ ∣

⎛

⎞

1

⎟ ⎟ ⎟ = ⎟ ⎟

αe

− 2

⎜ ⎜ ⎝

′′

(θ θ ) −

σ

2

0

⎟ ⎟

2 0

∑i

⎞

⎠

1

e

− 2

⎛ ⎜ ⎜

(yi θxi )

2

⎞

−

σ

⎝

⎟ ⎟

2

⎠

.

⎠

θ

Although this looks complicated, each exponent is a quadratic function of , so the sum of the two exponents is as well. Hence, the whole expression represents a Gaussian

θ

distribution for . Using algebraic manipulations very similar to those in Section 14.4 , we find

⎛

∣ ∣

P θd ⎜ ∣ ⎜ ⎜ ∣ ⎜ ⎜ ∣ ∣ ⎝ ∣

⎛

⎞ ⎟ ⎟ ⎟ = ⎟ ⎟

1

αe

− 2

′′′

⎜ ⎜

(θ θ N ) −

σN 2

⎝

2

⎞ ⎟ ⎟ ⎠

⎠

with “updated” mean and variance given by

θN

σθ σ σ σ 2

0

=

2

+

+

2 0

2

0

∑i xiyi ∑i xi 2

 and 

σN 2

σσ σ σ ∑ i xi 2

2

=

0

2

+

2

2

.

0

Let’s look at these formulas to see what they mean. When the data are narrowly

x

concentrated on a small region of the -axis near the origin, posterior variance

∑i xi will be small and the 2

σN will be large, roughly equal to the prior variance σ . This is as one 2

2

0

would expect: the data do little to constrain the rotation of the line around the origin. Conversely, when the data are widely spread along the axis,

∑i xi will be large and the 2

posterior variance

σN will be small, roughly equal to σ 2

2

/

(∑i

xi ), so the slope will be very 2

tightly constrained.

To make a prediction at a specific data point, we have to integrate over the possible values

θ

of , as suggested by Equation (20.2) :

P yx d ( |

,

)

=

∞

∫

P y x d θ P θ x d dθ ( |

,

,

)

( |

,

)

P y x θ P θ d dθ ( |

,

)

( |

)

−∞

−∞

α∫ e ∞

=

∞

∫

=

1

− 2

(

y θx σ

( −

)

2

2

)

e

(

1

−

(

θ θN ) σN −

2

)

2

2

dθ

−∞

θ

Again, the sum of the two exponents is a quadratic function of , so we have a Gaussian over

θ whose integral is 1. The remaining terms in y form another Gaussian: ⎛

∣ ∣

P yx d ⎜ ∣ ⎜ ⎜ ∣ ⎜ ⎜ ∣ ∣ ⎝ ∣

⎛

⎞

,

⎟ ⎟ ⎟ ∝ ⎟ ⎟

1

e

− 2

⎜ ⎜ ⎝

( y θ N x)

2

−

σ σN x 2

+

2

2

⎞ ⎟ ⎟ ⎠

.

⎠

y is θN x, that is, it is based on the posterior mean for θ. The variance of the prediction is given by the model noise σ plus a term proportional to x , which means that the standard deviation of the prediction Looking at this expression, we see that the mean prediction for

2

2

increases asymptotically linearly with the distance from the origin. Figure 20.7  illustrates this phenomenon. As noted at the beginning of this section, having greater uncertainty for predictions that are further from the observed data points makes perfect sense.

Figure 20.7

Bayesian linear regression with a model constrained to pass through the origin and fixed noise variance 2 = 0.2. Contours at ±1, ±2, and ±3 standard deviations are shown for the predictive density. (a) With

σ

σ

2 three data points near the origin, the slope is quite uncertain, with N ≈ 0.3861. Notice how the uncertainty increases with distance from the observed data points. (b) With two additional data points

θ

σN σ.

further away, the slope is very tightly constrained, with predictive density is almost entirely due to the fixed noise

≈ 0.0286.

2

The remaining variance in the

2

20.2.7 Learning Bayes net structures So far, we have assumed that the structure of the Bayes net is given and we are just trying to learn the parameters. The structure of the network represents basic causal knowledge about the domain that is often easy for an expert, or even a naive user, to supply. In some cases, however, the causal model may be unavailable or subject to dispute—for example, certain corporations have long claimed that smoking does not cause cancer and other corporations assert that CO2 concentrations have no effect on climate—so it is important to understand how the structure of a Bayes net can be learned from data. This section gives a brief sketch of the main ideas.

The most obvious approach is to search for a good model. We can start with a model containing no links and begin adding parents for each node, fitting the parameters with the methods we have just covered and measuring the accuracy of the resulting model. Alternatively, we can start with an initial guess at the structure and use hill climbing or simulated annealing search to make modifications, retuning the parameters after each change in the structure. Modifications can include reversing, adding, or deleting links. We must not introduce cycles in the process, so many algorithms assume that an ordering is given for the variables, and that a node can have parents only among those nodes that come earlier in the ordering (just as in the construction process in Chapter 13 ). For full generality, we also need to search over possible orderings.

There are two alternative methods for deciding when a good structure has been found. The first is to test whether the conditional independence assertions implicit in the structure are actually satisfied in the data. For example, the use of a naive Bayes model for the restaurant problem assumes that

P(Hungry, Bar|WillWait) = P(Hungry|WillWait) P(Bar|WillWait) and we can check in the data whether the same equation holds between the corresponding conditional frequencies. But even if the structure describes the true causal nature of the domain, statistical fluctuations in the data set mean that the equation will never be satisfied exactly, so we need to perform a suitable statistical test to see if there is sufficient evidence that the independence hypothesis is violated. The complexity of the resulting network will depend on the threshold used for this test—the stricter the independence test, the more links will be added and the greater the danger of overfitting.

An approach more consistent with the ideas in this chapter is to assess the degree to which the proposed model explains the data (in a probabilistic sense). We must be careful how we measure this, however. If we just try to find the maximum-likelihood hypothesis, we will end up with a fully connected network, because adding more parents to a node cannot decrease the likelihood (Exercise 20.MLPA). We are forced to penalize model complexity in some way. The MAP (or MDL) approach simply subtracts a penalty from the likelihood of each structure (after parameter tuning) before comparing different structures. The Bayesian approach places a joint prior over structures and parameters. There are usually far too many structures to sum over (superexponential in the number of variables), so most practitioners use MCMC to sample over structures.

Penalizing complexity (whether by MAP or Bayesian methods) introduces an important connection between the optimal structure and the nature of the representation for the conditional distributions in the network. With tabular distributions, the complexity penalty for a node’s distribution grows exponentially with the number of parents, but with, say, noisy-OR distributions, it grows only linearly. This means that learning with noisy-OR (or other compactly parameterized) models tends to produce learned structures with more parents than does learning with tabular distributions.

20.2.8 Density estimation with nonparametric models

It is possible to learn a probability model without making any assumptions about its structure and parameterization by adopting the nonparametric methods of Section 19.7 . The task of nonparametric density estimation is typically done in continuous domains, such as that shown in Figure 20.8(a) . The figure shows a probability density function on a space defined by two continuous variables. In Figure 20.8(b)  we see a sample of data points from this density function. The question is, can we recover the model from the samples?

Figure 20.8

(a) A 3D plot of the mixture of Gaussians from Figure 20.12(a) . (b) A 128-point sample of points from the mixture, together with two query points (small orange squares) and their 10-nearest-neighborhoods (large circle and smaller circle to the right).

Nonparametric density estimation

First we will consider k-nearest-neighbors models. (In Chapter 19  we saw nearestneighbor models for classification and regression; here we see them for density estimation.) Given a sample of data points, to estimate the unknown probability density at a query point

x we can simply measure the density of the data points in the neighborhood of x. Figure

20.8(b)  shows two query points (small squares). For each query point we have drawn the

smallest circle that encloses 10 neighbors—the 10-nearest-neighborhood. We can see that the central circle is large, meaning there is a low density there, and the circle on the right is

small, meaning there is a high density there. In Figure 20.9  we show three plots of density

k

k

estimation using -nearest-neighbors, for different values of . It seems clear that (b) is

k

k

about right, while (a) is too spiky ( is too small) and (c) is too smooth ( is too big).

Figure 20.9

k

k

Density estimation using -nearest-neighbors, applied to the data in Figure 20.8(b) , for = 3, 10, and 40 respectively. = 3 is too spiky, 40 is too smooth, and 10 is just about right. The best value for can be

k

k

chosen by cross-validation.

Another possibility is to use kernel functions, as we did for locally weighted regression. To apply a kernel model to density estimation, assume that each data point generates its own little density function. For example, we might use spherical Gaussians with standard deviation

w along each axis. Then estimated density at a query point x is the average of the

data kernels:

P

⎛ ⎜ ⎜ ⎝

⎞

= x⎟ ⎟ ⎠

1

N

⎞ N ⎛ x, xj ⎟ ∑ K⎜ ⎟ ⎜ j ⎠ ⎝ =1

where

K

⎛ ⎜ ⎜ ⎝

⎞

= x, xj ⎟ ⎟ ⎠

1

(w

2√

2

π)

de

D( x xj )

2

,

− 2

w

2

,

d is the number of dimensions in x and D is the Euclidean distance function. We still have the problem of choosing a suitable value for kernel width w; Figure 20.10  shows values that are too small, just right, and too large. A good value of w can be chosen by using where

cross-validation.

Figure 20.10

Density estimation using kernels for the data in Figure 20.8(b) , using Gaussian kernels with 0.07, and 0.20 respectively. = 0.07 is about right.

w

w

= 0.02,

20.3 Learning with Hidden Variables: The EM Algorithm

Latent variable

The preceding section dealt with the fully observable case. Many real-world problems have hidden variables (sometimes called latent variables), which are not observable in the data. For example, medical records often include the observed symptoms, the physician’s diagnosis, the treatment applied, and perhaps the outcome of the treatment, but they seldom contain a direct observation of the disease itself! (Note that the diagnosis is not the disease; it is a causal consequence of the observed symptoms, which are in turn caused by the disease.) One might ask, “If the disease is not observed, could we construct a model based only on the observed variables?” The answer appears in Figure 20.11 , which shows a small, fictitious diagnostic model for heart disease. There are three observable predisposing factors and three observable symptoms (which are too depressing to name). Assume that each variable has three possible values (e.g., none, moderate, and severe). Removing the hidden variable from the network in (a) yields the network in (b); the total number of parameters increases from 78 to 708. Thus, latent variables can dramatically reduce the number of parameters required to specify a Bayesian network. This, in turn, can dramatically reduce the amount of data needed to learn the parameters.

Figure 20.11

(a) A simple diagnostic network for heart disease, which is assumed to be a hidden variable. Each variable has three possible values and is labeled with the number of independent parameters in its conditional distribution; the total number is 78. (b) The equivalent network with removed.

HeartDisease

Note that the symptom variables are no longer conditionally independent given their parents. This network requires 708 parameters.

Hidden variables are important, but they do complicate the learning problem. In Figure 20.11(a) , for example, it is not obvious how to learn the conditional distribution for

HeartDisease, given its parents, because we do not know the value of HeartDisease in each case; the same problem arises in learning the distributions for the symptoms. This section describes an algorithm called expectation–maximization, or EM, that solves this problem in a very general way. We will show three examples and then provide a general description. The algorithm seems like magic at first, but once the intuition has been developed, one can find applications for EM in a huge range of learning problems.

Expectation–maximization

20.3.1 Unsupervised clustering: Learning mixtures of Gaussians Unsupervised clustering is the problem of discerning multiple categories in a collection of objects. The problem is unsupervised because the category labels are not given. For example, suppose we record the spectra of a hundred thousand stars; are there different types of stars revealed by the spectra, and, if so, how many types and what are their characteristics? We are all familiar with terms such as “red giant” and “white dwarf,” but the

stars do not carry these labels on their hats—astronomers had to perform unsupervised clustering to identify these categories. Other examples include the identification of species, genera, orders, phylum, and so on in the Linnaean taxonomy and the creation of natural kinds for ordinary objects (see Chapter 10 ).

Unsupervised clustering

Unsupervised clustering begins with data. Figure 20.12(b)  shows 500 data points, each of which specifies the values of two continuous attributes. The data points might correspond to stars, and the attributes might correspond to spectral intensities at two particular frequencies. Next, we need to understand what kind of probability distribution might have generated the data. Clustering presumes that the data are generated from a mixture distribution,

P . Such a distribution has k components, each of which is a distribution in its

own right. A data point is generated by first choosing a component and then generating a sample from that component. Let the random variable 1, … ,

k; then the mixture distribution is given by P (x)

∑ P (C k

=

i

=1

C denote the component, with values

i ) P (x C i ) ∣

=

∣ ∣ ∣

=

,

Mixture distribution

Component

where x refers to the values of the attributes for a data point. For continuous data, a natural choice for the component distributions is the multivariate Gaussian, which gives the socalled mixture of Gaussians family of distributions. The parameters of a mixture of

wi = P (C = i) (the weight of each component), μi (the mean of each component), and Σi (the covariance of each component). Figure 20.12(a)  shows a mixture Gaussians are

of three Gaussians; this mixture is in fact the source of the data in (b) as well as being the model shown in Figure 20.8(a)  on page 736.

Figure 20.12

(a) A Gaussian mixture model with three components; the weights (left-to-right) are 0.2, 0.3, and 0.5. (b) 500 data points sampled from the model in (a). (c) The model reconstructed by EM from the data in (b).

The unsupervised clustering problem, then, is to recover a Gaussian mixture model like the one in Figure 20.12(a)  from raw data like that in Figure 20.12(b) . Clearly, if we knew which component generated each data point, then it would be easy to recover the component Gaussians: we could just select all the data points from a given component and then apply (a multivariate version of) Equation (20.4)  (page 729) for fitting the parameters of a Gaussian to a set of data. On the other hand, if we knew the parameters of each component, then we could, at least in a probabilistic sense, assign each data point to a component.

The problem is that we know neither the assignments nor the parameters. The basic idea of EM in this context is to pretend that we know the parameters of the model and then to infer the probability that each data point belongs to each component. After that, we refit the components to the data, where each component is fitted to the entire data set with each point weighted by the probability that it belongs to that component. The process iterates until convergence. Essentially, we are “completing” the data by inferring probability distributions over the hidden variables—which component each data point belongs to—

based on the current model. For the mixture of Gaussians, we initialize the mixture-model parameters arbitrarily and then iterate the following two steps:

p

P (C = i∣xj ), the probability that datum xj was generated by component i. By Bayes’ rule, we have pij = αP (xj ∣C = i)P (C = i) . The term P (xj ∣C = i) is just the probability at xj of the ith Gaussian, and the term P (C = i) is just the weight parameter for the ith Gaussian. Define ni = ∑j pij , the effective number of data points currently assigned to component i.

1. E-STEP: Compute the probabilities ij =

2. M-STEP: Compute the new mean, covariance, and component weights using the following steps in sequence:

μi ← ∑j pij xj /ni ⊤

pij (xj − μi )(xj − μi ) /ni wi ← n i /N

Σi ← ∑j

N is the total number of data points. The E-step, or expectation step, can be viewed as computing the expected values pij of the hidden indicator variables Zij , where Zij is 1 if datum xj was generated by the ith component and 0 otherwise. The M-step, or where

maximization step, finds the new values of the parameters that maximize the log likelihood of the data, given the expected values of the hidden indicator variables.

Indicator variable

The final model that EM learns when it is applied to the data in Figure 20.12(a)  is shown in Figure 20.12(c) ; it is virtually indistinguishable from the original model from which the data were generated (horizontal line). Figure 20.13(a)  plots the log likelihood of the data according to the current model as EM progresses.

Figure 20.13

Graphs showing the log likelihood of the data, L, as a function of the EM iteration. The horizontal line shows the log likelihood according to the true model. (a) Graph for the Gaussian mixture model in Figure

20.12 . (b) Graph for the Bayesian network in Figure 20.14(a) .

There are two points to notice. First, the log likelihood for the final learned model slightly exceeds that of the original model, from which the data were generated. This might seem surprising, but it simply reflects the fact that the data were generated randomly and might not provide an exact reflection of the underlying model. The second point is that EM increases the log likelihood of the data at every iteration. This fact can be proved in general. Furthermore, under certain conditions (that hold in most cases), EM can be proven to reach a local maximum in likelihood. (In rare cases, it could reach a saddle point or even a local minimum.) In this sense, EM resembles a gradient-based hill-climbing algorithm, but notice that it has no “step size” parameter. Things do not always go as well as Figure 20.13(a)  might suggest. It can happen, for example, that one Gaussian component shrinks so that it covers just a single data point. Then its variance will go to zero and its likelihood will go to infinity! If we don’t know how many components are in the mixture we have to try different values of k and see which is best; that can be a source of error. Another problem is that two components can “merge,” acquiring identical means and variances and sharing their data points. These kinds of degenerate local maxima are serious problems, especially in high dimensions. One solution is to place priors on the model parameters and to apply the MAP version of EM. Another is to restart a component with new random parameters if it gets too small or too close to another component. Sensible initialization also helps.

20.3.2 Learning Bayes net parameter values for hidden variables

To learn a Bayesian network with hidden variables, we apply the same insights that worked for mixtures of Gaussians. Figure 20.14(a)  represents a situation in which there are two bags of candy that have been mixed together. Candies are described by three features: in addition to the

Flavor and the Wrapper, some candies have a Hole in the middle and some

do not. The distribution of candies in each bag is described by a naive Bayes model: the features are independent, given the bag, but the conditional probability distribution for each feature depends on the bag. The parameters are as follows:

θ

θ

θ is the prior probability that a

candy comes from Bag 1; F 1 and F 2 are the probabilities that the flavor is cherry, given that

θ

θ

the candy comes from Bag 1 or Bag 2 respectively; W 1 and W 2 give the probabilities that

θ

θ

the wrapper is red; and H 1 and H 2 give the probabilities that the candy has a hole.

Figure 20.14

(a) A mixture model for candy. The proportions of different flavors, wrappers, and presence of holes depend on the bag, which is not observed. (b) Bayesian network for a Gaussian mixture. The mean and covariance of the observable variables depend on the component .

X

C

The overall model is a mixture model: a weighted sum of two different distributions, each of which is a product of independent, univariate distributions. (In fact, we can also model the mixture of Gaussians as a Bayesian network, as shown in Figure 20.14(b) .) In the figure, the bag is a hidden variable because, once the candies have been mixed together, we no longer know which bag each candy came from. In such a case, can we recover the descriptions of the two bags by observing candies from the mixture? Let us work through an

iteration of EM for this problem. First, let’s look at the data. We generated 1000 samples from a model whose true parameters are as follows:

(20.9)

θ

= 0.5,  

θF

=

1

θW

=

1

θH

1

= 0.8,  

θF

2

=

θW

=

2

θH

2

= 0.3 .

That is, the candies are equally likely to come from either bag; the first is mostly cherry with red wrappers and holes; the second is mostly lime with green wrappers and no holes. The counts for the eight possible kinds of candy are as follows:

We start by initializing the parameters. For numerical simplicity, we arbitrarily choose5

5 It is better in practice to choose them randomly, to avoid local maxima due to symmetry.

(20.10)

θ

(0)

= 0.6,  

First, let us work on the

θF

(0) 1

=

θW

(0) 1

=

θH

(0) 1

= 0.6,  

θF

(0) 2

=

θW

(0) 2

=

θH

(0) 2

= 0.4 .

θ parameter. In the fully observable case, we would estimate this

directly from the observed counts of candies from bags 1 and 2. Because the bag is a hidden variable, we calculate the expected counts instead. The expected count

N (Bag ˆ

= 1

) is the

sum, over all candies, of the probability that the candy came from bag 1:

θ

(1)

=

N (Bag ˆ

= 1

)N /

N

=

∣

∑ P (Bag = 1∣∣flavorj , wrapperj , holesj )/N . j

=1

∣

These probabilities can be computed by any inference algorithm for Bayesian networks. For a naive Bayes model such as the one in our example, we can do the inference “by hand,” using Bayes’ rule and applying conditional independence:

θ

(1)

=

1

N

N

∑ j=1

P (flavorj Bag )P (wrapperj Bag )P (holesj Bag )P (Bag ) . ∑i P (flavorj Bag i)P (wrapperj Bag i)P (holesj Bag i)P (Bag i) ∣

∣

=1

∣

∣

=

∣

=1

=1

∣

=

=1

=

=

Applying this formula to, say, the 273 red-wrapped cherry candies with holes, we get a contribution of

θF θW θH θ ⋅ 1000 θ F θ W θ H θ + θ F θ W θ H (1 − θ (0) (0)

273

1

(0)

(0) (0) 1

1

1

(0)

(0)

1

(0)

1

(0) (0) 2

(0)

2

2

(0)

)

≈ 0.22797 .

Continuing with the other seven kinds of candy in the table of counts, we obtain

θ

(1)

= 0.6124.

θF

Now let us consider the other parameters, such as

1

. In the fully observable case, we

would estimate this directly from the observed counts of cherry and lime candies from bag 1. The expected count of cherry candies from bag 1 is given by

∑

P (Bag = 1|Flavorj = cherry, wrapperj , holesj ) .

j:Flavorj =cherry

Again, these probabilities can be calculated by any Bayes net algorithm. Completing this process, we obtain the new values of all the parameters:

(20.11)

θ θF

(1) (1) 2

θF = 0.3887,  θW

= 0.6124,  

(1) 1

(1) 2

= 0.6684, = 0.3817,  

θW

(1)

θH

(1) 2

1

= 0.6483,  

θH

(1) 1

= 0.6558,

= 0.3827 .

The log likelihood of the data increases from about −2044 initially to about −2021 after the first iteration, as shown in Figure 20.13(b) . That is, the update improves the likelihood

e ≈ 10 . By the tenth iteration, the learned model is a better fit than the original model (L = −1982.214). Thereafter, progress becomes very slow. This is itself by a factor of about

23

10

not uncommon with EM, and many practical systems combine EM with a gradient-based algorithm such as Newton–Raphson (see Chapter 4 ) for the last phase of learning.

The general lesson from this example is that the parameter updates for Bayesian network learning with hidden variables are directly available from the results of inference on each example. Moreover, only local posterior probabilities are needed for each parameter. Here, “local” means

Xi can be learned from posterior probabilities involving just Xi and its parents Ui . Defining θijk to be the CPT parameter P (Xi xij Ui uik ), the update is given by the normalized expected counts as that the conditional probability table (CPT) for each variable

∣

=

=

follows:

θijk

N (Xi xij Ui uik ) N (Ui uik ) ˆ

←

=

,

/ ˆ

=

=

.

The expected counts are obtained by summing over the examples, computing the probabilities

P (Xi xij Ui uik ) for each by using any Bayes net inference algorithm. For =

,

=

the exact algorithms—including variable elimination—all these probabilities are obtainable directly as a by-product of standard inference, with no need for extra computations specific to learning. Moreover, the information needed for learning is available locally for each parameter.

Standing back a little, we can think about what the EM algorithm is doing in this example as

θ θF θW θH θF θW θH ) from seven (

recovering seven parameters (

, 

1,

 

1,

 

1,

 

2,

 

2,

 

2

2

3

− 1)

observed

counts in the data. (The eighth count is fixed by the fact that the counts sum to 1000.) If each candy were described by two attributes rather than three (say, omitting the holes), we

θ θF θW θF θW

would have had five parameters (

, 

1,

 

1,

 

2,

 

2

) but only three (22

counts. In such a case it is not possible to recover the mixture weight

− 1)

observed

θ or the characteristics

of the two bags that were mixed together. We say that the two-attribute model is not identifiable.

Identifiability

Identifiability in Bayesian networks is a tricky issue. Note that even with three attributes and seven counts, we cannot uniquely recover the model, because there are two observationally equivalent models with the

Bag variable flipped. Depending on how the parameters are

initialized, EM will converge either to a model where bag 1 has mostly cherry and bag 2 mostly lime, or vice versa. This kind if non-identifiability is unavoidable with variables that are never observed.

20.3.3 Learning hidden Markov models Our final application of EM involves learning the transition probabilities in hidden Markov models (HMMs). Recall from Section 14.3  that a hidden Markov model can be represented by a dynamic Bayes net with a single discrete state variable, as illustrated in Figure 20.15 . Each data point consists of an observation sequence of finite length, so the problem is to learn the transition probabilities from a set of observation sequences (or from just one long sequence).

Figure 20.15

An unrolled dynamic Bayesian network that represents a hidden Markov model (repeat of Figure

14.16 ).

We have already seen how to learn Bayes nets, but there is a complication: in Bayes nets, each parameter is distinct; in a hidden Markov model, on the other hand, the individual

j at time t, θijt = P (Xt+1 = j∣Xt = i), are repeated across time—that is, θijt = θij for all t. To estimate the transition probability from state i to state j, we simply calculate the expected proportion of times that the system undergoes a transition to state j when in state i: i

transition probabilities from state to state

θij ←

∑ Nˆ(X t

t+1 = j, Xt = i)/

∑ Nˆ(X = i) . t

t

The expected counts are computed by an HMM inference algorithm. The forward– backward algorithm shown in Figure 14.4  can be modified very easily to compute the necessary probabilities. One important point is that the probabilities required are obtained by smoothing rather than filtering. Filtering gives the probability distribution of the current state given the past, but smoothing gives the distribution given all evidence, including what

happens after a particular transition occurred. The evidence in a murder case is usually obtained after the crime (i.e., the transition from state i to state j) has taken place.

20.3.4 The general form of the EM algorithm We have seen several instances of the EM algorithm. Each involves computing expected values of hidden variables for each example and then recomputing the parameters, using the expected values as if they were observed values. Let x be all the observed values in all the examples, let Z denote all the hidden variables for all the examples, and let θ be all the parameters for the probability model. Then the EM algorithm is

θ(i+1) = argmax θ

∑ P (Z z

=

z | x , θ i ) L( x , Z = z | θ ) . ( )

This equation is the EM algorithm in a nutshell. The E-step is the computation of the summation, which is the expectation of the log likelihood of the “completed” data with respect to the distribution P (Z = z∣∣x, θ(i) ), which is the posterior over the hidden variables, given the data. The M-step is the maximization of this expected log likelihood with respect to the parameters. For mixtures of Gaussians, the hidden variables are the Zij s, where Zij is

1 if example j was generated by component i. For Bayes nets, Zij is the value of unobserved variable Xi in example j. For HMMs, Zjt is the state of the sequence in example j at time t. Starting from the general form, it is possible to derive an EM algorithm for a specific application once the appropriate hidden variables have been identified.

As soon as we understand the general idea of EM, it becomes easy to derive all sorts of variants and improvements. For example, in many cases the E-step—the computation of posteriors over the hidden variables—is intractable, as in large Bayes nets. It turns out that one can use an approximate E-step and still obtain an effective learning algorithm. With a sampling algorithm such as MCMC (see Section 13.4 ), the learning process is very intuitive: each state (configuration of hidden and observed variables) visited by MCMC is treated exactly as if it were a complete observation. Thus, the parameters can be updated directly after each MCMC transition. Other forms of approximate inference, such as variational methods and loopy belief propagation, have also proved effective for learning very large networks.

20.3.5 Learning Bayes net structures with hidden variables

In Section 20.2.7 , we discussed the problem of learning Bayes net structures with complete data. When unobserved variables influence observed data, things get more difficult. In the simplest case, a human expert might tell the learning algorithm that certain hidden variables exist, leaving it to the algorithm to find a place for them in the network structure. For example, an algorithm might try to learn the structure shown in Figure 20.11(a)  on page 738, given the information that

HeartDisease (a three-valued variable) should be included

in the model. As in the complete-data case, the overall algorithm has an outer loop that searches over structures and an inner loop that fits the network parameters given the structure.

If the learning algorithm is not told which hidden variables exist, then there are two choices: either pretend that the data are really complete—which may force the algorithm to learn a parameter-intensive model such as the one in Figure 20.11(b) —or invent new hidden variables in order to simplify the model. The latter approach can be implemented by including new modification choices in the structure search: in addition to modifying links, the algorithm can add or delete a hidden variable or change its arity. Of course, the algorithm will not know that the new variable it has invented is called

HeartDisease; nor

will it have meaningful names for the values. Fortunately, newly invented hidden variables will usually be connected to preexisting variables, so a human expert can often inspect the local conditional distributions involving the new variable and ascertain its meaning.

As in the complete-data case, pure maximum-likelihood structure learning will result in a completely connected network (moreover, one with no hidden variables), so some form of complexity penalty is required. We can also apply MCMC to sample many possible network structures, thereby approximating Bayesian learning. For example, we can learn mixtures of Gaussians with an unknown number of components by sampling over the number; the approximate posterior distribution for the number of Gaussians is given by the sampling frequencies of the MCMC process.

For the complete-data case, the inner loop to learn the parameters is very fast—just a matter of extracting conditional frequencies from the data set. When there are hidden variables, the inner loop may involve many iterations of EM or a gradient-based algorithm, and each iteration involves the calculation of posteriors in a Bayes net, which is itself an NP-hard problem. To date, this approach has proved impractical for learning complex models.

One possible improvement is the so-called structural EM algorithm, which operates in much the same way as ordinary (parametric) EM except that the algorithm can update the structure as well as the parameters. Just as ordinary EM uses the current parameters to compute the expected counts in the E-step and then applies those counts in the M-step to choose new parameters, structural EM uses the current structure to compute expected counts and then applies those counts in the M-step to evaluate the likelihood for potential new structures. (This contrasts with the outer-loop/inner-loop method, which computes new expected counts for each potential structure.) In this way, structural EM may make several structural alterations to the network without once recomputing the expected counts, and is capable of learning nontrivial Bayes net structures. Structural EM has a search space over the space of structures rather than the space of structures and parameters. Nonetheless, much work remains to be done before we can say that the structure-learning problem is solved.

Structural EM

Summary Statistical learning methods range from simple calculation of averages to the construction of complex models such as Bayesian networks. They have applications throughout computer science, engineering, computational biology, neuroscience, psychology, and physics. This chapter has presented some of the basic ideas and given a flavor of the mathematical underpinnings. The main points are as follows:

Bayesian learning methods formulate learning as a form of probabilistic inference, using the observations to update a prior distribution over hypotheses. This approach provides a good way to implement Ockham’s razor, but quickly becomes intractable for complex hypothesis spaces. Maximum a posteriori (MAP) learning selects a single most likely hypothesis given the data. The hypothesis prior is still used and the method is often more tractable than full Bayesian learning. Maximum-likelihood learning simply selects the hypothesis that maximizes the likelihood of the data; it is equivalent to MAP learning with a uniform prior. In simple cases such as linear regression and fully observable Bayesian networks, maximumlikelihood solutions can be found easily in closed form. Naive Bayes learning is a particularly effective technique that scales well. When some variables are hidden, local maximum likelihood solutions can be found using the expectation maximization (EM) algorithm. Applications include unsupervised clustering using mixtures of Gaussians, learning Bayesian networks, and learning hidden Markov models. Learning the structure of Bayesian networks is an example of model selection. This usually involves a discrete search in the space of structures. Some method is required for trading off model complexity against degree of fit. Nonparametric models represent a distribution using the collection of data points. Thus, the number of parameters grows with the training set. Nearest-neighbors methods look at the examples nearest to the point in question, whereas kernel methods form a distance-weighted combination of all the examples.

Statistical learning continues to be a very active area of research. Enormous strides have been made in both theory and practice, to the point where it is possible to learn almost any

model for which exact or approximate inference is feasible.

Bibliographical and Historical Notes The application of statistical learning techniques in AI was an active area of research in the early years (see Duda and Hart, 1973) but became separated from mainstream AI as the latter field concentrated on symbolic methods. A resurgence of interest occurred shortly after the introduction of Bayesian network models in the late 1980s; at roughly the same time, a statistical view of neural network learning began to emerge. In the late 1990s, there was a noticeable convergence of interests in machine learning, statistics, and neural networks, centered on methods for creating large probabilistic models from data.

The naive Bayes model is one of the oldest and simplest forms of Bayesian network, dating back to the 1950s. Its origins were mentioned in Chapter 12 . Its surprising success is partially explained by Domingos and Pazzani (1997). A boosted form of naive Bayes learning won the first KDD Cup data mining competition (Elkan, 1997). Heckerman (1998) gives an excellent introduction to the general problem of Bayes net learning. Bayesian parameter learning with Dirichlet priors for Bayesian networks was discussed by Spiegelhalter et al. (1993). The beta distribution as a conjugate prior for a Bernoulli variable was first derived by Thomas (Bayes, 1763) and later reintroduced by Karl Pearson (1895) as a model for skewed data; for many years it was known as a “Pearson Type I distribution.” Bayesian linear regression is discussed in the text by Box and Tiao (1973); Minka (2010) provides a concise summary of the derivations for the general multivariate case.

Several software packages incorporate mechanisms for statistical learning with Bayes net models. These include BUGS (Bayesian inference Using Gibbs Sampling) (Gilks et al., 1994; Lunn et al., 2000, 2013), JAGS (Just Another Gibbs Sampler) (Plummer, 2003), and STAN (Carpenter et al., 2017).

The first algorithms for learning Bayes net structures used conditional independence tests (Pearl, 1988; Pearl and Verma, 1991). Spirtes et al. (1993) implemented a comprehensive approach in the TETRAD package for Bayes net learning. Algorithmic improvements since then led to a clear victory in the 2001 KDD Cup data mining competition for a Bayes net learning method (Cheng et al., 2002). (The specific task here was a bioinformatics problem with 139,351 features!) A structure-learning approach based on maximizing likelihood was developed by Cooper and Herskovits (1992) and improved by Heckerman et al. (1994).

More recent algorithms have achieved quite respectable performance in the complete-data case (Moore and Wong, 2003; Teyssier and Koller, 2005). One important component is an efficient data structure, the AD-tree, for caching counts over all possible combinations of variables and values (Moore and Lee, 1997). Friedman and Goldszmidt (1996) pointed out the influence of the representation of local conditional distributions on the learned structure.

The general problem of learning probability models with hidden variables and missing data was addressed by Hartley (1958), who described the general idea of what was later called EM and gave several examples. Further impetus came from the Baum–Welch algorithm for HMM learning (Baum and Petrie, 1966), which is a special case of EM. The paper by Dempster, Laird, and Rubin (1977), which presented the EM algorithm in general form and analyzed its convergence, is one of the most cited papers in both computer science and statistics. (Dempster himself views EM as a schema rather than an algorithm, since a good deal of mathematical work may be required before it can be applied to a new family of distributions.) McLachlan and Krishnan (1997) devote an entire book to the algorithm and its properties. The specific problem of learning mixture models, including mixtures of Gaussians, is covered by Titterington et al. (1985).

Within AI, AUTOCLASS (Cheeseman et al., 1988; Cheeseman and Stutz, 1996) was the first successful system that used EM for mixture modeling. AUTOCLASS was applied to a number of real-world scientific classification tasks, including the discovery of new types of stars from spectral data (Goebel et al., 1989) and new classes of proteins and introns in DNA/protein sequence databases (Hunter and States, 1992).

For maximum-likelihood parameter learning in Bayes nets with hidden variables, EM and gradient-based methods were introduced around the same time by Lauritzen (1995) and Russell et al. (1995). The structural EM algorithm was developed by Friedman (1998) and applied to maximum-likelihood learning of Bayes net structures with latent variables. Friedman and Koller (2003) describe Bayesian structure learning. Daly et al. (2011) review the field of Bayes net learning, providing extensive citations to the literature.

The ability to learn the structure of Bayesian networks is closely connected to the issue of recovering causal information from data. That is, is it possible to learn Bayes nets in such a way that the recovered network structure indicates real causal influences? For many years,

statisticians avoided this question, believing that observational data (as opposed to data generated from experimental trials) could yield only correlational information—after all, any two variables that appear related might in fact be influenced by a third, unknown causal factor rather than influencing each other directly. Pearl (2000) has presented convincing arguments to the contrary, showing that there are in fact many cases where causality can be ascertained and developing the causal network formalism to express causes and the effects of intervention as well as ordinary conditional probabilities.

Nonparametric density estimation, also called Parzen window density estimation, was investigated initially by Rosenblatt (1956) and Parzen (1962). Since that time, a huge literature has developed investigating the properties of various estimators. Devroye (1987) gives a thorough introduction. There is also a rapidly growing literature on nonparametric Bayesian methods, originating with the seminal work of Ferguson (1973) on the Dirichlet process, which can be thought of as a distribution over Dirichlet distributions. These methods are particularly useful for mixtures with unknown numbers of components. Ghahramani (2005) and Jordan (2005) provide useful tutorials on the many applications of these ideas to statistical learning. The text by Rasmussen and Williams (2006) covers the Gaussian process, which gives a way of defining prior distributions over the space of continuous functions.

Dirichlet process

Gaussian process

The material in this chapter brings together work from the fields of statistics and pattern recognition, so the story has been told many times in many ways. Good texts on Bayesian statistics include those by DeGroot (1970), Berger (1985), and Gelman et al. (1995). Bishop (2007), Hastie et al. (2009), Barber (2012), and Murphy (2012) provide excellent introductions to statistical machine learning. For pattern classification, the classic text for many years has been Duda and Hart (1973), now updated (Duda et al., 2001). The annual

NeurIPS (Neural Information Processing Systems, formerly NIPS) conference, whose proceedings are published as the series Advances in Neural Information Processing Systems, includes many Bayesian learning papers, as does the annual conference on Artificial Intelligence and Statistics. Specifically Bayesian venues include the Valencia International Meetings on Bayesian Statistics and the journal Bayesian Analysis.

Chapter 21 Deep Learning In which gradient descent learns multistep programs, with significant implications for the major subfields of artificial intelligence.

Deep learning is a broad family of techniques for machine learning in which hypotheses take the form of complex algebraic circuits with tunable connection strengths. The word “deep” refers to the fact that the circuits are typically organized into many layers, which means that computation paths from inputs to outputs have many steps. Deep learning is currently the most widely used approach for applications such as visual object recognition, machine translation, speech recognition, speech synthesis, and image synthesis; it also plays a significant role in reinforcement learning applications (see Chapter 22 ).

Deep learning

Layer

Deep learning has its origins in early work that tried to model networks of neurons in the brain (McCulloch and Pitts, 1943) with computational circuits. For this reason, the networks trained by deep learning methods are often called neural networks, even though the resemblance to real neural cells and structures is superficial.

Neural network

While the true reasons for the success of deep learning have yet to be fully elucidated, it has self-evident advantages over some of the methods covered in Chapter 19 —particularly for high-dimensional data such as images. For example, although methods such as linear and logistic regression can handle a large number of input variables, the computation path from each input to the output is very short: multiplication by a single weight, then adding into the aggregate output. Moreover, the different input variables contribute independently to the output, without interacting with each other (Figure 21.1(a) ). This significantly limits the expressive power of such models. They can represent only linear functions and boundaries in the input space, whereas most real-world concepts are far more complex.

Figure 21.1

(a) A shallow model, such as linear regression, has short computation paths between inputs and output. (b) A decision list network (page 674) has some long paths for some possible input values, but most paths are short. (c) A deep learning network has longer computation paths, allowing each variable to interact with all the others.

Decision lists and decision trees, on the other hand, allow for long computation paths that can depend on many input variables—but only for a relatively small fraction of the possible input vectors (Figure 21.1(b) ). If a decision tree has long computation paths for a significant fraction of the possible inputs, it must be exponentially large in the number of input variables. The basic idea of deep learning is to train circuits such that the computation paths are long, allowing all the input variables to interact in complex ways (Figure

21.1(c) ). These circuit models turn out to be sufficiently expressive to capture the complexity of real-world data for many important kinds of learning problems. Section 21.1  describes simple feedforward networks, their components, and the essentials of learning in such networks. Section 21.2  goes into more detail on how deep networks are put together, and Section 21.3  covers a class of networks called convolutional neural networks that are especially important in vision applications. Sections 21.4  and 21.5  go into more detail on algorithms for training networks from data and methods for improving generalization. Section 21.6  covers networks with recurrent structure, which are well suited for sequential data. Section 21.7  describes ways to use deep learning for tasks other than supervised learning. Finally, Section 21.8  surveys the range of applications of deep learning.

21.1 Simple Feedforward Networks A feedforward network, as the name suggests, has connections only in one direction—that is, it forms a directed acyclic graph with designated input and output nodes. Each node computes a function of its inputs and passes the result to its successors in the network. Information flows through the network from the input nodes to the output nodes, and there are no loops. A recurrent network, on the other hand, feeds its intermediate or final outputs back into its own inputs. This means that the signal values within the network form a dynamical system that has internal state or memory. We will consider recurrent networks in Section 21.6 .

Feedforward network

Recurrent network

Boolean circuits, which implement Boolean functions, are an example of feedforward networks. In a Boolean circuit, the inputs are limited to 0 and 1, and each node implements a simple Boolean function of its inputs, producing a 0 or a 1. In neural networks, input values are typically continuous, and nodes take continuous inputs and produce continuous outputs. Some of the inputs to nodes are parameters of the network; the network learns by adjusting the values of these parameters so that the network as a whole fits the training data.

21.1.1 Networks as complex functions

Unit

Each node within a network is called a unit. Traditionally, following the design proposed by McCulloch and Pitts, a unit calculates the weighted sum of the inputs from predecessor nodes and then applies a nonlinear function to produce its output. Let aj denote the output of unit j and let wi,j be the weight attached to the link from unit i to unit j; then we have

aj

= gj (∑i wi,j ai ) ≡ gj (inj ),

where gj is a nonlinear activation function associated with unit j and inj is the weighted sum of the inputs to unit j.

Activation function

As in Section 19.6.3  (page 679), we stipulate that each unit has an extra input from a dummy unit 0 that is fixed to +1 and a weight w0,j for that input. This allows the total weighted input inj to unit j to be nonzero even when the outputs of the preceding layer are all zero. With this convention, we can write the preceding equation in vector form:

(21.1)

aj where

= gj (

w x) ⊤

w is the vector of weights leading into unit j (including w

0,j

) and

x is the vector of

inputs to unit j (including the +1).

The fact that the activation function is nonlinear is important because if it were not, any composition of units would still represent a linear function. The nonlinearity is what allows sufficiently large networks of units to represent arbitrary functions. The universal approximation theorem states that a network with just two layers of computational units, the first nonlinear and the second linear, can approximate any continuous function to an arbitrary degree of accuracy. The proof works by showing that an exponentially large network can represent exponentially many “bumps” of different heights at different locations in the input space, thereby approximating the desired function. In other words,

sufficiently large networks can implement a lookup table for continuous functions, just as sufficiently large decision trees implement a lookup table for Boolean functions.

A variety of different activation functions are used. The most common are the following:

The logistic or sigmoid function, which is also used in logistic regression (see page 685):

σ(x) = 1/(1 + e−x ).

Sigmoid

The ReLU function, whose name is an abbreviation for rectified linear unit: ReLU(x) = max(0,x).

ReLU

The softplus function, a smooth version of the ReLU function: softplus(x) = log(1 + ex ).

Softplus

The derivative of the softplus function is the sigmoid function. The tanh function:

tanh(x) =

e2x −1 . e2x +1

Tanh

Note that the range of tanh is (−1, + 1). Tanh is a scaled and shifted version of the sigmoid, as tanh(x) = 2σ(2x) − 1. These functions are shown in Figure 21.2 . Notice that all of them are monotonically nondecreasing, which means that their derivatives g ′ are nonnegative. We will have more to say about the choice of activation function in later sections.

Figure 21.2

Activation functions commonly used in deep learning systems: (a) the logistic or sigmoid function; (b) the ReLU function and the softplus function; (c) the tanh function.

Coupling multiple units together into a network creates a complex function that is a composition of the algebraic expressions represented by the individual units. For example,

x

the network shown in Figure 21.3(a)  represents a function hw ( ), parameterized by weights

w, that maps a two-element input vector x to a scalar output value yˆ. The internal

structure of the function mirrors the structure of the network. For example, we can write an expression for the output yˆ as follows: (21.2)

yˆ = g5 (in5 ) = g5 (w0,5 + w3,5 a3 + w4,5 a4 ) = g5 (w0,5 + w3,5 g3 (in3 ) + w4,5 g4 (in4 )) =

g5 (w0,5 + w3,5 g3 (w0,3 + w1,3 x1 + w2,3 x2 ) + w4,5 g4 (w0,4 + w1,4 x1 + w2,4 x2 )).

Figure 21.3

(a) A neural network with two inputs, one hidden layer of two units, and one output unit. Not shown are the dummy inputs and their associated weights. (b) The network in (a) unpacked into its full computation graph.

x

Thus, we have the output yˆ expressed as a function hw ( ) of the inputs and the weights. Figure 21.3(a)  shows the traditional way a network might be depicted in a book on neural networks. A more general way to think about the network is as a computation graph or dataflow graph—essentially a circuit in which each node represents an elementary computation. Figure 21.3(b)  shows the computation graph corresponding to the network in Figure 21.3(a) ; the graph makes each element of the overall computation explicit. It also distinguishes between the inputs (in blue) and the weights (in light mauve): the weights can be adjusted to make the output yˆ agree more closely with the true value y in the training data. Each weight is like a volume control knob that determines how much the next node in the graph hears from that particular predecessor in the graph.

Computation graph

Dataflow graph

Just as Equation (21.1)  described the operation of a unit in vector form, we can do

W w w g g

something similar for the network as a whole. We will generally use matrix; for this network,

W

(1)

denotes the weights in the first layer (

denotes the weights in the second layer (w3,5 etc.). Finally, let

(1)

and

to denote a weight

1,3

,

(2)

1,4

, etc.) and

W

(2)

denote the

activation functions in the first and second layers. Then the entire network can be written as follows:

(21.3)

hw (

x g W g W x )=

(2)

(

(2)

(1)

(

(1)

)).

Like Equation (21.2) , this expression corresponds to a computation graph, albeit a much simpler one than the graph in Figure 21.3(b) : here, the graph is simply a chain with weight matrices feeding into each layer. The computation graph in Figure 21.3(b)  is relatively small and shallow, but the same idea applies to all forms of deep learning: we construct computation graphs and adjust their weights to fit the data. The graph in Figure 21.3(b)  is also fully connected, meaning that every node in each layer is connected to every node in the next layer. This is in some sense the default, but we will see in Section 21.3  that choosing the connectivity of the network is also important in achieving effective learning.

Fully connected

21.1.2 Gradients and learning In Section 19.6 , we introduced an approach to supervised learning based on gradient descent: calculate the gradient of the loss function with respect to the weights, and adjust the weights along the gradient direction to reduce the loss. (If you have not already read

Section 19.6 , we recommend strongly that you do so before continuing.) We can apply exactly the same approach to learning the weights in computation graphs. For the weights leading into units in the output layer—the ones that produce the output of the network, the gradient calculation is essentially identical to the process in Section 19.6 . For weights leading into units in the hidden layers, which are not directly connected to the outputs, the process is only slightly more complicated.

Output layer

Hidden layer

For now, we will use the squared loss function, L2 , and we will calculate the gradient for the

x

network in Figure 21.3  with respect to a single training example ( ,y). (For multiple

examples, the gradient is just the sum of the gradients for the individual examples.) The network outputs a prediction yˆ =

hw (x) and the true value is y, so we have

Loss(hw ) = L2 (y,hw (x)) =∥ y − hw (x) ∥2 = (y − yˆ)2 . To compute the gradient of the loss with respect to the weights, we need the same tools of calculus we used in Chapter 19 —principally the chain rule,

gfx

∂ ( ( ))/∂

x = g ′ (f (x))∂f (x)/∂x. We’ll start with the easy case: a weight such as w3,5 that is

connected to the output unit. We operate directly on the network-defining expressions from Equation (21.2) : (21.4)

∂

∂

w3,5

Loss(hw ) =

∂

∂

w3,5

y y

y y

( − ˆ)2 = −2( − ˆ)

y w3,5 ∂ˆ

∂

∂

∂

y y ∂w g5 (in5 ) = −2(y − yˆ)g5′ (in5 ) ∂w in5 3,5 3,5

= −2( − ˆ)

∂

y y g5′ (in5 ) ∂w

= −2( − ˆ)

y y g5′ (in5 )a3 .

3,5

(w0,5 + w3,5 a3 + w4,5 a4 )

= −2( − ˆ)

The simplification in the last line follows because w0,5 and w4,5 a4 do not depend on w3,5 , nor does the coefficient of w3,5 , a3 .

The slightly more difficult case involves a weight such as w1,3 that is not directly connected to the output unit. Here, we have to apply the chain rule one more time. The first few steps are identical, so we omit them:

(21.5) ∂

∂

Loss(hw ) = −2(y − yˆ)g5′ (in5 ) ∂w (w0,5 + w3,5 a3 + w4,5 a4 ) ∂w1,3 1,3 ∂

y y g5′ (in5 )w3,5 ∂w a3

= −2( − ˆ)

1,3

∂

y y g5′ (in5 )w3,5 ∂w g3 (in3 )

= −2( − ˆ)

1,3

∂

y y g5′ (in5 )w3,5 g3′ (in3 ) ∂w in3

= −2( − ˆ)

1,3

∂

y y g5′ (in5 )w3,5 g3′ (in3 ) ∂w

= −2( − ˆ)

y y g5′ (in5 )w3,5 g3′ (in3 )x1 .

1,3

(w0,3 + w1,3 x1 + w2,3 x2 )

= −2( − ˆ)

So, we have fairly simple expressions for the gradient of the loss with respect to the weights

w3,5 and w1,3 . If we define Δ5 = 2(yˆ − y)g5′ (in5 ) as a sort of “perceived error” at the point where unit 5 receives its input, then the gradient with respect to w3,5 is just Δ5 a3 . This makes perfect

sense: if Δ5 is positive, that means yˆ is too big (recall that g ′ is always nonnegative); if a3 is also positive, then increasing w3,5 will only make things worse, whereas if a3 is negative,

w3,5 will reduce the error. The magnitude of a3 also matters: if a3 is small for this training example, then w3,5 didn’t play a major role in producing the error and doesn’t then increasing

need to be changed much.

If we also define Δ3 = Δ5 w3,5 g3′ (in3 ), then the gradient for w1,3 becomes just Δ3 x1 . Thus, the perceived error at the input to unit 3 is the perceived error at the input to unit 5, multiplied by information along the path from 5 back to 3. This phenomenon is completely general, and gives rise to the term back-propagation for the way that the error at the output is passed back through the network.

Back-propagation

Another important characteristic of these gradient expressions is that they have as factors the local derivatives gj′ (inj ). As noted earlier, these derivatives are always nonnegative, but they can be very close to zero (in the case of the sigmoid, softplus, and tanh functions) or exactly zero (in the case of ReLUs), if the inputs from the training example in question happen to put unit j in the flat operating region. If the derivative gj′ is small or zero, that means that changing the weights leading into unit j will have a negligible effect on its output. As a result, deep networks with many layers may suffer from a vanishing gradient— the error signals are extinguished altogether as they are propagated back through the network. Section 21.3.3  provides one solution to this problem.

Vanishing gradient

We have shown that gradients in our tiny example network are simple expressions that can be computed by passing information back through the network from the output units. It turns out that this property holds more generally. In fact, as we show in Section 21.4.1 , the gradient computations for any feedforward computation graph have the same structure as the underlying computation graph. This property follows straightforwardly from the rules of differentiation.

We have shown the gory details of a gradient calculation, but worry not: there is no need to redo the derivations in Equations (21.4)  and (21.5)  for each new network structure! All

such gradients can be computed by the method of automatic differentiation, which applies the rules of calculus in a systematic way to calculate gradients for any numeric program.1 In fact, the method of back-propagation in deep learning is simply an application of reverse mode differentiation, which applies the chain rule “from the outside in” and gains the efficiency advantages of dynamic programming when the network in question has many inputs and relatively few outputs.

1 Automatic differentiation methods were originally developed in the 1960s and 1970s for optimizing the parameters of systems defined by large, complex Fortran programs.

Automatic differentiation

Reverse mode

All of the major packages for deep learning provide automatic differentiation, so that users can experiment freely with different network structures, activation functions, loss functions, and forms of composition without having to do lots of calculus to derive a new learning algorithm for each experiment. This has encouraged an approach called end-to-end learning, in which a complex computational system for a task such as machine translation can be composed from several trainable subsystems; the entire system is then trained in an end-to-end fashion from input/output pairs. With this approach, the designer need have only a vague idea about how the overall system should be structured; there is no need to know in advance exactly what each subsystem should do or how to label its inputs and outputs.

End-to-end learning

21.2 Computation Graphs for Deep Learning We have established the basic ideas of deep learning: represent hypotheses as computation graphs with tunable weights and compute the gradient of the loss function with respect to those weights in order to fit the training data. Now we look at how to put together computation graphs. We begin with the input layer, which is where the training or test example x is encoded as values of the input nodes. Then we consider the output layer,

ˆ

where the outputs y are compared with the true values y to derive a learning signal for tuning the weights. Finally, we look at the hidden layers of the network.

21.2.1 Input encoding The input and output nodes of a computational graph are the ones that connect directly to the input data x and the output data y. The encoding of input data is usually straightforward, at least for the case of factored data where each training example contains values for n input attributes. If the attributes are Boolean, we have n input nodes; usually

false is mapped to an input of 0 and true is mapped to 1, although sometimes −1 and +1

are used. Numeric attributes, whether integer or real-valued, are typically used as is, although they may be scaled to fit within a fixed range; if the magnitudes for different examples vary enormously, the values can be mapped onto a log scale.

Images do not quite fit into the category of factored data; although an RGB image of size

X×Y range

pixels can be thought of as

3XY integer-valued attributes (typically with values in the

{0, … ,255}), this would ignore the fact that the RGB triplets belong to the same pixel

in the image and the fact that pixel adjacency really matters. Of course, we can map adjacent pixels onto adjacent input nodes in the network, but the meaning of adjacency is completely lost if the internal layers of the network are fully connected. In practice, networks used with image data have array-like internal structures that aim to reflect the semantics of adjacency. We will see this in more detail in Section 21.3 . Categorical attributes with more than two values—like the T ype attribute in the restaurant

problem from Chapter 19 , which has values French, Italian, Thai, or burger)—are usually encoded using the so-called one-hot encoding. An attribute with d possible values is represented by d separate input bits. For any given value, the corresponding input bit is set

to 1 and all the others are set to 0. This generally works better than mapping the values to integers. If we used integers for the

Type attribute, Thai would be 3 and burger would be 4.

Because the network is a composition of continuous functions, it would have no choice but to pay attention to numerical adjacency, but in this case the numerical adjacency between Thai and burger is semantically meaningless.

21.2.2 Output layers and loss functions On the output side of the network, the problem of encoding the raw data values into actual values

y for the output nodes of the graph is much the same as the input encoding problem.

For example, if the network is trying to predict the Weather variable from Chapter 12 ,

which has values {sun, rain, cloud, snow}, we would use a one-hot encoding with four bits.

y

y

So much for the data values . What about the prediction ˆ ? Ideally, it would exactly match

y

the desired value , and the loss would be zero, and we’d be done. In practice, this seldom happens—especially before we have started the process of adjusting the weights! Thus, we need to think about what an incorrect output value means, and how to measure the loss. In deriving the gradients in Equations (21.4)  and (21.5) , we began with the squared-error loss function. This keeps the algebra simple, but it is not the only possibility. In fact, for

y

most deep learning applications, it is more common to interpret the output values ˆ as probabilities and to use the negative log likelihood as the loss function—exactly as we did with maximum likelihood learning in Chapter 20 .

Maximum likelihood learning finds the value of

w that maximizes the probability of the

observed data. And because the log function is monotonic, this is equivalent to maximizing the log likelihood of the data, which is equivalent in turn to minimizing a loss function defined as the negative log likelihood. (Recall from page 725 that taking logs turns products of probabilities into sums, which are more amenable for taking derivatives.) In other words, we are looking for

w

∗

that minimizes the sum of negative log probabilities of the

examples:

(21.6)

w

∗

∑ N

= argmin −

w

j=1

log 

Pw (yj ∣∣xj ).

N

In the deep learning literature, it is common to talk about minimizing the cross-entropy loss. Cross-entropy, written as distributions

H (P ,Q), is a kind of measure of dissimilarity between two

P and Q.2 The general definition is

H (P ,P ) is not zero; rather, it equals the entropy H (P ). It is easy to show H (P ,Q) = H (P ) + DKL(P ∥ Q), where DKL is the Kullback–Leibler divergence, which does satisfy DKL(P ∥ P ) = 0. Thus, for fixed P , varying Q to minimize the cross-entropy also minimizes the KL divergence. 2 Cross-entropy is not a distance in the usual sense because that

Cross-entropy

(21.7)

H ( P ,Q ) = E z

Pz

∼ ( ) [log

Q(z)] = ∫ P (z) log Q(z)dz.

P being the true distribution over training examples, P (x,y), and Q being the predictive hypothesis Pw (y|x). Minimizing the cross-entropy H (P (x,y),Pw (y|x)) by adjusting w makes the hypothesis agree as closely as In machine learning, we typically use this definition with ∗

∗

possible with the true distribution. In reality, we cannot minimize this cross-entropy because we do not have access to the true data distribution to samples from

P (x,y); but we do have access ∗

P (x,y), so the sum over the actual data in Equation (21.6)  approximates ∗

the expectation in Equation (21.7) .

To minimize the negative log likelihood (or the cross-entropy), we need to be able to interpret the output of the network as a probability. For example, if the network has one output unit with a sigmoid activation function and is learning a Boolean classification, we can interpret the output value directly as the probability that the example belongs to the positive class. (Indeed, this is exactly how logistic regression is used; see page 684.) Thus, for Boolean classification problems, we commonly use a sigmoid output layer.

Multiclass classification problems are very common in machine learning. For example, classifiers used for object recognition often need to recognize thousands of distinct categories of objects. Natural language models that try to predict the next word in a sentence may have to choose among tens of thousands of possible words. For this kind of

prediction, we need the network to output a categorical distribution—that is, if there are d possible answers, we need d output nodes that represent probabilities summing to 1. To achieve this, we use a softmax layer, which outputs a vector of d values given a vector of input values in = ⟨in1 , … ,ind ⟩. The kth element of that output vector is given by

softmax(in)k =

∑

eink

d ink′ k′ =1 e

.

Softmax

By construction, the softmax function outputs a vector of nonnegative numbers that sum to 1. As usual, the input ink to each of the output nodes will be a weighted linear combination of the outputs of the preceding layer. Because of the exponentials, the softmax layer accentuates differences in the inputs: for example, if the vector of inputs is given by in

= ⟨5,2,0, − 2⟩, then the outputs are ⟨0.946,0.047,0.006,0.001⟩. The softmax, is, nonetheless,

smooth and differentiable (Exercise 21.SOFG), unlike the max function. It is easy to show (Exercise 21.SMSG) that the sigmoid is a softmax with d = 2. In other words, just as sigmoid units propagate binary class information through a network, softmax units propagate multiclass information.

For a regression problem, where the target value y is continuous, it is common to use a linear output layer—in other words, yˆj = inj , without any activation function g—and to interpret this as the mean of a Gaussian prediction with fixed variance. As we noted on page 729, maximizing likelihood (i.e., minimizing negative log likelihood) with a fixed-variance Gaussian is the same as minimizing squared error. Thus, a linear output layer interpreted in this way does classical linear regression. The input features to this linear regression are the outputs from the preceding layer, which typically result from multiple nonlinear transformations of the original inputs to the network.

Many other output layers are possible. For example, a mixture density layer represents the outputs using a mixture of Gaussian distributions. (See Section 20.3.1  for more details on

Gaussian mixtures.) Such layers predict the relative frequency of each mixture component, the mean of each component, and the variance of each component. As long as these output values are interpreted appropriately by the loss function as defining the probability for the true output value y, the network will, after training, fit a Gaussian mixture model in the space of features defined by the preceding layers.

Mixture density

21.2.3 Hidden layers During the training process, a neural network is shown many input values x and many corresponding output values y. While processing an input vector x, the neural network performs several intermediate computations before producing the output y. We can think of the values computed at each layer of the network as a different representation for the input x. Each layer transforms the representation produced by the preceding layer to produce a new representation. The composition of all these transformations succeeds—if all goes well—in transforming the input into the desired output. Indeed, one hypothesis for why deep learning works well is that the complex end-to-end transformation that maps from input to output—say, from an input image to the output category “giraffe”—is decomposed by the many layers into the composition of many relatively simple transformations, each of which is fairly easy to learn by a local updating process.

In the process of forming all these internal transformations, deep networks often discover meaningful intermediate representations of the data. For example, a network learning to recognize complex objects in images may form internal layers that detect useful subunits: edges, corners, ellipses, eyes, faces—cats. Or it may not—deep networks may form internal layers whose meaning is opaque to humans, even though the output is still correct.

The hidden layers of neural networks are typically less diverse than the output layers. For the first 25 years of research with multilayer networks (roughly 1985–2010), internal nodes used sigmoid and tanh activation functions almost exclusively. From around 2010 onwards, the ReLU and softplus become more popular, partly because they are believed to avoid the problem of vanishing gradients mentioned in Section 21.1.2 . Experimentation with

increasingly deep networks suggested that, in many cases, better learning was obtained with deep and relatively narrow networks rather than shallow, wide networks, given a fixed total number of weights. A typical example of this is shown in Figure 21.7  on page 769.

Figure 21.7

Test-set error as a function of layer width (as measured by total number of weights) for three-layer and eleven-layer convolutional networks. The data come from early versions of Google’s system for transcribing addresses in photos taken by Street View cars (Goodfellow et al., 2014).

There are, of course, many other structures to consider for computation graphs, besides just playing with width and depth. At the time of writing, there is little understanding as to why some structures seem to work better than others for some particular problem. With experience, practitioners gain some intuition as to how to design networks and how to fix them when they don’t work, just as chefs gain intuition for how to design recipes and how to fix them when they taste unpleasant. For this reason, tools that facilitate rapid exploration and evaluation of different structures are essential for success in real-world problems.

21.3 Convolutional Networks We mentioned in Section 21.2.1  that an image cannot be thought of as a simple vector of input pixel values, primarily because adjacency of pixels really matters. If we were to construct a network with fully connected layers and an image as input, we would get the same result whether we trained with unperturbed images or with images all of whose pixels had been randomly permuted. Furthermore, suppose there are n pixels and n units in the first hidden layer, to which the pixels provide input. If the input and the first hidden layer are fully connected, that means n2 weights; for a typical megapixel RGB image, that’s 9 trillion weights. Such a vast parameter space would require correspondingly vast numbers of training images and a huge computational budget to run the training algorithm.

These considerations suggest that we should construct the first hidden layer so that each hidden unit receives input from only a small, local region of the image. This kills two birds with one stone. First, it respects adjacency, at least locally. (And we will see later that if subsequent layers have the same locality property, then the network will respect adjacency in a global sense.) Second, it cuts down the number of weights: if each local region has l

≪ n pixels, then there will be ln ≪ n2 weights in all.

So far, so good. But we are missing another important property of images: roughly speaking, anything that is detectable in one small, local region of the image—perhaps an eye or a blade of grass—would look the same if it appeared in another small, local region of the image. In other words, we expect image data to exhibit approximate spatial invariance, at least at small to moderate scales.3 We don’t necessarily expect the top halves of photos to look like bottom halves, so there is a scale beyond which spatial invariance no longer holds.

3 Similar ideas can be applied to process time-series data sources such as audio waveforms. These typically exhibit temporal invariance—a word sounds the same no matter what time of day it is uttered. Recurrent neural networks (Section 21.6 ) automatically exhibit temporal invariance.

Spatial invariance

Local spatial invariance can be achieved by constraining the l weights connecting a local region to a unit in the hidden layer to be the same for each hidden unit. (That is, for hidden units i and j, the weights w1,i , … ,wl,i are the same as w1,j , … ,wl,j .) This makes the hidden units into feature detectors that detect the same feature wherever it appear in the image. Typically, we want the first hidden layer to detect many kinds of features, not just one; so for each local image region we might have d hidden units with d distinct sets of weights. This means that there are dl weights in all—a number that is not only far smaller than n2 , but is actually independent of n, the image size. Thus, by injecting some prior knowledge— namely, knowledge of adjacency and spatial invariance—we can develop models that have far fewer parameters and can learn much more quickly.

A convolutional neural network (CNN) is one that contains spatially local connections, at least in the early layers, and has patterns of weights that are replicated across the units in each layer. A pattern of weights that is replicated across multiple local regions is called a kernel and the process of applying the kernel to the pixels of the image (or to spatially organized units in a subsequent layer) is called convolution.4

4 In the terminology of signal processing, we would call this operation a cross-correlation, not a convolution. But “convolution” is used within the field of neural networks.

Convolutional neural network (CNN)

Kernel

Convolution

Kernels and convolutions are easiest to illustrate in one dimension rather than two or more, so we will assume an input vector x of size n, corresponding to n pixels in a one-

dimensional image, and a vector kernel k of size l. (For simplicity we will assume that l is an odd number.) All the ideas carry over straightforwardly to higher-dimensional cases. We write the convolution operation using the ∗ symbol, for example: z =

x ∗ k. The

operation is defined as follows:

(21.8)

∑ l

zi =

kj xj+i−(l+1)/2 .

j=1

In other words, for each output position i, we take the dot product between the kernel k and a snippet of x centered on xi with width l. The process is illustrated in Figure 21.4  for a kernel vector [+1,−1,+1], which detects a darker point in the 1D image. (The 2D version might detect a darker line.) Notice that in this example the pixels on which the kernels are centered are separated by a distance of 2 pixels; we say the kernel is applied with a stride s = 2. Notice that the output layer has fewer pixels: because of the stride, the number of pixels is reduced from n to roughly n/s. (In two dimensions, the number of pixels would be roughly n/sx sy , where sx and sy are the strides in the x and y directions in the image.) We say “roughly” because of what happens at the edge of the image: in Figure 21.4  the convolution stops at the edges of the image, but one can also pad the input with extra pixels (either zeroes or copies of the outer pixels) so that the kernel can be applied exactly ⌊n/s⌋ times. For small kernels, we typically use s = 1, so the output has the same dimensions as the image (see Figure 21.5 ).

Figure 21.4

An example of a one-dimensional convolution operation with a kernel of size l = 3 and a stride s = 2. The peak response is centered on the darker (lower intensity) input pixel. The results would usually be fed through a nonlinear activation function (not shown) before going to the next hidden layer.

Figure 21.5

=3

=1

The first two layers of a CNN for a 1D image with a kernel size l and a stride s . Padding is added at the left and right ends in order to keep the hidden layers the same size as the input. Shown in red is the receptive field of a unit in the second hidden layer. Generally speaking, the deeper the unit, the larger the receptive field.

Stride

The operation of applying a kernel across an image can be implemented in the obvious way by a program with suitable nested loops; but it can also be formulated as a single matrix operation, just like the application of the weight matrix in Equation (21.1) . For example, the convolution illustrated in Figure 21.4  can be viewed as the following matrix multiplication:

(21.9)

⎛

⎛ ⎜ ⎝

+1 −1 +1 0 0 0 0 0 0 +1 −1 +1 0 0 0 0 0 0 +1 −1 +1

⎜ ⎜ ⎞⎜ ⎜ ⎟⎜ ⎜ ⎠⎜ ⎜ ⎜ ⎜ ⎜ ⎝

5 6 6 5 2 = 9 . 5 4 6 5 ⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟

⎛

⎞

⎜

⎟

⎝

⎠

⎠

In this weight matrix, the kernel appears in each row, shifted according to the stride relative to the previous row, One wouldn’t necessarily construct the weight matrix explicitly—it is

mostly zeroes, after all—but the fact that convolution is a linear matrix operation serves as a reminder that gradient descent can be applied easily and effectively to CNNs, just as it can to plain vanilla neural networks.

As mentioned earlier, there will be

d kernels, not just one; so, with a stride of 1, the output

d times larger. This means that a two-dimensional input array becomes a threedimensional array of hidden units, where the third dimension is of size d. It is important to will be

organize the hidden layer this way, so that all the kernel outputs from a particular image location stay associated with that location. Unlike the spatial dimensions of the image, however, this additional “kernel dimension” does not have any adjacency properties, so it does not make sense to run convolutions along it.

CNNs were inspired originally by models of the visual cortex proposed in neuroscience. In those models, the receptive field of a neuron is the portion of the sensory input that can affect that neuron’s activation. In a CNN, the receptive field of a unit in the first hidden layer

l

is small—just the size of the kernel, i.e., pixels. In the deeper layers of the network, it can be much larger. Figure 21.5  illustrates this for a unit in the second hidden layer, whose

mth hidden layer will have a receptive field of size (l − 1)m + 1; so the growth is linear in m. (In a 2D image, each dimension of the receptive field grows linearly with m, so the area grows quadratically.) When the stride is larger than 1, each pixel in layer m represents s pixels in layer m − 1; therefore, the receptive field grows as O(lsm )—that is, exponentially with depth. receptive field contains five pixels. When the stride is 1, as in the figure, a node in the

The same effect occurs with pooling layers, which we discuss next.

Receptive field

21.3.1 Pooling and downsampling A pooling layer in a neural network summarizes a set of adjacent units from the preceding

l

layer with a single value. Pooling works just like a convolution layer, with a kernel size and

s

stride , but the operation that is applied is fixed rather than learned. Typically, no activation function is associated with the pooling layer. There are two common forms of pooling:

Pooling

Downsampling

Average-pooling computes the average value of its l inputs. This is identical to convolution with a uniform kernel vector k

= [1/l, … ,1/l].

If we set l

=

s, the effect is to

coarsen the resolution of the image—to downsample it—by a factor of s. An object that occupied, say, 10s pixels would now occupy only

10

pixels after pooling. The same

learned classifier that would be able to recognize the object at a size of 10 pixels in the original image would now be able to recognize that object in the pooled image, even if it was too big to recognize in the original image. In other words, average-pooling facilitates multiscale recognition. It also reduces the number of weights required in subsequent layers, leading to lower computational cost and possibly faster learning. Max-pooling computes the maximum value of its l inputs. It can also be used purely for downsampling, but it has a somewhat different semantics. Suppose we applied maxpooling to the hidden layer [5,9,4] in Figure 21.4 : the result would be a 9, indicating that somewhere in the input image there is a darker dot that is detected by the kernel. In other words, max-pooling acts as a kind of logical disjunction, saying that a feature exists somewhere in the unit’s receptive field.

If the goal is to classify the image into one of c categories, then the final layer of the network will be a softmax with c output units. The early layers of the CNN are image-sized, so somewhere in between there must be significant reductions in layer size. Convolution layers and pooling layers with stride larger than 1 all serve to reduce the layer size. It’s also possible to reduce the layer size simply by having a fully connected layer with fewer units than the preceding layer. CNNs often have one or two such layers preceding the final softmax layer.

21.3.2 Tensor operations in CNNs We saw in Equations (21.1)  and (21.3)  that the use of vector and matrix notation can be helpful in keeping mathematical derivations simple and elegant and providing concise

descriptions of computation graphs. Vectors and matrices are one-dimensional and twodimensional special cases of tensors, which (in deep learning terminology) are simply multidimensional arrays of any dimension.5

5 The proper mathematical definition of tensors requires that certain invariances hold under a change of basis.

Tensor

For CNNs, tensors are a way of keeping track of the “shape” of the data as it progresses through the layers of the network. This is important because the whole notion of convolution depends on the idea of adjacency: adjacent data elements are assumed to be semantically related, so it makes sense to apply operators to local regions of the data. Moreover, with suitable language primitives for constructing tensors and applying operators, the layers themselves can be described concisely as maps from tensor inputs to tensor outputs.

A final reason for describing CNNs in terms of tensor operations is computational efficiency: given a description of a network as a sequence of tensor operations, a deep learning software package can generate compiled code that is highly optimized for the underlying computational substrate. Deep learning workloads are often run on GPUs (graphics processing units) or TPUs (tensor processing units), which make available a high degree of parallelism. For example, one of Google’s third-generation TPU pods has throughput equivalent to about ten million laptops. Taking advantage of these capabilities is essential if one is training a large CNN on a large database of images. Thus, it is common to process not one image at a time but many images in parallel; as we will see in Section 21.4 , this also aligns nicely with the way that the stochastic gradient descent algorithm calculates gradients with respect to a minibatch of training examples.

Let us put all this together in the form of an example. Suppose we are training on 256 × 256 RGB images with a minibatch size of 64. The input in this case will be a four-dimensional tensor of size 256 × 256 × 3 × 64. Then we apply 96 kernels of size 5 × 5 × 3 with a stride of 2 in both x and y directions in the image. This gives an output tensor of size

128 × 128 × 96 × 64. Such a tensor is often called a feature map, since it shows how each

feature extracted by a kernel appears across the entire image; in this case it is composed of 96 channels, where each channel carries information from one feature. Notice that unlike the input tensor, this feature map no longer has dedicated color channels; nonetheless, the color information may still be present in the various feature channels if the learning algorithm finds color to be useful for the final predictions of the network.

Feature map

Channel

21.3.3 Residual networks Residual networks are a popular and successful approach to building very deep networks that avoid the problem of vanishing gradients.

Residual network

Typical deep models use layers that learn a new representation at layer i by completely replacing the representation at layer i − 1. Using the matrix–vector notation that we

introduced in Equation (21.3) , with z(i) being the values of the units in layer i, we have

z(i) = f (z(i−1) ) = g(i) (W(i) z(i−1) ). Because each layer completely replaces the representation from the preceding layer, all of the layers must learn to do something useful. Each layer must, at the very least, preserve the task-relevant information contained in the preceding layer. If we set W the entire network ceases to function. If we also set W

(i−1)

=

(i)

=

0 for any layer i,

0, the network would not even

be able to learn: layer i would not learn because it would observe no variation in its input from layer i − 1, and layer i − 1 would not learn because the back-propagated gradient from layer i would always be zero. Of course, these are extreme examples, but they illustrate the need for layers to serve as conduits for the signals passing through the network.

The key idea of residual networks is that a layer should perturb the representation from the previous layer rather than replace it entirely. If the learned perturbation is small, the next layer is close to being a copy of the previous layer. This is achieved by the following equation for layer i in terms of layer i − 1: (21.10)

z(i) = g(ri) (z(i−1) + f (z(i−1) )), where gr denotes the activation functions for the residual layer. Here we think of f as the residual, perturbing the default behavior of passing layer i − 1 through to layer i. The function used to compute the residual is typically a neural network with one nonlinear layer combined with one linear layer:

f (z) = Vg(Wz),

Residual

where W and V are learned weight matrices with the usual bias weights added.

Residual networks make it possible to learn significantly deeper networks reliably. Consider what happens if we set V =

0 for a particular layer in order to disable that layer. Then the

residual f disappears and Equation (21.10)  simplifies to

z(i) = gr (z(i−1) ). Now suppose that gr consists of ReLU activation functions and that z(i−1) also applies a ReLU function to its inputs: z(i−1) = ReLU(in(i−1) ). In that case we have

z(i) = gr (z(i−1) ) = ReLU(z(i−1) ) = ReLU(ReLU(in(i−1) )) = ReLU(in(i−1) ) = z(i−1) , where the penultimate step follows because ReLU(ReLU(x)) = ReLU(x). In other words, in residual nets with ReLU activations, a layer with zero weights simply passes its inputs through with no change. The rest of the network functions just as if the layer had never existed. Whereas traditional networks must learn to propagate information and are subject to catastrophic failure of information propagation for bad choices of the parameters, residual networks propagate information by default.

Residual networks are often used with convolutional layers in vision applications, but they are in fact a general-purpose tool that makes deep networks more robust and allows researchers to experiment more freely with complex and heterogeneous network designs. At the time of writing, it is not uncommon to see residual networks with hundreds of layers. The design of such networks is evolving rapidly, so any additional specifics we might provide would probably be outdated before reaching printed form. Readers desiring to know the best architectures for specific applications should consult recent research publications.

21.4 Learning Algorithms Training a neural network consists of modifying the network’s parameters so as to minimize the loss function on the training set. In principle, any kind of optimization algorithm could be used. In practice, modern neural networks are almost always trained with some variant of stochastic gradient descent (SGD). We covered standard gradient descent and its stochastic version in Section 19.6.2 . Here,

w

the goal is to minimize the loss L( ), where

w represents all of the parameters of the

network. Each update step in the gradient descent process looks like this:

w ← w − α∇w L(w), where α is the learning rate. For standard gradient descent, the loss L is defined with

respect to the entire training set. For SGD, it is defined with respect to a minibatch of m examples chosen randomly at each step. As noted in Section 4.2 , the literature on optimization methods for high-dimensional continuous spaces includes innumerable enhancements to basic gradient descent. We will not cover all of them here, but it is worth mentioning a few important considerations that are particularly relevant to training neural networks:

For most networks that solve real-world problems, both the dimensionality of

w and the

size of the training set are very large. These considerations militate strongly in favor of using SGD with a relatively small minibatch size m: stochasticity helps the algorithm escape small local minima in the high-dimensional weight space (as in simulated annealing—see page 114); and the small minibatch size ensures that the computational cost of each weight update step is a small constant, independent of the training set size. Because the gradient contribution of each training example in the SGD minibatch can be computed independently, the minibatch size is often chosen so as to take maximum advantage of hardware parallelism in GPUs or TPUs. To improve convergence, it is usually a good idea to use a learning rate that decreases over time. Choosing the right schedule is usually a matter of trial and error.

Near a local or global minimum of the loss function with respect to the entire training set, the gradients estimated from small minibatches will often have high variance and may point in entirely the wrong direction, making convergence difficult. One solution is to increase the minibatch size as training proceeds; another is to incorporate the idea of momentum, which keeps a running average of the gradients of past minibatches in order to compensate for small minibatch sizes.

Momentum

Care must be taken to mitigate numerical instabilities that may arise due to overflow, underflow, and rounding error. These are particularly problematic with the use of exponentials in softmax, sigmoid, and tanh activation functions, and with the iterated computations in very deep networks and recurrent networks (Section 21.6 ) that lead to vanishing and exploding activations and gradients.

Overall, the process of learning the weights of the network is usually one that exhibits diminishing returns. We run until it is no longer practical to decrease the test error by running longer. Usually this does not mean we have reached a global or even a local minimum of the loss function. Instead, it means we would have to make an impractically large number of very small steps to continue reducing the cost, or that additional steps would only cause overfitting, or that estimates of the gradient are too inaccurate to make further progress.

21.4.1 Computing gradients in computation graphs On page 755, we derived the gradient of the loss function with respect to the weights in a specific (and very simple) network. We observed that the gradient could be computed by back-propagating error information from the output layer of the network to the hidden layers. We also said that this result holds in general for any feedforward computation graph. Here, we explain how this works. Figure 21.6  shows a generic node in a computation graph. (The node h has in-degree and out-degree 2, but nothing in the analysis depends on this.) During the forward pass, the

node computes some arbitrary function h from its inputs, which come from nodes f and g. In turn, h feeds its value to nodes j and k.

Figure 21.6

Illustration of the back-propagation of gradient information in an arbitrary computation graph. The forward computation of the output of the network proceeds from left to right, while the back-propagation of gradients proceeds from right to left.

The back-propagation process passes messages back along each link in the network. At each node, the incoming messages are collected and new messages are calculated to pass back to the next layer. As the figure shows, the messages are all partial derivatives of the loss L. For example, the backward message

∂L/∂hj is the partial derivative of L with respect to j’s first

input, which is the forward message from h to j. Now, h affects L through both j and k, so we have

(21.11)

∂L/∂h = ∂L/∂hj + ∂L/∂hk . With this equation, the node h can compute the derivative of L with respect to h by

summing the incoming messages from j and k. Now, to compute the outgoing messages

∂L/∂fh and ∂L/∂gh , we use the following equations:

(21.12)

L fh

∂ ∂

=

∂

L ∂h h ∂fh

∂

and

L gh

∂ ∂

=

∂

L ∂h h ∂ gh .

∂

In Equation (21.12) , ∂L/∂h was already computed by Equation (21.11) , and ∂h/∂fh and

h gh are just the derivatives of h with respect to its first and second arguments, respectively. For example, if h is a multiplication node—that is, h(f ,g) = f ⋅ g—then ∂h/∂fh = g and ∂h/∂gh = f . Software packages for deep learning typically come with a ∂ /∂

library of node types (addition, multiplication, sigmoid, and so on), each of which knows how to compute its own derivatives as needed for Equation (21.12) .

The back-propagation process begins with the output nodes, where each initial message

L y

∂ /∂ ˆj is calculated directly from the expression for

L in terms of the predicted value yˆ and

the true value y from the training data. At each internal node, the incoming backward

messages are summed according to Equation (21.11)  and the outgoing messages are generated from Equation (21.12) . The process terminates at each node in the computation graph that represents a weight w (e.g., the light mauve ovals in Figure 21.3(b) ). At that

point, the sum of the incoming messages to w is ∂L/∂w—precisely the gradient we need to update w. Exercise 21.BPRE asks you to apply this process to the simple network in Figure 21.3  in order to rederive the gradient expressions in Equations (21.4)  and (21.5) .

Weight-sharing, as used in convolutional networks (Section 21.3 ) and recurrent networks (Section 21.6 ), is handled simply by treating each shared weight as a single node with multiple outgoing arcs in the computation graph. During back-propagation, this results in multiple incoming gradient messages. By Equation (21.11) , this means that the gradient for the shared weight is the sum of the gradient contributions from each place it is used in the network.

It is clear from this description of the back-propagation process that its computational cost is linear in the number of nodes in the computation graph, just like the cost of the forward computation. Furthermore, because the node types are typically fixed when the network is designed, all of the gradient computations can be prepared in symbolic form in advance and compiled into very efficient code for each node in the graph. Note also that the messages in Figure 21.6  need not be scalars: they could equally be vectors, matrices, or higher-

dimensional tensors, so that the gradient computations can be mapped onto GPUs or TPUs to benefit from parallelism.

One drawback of back-propagation is that it requires storing most of the intermediate values that were computed during forward propagation in order to calculate gradients in the backward pass. This means that the total memory cost of training the network is proportional to the number of units in the entire network. Thus, even if the network itself is represented only implicitly by propagation code with lots of loops, rather than explicitly by a data structure, all of the intermediate results of that propagation code have to be stored explicitly.

21.4.2 Batch normalization Batch normalization is a commonly used technique that improves the rate of convergence of SGD by rescaling the values generated at the internal layers of the network from the examples within each minibatch. Although the reasons for its effectiveness are not well understood at the time of writing, we include it because it confers significant benefits in practice. To some extent, batch normalization seems to have effects similar to those of the residual network.

Batch normalization

Consider a node z somewhere in the network: the values of z for the m examples in a minibatch are z1

, … ,z

m

ˆ

. Batch normalization replaces each zi with a new quantity z i :

ˆ = γ z − μ 2 + β, √ϵ + σ

zi

i

where μ is the mean value of z across the minibatch, σ is the standard deviation of z1 ϵ

, … ,z

m

,

is a small constant added to prevent division by zero, and γ and β are learned parameters.

Batch normalization standardizes the mean and variance of the values, as determined by the values of β and γ. This makes it much simpler to train a deep network. Without batch

normalization, information can get lost if a layer’s weights are too small, and the standard deviation at that layer decays to near zero. Batch normalization prevents this from happening. It also reduces the need for careful initialization of all the weights in the network to make sure that the nodes in each layer are in the right operating region to allow information to propagate.

With batch normalization, we usually include β and γ, which may be node-specific or layerspecific, among the parameters of the network, so that they are included in the learning process. After training, β and γ are fixed at their learned values.

21.5 Generalization So far we have described how to fit a neural network to its training set, but in machine learning the goal is to generalize to new data that has not been seen previously, as measured by performance on a test set. In this section, we focus on three approaches to improving generalization performance: choosing the right network architecture, penalizing large weights, and randomly perturbing the values passing through the network during training.

21.5.1 Choosing a network architecture A great deal of effort in deep learning research has gone into finding network architectures that generalize well. Indeed, for each particular kind of data—images, speech, text, video, and so on—a good deal of the progress in performance has come from exploring different kinds of network architectures and varying the number of layers, their connectivity, and the types of node in each layer.6

6 Noting that much of this incremental, exploratory work is carried out by graduate students, some have called the process graduate student descent (GSD).

Some neural network architectures are explicitly designed to generalize well on particular types of data: convolutional networks encode the idea that the same feature extractor is useful at all locations across a spatial grid, and recurrent networks encode the idea that the same update rule is useful at all points in a stream of sequential data. To the extent that these assumptions are valid, we expect convolutional architectures to generalize well on images and recurrent networks to generalize well on text and audio signals.

One of the most important empirical findings in the field of deep learning is that when comparing two networks with similar numbers of weights, the deeper network usually gives better generalization performance. Figure 21.7  shows this effect for at least one real-world application—recognizing house numbers. The results show that for any fixed number of parameters, an eleven-layer network gives much lower test-set error than a three-layer network.

Deep learning systems perform well on some but not all tasks. For tasks with highdimensional inputs—images, video, speech signals, etc.—they perform better than any other pure machine learning approaches. Most of the algorithms described in Chapter 19  can handle high-dimensional input only if it is preprocessed using manually designed features to reduce the dimensionality. This preprocessing approach, which prevailed prior to 2010, has not yielded performance comparable to that achieved by deep learning systems.

Clearly, deep learning models are capturing some important aspects of these tasks. In particular, their success implies that the tasks can be solved by parallel programs with a relatively small number of steps (10 to 103 rather than, say, 107 ). This is perhaps not surprising, because these tasks are typically solved by the brain in less than a second, which is time enough for only a few tens of sequential neuron firings. Moreover, by examining the internal-layer representations learned by deep convolutional networks for vision tasks, we find evidence that the processing steps seem to involve extracting a sequence of increasingly abstract representations of the scene, beginning with tiny edges, dots, and corner features and ending with entire objects and arrangements of multiple objects.

On the other hand, because they are simple circuits, deep learning models lack the compositional and quantificational expressive power that we see in first-order logic (Chapter 8 ) and context-free grammars (Chapter 23 ).

Although deep learning models generalize well in many cases, they may also produce unintuitive errors. They tend to produce input–output mappings that are discontinuous, so that a small change to an input can cause a large change in the output. For example, it may be possible to alter just a few pixels in an image of a dog and cause the network to classify the dog as an ostrich or a school bus—even though the altered image still looks exactly like a dog. An altered image of this kind is called an adversarial example.

Adversarial example

In low-dimensional spaces it is hard to find adversarial examples. But for an image with a million pixel values, it is often the case that even though most of the pixels contribute to the

image being classified in the middle of the “dog” region of the space, there are a few dimensions where the pixel value is near the boundary to another category. An adversary with the ability to reverse engineer the network can find the smallest vector difference that would move the image over the border.

When adversarial examples were first discovered, they set off two worldwide scrambles: one to find learning algorithms and network architectures that would not be susceptible to adversarial attack, and another to create ever-more-effective adversarial attacks against all kinds of learning systems. So far the attackers seem to be ahead. In fact, whereas it was assumed initially that one would need access to the internals of the trained network in order to construct an adversarial example specifically for that network, it has turned out that one can construct robust adversarial examples that fool multiple networks with different architectures, hyperparameters, and training sets. These findings suggest that deep learning models recognize objects in ways that are quite different from the human visual system.

21.5.2 Neural architecture search Unfortunately, we don’t yet have a clear set of guidelines to help you choose the best network architecture for a particular problem. Success in deploying a deep learning solution requires experience and good judgment.

From the earliest days of neural network research, attempts have been made to automate the process of architecture selection. We can think of this as a case of hyperparameter tuning (Section 19.4.4 ), where the hyperparameters determine the depth, width, connectivity, and other attributes of the network. However, there are so many choices to be made that simple approaches like grid search can’t cover all possibilities in a reasonable amount of time.

Neural architecture search

Therefore, it is common to use neural architecture search to explore the state space of possible network architectures. Many of the search techniques and learning techniques we covered earlier in the book have been applied to neural architecture search.

Evolutionary algorithms have been popular because it is sensible to do both recombination (joining parts of two networks together) and mutation (adding or removing a layer or changing a parameter value). Hill climbing can also be used with these same mutation operations. Some researchers have framed the problem as reinforcement learning, and some as Bayesian optimization. Another possibility is to treat the architectural possibilities as a continuous differentiable space and use gradient descent to find a locally optimal solution.

For all these search techniques, a major challenge is estimating the value of a candidate network. The straightforward way to evaluate an architecture is to train it on a test set for multiple batches and then evaluate its accuracy on a validation set. But with large networks that could take many GPU-days.

Therefore, there have been many attempts to speed up this estimation process by eliminating or at least reducing the expensive training process. We can train on a smaller data set. We can train for a small number of batches and predict how the network would improve with more batches. We can use a reduced version of the network architecture that we hope retains the properties of the full version. We can train one big network and then search for subgraphs of the network that perform better; this search can be fast because the subgraphs share parameters and don’t have to be retrained.

Another approach is to learn a heuristic evaluation function (as was done for A* search). That is, start by choosing a few hundred network architectures and train and evaluate them. That gives us a data set of (network, score) pairs. Then learn a mapping from the features of a network to a predicted score. From that point on we can generate a large number of candidate networks and quickly estimate their value. After a search through the space of networks, the best one(s) can be fully evaluated with a complete training procedure.

21.5.3 Weight decay In Section 19.4.3  we saw that regularization—limiting the complexity of a model—can aid generalization. This is true for deep learning models as well. In the context of neural networks we usually call this approach weight decay.

Weight decay

Weight decay consists of adding a penalty λ

∑i j Wi j to the loss function used to train the 2 ,

,

neural network, where λ is a hyperparameter controlling the strength of the penalty and the sum is usually taken over all of the weights in the network. Using λ = 0 is equivalent to not using weight decay, while using larger values of λ encourages the weights to become small. It is common to use weight decay with λ near 10−4 .

Choosing a specific network architecture can be seen as an absolute constraint on the hypothesis space: a function is either representable within that architecture or it is not. Loss function penalty terms such as weight decay offer a softer constraint: functions represented with large weights are in the function family, but the training set must provide more evidence in favor of these functions than is required to choose a function with small weights.

It is not straightforward to interpret the effect of weight decay in a neural network. In networks with sigmoid activation functions, it is hypothesized that weight decay helps to keep the activations near the linear part of the sigmoid, avoiding the flat operating region that leads to vanishing gradients. With ReLU activation functions, weight decay seems to be beneficial, but the explanation that makes sense for sigmoids no longer applies because the ReLU’s output is either linear or zero. Moreover, with residual connections, weight decay encourages the network to have small differences between consecutive layers rather than small absolute weight values. Despite these differences in the behavior of weight decay across many architectures, weight decay is still widely useful.

One explanation for the beneficial effect of weight decay is that it implements a form of maximum a posteriori (MAP) learning (see page 723). Letting

X and y stand for the inputs

and outputs across the entire training set, the maximum a posteriori hypothesis hMAP satisfies

y | X , W ) P ( W) w = argmin [− log P (y|X,W) − log P (W)]. w

hMAP = argmax P (

The first term is the usual cross-entropy loss; the second term prefers weights that are likely under a prior distribution. This aligns exactly with a regularized loss function if we set

log

P(

W

)=−

λ

∑W i,j

2

i,j ,

which means that P (

W is a zero-mean Gaussian prior. )

21.5.4 Dropout Another way that we can intervene to reduce the test-set error of a network—at the cost of making it harder to fit the training set—is to use dropout. At each step of training, dropout applies one step of back-propagation learning to a new version of the network that is created by deactivating a randomly chosen subset of the units. This is a rough and very lowcost approximation to training a large ensemble of different networks (see Section 19.8 ).

Dropout

More specifically, let us suppose we are using stochastic gradient descent with minibatch size m. For each minibatch, the dropout algorithm applies the following process to every node in the network: with probability p, the unit output is multiplied by a factor of 1/p; otherwise, the unit output is fixed at zero. Dropout is typically applied to units in the hidden layers with p = 0.5; for input units, a value of p = 0.8 turns out to be most effective. This process produces a thinned network with about half as many units as the original, to which back-propagation is applied with the minibatch of m training examples. The process repeats in the usual way until training is complete. At test time, the model is run with no dropout.

We can think of dropout from several perspectives:

By introducing noise at training time, the model is forced to become robust to noise. As noted above, dropout approximates the creation of a large ensemble of thinned networks. This claim can be verified analytically for linear models, and appears to hold experimentally for deep learning models. Hidden units trained with dropout must learn not only to be useful hidden units; they must also learn to be compatible with many other possible sets of other hidden units that may or may not be included in the full model. This is similar to the selection processes that guide the evolution of genes: each gene must not only be effective in its own function, but must work well with other genes, whose identity in future organisms may vary considerably.

Dropout applied to later layers in a deep network forces the final decision to be made robustly by paying attention to all of the abstract features of the example rather than focusing on just one and ignoring the others. For example, a classifier for animal images might be able to achieve high performance on the training set just by looking at the animal’s nose, but would presumably fail on a test case where the nose was obscured or damaged. With dropout, there will be training cases where the internal “nose unit” is zeroed out, causing the learning process to find additional identifying features. Notice that trying to achieve the same degree of robustness by adding noise to the input data would be difficult: there is no easy way to know in advance that the network is going to focus on noses, and no easy way to delete noses automatically from each image.

Altogether, dropout forces the model to learn multiple, robust explanations for each input. This causes the model to generalize well, but also makes it more difficult to fit the training set—it is usually necessary to use a larger model and to train it for more iterations.

21.6 Recurrent Neural Networks Recurrent neural networks (RNNs) are distinct from feedforward networks in that they allow cycles in the computation graph. In all the cases we will consider, each cycle has a delay, so that units may take as input a value computed from their own output at an earlier step in the computation. (Without the delay, a cyclic circuit may reach an inconsistent state.) This allows the RNN to have internal state, or memory: inputs received at earlier time steps affect the RNN’s response to the current input.

Memory

RNNs can also be used to perform more general computations—after all, ordinary computers are just Boolean circuits with memory—and to model real neural systems, many of which contain cyclic connections. Here we focus on the use of RNNs to analyze sequential data, where we assume that a new input vector

xt arrives at each time step.

As tools for analyzing sequential data, RNNs can be compared to the hidden Markov models, dynamic Bayesian networks, and Kalman filters described in Chapter 14 . (The reader may find it helpful to refer back to that chapter before proceeding.) Like those models, RNNs make a Markov assumption (see page 463): the hidden state

zt of the

network suffices to capture the information from all previous inputs. Furthermore, suppose we describe the RNN’s update process for the hidden state by the equation

z t = fw ( z t , x t ) −1

for some parameterized function fw . Once trained, this function represents a timehomogeneous process (page 463)—effectively a universally quantified assertion that the dynamics represented by fw hold for all time steps. Thus, RNNs add expressive power compared to feedforward networks, just as convolutional networks do, and just as dynamic Bayes nets add expressive power compared to regular Bayes nets. Indeed, if you tried to use a feedforward network to analyze sequential data, the fixed size of the input layer would force the network to examine only a finite-length window of data, in which case the network would fail to detect long-distance dependencies.

21.6.1 Training a basic RNN

x

The basic model we will consider has an input layer , a hidden layer

y

z

with recurrent

connections, and an output layer , as shown in Figure 21.8(a) . We assume that both and

y

x

are observed in the training data at each time step. The equations defining the model

refer to the values of the variables indexed by time step t: (21.13)

z fw z x g W z y g W z g in t =

ˆt =

( t−1 , t ) =

y(

z ,y t ) ≡

z(

y(

z,z t−1 +

y,t ),

W x g in x, z t ) ≡

z(

z ,t )

Figure 21.8

(a) Schematic diagram of a basic RNN where the hidden layer

z

has recurrent connections; the Δ symbol

indicates a delay. (b) The same network unrolled over three time steps to create a feedforward network. Note that the weights are shared across all time steps.

g

g

where z and y denote the activation functions for the hidden and output layers, respectively. As usual, we assume an extra dummy input fixed at +1 for each unit as well as bias weights associated with those inputs.

Given a sequence of input vectors

x

1,

x W W

… , T and observed outputs

y

1

y

, … , T , we can turn

this model into a feedforward network by “unrolling” it for T steps, as shown in Figure 21.8(b) . Notice that the weight matrices

x, z ,

z,z , and

W

z,y are shared across all time

steps. In the unrolled network, it is easy to see that we can calculate gradients to train the weights in the usual way; the only difference is that the sharing of weights across layers makes the gradient computation a little more complicated.

To keep the equations simple, we will show the gradient calculation for an RNN with just one input unit, one hidden unit, and one output unit. For this case, making the bias weights explicit, we have zt = gz (wz,z zt−1 + wx,z xt + w0,z ) and yˆt = gy (wz,y zt + w0,y ). As in Equations

(21.4)  and (21.5) , we will assume a squared-error loss L—in this case, summed over the time steps. The derivations for the input-layer and output-layer weights wx,z and wz,y are

essentially identical to Equation (21.4) , so we leave them as an exercise. For the hiddenlayer weight wz,z , the first few steps also follow the same pattern as Equation (21.4) : (21.14)

L ∂ wz , z ∂

= = = =

∑ yt y t ∑ yt y t wz z ∑ yt y t wz z gy iny t ∑ ∑ yt y t gy iny t wz z wz yzt ∑ yt y t gy iny t wz y wzzt z T

∂

∂ T

t=1 T

t=1 T

t=1

,

t=1

−2(

−2(

−2(

(

− ˆ )2 =

−ˆ)

−ˆ)

−ˆ)

T

t=1

∂

∂ ′

′

(

,

,

(

(

,

)

,

)

−ˆ)

−2(

)=

∂

∂

(

,

∂

t=1 ,

∂

,

T

y ∂ wz , z ∂ ˆt

−2( +

yt − yˆt )gy′ (iny,t )

∂

∂

wz ,z

iny,t

w0,y )

.

,

Now the gradient for the hidden unit zt can be obtained from the previous time step as follows:

(21.15)

zt ∂ wz , z ∂

∂

∂

∂

gz (inz,t ) = gz′ (inz,t ) ∂w inz,t = gz′ (inz,t ) ∂w (wz,z zt−1 + wx,z xt + w0,z ) ∂wz ,z z ,z z ,z ∂ z t−1 = gz′ (inz,t )(zt−1 + wz,z ), ∂wz ,z =

where the last line uses the rule for derivatives of products: ∂(uv)/∂x = v∂u/∂x + u∂v/∂x. Looking at Equation (21.15) , we notice two things. First, the gradient expression is recursive: the contribution to the gradient from time step t is calculated using the

contribution from time step t − 1. If we order the calculations in the right way, the total run time for computing the gradient will be linear in the size of the network. This algorithm is called back-propagation through time, and is usually handled automatically by deep learning software systems. Second, if we iterate the recursive calculation, we see that

T

gradients at T will include terms proportional to wz,z ∏t=1 gz′ (inz,t ). For sigmoids, tanhs, and ReLUs, g ′ ≤ 1, so our simple RNN will certainly suffer from the vanishing gradient problem

(see page 756) if

wz z ,

< 1.

On the other hand, if

wz z ,

> 1,

we may experience the exploding

gradient problem. (For the general case, these outcomes depend on the first eigenvalue of the weight matrix

Wz z ,

.) The next section describes a more elaborate RNN design intended

to mitigate this issue.

Back-propagation through time

Exploding gradient

21.6.2 Long short-term memory RNNs Several specialized RNN architectures have been designed with the goal of enabling information to be preserved over many time steps. One of the most popular is the long short-term memory or LSTM. The long-term memory component of an LSTM, called the

c

memory cell and denoted by , is essentially copied from time step to time step. (In contrast, the basic RNN multiplies its memory by a weight matrix at every time step, as shown in Equation (21.13) .) New information enters the memory by adding updates; in this way, the gradient expressions do not accumulate multiplicatively over time. LSTMs also include gating units, which are vectors that control the flow of information in the LSTM via elementwise multiplication of the corresponding information vector:

f

The forget gate determines if each element of the memory cell is remembered (copied to the next time step) or forgotten (reset to zero).

Forget gate

i

The input gate determines if each element of the memory cell is updated additively by new information from the input vector at the current time step.

Input gate

The output gate

o

determines if each element of the memory cell is transferred to the

z

short-term memory , which plays a similar role to the hidden state in basic RNNs.

Output gate

Long short-term memory

Memory cell

Gating unit

Whereas the word “gate” in circuit design usually connotes a Boolean function, gates in LSTMs are soft—for example, elements of the memory cell vector will be partially forgotten if the corresponding elements of the forget-gate vector are small but not zero. The values for the gating units are always in the range [0,1] and are obtained as the outputs of a sigmoid function applied to the current input and the previous hidden state. In detail, the update equations for the LSTM are as follows:

f W x i Wx o W x c c f z c

W z Wz Wz i W x Wz o

t

= σ(

x, f

t

+

z,f t−1 )

t

= σ(

x, i

t

+

z,i t−1 )

t

= σ(

x, o

t

+

z,o t−1 )

t

=

t

= tanh(

t−1

⊙

t

+

t)

⊙

t

⊙ tanh(

t,

x, c

t

+

z,c t−1 )

where the subscripts on the various weight matrices of the corresponding links. The

W

indicate the origin and destination

⊙ symbol denotes elementwise multiplication.

LSTMs were among the first practically usable forms of RNN. They have demonstrated excellent performance on a wide range of tasks including speech recognition and handwriting recognition. Their use in natural language processing is discussed in Chapter 24 .

21.7 Unsupervised Learning and Transfer Learning The deep learning systems we have discussed so far are based on supervised learning, which requires each training example to be labeled with a value for the target function. Although such systems can reach a high level of test-set accuracy—as shown by the ImageNet competition results, for example—they often require far more labeled data than a human would for the same task. For example, a child needs to see only one picture of a giraffe, rather than thousands, in order to be able to recognize giraffes reliably in a wide range of settings and views. Clearly, something is missing in our deep learning story; indeed, it may be the case that our current approach to supervised deep learning renders some tasks completely unattainable because the requirements for labeled data would exceed what the human race (or the universe) can supply. Moreover, even in cases where the task is feasible, labeling large data sets usually requires scarce and expensive human labor.

For these reasons, there is intense interest in several learning paradigms that reduce the dependence on labeled data. As we saw in Chapter 19 , these paradigms include unsupervised learning, transfer learning, and semisupervised learning. Unsupervised learning algorithms learn solely from unlabeled inputs x, which are often more abundantly available than labeled examples. Unsupervised learning algorithms typically produce generative models, which can produce realistic text, images, audio, and video, rather than simply predicting labels for such data. Transfer learning algorithms require some labeled examples but are able to improve their performance further by studying labeled examples for different tasks, thus making it possible to draw on more existing sources of data. Semisupervised learning algorithms require some labeled examples but are able to improve their performance further by also studying unlabeled examples. This section covers deep learning approaches to unsupervised and transfer learning; while semisupervised learning is also an active area of research in the deep learning community, the techniques developed so far have not proven broadly effective in practice, so we do not cover them.

21.7.1 Unsupervised learning Supervised learning algorithms all have essentially the same goal: given a training set of inputs x and corresponding outputs y = f (x), learn a function h that approximates f well. Unsupervised learning algorithms, on the other hand, take a training set of unlabeled

examples x. Here we describe two things that such an algorithm might try to do. The first is to learn new representations—for example, new features of images that make it easier to identify the objects in an image. The second is to learn a generative model—typically in the form of a probability distribution from which new samples can be generated. (The algorithms for learning Bayes nets in Chapter 20  fall in this category.) Many algorithms are capable of both representation learning and generative modeling. Suppose we learn a joint model PW (x,z), where z is a set of latent, unobserved variables that represent the content of the data x in some way. In keeping with the spirit of the chapter, we do not predefine the meanings of the z variables; the model is free to learn to associate z with x however it chooses. For example, a model trained on images of handwritten digits might choose to use one direction in z space to represent the thickness of pen strokes, another to represent ink color, another to represent background color, and so on. With images of faces, the learning algorithm might choose one direction to represent gender and another to capture the presence or absence of glasses, as illustrated in Figure 21.9 .

Figure 21.9

A demonstration of how a generative model has learned to use different directions in z space to represent different aspects of faces. We can actually perform arithmetic in z space. The images here are all generated from the learned model and show what happens when we decode different points in z space. We start with the coordinates for the concept of “man with glasses,” subtract off the coordinates for “man,” add the coordinates for “woman,” and obtain the coordinates for “woman with glasses.” Images reproduced with permission from (Radford et al., 2015).

A learned probability model PW (x,z) achieves both representation learning (it has constructed meaningful z vectors from the raw x vectors) and generative modeling: if we integrate z out of PW (x,z) we obtain PW (x).

Probabilistic PCA: A simple generative model

PPCA

There have been many proposals for the form that

PW (x,z) might take. One of the simplest

is the probabilistic principal components analysis (PPCA) model.7 In a PPCA model, z is chosen from a zero-mean, spherical Gaussian, then x is generated from z by applying a weight matrix W and adding spherical Gaussian noise:

7 Standard PCA involves fitting a multivariate Gaussian to the raw input data and then selecting out the longest axes—the principal components—of that ellipsoidal distribution.

P ( z ) = N ( z ; 0 ,I) PW (x|z) = N (x; Wz,σ I). 2

The weights W (and optionally the noise parameter

σ ) can be learned by maximizing the 2

likelihood of the data, given by

(21.16)

PW (x) = ∫ PW (x,z)dz = N (x; 0,WW

⊤

+

σ I). 2

The maximization with respect to W can be done by gradient methods or by an efficient

iterative EM algorithm (see Section 20.3 ). Once W has been learned, new data samples can be generated directly from

PW (x) using Equation (21.16) . Moreover, new observations

x that have very low probability according to Equation (21.16)  can be flagged as potential anomalies. With PPCA, we usually assume that the dimensionality of z is much less than the

dimensionality of x, so that the model learns to explain the data as well as possible in terms of a small number of features. These features can be extracted for use in standard classifiers ˆ, the expectation of by computing z

PW ( z |x ) .

Generating data from a probabilistic PCA model is straightforward: first sample fixed Gaussian prior, then sample

x

from a Gaussian with mean

Wz

z

from its

. As we will see shortly,

many other generative models resemble this process, but use complicated mappings defined

z

x

by deep models rather than linear mappings from -space to -space.

Autoencoders Many unsupervised deep learning algorithms are based on the idea of an autoencoder. An autoencoder is a model containing two parts: an encoder that maps from

z

z f x gfx

x

to a

x

representation ˆ and a decoder that maps from a representation ˆ to observed data . In general, the encoder is just a parameterized function

and the decoder is just a

parameterized function g. The model is trained so that

≈ ( ( )), so that the encoding

process is roughly inverted by the decoding process. The functions f and g can be simple linear models parameterized by a single matrix or they can be represented by a deep neural network.

Autoencoder

A very simple autoencoder is the linear autoencoder, where both f and g are linear with a shared weight matrix

W

:

z f x Wx x gz W z ˆ=

( )=

=

(ˆ) =

⊤

ˆ.

One way to train this model is to minimize the squared error

x

x

≈ g(f ( )). The idea is to train

W

∑ x

z

j ∥

j−

g (f (

x

2

∥ so that

j ))∥

so that a low-dimensional ˆ will retain as much

x

information as possible to reconstruct the high-dimensional data . This linear autoencoder turns out to be closely connected to classical principal components analysis (PCA). When is m-dimensional, the matrix

W

should learn to span the m principal components of the

data—in other words, the set of m orthogonal directions in which the data has highest variance, or equivalently the m eigenvectors of the data covariance matrix that have the largest eigenvalues—exactly as in PCA.

z

The PCA model is a simple generative model that corresponds to a simple linear autoencoder. The correspondence suggests that there may be a way to capture more complex kinds of generative models using more complex kinds of autoencoders. The variational autoencoder (VAE) provides one way to do this.

Variational autoencoder

Variational methods were introduced briefly on page 458 as a way to approximate the posterior distribution in complex probability models, where summing or integrating out a large number of hidden variables is intractable. The idea is to use a variational posterior

Q(z), drawn from a computationally tractable family of distributions, as an approximation to the true posterior. For example, we might choose Q from the family of Gaussian distributions with a diagonal covariance matrix. Within the chosen family of tractable distributions,

P ( z |x ) .

Q is optimized to be as close as possible to the true posterior distribution

Variational posterior

For our purposes, the notion of “as close as possible” is defined by the KL divergence, which we mentioned on page 758. This is given by

Q( z) DKL (Q(z) ∥ P (z|x)) = ∫ Q(z) log P (z|x) dz, Q) of the log ratio between Q and P . It is easy to see that DKL (Q(z) ∥ P (z|x)) ≥ 0, with equality when Q and P coincide. We can then define the variational lower bound L (sometimes called the evidence lower bound, or ELBO) on the which is an average (with respect to

log likelihood of the data:

(21.17)

L(x,Q) = log P (x) − DKL (Q(z) ∥ P (z|x)).

Variational lower bound

ELBO

L is a lower bound for log P because the KL divergence is nonnegative. Variational learning maximizes L with respect to parameters w rather than maximizing log P (x), in the hope that the solution found, w , is close to maximizing log P (x) as well. We can see that

∗

As written,

L does not yet seem to be any easier to maximize than log P . Fortunately, we can

rewrite Equation (21.17)  to reveal improved computational tractability:

L

=

log

=

−∫

=

Q(z) P (x) − ∫ Q(z) log P (z|x) dz

Q(z) log Q(z)dz + ∫ Q(z) log P (x)P (z∣∣x)dz H (Q) + Ez Q log P (z,x) ∼

H (Q) is the entropy of the Q distribution. For some variational families Q (such as Gaussian distributions), H (Q) can be evaluated analytically. Moreover, the expectation, Ez Q log P (z,x), admits an efficient unbiased estimate via samples of z from Q. For each sample, P (z,x) can usually be evaluated efficiently—for example, if P is a Bayes net, P (z,x) where

∼

is just a product of conditional probabilities because

z and x comprise all the variables.

Variational autoencoders provide a means of performing variational learning in the deep

L with respect to the parameters of both P and Q. For a variational autoencoder, the decoder g(z) is interpreted as defining log P (x|z). For example, the output of the decoder might define the mean of a conditional Gaussian. Similarly, the output of the encoder f (x) is interpreted as defining the parameters of Q—for example, Q might be a Gaussian with mean f (x). Training the variational autoencoder then consists of maximizing L with respect to the parameters of both the learning setting. Variational learning involves maximizing

encoder

f and the decoder g, which can themselves be arbitrarily complicated deep

networks.

Deep autoregressive models An autoregressive model (or AR model) is one in which each element xi of the data vector

x is predicted based on other elements of the vector. Such a model has no latent variables. If x is of fixed size, an AR model can be thought of as a fully observable and possibly fully connected Bayes net. This means that calculating the likelihood of a given data vector according to an AR model is trivial; the same holds for predicting the value of a single missing variable given all the others, and for sampling a data vector from the model.

Autoregressive model

The most common application of autoregressive models is in the analysis of time series data, where an AR model of order k predicts xt given xt−k , … ,xt−1 . In the terminology of Chapter

14 , an AR model is a non-hidden Markov model. In the terminology of Chapter 23 , an ngram model of letter or word sequences is an AR model of order n − 1.

In classical AR models, where the variables are real-valued, the conditional distribution

P (xt |xt−k , … ,xt−1 ) is a linear–Gaussian model with fixed variance whose mean is a weighted linear combination of xt−k , … ,xt−1 —in other words, a standard linear regression model. The maximum likelihood solution is given by the Yule–Walker equations, which are closely related to the normal equations on page 680.

Yule–Walker equations

A deep autoregressive model is one in which the linear–Gaussian model is replaced by an arbitrary deep network with a suitable output layer depending on whether xt is discrete or continuous. Recent applications of this autoregressive approach include DeepMind’s

WaveNet model for speech generation (van den Oord et al., 2016a). WaveNet is trained on raw acoustic signals, sampled 16,000 times per second, and implements a nonlinear AR model of order 4800 with a multilayer convolutional structure. In tests it proves to be substantially more realistic than previous state-of-the-art speech generation systems.

Deep autoregressive model

Generative adversarial networks A generative adversarial network (GAN) is actually a pair of networks that combine to form

z to x in order to produce samples from the distribution Pw (x). A typical scheme samples z from a unit Gaussian of moderate dimension and then passes it through a deep network hw to obtain x. The other network, the discriminator, is a classifier trained to classify inputs x as real a generative system. One of the networks, the generator, maps values from

(drawn from the training set) or fake (created by the generator). GANs are a kind of implicit model in the sense that samples can be generated but their probabilities are not readily available; in a Bayes net, on the other hand, the probability of a sample is just the product of the conditional probabilities along the sample generation path.

Generative adversarial network (GAN)

Generator

Discriminator

Implicit model

The generator is closely related to the decoder from the variational autoencoder framework. The challenge in implicit modeling is to design a loss function that makes it possible to train the model using samples from the distribution, rather than maximizing the likelihood assigned to training examples from the data set.

Both the generator and the discriminator are trained simultaneously, with the generator learning to fool the discriminator and the discriminator learning to accurately separate real from fake data. The competition between generator and discriminator can be described in the language of game theory (see Chapter 18 ). The idea is that in the equilibrium state of the game, the generator should reproduce the training distribution perfectly, such that the discriminator cannot perform better than random guessing. GANs have worked particularly well for image generation tasks. For example, GANs can create photorealistic, highresolution images of people who have never existed (Karras et al., 2017).

Unsupervised translation Translation tasks, broadly construed, consist of transforming an input x that has rich structure into an output y that also has rich structure. In this context, “rich structure” means that the data are multidimensional and have interesting statistical dependencies among the various dimensions. Images and natural language sentences have a rich structure, but a single number, such as a class ID, does not. Transforming a sentence from English to French or converting a photo of a night scene into an equivalent photo taken during the daytime are both examples of translation tasks. Supervised translation consists of gathering many (x,y) pairs and training the model to map each x to the corresponding y. For example, machine translation systems are often trained on pairs of sentences that have been translated by professional human translators. For other kinds of translation, supervised training data may not be available. For example, consider a photo of a night scene containing many moving cars and pedestrians. It is presumably not feasible to find all of the cars and pedestrians and return them to their original positions in the night-time photo in order to retake the same photo in the daytime. To overcome this difficulty, it is possible to use unsupervised translation techniques that are capable of

training on many examples of x and many separate examples of y but no corresponding (x,y) pairs.

Unsupervised translation

These approaches are generally based on GANs; for example, one can train a GAN generator to produce a realistic example of y when conditioned on x, and another GAN generator to perform the reverse mapping. The GAN training framework makes it possible to train a generator to generate any one of many possible samples that the discriminator accepts as a realistic example of y given x, without any need for a specific paired y as is traditionally needed in supervised learning. More detail on unsupervised translation for images is given in Section 25.7.5 .

21.7.2 Transfer learning and multitask learning In transfer learning, experience with one learning task helps an agent learn better on another task. For example, a person who has already learned to play tennis will typically find it easier to learn related sports such as racquetball and squash; a pilot who has learned to fly one type of commercial passenger airplane will very quickly learn to fly another type; a student who has already learned algebra finds it easier to learn calculus.

Transfer learning

We do not yet know the mechanisms of human transfer learning. For neural networks, learning consists of adjusting weights, so the most plausible approach for transfer learning is to copy over the weights learned for task A to a network that will be trained for task B. The weights are then updated by gradient descent in the usual way using data for task B. It may be a good idea to use a smaller learning rate in task B, depending on how similar the tasks are and how much data was used in task A.

Notice that this approach requires human expertise in selecting the tasks: for example, weights learned during algebra training may not be very useful in a network intended for racquetball. Also, the notion of copying weights requires a simple mapping between the input spaces for the two tasks and essentially identical network architectures.

One reason for the popularity of transfer learning is the availability of high-quality pretrained models. For example, you could download a pretrained visual object recognition model such as the ResNet-50 model trained on the COCO data set, thereby saving yourself weeks of work. From there you can modify the model parameters by supplying additional images and object labels for your specific task.

Suppose you want to classify types of unicycles. You have only a few hundred pictures of different unicycles, but the COCO data set has over 3,000 images in each of the categories of bicycles, motorcycles, and skateboards. This means that a model pretrained on COCO already has experience with wheels and roads and other relevant features that will be helpful in interpreting the unicycle images.

Often you will want to freeze the first few layers of the pretrained model—these layers serve as feature detectors that will be useful for your new model. Your new data set will be allowed to modify the parameters of the higher levels only; these are the layers that identify problem-specific features and do classification. However, sometimes the difference between sensors means that even the lowest-level layers need to be retrained.

As another example, for those building a natural language system, it is now common to start with a pretrained model such as the ROBERTA model (see Section 24.6 ), which already “knows” a great deal about the vocabulary and syntax of everyday language. The next step is to fine-tune the model in two ways. First, by giving it examples of the specialized vocabulary used in the desired domain; perhaps a medical domain (where it will learn about “mycardial infarction”) or perhaps a financial domain (where it will learn about “fiduciary responsibility”). Second, by training the model on the task it is to perform. If it is to do question answering, train it on question/answer pairs.

One very important kind of transfer learning involves transfer between simulations and the real world. For example, the controller for a self-driving car can be trained on billions of

miles of simulated driving, which would be impossible in the real world. Then, when the controller is transitioned to the real vehicle, it adapts quickly to the new environment.

Multitask learning is a form of transfer learning in which we simultaneously train a model on multiple objectives. For example, rather than training a natural language system on partof-speech tagging and then transferring the learned weights to a new task such as document classification, we train one system simultaneously on part-of-speech tagging, document classification, language detection, word prediction, sentence difficulty modeling, plagiarism detection, sentence entailment, and question answering. The idea is that to solve any one of these tasks, a model might be able to take advantage of superficial features of the data. But to solve all eight at once with a common representation layer, the model is more likely to create a common representation that reflects real natural language usage and content.

Multitask learning

21.8 Applications Deep learning has been applied successfully to many important problem areas in AI. For indepth explanations, we refer the reader to the relevant chapters: Chapter 22  for the use of deep learning in reinforcement learning systems, Chapter 24  for natural language processing, Chapter 25  (particularly Section 25.4 ) for computer vision, and Chapter 26  for robotics.

21.8.1 Vision We begin with computer vision, which is the application area that has arguably had the biggest impact on deep learning, and vice versa. Although deep convolutional networks had been in use since the 1990s for tasks such as handwriting recognition, and neural networks had begun to surpass generative probability models for speech recognition by around 2010, it was the success of the AlexNet deep learning system in the 2012 ImageNet competition that propelled deep learning into the limelight.

The ImageNet competition was a supervised learning task with 1,200,000 images in 1,000 different categories, and systems were evaluated on the “top-5” score—how often the correct category appears in the top five predictions. AlexNet achieved an error rate of 15.3%, whereas the next best system had an error rate of more than 25%. AlexNet had five convolutional layers interspersed with max-pooling layers, followed by three fully connected layers. It used ReLU activation functions and took advantage of GPUs to speed up the process of training 60 million weights.

Since 2012, with improvements in network design, training methods, and computing resources, the top-5 error rate has been reduced to less than 2%—well below the error rate of a trained human (around 5%). CNNs have been applied to a wide range of vision tasks, from self-driving cars to grading cucumbers.8 Driving, which is covered in Section 25.7.6  and in several sections of Chapter 26 , is among the most demanding of vision tasks: not only must the algorithm detect, localize, track, and recognize pigeons, paper bags, and pedestrians, but it has to do it in real time with near-perfect accuracy.

8 The widely known tale of the Japanese cucumber farmer who built his own cucumber-sorting robot using TensorFlow is, it turns out, mostly mythical. The algorithm was developed by the farmer’s son, who worked previously as a software engineer at Toyota, and

its low accuracy—about 70%—meant that the cucumbers still had to be sorted by hand (Zeeberg, 2017).

21.8.2 Natural language processing Deep learning has also had a huge impact on natural language processing (NLP) applications such as machine translation and speech recognition. Some advantages of deep learning for these applications include the possibility of end-to-end learning, the automatic generation of internal representations for the meanings of words, and the interchangeability of learned encoders and decoders.

End-to-end learning refers to the construction of entire systems as a single, learned function

f . For example, an f for machine translation might take as input an English sentence SE and produce an equivalent Japanese sentence SJ = f (SE ). Such an f can be learned from training data in the form of human-translated pairs of sentences (or even pairs of texts, where the alignment of corresponding sentences or phrases is part of the problem to be solved). A more classical pipeline approach might first parse SE , then extract its meaning, then reexpress the meaning in Japanese as SJ , then post-edit SJ using a language model for Japanese. This pipeline approach has two major drawbacks: first, errors are compounded at each stage; and second, humans have to determine what constitutes a “parse tree” and a “meaning representation,” but there is no easily accessible ground truth for these notions, and our theoretical ideas about them are almost certainly incomplete.

At our present stage of understanding, then, the classical pipeline approach—which, at least naively, seems to correspond to how a human translator works—is outperformed by the end-to-end method made possible by deep learning. For example, Wu et al. (2016b) showed that end-to-end translation using deep learning reduced translation errors by 60% relative to a previous pipeline-based system. As of 2020, machine translation systems are approaching human performance for language pairs such as French and English for which very large paired data sets are available, and they are usable for other language pairs covering the majority of Earth’s population. There is even some evidence that networks trained on multiple languages do in fact learn an internal meaning representation: for example, after learning to translate Portuguese to English and English to Spanish, it is possible to translate Portuguese directly into Spanish without any Portuguese/Spanish sentence pairs in the training set.

One of the most significant findings to emerge from the application of deep learning to language tasks is that a great deal deal of mileage comes from re-representing individual words as vectors in a high-dimensional space—so-called word embeddings (see Section 24.1 ). The vectors are usually extracted from the weights of the first hidden layer of a network trained on large quantities of text, and they capture the statistics of the lexical contexts in which words are used. Because words with similar meanings are used in similar contexts, they end up close to each other in the vector space. This allows the network to generalize effectively across categories of words, without the need for humans to predefine those categories. For example, a sentence beginning “John bought a watermelon and two pounds of ...” is likely to continue with “apples” or “bananas” but not with “thorium” or “geography.” Such a prediction is much easier to make if “apples” and “bananas” have similar representations in the internal layer.

21.8.3 Reinforcement learning In reinforcement learning (RL), a decision-making agent learns from a sequence of reward signals that provide some indication of the quality of its behavior. The goal is to optimize the sum of future rewards. This can be done in several ways: in the terminology of Chapter 17 , the agent can learn a value function, a Q-function, a policy, and so on. From the point of view of deep learning, all these are functions that can be represented by computation graphs. For example, a value function in Go takes a board position as input and returns an estimate of how advantageous the position is for the agent. While the methods of training in RL differ from those of supervised learning, the ability of multilayer computation graphs to represent complex functions over large input spaces has proved to be very useful. The resulting field of research is called deep reinforcement learning.

Deep reinforcement learning

In the 1950s, Arthur Samuel experimented with multilayer representations of value functions in his work on reinforcement learning for checkers, but he found that in practice a linear function approximator worked best. (This may have been a consequence of working with a computer roughly 100 billion times less powerful than a modern tensor processing unit.) The first major successful demonstration of deep RL was DeepMind’s Atari-playing

agent, DQN (Mnih et al., 2013). Different copies of this agent were trained to play each of several different Atari video games, and demonstrated skills such as shooting alien spaceships, bouncing balls with paddles, and driving simulated racing cars. In each case, the agent learned a Q-function from raw image data with the reward signal being the game score. Subsequent work has produced deep RL systems that play at a superhuman level on the majority of the 57 different Atari games. DeepMind’s ALPHAGO system also used deep RL to defeat the best human players at the game of Go (see Chapter 5 ).

Despite its impressive successes, deep RL still faces significant obstacles: it is often difficult to get good performance, and the trained system may behave very unpredictably if the environment differs even a little from the training data (Irpan, 2018). Compared to other applications of deep learning, deep RL is rarely applied in commercial settings. It is, nonetheless, a very active area of research.

Summary This chapter described methods for learning functions represented by deep computational graphs. The main points were:

Neural networks represent complex nonlinear functions with a network of parameterized linear-threshold units. The back-propagation algorithm implements a gradient descent in parameter space to minimize the loss function. Deep learning works well for visual object recognition, speech recognition, natural language processing, and reinforcement learning in complex environments. Convolutional networks are particularly well suited for image processing and other tasks where the data have a grid topology. Recurrent networks are effective for sequence-processing tasks including language modeling and machine translation.

Bibliographical and Historical Notes The literature on neural networks is vast. Cowan and Sharp (1988b, 1988a) survey the early history, beginning with the work of McCulloch and Pitts (1943). (As mentioned in Chapter 1 , John McCarthy has pointed to the work of Nicolas Rashevsky (1936, 1938) as the earliest mathematical model of neural learning.) Norbert Wiener, a pioneer of cybernetics and control theory (Wiener, 1948), worked with McCulloch and Pitts and influenced a number of young researchers, including Marvin Minsky, who may have been the first to develop a working neural network in hardware, in 1951 (see Minsky and Papert, 1988, pp. ix–x). Alan Turing (1948) wrote a research report titled Intelligent Machinery that begins with the sentence “I propose to investigate the question as to whether it is possible for machinery to show intelligent behaviour” and goes on to describe a recurrent neural network architecture he called “B-type unorganized machines” and an approach to training them. Unfortunately, the report went unpublished until 1969, and was all but ignored until recently.

The perceptron, a one-layer neural network with a hard-threshold activation function, was popularized by Frank Rosenblatt (1957). After a demonstration in July 1958, the New York Times described it as “the embryo of an electronic computer that [the Navy] expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.” Rosenblatt (1960) later proved the perceptron convergence theorem, although it had been foreshadowed by purely mathematical work outside the context of neural networks (Agmon, 1954; Motzkin and Schoenberg, 1954). Some early work was also done on multilayer networks, including Gamba perceptrons (Gamba et al., 1961) and madalines (Widrow, 1962). Learning Machines (Nilsson, 1965) covers much of this early work and more. The subsequent demise of early perceptron research efforts was hastened (or, the authors later claimed, merely explained) by the book Perceptrons (Minsky and Papert, 1969), which lamented the field’s lack of mathematical rigor. The book pointed out that single-layer perceptrons could represent only linearly separable concepts and noted the lack of effective learning algorithms for multilayer networks. These limitations were already well known (Hawkins, 1961) and had been acknowledged by Rosenblatt himself (Rosenblatt, 1962).

The papers collected by Hinton and Anderson (1981), based on a conference in San Diego in 1979, can be regarded as marking a renaissance of connectionism. The two-volume “PDP”

(Parallel Distributed Processing) anthology (Rumelhart and McClelland, 1986) helped to spread the gospel, so to speak, particularly in the psychology and cognitive science communities. The most important development of this period was the back-propagation algorithm for training multilayer networks.

The back-propagation algorithm was discovered independently several times in different contexts (Kelley, 1960; Bryson, 1962; Dreyfus, 1962; Bryson and Ho, 1969; Werbos, 1974; Parker, 1985) and Stuart Dreyfus (1990) calls it the “Kelley–Bryson gradient procedure.” Although Werbos had applied it to neural networks, this idea did not become widely known until a paper by David Rumelhart, Geoff Hinton, and Ron Williams (1986) appeared in Nature giving a nonmathematical presentation of the algorithm. Mathematical respectability was enhanced by papers showing that multilayer feedforward networks are (subject to technical conditions) universal function approximators (Cybenko, 1988, Cybenko, 1989). The late 1980s and early 1990s saw a huge growth in neural network research: the number of papers mushroomed by a factor of 200 between 1980–84 and 1990–94.

In the late 1990s and early 2000s, interest in neural networks waned as other techniques such as Bayes nets, ensemble methods, and kernel machines came to the fore. Interest in deep models was sparked when Geoff Hinton’s research on deep Bayesian networks— generative models with category variables at the root and evidence variables at the leaves— began to bear fruit, outperforming kernel machines on small benchmark data sets (Hinton et al., 2006). Interest in deep learning exploded when Krizhevsky et al., 2013 used deep convolutional networks to win the ImageNet competition (Russakovsky et al., 2015).

Commentators often cite the availability of “big data” and the processing power of GPUs as the main contributing factors in the emergence of deep learning. Architectural improvements were also important, including the adoption of the ReLU activation function instead of the logistic sigmoid (Jarrett et al., 2009; Nair and Hinton, 2010; Glorot et al., 2011) and later the development of residual networks (He et al., 2016).

On the algorithmic side, the use of stochastic gradient descent (SGD) with small batches was essential in allowing neural networks to scale to large data sets (Bottou and Bousquet, 2008). Batch normalization (Ioffe and Szegedy, 2015) also helped in making the training process faster and more reliable and has spawned several additional normalization techniques (Ba et al., 2016; Wu and He, 2018; Miyato et al., 2018). Several papers have

studied the empirical behavior of SGD on large networks and large data sets (Dauphin et al., 2015; Choromanska et al., 2014; Goodfellow et al., 2015b). On the theoretical side, some progress has been made on explaining the observation that SGD applied to overparameterized networks often reaches a global minimum with a training error of zero, although so far the theorems to this effect assume a network with layers far wider than would ever occur in practice (Allen-Zhu et al., 2018; Du et al., 2018). Such networks have more than enough capacity to function as lookup tables for the training data.

The last piece of the puzzle, at least for vision applications, was the use of convolutional networks. These had their origins in the descriptions of the mammalian visual system by neurophysiologists David Hubel and Torsten Wiesel (Hubel and Wiesel, 1959, 1962, 1968). They described “simple cells” in the visual system of a cat that resemble edge detectors, as well as “complex cells” that are invariant to some transformations such as small spatial translations. In modern convolutional networks, the output of a convolution is analogous to a simple cell while the output of a pooling layer is analogous to a complex cell.

The work of Hubel and Wiesel inspired many of the early connectionist models of vision (Marr and Poggio, 1976). The neocognitron (Fukushima, 1980; Fukushima and Miyake, 1982), designed as a model of the visual cortex, was essentially a convolutional network in terms of model architecture, although an effective training algorithm for such networks had to wait until Yann LeCun and collaborators showed how to apply back-propagation (LeCun et al., 1995). One of the early commercial successes of neural networks was handwritten digit recognition using convolutional networks (LeCun et al., 1995).

Recurrent neural networks (RNNs) were commonly proposed as models of brain function in the 1970s, but no effective learning algorithms were associated with these proposals. The method of back-propagation through time appears in the PhD thesis of Paul Werbos (1974), and his later review paper (Werbos, 1990) gives several additional references to rediscoveries of the method in the 1980s. One of the most influential early works on RNNs was due to Jeff Elman (1990), building on an RNN architecture suggested by Michael Jordan (1986). Williams and Zipser (1989) present an algorithm for online learning in RNNs. Bengio et al., 1994 analyzed the problem of vanishing gradients in recurrent networks. The long short-term memory (LSTM) architecture (Hochreiter, 1991; Hochreiter and Schmidhuber, 1997; Gers et al., 2000) was proposed as a way of avoiding this problem. More

recently, effective RNN designs have been derived automatically (Jozefowicz et al., 2015; Zoph and Le, 2016).

Many methods have been tried for improving generalization in neural networks. Weight decay was suggested by Hinton (1987) and analyzed mathematically by Krogh and Hertz (1992). The dropout method is due to Srivastava et al. (2014a). Szegedy et al., 2013 introduced the idea of adversarial examples, spawning a huge literature.

Poole et al., 2017 showed that deep networks (but not shallow ones) can disentangle complex functions into flat manifolds in the space of hidden units. Rolnick and Tegmark (2018) showed that the number of units required to approximate a certain class of polynomials of

n variables grows exponentially for shallow networks but only linearly for

deep networks.

White et al., 2019 showed that their BANANAS system could do neural architecture search (NAS) by predicting the accuracy of a network to within 1% after training on just 200 random sample architectures. Zoph and Le (2016) use reinforcement learning to search the space of neural network architectures. Real et al., 2018 use an evolutionary algorithm to do model selection, Liu et al., 2017 use evolutionary algorithms on hierarchical representations, and Jaderberg et al., 2017 describe population-based training. Liu et al., 2019 relax the space of architectures to a continuous differentiable space and use gradient descent to find a locally optimal solution. Pham et al., 2018 describe the ENAS (Efficient Neural Architecture Search) system, which searches for optimal subgraphs of a larger graph. It is fast because it does not need to retrain parameters. The idea of searching for a subgraph goes back to the “optimal brain damage” algorithm of LeCun et al., (1990).

Despite this impressive array of approaches, there are critics who feel the field has not yet matured. Yu et al., 2019 show that in some cases these NAS algorithms are no more efficient than random architecture selection. For a survey of recent results in neural architecture search, see Elsken et al., 2018.

Unsupervised learning constitutes a large subfield within statistics, mostly under the heading of density estimation. Silverman (1986) and Murphy (2012) are good sources for classical and modern techniques in this area. Principal components analysis (PCA) dates back to Pearson (1901); the name comes from independent work by Hotelling (1933). The

probabilistic PCA model (Tipping and Bishop, 1999) adds a generative model for the principal components themselves. The variational autoencoder is due to Kingma and Welling (2013) and Rezende et al., 2014; Jordan et al., 1999 provide an introduction to variational methods for inference in graphical models.

For autoregressive models, the classic text is by Box et al., 2016. The Yule–Walker equations for fitting AR models were developed independently by Yule (1927) and Walker (1931). Autoregressive models with nonlinear dependencies were developed by several authors (Frey, 1998; Bengio and Bengio, 2001; Larochelle and Murray, 2011). The autoregressive WaveNet model (van den Oord et al., 2016a) was based on earlier work on autoregressive image generation (van den Oord et al., 2016b). Generative adversarial networks, or GANs, were first proposed by Goodfellow et al. (2015a), and have found many applications in AI. Some theoretical understanding of their properties is emerging, leading to improved GAN models and algorithms (Li and Malik, 2018b, 2018a; Zhu et al., 2019). Part of that understanding involves protecting against adversarial attacks (Carlini et al., 2019).

Several branches of research into neural networks have been popular in the past but are not actively explored today. Hopfield networks (Hopfield, 1982) have symmetric connections between each pair of nodes and can learn to store patterns in an associative memory, so that an entire pattern can be retrieved by indexing into the memory using a fragment of the pattern. Hopfield networks are deterministic; they were later generalized to stochastic Boltzmann machines (Hinton and Sejnowski, 1983, 1986). Boltzmann machines are possibly the earliest example of a deep generative model. The difficulty of inference in Boltzmann machines led to advances in both Monte Carlo techniques and variational techniques (see Section 13.4 ).

Hopfield network

Boltzmann machine

Research on neural networks for AI has also been intertwined to some extent with research into biological neural networks. The two topics coincided in the 1940s, and ideas for convolutional networks and reinforcement learning can be traced to studies of biological systems; but at present, new ideas in deep learning tend to be based on purely computational or statistical concerns. The field of computational neuroscience aims to build computational models that capture important and specific properties of actual biological systems. Overviews are given by Dayan and Abbott (2001) and Trappenberg (2010).

Computational neuroscience

For modern neural nets and deep learning, the leading textbooks are those by Goodfellow et al., 2016 and Charniak (2018). There are also many hands-on guides associated with the various open-source software packages for deep learning. Three of the leaders of the field— Yann LeCun, Yoshua Bengio, and Geoff Hinton—introduced the key ideas to non-AI researchers in an influential Nature article (2015). The three were recipients of the 2018 Turing Award. Schmidhuber (2015) provides a general overview, and Deng et al., 2014 focus on signal processing tasks.

The primary publication venues for deep learning research are the conference on Neural Information Processing Systems (NeurIPS), the International Conference on Machine Learning (ICML), and the International Conference on Learning Representations (ICLR). The main journals are Machine Learning, the Journal of Machine Learning Research, and Neural Computation. Increasingly, because of the fast pace of research, papers appear first on arXiv.org and are often described in the research blogs of the major research centers.

Chapter 22 Reinforcement Learning In which we see how experiencing rewards and punishments can teach an agent how to maximize rewards in the future.

With supervised learning, an agent learns by passively observing example input/output pairs provided by a “teacher.” In this chapter, we will see how agents can actively learn from their own experience, without a teacher, by considering their own ultimate success or failure.

22.1 Learning from Rewards Consider the problem of learning to play chess. Let’s imagine treating this as a supervised learning problem using the methods of Chapters 19 –21 . The chess-playing agent function takes as input a board position and returns a move, so we train this function by supplying examples of chess positions, each labeled with the correct move. Now, it so happens that we have available databases of several million grandmaster games, each a sequence of positions and moves. The moves made by the winner are, with few exceptions, assumed to be good, if not always perfect. Thus, we have a promising training set. The problem is that there are relatively few examples (about 108 ) compared to the space of all possible chess positions (about 1040 ). In a new game, one soon encounters positions that are significantly different from those in the database, and the trained agent function is likely to fail miserably—not least because it has no idea of what its moves are supposed to achieve (checkmate) or even what effect the moves have on the positions of the pieces. And of course chess is a tiny part of the real world. For more realistic problems, we would need much vaster grandmaster databases, and they simply don’t exist.1

1 As Yann LeCun and Alyosha Efros have pointed out, “the AI revolution will not be supervised.”

An alternative is reinforcement learning (RL), in which an agent interacts with the world and periodically receives rewards (or, in the terminology of psychology, reinforcements) that reflect how well it is doing. For example, in chess the reward is 1 for winning, 0 for losing, and

1 2

for a draw. We have already seen the concept of rewards in Chapter 17  for

Markov decision processes (MDPs). Indeed, the goal is the same in reinforcement learning: maximize the expected sum of rewards. Reinforcement learning differs from “just solving an MDP” because the agent is not given the MDP as a problem to solve; the agent is in the MDP. It may not know the transition model or the reward function, and it has to act in order to learn more. Imagine playing a new game whose rules you don’t know; after a hundred or so moves, the referee tells you “You lose.” That is reinforcement learning in a nutshell.

Reinforcement learning

From our point of view as designers of AI systems, providing a reward signal to the agent is usually much easier than providing labeled examples of how to behave. First, the reward function is often (as we saw for chess) very concise and easy to specify: it requires only a few lines of code to tell the chess agent if it has won or lost the game or to tell the car-racing agent that it has won or lost the race or has crashed. Second, we don’t have to be experts, capable of supplying the correct action in any situation, as would be the case if we tried to apply supervised learning.

It turns out, however, that a little bit of expertise can go a long way in reinforcement learning. The two examples in the preceding paragraph—the win/loss rewards for chess and racing—are what we call sparse rewards, because in the vast majority of states the agent is given no informative reward signal at all. In games such as tennis and cricket, we can easily supply additional rewards for each point won or for each run scored. In car racing, we could reward the agent for making progress around the track in the right direction. When learning to crawl, any forward motion is an achievement. These intermediate rewards make learning much easier.

Sparse

As long as we can provide the correct reward signal to the agent, reinforcement learning provides a very general way to build AI systems. This is particularly true for simulated environments, where there is no shortage of opportunities to gain experience. The addition of deep learning as a tool within RL systems has also made new applications possible, including learning to play Atari video games from raw visual input (Mnih et al., 2013), controlling robots (Levine et al., 2016), and playing poker (Brown and Sandholm, 2017).

Literally hundreds of different reinforcement learning algorithms have been devised, and many of them can employ as tools a wide range of learning methods from Chapters 19 – 21 . In this chapter, we cover the basic ideas and give some sense of the variety of approaches through a few examples. We categorize the approaches as follows:

MODEL-BASED REINFORCEMENT LEARNING: In these approaches the agent uses a transition model of the environment to help interpret the reward signals and to make decisions about how to act. The model may be initially unknown, in which case the agent learns the model from observing the effects of its actions, or it may already be known—for example, a chess program may know the rules of chess even if it does not know how to choose good moves. In partially observable environments, the transition model is also useful for state estimation (see Chapter 14 ). Model-based reinforcement learning systems often learn a utility function of the sum of rewards from state s onward.2

U (s), defined (as in Chapter 17 ) in terms

2 In the RL literature, which draws more on operations research than economics, utility functions are often called value functions and denoted V (s).

Model-based reinforcement learning

MODEL-FREE REINFORCEMENT LEARNING: In these approaches the agent neither knows nor learns a transition model for the environment. Instead, it learns a more direct representation of how to behave. This comes in one of two varieties:

Model-free reinforcement learning

ACTION-UTILITY LEARNING: We introduced action-utility functions in Chapter 17 . The most common form of action-utility learning is Q-learning, where the agent learns a Q-function, or quality-function, Q(s,a), denoting the sum of rewards

from state s onward if action a is taken. Given a Q-function, the agent can choose what to do in s by finding the action with the highest Q-value.

Action-utility learning

Q-learning

Q-function

POLICY SEARCH: The agent learns a policy π(s) that maps directly from states to actions. In the terminology of Chapter 2 , this a reflex agent.

Policy search

We begin in Section 22.2  with passive reinforcement learning, where the agent’s policy is fixed and the task is to learn the utilities of states (or of state–action pairs); this could also involve learning a model of the environment. (An understanding of Markov decision processes, as described in Chapter 17 , is essential for this section.) Section 22.3  covers active reinforcement learning, where the agent must also figure out what to do. The principal issue is exploration: an agent must experience as much as possible of its environment in order to learn how to behave in it. Section 22.4  discusses how an agent can use inductive learning (including deep learning methods) to learn much faster from its experiences. We also discuss other approaches that can help scale up RL to solve real problems, including providing intermediate pseudorewards to guide the learner and organizing behavior into a hierarchy of actions. Section 22.5  covers methods for policy search. In Section 22.6 , we explore apprenticeship learning: training a learning agent using demonstrations rather than reward signals. Finally, Section 22.7  reports on applications of reinforcement learning.

Passive reinforcement learning

Active reinforcement learning

22.2 Passive Reinforcement Learning We start with the simple case of a fully observable environment with a small number of actions and states, in which an agent already has a fixed policy π(s) that determines its actions. The agent is trying to learn the utility function U π (s)—the expected total discounted reward if policy

π is executed beginning in state s. We call this a passive learning agent.

Passive learning agent

The passive learning task is similar to the policy evaluation task, part of the policy iteration algorithm described in Section 17.2.2 . The difference is that the passive learning agent does not know the transition model P (s′ |s,a), which specifies the probability of reaching state s′ from state s after doing action a; nor does it know the reward function R(s,a,s′ ), which specifies the reward for each transition. We will use as our example the 4 × 3 world introduced in Chapter 17 . Figure 22.1  shows the optimal policies for that world and the corresponding utilities. The agent executes a set of trials in the environment using its policy π. In each trial, the agent starts in state (1,1) and experiences a sequence of state transitions until it reaches one of the terminal states, (4,2) or (4,3). Its percepts supply both the current state and the reward received for the transition that just occurred to reach that state. Typical trials might look like this:

-.04

-.04

-.04

-.04

-.04

-.04

+1

Up

Right

Up

Right

Right

Right

(1,1) −→ (1,2) −→ (1,3) −−→ (1,2) −→ (1,3) −−→ (2,3) −−→ (3,3) −−→ (4,3)

Up

-.04

-.04

-.04

(1,1) −→ (1,2) −→ (1,3) −−→ (2,3)

Up

Up

-.04

-.04

Up

Up

Right

-.04

-.04

−−→

Right

-.04

-.04

-.04

Right

Up

-.04

-1

Right

Up

(1,1) −→ (1,2) −→ (1,3) −−→ (2,3) −−→ (3,3) −−→ (3,2) → (4,2)

Figure 22.1

Right

Right

+1

(3,3) −−→ (3,2) −→ (3,3) −−→ (4,3)

Right

R(s,a,s ) = −0.04 for transitions between nonterminal states. There are two policies because in state (3,1) both Left and Up are optimal. We saw this before in Figure 17.2 . (b) The utilities of the states in the 4 × 3 world, given policy π. (a) The optimal policies for the stochastic environment with

′

Note that each transition is annotated with both the action taken and the reward received at the next state. The object is to use the information about rewards to learn the expected

U π (s) associated with each nonterminal state s. The utility is defined to be the expected sum of (discounted) rewards obtained if policy π is followed. As in Equation utility

(17.2)  on page 567, we write (22.1)

⎡ U π ( s) = E ⎣

∑ ∞

t=0

γ t R(St ,π(St ),St

+1 )

⎤ , ⎦

R(St ,π(St ),St ) is the reward received when action π(St ) is taken in state St and reaches state St . Note that St is a random variable denoting the state reached at time t when executing policy π, starting from state S = s. We will include a discount factor γ in all of our equations, but for the 4 × 3 world we will set γ = 1, which means no discounting. where

+1

+1

0

22.2.1 Direct utility estimation The idea of direct utility estimation is that the utility of a state is defined as the expected total reward from that state onward (called the expected reward-to-go), and that each trial provides a sample of this quantity for each state visited. For example, the first of the three trials shown earlier provides a sample total reward of 0.76 for state (1,1), two samples of 0.80 and 0.88 for (1,2), two samples of 0.84 and 0.92 for (1,3), and so on. Thus, at the end of

each sequence, the algorithm calculates the observed reward-to-go for each state and updates the estimated utility for that state accordingly, just by keeping a running average for each state in a table. In the limit of infinitely many trials, the sample average will converge to the true expectation in Equation (22.1) .

Direct utility estimation

Reward-to-go

This means that we have reduced reinforcement learning to a standard supervised learning problem in which each example is a (state, reward-to-go) pair. We have a lot of powerful algorithms for supervised learning, so this approach seems promising, but it ignores an important constraint: The utility of a state is determined by the reward and the expected utility of the successor states. More specifically, the utility values obey the Bellman equations for a fixed policy (see also Equation (17.14) ): (22.2)

Ui s

( ) =

∑P s sπ s ∣

(

s

′

′∣

∣ ∣

,

i(

))[

R s πi s s ( ,

( ),

′

) +

γUi s (

′

)].

By ignoring the connections between states, direct utility estimation misses opportunities for learning. For example, the second of the three trials given earlier reaches the state (3,2), which has not previously been visited. The next transition reaches (3,3), which is known from the first trial to have a high utility. The Bellman equation suggests immediately that (3,2) is also likely to have a high utility, because it leads to (3,3), but direct utility estimation learns nothing until the end of the trial. More broadly, we can view direct utility estimation as searching for U in a hypothesis space that is much larger than it needs to be, in that it includes many functions that violate the Bellman equations. For this reason, the algorithm often converges very slowly.

22.2.2 Adaptive dynamic programming An adaptive dynamic programming (or ADP) agent takes advantage of the constraints among the utilities of states by learning the transition model that connects them and solving the corresponding Markov decision process using dynamic programming. For a passive learning agent, this means plugging the learned transition model observed rewards

P (s |s,π(s)) and the ′

R(s,π(s),s ) into Equation (22.2)  to calculate the utilities of the states. As ′

we remarked in our discussion of policy iteration in Chapter 17 , these Bellman equations are linear when the policy

π is fixed, so they can be solved using any linear algebra package.

Adaptive dynamic programming

Alternatively, we can adopt the approach of modified policy iteration (see page 578), using a simplified value iteration process to update the utility estimates after each change to the learned model. Because the model usually changes only slightly with each observation, the value iteration process can use the previous utility estimates as initial values and typically converge very quickly.

Learning the transition model is easy, because the environment is fully observable. This means that we have a supervised learning task where the input for each training example is

sa

a state–action pair, ( , ), and the output is the resulting state,

s . The transition model ′

P (s |s,a) is represented as a table and it is estimated directly from the counts that are accumulated in Ns sa . The counts record how often state s is reached when executing a in s. ′

′

′

|

For example, in the three trials given on page 792, Right is executed four times in (3,3) and the resulting state is (3,2) twice and (4,3) twice, so

P ((4,3)|(3,3),Right) are both estimated to be

1 2

P ((3,2)|(3,3),Right) and

.

The full agent program for a passive ADP agent is shown in Figure 22.2 . Its performance on the 4 × 3 world is shown in Figure 22.3 . In terms of how quickly its value estimates improve, the ADP agent is limited only by its ability to learn the transition model. In this sense, it provides a standard against which to measure any other reinforcement learning algorithms. It is, however, intractable for large state spaces. In backgammon, for example, it would involve solving roughly 1020 equations in 1020 unknowns.

Figure 22.2

A passive reinforcement learning agent based on adaptive dynamic programming. The agent chooses a value for γ and then incrementally computes the P and R values of the MDP. The POLICY-EVALUATION function solves the fixed-policy Bellman equations, as described on page 577.

Figure 22.3

The passive ADP learning curves for the 4 × 3 world, given the optimal policy shown in Figure 22.1 . (a) The utility estimates for a selected subset of states, as a function of the number of trials. Notice that it takes 14 and 23 trials respectively before the rarely visited states (2,1) and (3,2) “discover” that they connect to the +1 exit state at (4,3). (b) The root-mean-square error (see Appendix A ) in the estimate for U (1,1), averaged over 50 runs of 100 trials each.

22.2.3 Temporal-difference learning

Solving the underlying MDP as in the preceding section is not the only way to bring the Bellman equations to bear on the learning problem. Another way is to use the observed transitions to adjust the utilities of the observed states so that they agree with the constraint equations. Consider, for example, the transition from (1,3) to (2,3) in the second trial on page 792. Suppose that as a result of the first trial, the utility estimates are and

U π (1,3) = 0.88

U π (2,3) = 0.96. Now, if this transition from (1,3) to (2,3) occurred all the time, we

would expect the utilities to obey the equation

U π (1,3) = −0.04 + U π (2,3), U π (1,3) would be 0.92. Thus, its current estimate of 0.84 might be a little low and should be increased. More generally, when a transition occurs from state s to state s via action π(s) , we apply the following update to U π (s): so

′

(22.3)

U π (s) ← U π (s) + α[R(s,π(s),s ) + γU π (s ) − U π (s)]. ′

Here,

′

α is the learning rate parameter. Because this update rule uses the difference in

utilities between successive states (and thus successive times), it is often called the temporal-difference (TD) equation. Just as in the weight update rules from Chapter 19  (e.g., Equation (19.6)  on page 680), the TD term

R(s,π(s),s ) + γU π (s ) − U π (s) is ′

′

effectively an error signal, and the update is intended to reduce the error.

Temporal-difference

All temporal-difference methods work by adjusting the utility estimates toward the ideal equilibrium that holds locally when the utility estimates are correct. In the case of passive learning, the equilibrium is given by Equation (22.2) . Now Equation (22.3)  does in fact cause the agent to reach the equilibrium given by Equation (22.2) , but there is some subtlety involved. First, notice that the update involves only the observed successor

s, ′

whereas the actual equilibrium conditions involve all possible next states. One might think that this causes an improperly large change in

U π (s) when a very rare transition occurs; but,

in fact, because rare transitions occur only rarely, the average value of U π (s) will converge to the correct quantity in the limit, even if the value itself continues to fluctuate. Furthermore, if we turn the parameter α into a function that decreases as the number of times a state has been visited increases, as shown in Figure 22.4 , then U π (s) itself will

converge to the correct value.3 Figure 22.5  illustrates the performance of the passive TD

agent on the 4 × 3 world. It does not learn quite as fast as the ADP agent and shows much higher variability, but it is much simpler and requires much less computation per observation. Notice that TD does not need a transition model to perform its updates. The environment itself supplies the connection between neighboring states in the form of observed transitions. 3 The technical conditions are given on page 684. In Figure 22.5  we have used α(n) = 60/(59 + n), which satisfies the conditions.

Figure 22.4

A passive reinforcement learning agent that learns utility estimates using temporal differences. The stepsize function α(n) is chosen to ensure convergence.

Figure 22.5

The TD learning curves for the 4 × 3 world. (a) The utility estimates for a selected subset of states, as a function of the number of trials, for a single run of 500 trials. Compare with the run of 100 trials in Figure 22.3(a) . (b) The root-mean-square error in the estimate for U (1,1), averaged over 50 runs of 100 trials each.

The ADP and TD approaches are closely related. Both try to make local adjustments to the utility estimates in order to make each state “agree” with its successors. One difference is that TD adjusts a state to agree with its observed successor (Equation (22.3) ), whereas ADP adjusts the state to agree with all of the successors that might occur, weighted by their probabilities (Equation (22.2) ). This difference disappears when the effects of TD adjustments are averaged over a large number of transitions, because the frequency of each successor in the set of transitions is approximately proportional to its probability. A more important difference is that whereas TD makes a single adjustment per observed transition, ADP makes as many as it needs to restore consistency between the utility estimates U and the transition model P . Although the observed transition makes only a local change in P , its effects might need to be propagated throughout U. Thus, TD can be viewed as a crude but efficient first approximation to ADP.

Each adjustment made by ADP could be seen, from the TD point of view, as a result of a pseudoexperience generated by simulating the current transition model. It is possible to extend the TD approach to use a transition model to generate several pseudoexperiences— transitions that the TD agent can imagine might happen, given its current model. For each observed transition, the TD agent can generate a large number of imaginary transitions. In this way, the resulting utility estimates will approximate more and more closely those of ADP—of course, at the expense of increased computation time.

Pseudoexperience

In a similar vein, we can generate more efficient versions of ADP by directly approximating the algorithms for value iteration or policy iteration. Even though the value iteration algorithm is efficient, it is intractable if we have, say, 10100 states. However, many of the necessary adjustments to the state values on each iteration will be extremely tiny. One possible approach to generating reasonably good answers quickly is to bound the number of adjustments made after each observed transition. One can also use a heuristic to rank the possible adjustments so as to carry out only the most significant ones. The prioritized sweeping heuristic prefers to make adjustments to states whose likely successors have just undergone a large adjustment in their own utility estimates.

Prioritized sweeping

Using heuristics like this, approximate ADP algorithms can learn roughly as fast as full ADP, in terms of the number of training sequences, but can be orders of magnitude more efficient in terms of total computation (see Exercise 22.PRSW). This enables them to handle state spaces that are far too large for full ADP. Approximate ADP algorithms have an additional advantage: in the early stages of learning a new environment, the transition model P often will be far from correct, so there is little point in calculating an exact utility function to match it. An approximation algorithm can use a minimum adjustment size that decreases as the transition model becomes more accurate. This eliminates the very long runs of value iteration that can occur early in learning due to large changes in the model.

22.3 Active Reinforcement Learning A passive learning agent has a fixed policy that determines its behavior. An active learning agent gets to decide what actions to take. Let us begin with the adaptive dynamic programming (ADP) agent and consider how it can be modified to take advantage of this new freedom.

First, the agent will need to learn a complete transition model with outcome probabilities for all actions, rather than just the model for the fixed policy. The learning mechanism used by PASSIVE-ADP-AGENT will do just fine for this. Next, we need to take into account the fact that the agent has a choice of actions. The utilities it needs to learn are those defined by the optimal policy; they obey the Bellman equations (which we repeat here):

(22.4)

U (s) = amax As

∈ ( )

∑P s sa R sas ′

s

′

( | , )[ ( , , ) +

γU (s )]. ′

′

These equations can be solved to obtain the utility function policy iteration algorithms from Chapter 17 .

U using the value iteration or

The final issue is what to do at each step. Having obtained a utility function

U that is

optimal for the learned model, the agent can extract an optimal action by one-step lookahead to maximize the expected utility; alternatively, if it uses policy iteration, the optimal policy is already available, so it could simply execute the action the optimal policy recommends. But should it?

22.3.1 Exploration Figure 22.6  shows the results of one sequence of trials for an ADP agent that follows the recommendation of the optimal policy for the learned model at each step. The agent does not learn the true utilities or the true optimal policy! What happens instead is that in the third trial, it finds a policy that reaches the +1 reward along the lower route via (2,1), (3,1), (3,2), and (3,3). (See Figure 22.6(b) .) After experimenting with minor variations, from the eighth trial onward it sticks to that policy, never learning the utilities of the other states and never

finding the optimal route via (1,2), (1,3), and (2,3). We will call this agent a greedy agent, because it greedily takes the action that it currently believes to be optimal at each step. Sometimes greed pays off and the agent converges to the optimal policy, but often it does not.

Figure 22.6

Performance of a greedy ADP agent that executes the action recommended by the optimal policy for the learned model. (a) The root mean square (RMS) error averaged across all nine nonterminal squares and the policy loss in (1,1). We see that the policy converges quickly, after just eight trials, to a suboptimal policy with a loss of 0.235. (b) The suboptimal policy to which the greedy agent converges in this particular sequence of trials. Notice the action in (1,2).

Down

Greedy agent

How can it be that choosing the optimal action leads to suboptimal results? The answer is that the learned model is not the same as the true environment; what is optimal in the learned model can therefore be suboptimal in the true environment. Unfortunately, the agent does not know what the true environment is, so it cannot compute the optimal action for the true environment. What, then, should it do?

The greedy agent has overlooked the fact that actions do more than provide rewards; they also provide information in the form of percepts in the resulting states. As we saw with bandit problems in Section 17.3 , an agent must make a tradeoff between exploitation of

the current best action to maximize its short-term reward and exploration of previously unknown states to gain information that can lead to a change in policy (and to greater rewards in the future). In the real world, one constantly has to decide between continuing in a comfortable existence, versus striking out into the unknown in the hopes of a better life.

Although bandit problems are difficult to solve exactly to obtain an optimal exploration scheme, it is nonetheless possible to come up with a scheme that will eventually discover an optimal policy, even if it might take longer to do so than is optimal. Any such scheme should not be greedy in terms of the immediate next move, but should be what is called “greedy in the limit of infinite exploration,” or GLIE. A GLIE scheme must try each action in each state an unbounded number of times to avoid having a finite probability that an optimal action is missed. An ADP agent using such a scheme will eventually learn the true transition model, and can then operate under exploitation.

GLIE

There are several GLIE schemes; one of the simplest is to have the agent choose a random action at time step

t with probability 1/t and to follow the greedy policy otherwise. While

this does eventually converge to an optimal policy, it can be slow. A better approach would give some weight to actions that the agent has not tried very often, while tending to avoid actions that are believed to be of low utility (as we did with Monte Carlo tree search in Section 5.4 ). This can be implemented by altering the constraint equation (22.4)  so that it assigns a higher utility estimate to relatively unexplored state–action pairs.

This amounts to an optimistic prior over the possible environments and causes the agent to behave initially as if there were wonderful rewards scattered all over the place. Let us use

U (s) to denote the optimistic estimate of the utility (i.e., the expected reward-to-go) of the state s, and let N (s,a) be the number of times action a has been tried in state s. Suppose we +

are using value iteration in an ADP learning agent; then we need to rewrite the update equation (Equation (17.10)  on page 573) to incorporate the optimistic estimate: (22.5)

U (s) ← max a  f ( +

∑P s sa R sas s

( ′ | , )[ ( , , ′ ) +

γU (s )], N (s,a)) . +

′

′

f is the exploration function. The function f (u,n) determines how greed (preference for high values of the utility u) is traded off against curiosity (preference for actions that have not been tried often and have a low count n). The function should be increasing in u and decreasing in n. Obviously, there are many possible functions that fit these conditions.

Here,

One particularly simple definition is

f (u,n) = { uR

+

if 

n < Ne

otherwise,

Exploration function

where

R

+

is an optimistic estimate of the best possible reward obtainable in any state and

Ne is a fixed parameter. This will have the effect of making the agent try each state–action pair at least Ne times. The fact that U rather than U appears on the right-hand side of +

Equation (22.5)  is very important. As exploration proceeds, the states and actions near the

start state might well be tried a large number of times. If we used

U , the more pessimistic

utility estimate, then the agent would soon become disinclined to explore further afield. The use of

U

+

means that the benefits of exploration are propagated back from the edges of

unexplored regions, so that actions that lead toward unexplored regions are weighted more highly, rather than just actions that are themselves unfamiliar. The effect of this exploration policy can be seen clearly in Figure 22.7(b) , which shows a rapid convergence toward zero policy loss, unlike with the greedy approach. A very nearly optimal policy is found after just 18 trials. Notice that the RMS error in the utility estimates does not converge as quickly. This is because the agent stops exploring the unrewarding parts of the state space fairly soon, visiting them only “by accident” thereafter. However, it makes perfect sense for the agent not to care about the exact utilities of states that it knows are undesirable and can be avoided. There is not much point in learning about the best radio station to listen to while falling off a cliff.

Figure 22.7

R =2

N =5

Performance of the exploratory ADP agent using + and e . (a) Utility estimates for selected states over time. (b) The RMS error in utility values and the associated policy loss.

22.3.2 Safe exploration So far we have assumed that an agent is free to explore as it wishes—that any negative rewards serve only to improve its model of the world. That is, if we play a game of chess and lose, we suffer no damage (except perhaps to our pride), and whatever we learned will make us a better player in the next game. Similarly, in a simulation environment for a selfdriving car, we could explore the limits of the car’s performance, and any accidents give us more information. If the car crashes, we just hit the reset button.

Unfortunately, the real world is less forgiving. If you are a baby sunfish, your probability of surviving to adulthood is about 0.00000001. Many actions are irreversible, in the sense defined for online search agents in Section 4.5 : no subsequent sequence of actions can restore the state to what it was before the irreversible action was taken. In the worst case, the agent enters an absorbing state where no actions have any effect and no rewards are received.

Absorbing state

In many practical settings, we cannot afford to have our agents taking irreversible actions or entering absorbing states. For example, an agent learning to drive in a real car should avoid taking actions that might lead to any of the following:

states with large negative rewards, such as serious car crashes; states from which there is no escape, such as driving the car into a deep ditch; states that permanently limit future rewards, such as damaging the car’s engine so that its maximum speed is reduced.

We can end up in a bad state either because our model is unknown, and we actively choose to explore in a direction that turns out to be bad, or because our model is incorrect and we don’t know that a given action can have a disastrous result. Note that the algorithm in Figure 22.2  is using maximum-likelihood estimation (see Chapter 20 ) to learn the transition model; moreover, by choosing a policy based solely on the estimated model, it is acting as if the model were correct. This is not necessarily a good idea! For example, a taxi agent that didn’t know how traffic lights work might ignore a red light once or twice with no ill effects and then formulate a policy to ignore all red lights from then on.

A better idea would be to choose a policy that works reasonably well for the whole range of models that have a reasonable chance of being the true model, even if the policy happens to be suboptimal for the maximum-likelihood model. There are three mathematical approaches that have this flavor. The first approach, Bayesian reinforcement learning, assumes a prior probability P (h) over hypotheses h about what the true model is; the posterior probability P (h|e) is obtained in the usual way by Bayes’ rule given the observations to date. Then, if the agent has decided to stop learning, the optimal policy is the one that gives the highest expected utility. Let Uhπ

be the expected utility, averaged over all possible start states, obtained by executing policy π in model h. Then we have

π ∗ = argmax π

Bayesian reinforcement learning

∑P h h

U

( |e) hπ .

In some special cases, this policy can even be computed! If the agent will continue learning in the future, however, then finding an optimal policy becomes considerably more difficult, because the agent must consider the effects of future observations on its beliefs about the transition model. The problem becomes an exploration POMDP whose belief states are distributions over models. In principle, this exploration POMDP can be formulated and solved before the agent ever sets foot in the world. (Exercise 22.EPOM asks you to do this for the Minesweeper game to find the best first move.) The result is a complete strategy that tells the agent what to do next given any possible percept sequence. Solving the exploration POMDP is usually wildly intractable, but the concept provides an analytical foundation for understanding the exploration problem described in Section 22.3 .

Exploration POMDP

It is worth noting that being perfectly Bayesian will not protect the agent from an untimely death. Unless the prior gives some indication of percepts that suggest danger, there is nothing to prevent the agent from taking an exploratory action that leads to an absorbing state. For example, it used to be thought that human infants had an innate fear of heights and would not crawl off a cliff, but this turns out not to be true (Adolph et al., 2014).

The second approach, derived from robust control theory, allows for a set of possible

H without assigning probabilities to them, and defines an optimal robust policy as one that gives the best outcome in the worst case over H : models

π

∗

= argmax min

π

h

Uhπ .

Robust control theory

Often, the set

H will be the set of models that exceed some likelihood threshold on P (h|e),

so the robust and Bayesian approaches are related.

The robust control approach can be considered as a game between the agent and an adversary, where the adversary gets to pick the worst possible result for any action, and the policy we get is the minimax solution for the game. Our logical wumpus agent (see Section 7.7 ) is a robust control agent in this way: it considers all models that are logically possible, and does not explore any locations that could possibly contain a pit or a wumpus, so it is finding the action with maximum utility in the worst case over all possible hypotheses.

The problem with the worst-case assumption is that it results in overly conservative behavior. A self-driving car that assumes that every other driver will try to collide with it has no choice but to stay in the garage. Real life is full of such risk–reward tradeoffs.

Although one reason for venturing into reinforcement learning was to escape the need for a human teacher (as in supervised learning), it turns out that human knowledge can help keep a system safe. One way is to record a series of actions by an experienced teacher, so that the system will act reasonably from the start, and can learn to improve from there. A second way is for a human to write down constraints on what a system can do, and have a program outside of the reinforcement learning system enforce those constraints. For example, when training an autonomous helicopter, a partial policy can be provided that takes over control when the helicopter enters a state from which any further unsafe actions would lead to an irrecoverable state—one in which the safety controller cannot guarantee that the absorbing state will be avoided. In all other states, the learning agent is free to do as it pleases.

22.3.3 Temporal-difference Q-learning Now that we have an active ADP agent, let us consider how to construct an active temporaldifference (TD) learning agent. The most obvious change is that the agent will have to learn a transition model so that it can choose an action based on

U (s) via one-step look-ahead.

The model acquisition problem for the TD agent is identical to that for the ADP agent, and the TD update rule remains unchanged. Once again, it can be shown that the TD algorithm will converge to the same values as ADP, as the number of training sequences tends to infinity.

The Q-learning method avoids the need for a model by learning an action-utility function

Q(s,a) instead of a utility function U (s). Q(s,a) denotes the expected total discounted reward if the agent takes action a in s and acts optimally thereafter. Knowing the Q-function

enables the agent to act optimally simply by choosing arg maxa

Q(s,a), with no need for

look-ahead based on a transition model.

We can also derive a model-free TD update for the Q-values. We begin with the Bellman equation for

Q(s,a), repeated here from Equation (17.8) :

(22.6)

Q ( s,a ) =

∑P s sa R sas s

γ

( ′ | , )[ ( , , ′ ) +  max 

′

a

′

Q(s ,a )] ′

′

From this, we can write down the Q-learning TD update, by analogy to the TD update for utilities in Equation (22.3) : (22.7)

Q(s,a) ← Q(s,a) + α[R(s,a,s ) + γ max  Q(s ,a ) − Q(s,a)]. a ′

′

′

′

a is executed in state s leading to state s . As in Equation (22.3) , the term R(s,a,s ) + γ maxa  Q(s ,a ) − Q(s,a) represents an error that the ′

This update is calculated whenever action ′

′

′

′

update is trying to minimize.

The important part of this equation is what it does not contain: a TD Q-learning agent does not need a transition model,

P (s |s,a), either for learning or for action selection. As noted at the ′

beginning of the chapter, model-free methods can be applied even in very complex domains because no model need be provided or learned. On the other hand, the Q-learning agent has no means of looking into the future, so it may have difficulty when rewards are sparse and long action sequences must be constructed to reach them.

The complete agent design for an exploratory TD Q-learning agent is shown in Figure 22.8 . Notice that it uses exactly the same exploration function

f as that used by the

exploratory ADP agent—hence the need to keep statistics on actions taken (the table

N ). If a

simpler exploration policy is used—say, acting randomly on some fraction of steps, where the fraction decreases over time—then we can dispense with the statistics.

Figure 22.8

Q(s,a) of each action in each f as the exploratory ADP agent, but avoids having to

An exploratory Q-learning agent. It is an active learner that learns the value situation. It uses the same exploration function learn the transition model.

Q-learning has a close relative called SARSA (for state, action, reward, state, action). The update rule for SARSA is very similar to the Q-learning update rule (Equation (22.7) ), except that SARSA updates with the Q-value of the action

a

′

that is actually taken:

(22.8)

Q(s,a) ← Q(s,a) + α[R(s,a,s ) + γ Q(s ,a ) − Q(s,a)], ′

′

′

SARSA

sars a

The rule is applied at the end of each ,  ,  ,  ′ , 

′

quintuplet—hence the name. The

difference from Q-learning is quite subtle: whereas Q-learning backs up the Q-value from the best action in

s , SARSA waits until an action is taken and backs up the Q-value for that ′

action. If the agent is greedy and always takes the action with the best Q-value, the two algorithms are identical. When exploration is happening, however, they differ: if the exploration yields a negative reward, SARSA penalizes the action, while Q-learning does not.

Q-learning is an off-policy learning algorithm, because it learns Q-values that answer the question “What would this action be worth in this state, assuming that I stop using whatever policy I am using now, and start acting according to a policy that chooses the best action

(according to my estimates)?” SARSA is an on-policy algorithm: it learns Q-values that answer the question “What would this action be worth in this state, assuming I stick with my policy?” Q-learning is more flexible than SARSA, in the sense that a Q-learning agent can learn how to behave well when under the control of a wide variety of exploration policies. On the other hand, SARSA is appropriate if the overall policy is even partly controlled by other agents or programs, in which case it is better to learn a Q-function for what will actually happen rather than what would happen if the agent got to pick estimated best actions. Both Q-learning and SARSA learn the optimal policy for the 4 × 3 world, but they do so at a much slower rate than the ADP agent. This is because the local updates do not enforce consistency among all the Q-values via the model.

Off-policy

On-policy

22.4 Generalization in Reinforcement Learning So far, we have assumed that utility functions and Q-functions are represented in tabular form with one output value for each state. This works for state spaces with up to about 106 states, which is more than enough for our toy two-dimensional grid environments. But in real-world environments with many more states, convergence will be too slow. Backgammon is simpler than most real-world applications, yet it has about 10

20

states. We

cannot easily visit them all in order to learn how to play the game. Chapter 5  introduced the idea of an evaluation function as a compact measure of desirability for potentially vast state spaces. In the terminology of this chapter, the evaluation function is an approximate utility function; we use the term function approximation for the process of constructing a compact approximation of the true utility function or Q-function. For example, we might approximate the utility function using a weighted linear combination of features f1 , … ,fn :

Uˆθ (s) = θ1 f1 (s) + θ2 f2 (s) + ⋯ + θn fn (s).

Function approximation

Instead of learning 1020 state values in a table, a reinforcement learning algorithm can learn, say, 20 values for the parameters θ =

θ1 , … ,θ20 that make Uˆθ a good approximation to the

true utility function. Sometimes this approximate utility function is combined with lookahead search to produce more accurate decisions. Adding look-ahead search means that effective behavior can be generated from a much simpler utility function approximator that is learnable from far fewer experiences.

Function approximation makes it practical to represent utility (or Q) functions for very large state spaces, but more importantly, it allows for inductive generalization: the agent can generalize from states it has visited to states it has not yet visited. Tesauro (1992) used this

technique to build a backgammon-playing program that played at human champion level, even though it explored only a trillionth of the complete state space of backgammon.

22.4.1 Approximating direct utility estimation The method of direct utility estimation (Section 22.2 ) generates trajectories in the state space and extracts, for each state, the sum of rewards received from that state onward until termination. The state and the sum of rewards received constitute a training example for a supervised learning algorithm. For example, suppose we represent the utilities for the 4 × 3 world using a simple linear function, where the features of the squares are just their

x and y

coordinates. In that case, we have

(22.9)

Uˆθ (x,y) = θ θ θ θ

0

+

θ x + θ y. 1

2

U

Thus, if ( 0 , 1 , 2 ) = (0.5,0.2,0.1), then ˆ θ (1,1) = 0.8. Given a collection of trials, we obtain a

U xy

set of sample values of ˆ θ ( , ), and we can find the best fit, in the sense of minimizing the squared error, using standard linear regression (see Chapter 19 ).

For reinforcement learning, it makes more sense to use an online learning algorithm that updates the parameters after each trial. Suppose we run a trial and the total reward obtained

U

starting at (1,1) is 0.4. This suggests that ˆ θ (1,1), currently 0.8, is too large and must be reduced. How should the parameters be adjusted to achieve this? As with neural-network learning, we write an error function and compute its gradient with respect to the

u s

parameters. If j ( ) is the observed total reward from state

s onward in the jth trial, then the

error is defined as (half) the squared difference of the predicted total and the actual total:

Ej (s) = (Uˆθ (s) − uj (s)) /2. The rate of change of the error with respect to each parameter θi is ∂ Ej /∂ θi , so to move the parameter in the direction of decreasing the error, we want 2

(22.10)

θi ← θi − α

∂

Ej ( s ) ∂ θi

=

θi + α[uj (s) − Uˆθ (s)]

U s ∂ θi

∂ ˆθ ( )

.

This is called the Widrow–Hoff rule, or the delta rule, for online least-squares. For the linear function approximator ˆ θ ( ) in Equation (22.9) , we get three simple update rules:

U s

θ0 ← θ0 + α[uj (s) − Uˆθ (s)], θ1 ← θ1 + α[uj (s) − Uˆθ (s)]x, θ2 ← θ2 + α[uj (s) − Uˆθ (s)]y.

Widrow–Hoff rule

Delta rule

ˆ (1,1) is 0.8 and u (1,1) is 0.4. Parameters We can apply these rules to the example where U θ j

θ0 , θ1 , and θ2 are all decreased by 0.4α, which reduces the error for (1,1). Notice that changing the parameters θi in response to an observed transition between two states also changes ˆ for every other state! This is what we mean by saying that function the values of U θ approximation allows a reinforcement learner to generalize from its experiences.

The agent will learn faster if it uses a function approximator, provided that the hypothesis space is not too large and includes some functions that are a reasonably good fit to the true utility function. Exercise 22.APLM asks you to evaluate the performance of direct utility estimation, both with and without function approximation. The improvement in the 4 × 3 world is noticeable but not dramatic, because this is a very small state space to begin with. The improvement is much greater in a 10 × 10 world with a +1 reward at (10,10).

The 10 × 10 world is well suited for a linear utility function because the true utility function is smooth and nearly linear: it is basically a diagonal slope with its lower corner at (1,1) and its upper corner at (10,10). (See Exercise 22.TENX.) On the other hand, if we put the +1 reward at (5,5), the true utility is more like a pyramid and the function approximator in Equation (22.9)  will fail miserably.

All is not lost, however! Remember that what matters for linear function approximation is that the function be linear in the features. But we can choose the features to be arbitrary nonlinear functions of the state variables. Hence, we can include a feature such as

f ( x, y ) = √ ( x − xg ) 3

2

y yg )

+( −

2

that measures the distance to the goal. With this new

feature, the linear function approximator does well.

22.4.2 Approximating temporal-difference learning We can apply these ideas equally well to temporal-difference learners. All we need do is adjust the parameters to try to reduce the temporal difference between successive states. The new versions of the TD and Q-learning equations (22.3  on page 795 and 22.7  on page 802) are given by

(22.11)

θi ← θi + α[R(s,a,s ) + γUˆθ (s ) − Uˆθ (s)] ∂U∂θθ(s) ′

ˆ

′

i

for utilities and

(22.12) ∂ Q θ ( s ,a ) ˆ ( s ,a ) − Q ˆ (s,a)]  Q θi ← θi + α[R(s,a,s ) + γ max θ θ ∂θ a ′

′

′

′

ˆ

i

for Q-values. For passive TD learning, the update rule can be shown to converge to the closest possible approximation to the true function when the function approximator is linear in the features.4 With active learning and nonlinear functions such as neural networks, nearly all bets are off: there are some very simple cases in which the parameters can go off to infinity with these update rules, even though there are good solutions in the hypothesis space. There are more sophisticated algorithms that can avoid these problems, but at present reinforcement learning with general function approximators remains a delicate art.

4 The definition of distance between utility functions is rather technical; see Tsitsiklis and Van Roy (1997).

In addition to parameters diverging to infinity, there is a more surprising problem called catastrophic forgetting. Suppose you are training an autonomous vehicle to drive along (simulated) roads safely without crashing. You assign a high negative reward for crossing the edge of the road, and you use quadratic features of the road position so that the car can learn that the utility of being in the middle of the road is higher than being close to the edge. All goes well, and the car learns to drive perfectly down the middle of the road. After

a few minutes of this, you are starting to get bored and are about to halt the simulation and write up the excellent results. All of a sudden, the vehicle swerves off the road and crashes. Why? What has happened is that the car has learned too well: because it has learned to steer away from the edge, it has learned that the entire central region of the road is a safe place to be, and it has forgotten that the region closer to the edge is dangerous. The central region therefore has a flat value function, so the quadratic features get zero weight; then, any nonzero weight on the linear features causes the car to slide off the road to one side or the other.

Catastrophic forgetting

One solution to this problem, called experience replay, ensures that the car keeps reliving its youthful crashing behavior at regular intervals. The learning algorithm can retain trajectories from the entire learning process and replay those trajectories to ensure that its value function is still accurate for parts of the state space it no longer visits.

Experience replay

For model-based reinforcement learning systems, function approximation can also be very helpful for learning a model of the environment. Remember that learning a model for an observable environment is a supervised learning problem, because the next percept gives the outcome state. Any of the supervised learning methods in Chapters 19 –21  can be used, with suitable adjustments for the fact that we need to predict a complete state description rather than just a Boolean classification or a single real value. With a learned model, the agent can do a look-ahead search to improve its decisions and can carry out internal simulations to improve its approximate representations of and potentially expensive real-world experiences.

U or Q rather than requiring slow

For a partially observable environment, the learning problem is much more difficult because the next percept is no longer a label for the state prediction problem. If we know what the hidden variables are and how they are causally related to each other and to the observable variables, then we can fix the structure of a dynamic Bayesian network and use the EM algorithm to learn the parameters, as was described in Chapter 20 . Learning the internal structure of dynamic Bayesian networks and creating new state variables is still considered a difficult problem. Deep recurrent neural networks (Section 21.6 ) have in some cases been successful at inventing the hidden structure.

22.4.3 Deep reinforcement learning There are two reasons why we need to go beyond linear function approximators: first, there may be no good linear function that comes close to approximating the utility function or the Q-function; second, we may not be able to invent the necessary features, particularly in new domains. If you think about it, these are really the same reason: it is always possible to

U or Q as linear combinations of features, especially if we have features such as f (s) = U (s) or f (s,a) = Q(s,a), but unless we can come up with such features (in an represent 1

2

efficiently computable form) the linear function approximator may be insufficient.

For these reasons (or reason), researchers have explored more complex, nonlinear function approximators since the earliest days of reinforcement learning. Currently, deep neural networks (Chapter 21 ) are very popular in this role and have proved to be effective even when the input is a raw image with no human-designed feature extraction at all. If all goes well, the deep neural network in effect discovers the useful features for itself. And if the final layer of the network is linear, then we can see what features the network is using to build its own linear function approximator. A reinforcement learning system that uses a deep network as a function approximator is called a deep reinforcement learning system.

θ

Just as in Equation (22.9) , the deep network is a function parameterized by , except that now the function is much more complicated. The parameters are all the weights in all the layers of the network. Nonetheless, the gradients required for Equations (22.11)  and (22.12)  are just the same as the gradients required for supervised learning, and they can be computed by the same back-propagation process described in Section 21.4 . As we explain in Section 22.7 , deep RL has achieved very significant results, including learning to play a wide range of video games at an expert level, defeating the human world

champion at Go, and training robots to perform complex tasks.

Despite its impressive successes, deep RL still faces significant obstacles: it is often difficult to get good performance and the trained system may behave very unpredictably if the environment differs even a little from the training data. Compared to other applications of deep learning, deep RL is rarely applied in commercial settings. It is, nonetheless, a very active area of research.

22.4.4 Reward shaping As noted in the introduction to this chapter, real-world environments may have very sparse rewards: many primitive actions are required to achieve any nonzero reward. For example, a soccer-playing robot might send a hundred thousand motor control commands to its various joints before conceding a goal. Now it has to work out what it did wrong. The technical term for this is the credit assignment problem. Other than playing trillions of soccer games so that the negative reward eventually propagates back to the actions responsible for it, is there a good solution?

Credit assignment

One common method, originally used in animal training, is called reward shaping. This involves supplying the agent with additional rewards, called pseudorewards, for “making progress.” For example, we might give pseudorewards to the robot for making contact with the ball or for advancing it toward the goal. Such rewards can speed up learning enormously and are simple to provide, but there is a risk that the agent will learn to maximize the pseudorewards rather than the true rewards; for example, standing next to the ball and “vibrating” causes many contacts with the ball.

Reward shaping

Pseudoreward

In Chapter 17  (page 569), we saw a way to modify the reward function without changing

s

the optimal policy. For any potential function Φ( ) and any reward function a new reward function

R

′

R, we can create

as follows:

R (s,a,s ) = R(s,a,s ) + γΦ(s ) − Φ(s). ′

′

′

′

The potential fuction Φ can be constructed to reflect any desirable aspects of the state, such as achievement of subgoals or distance to a desired terminal state. For example, Φ for the soccer-playing robot could add a constant bonus for states where the robot’s team has possession and another bonus for reducing the distance of the ball from the opponents’ goal. This will result in faster learning overall, but will not prevent the robot from, say, learning to pass back to the goalkeeper when danger threatens.

22.4.5 Hierarchical reinforcement learning

Hierarchical reinforcement learning

Another way to cope with very long action sequences is to break them up into a few smaller pieces, and then break those into smaller pieces still, and so on until the action sequences are short enough to make learning easy. This approach is called hierarchical reinforcement learning (HRL), and it has much in common with the HTN planning methods described in Chapter 11 . For example, scoring a goal in soccer can be broken down into obtaining possession, passing to a teammate, receiving the ball from a team-mate, dribbling toward the goal, and shooting; each of these can be broken down further into lower-level motor behaviors. Obviously, there are multiple ways of obtaining possession and shooting, multiple teammates one could pass to, and so on, so each higher-level action may have many different lower-level implementations.

To illustrate these ideas, we’ll use a simplified soccer game called keepaway, in which one team of three players tries to keep possession of the ball for as long as possible by dribbling and passing amongst themselves while the other team of two players tries to take possession by intercepting a pass or tackling a player in possession.5 The game is implemented within the RoboCup 2D simulator, which provides detailed continuous-state motion models with 100ms time steps and has proved to be a good testbed for RL systems.

5 Rumors that keepaway was inspired by the real-world tactics of Barcelona FC are probably unfounded.

Keepaway

A hierarchical reinforcement learning agent begins with a partial program that outlines a hierarchical structure for the agent’s behavior. The partial-programming language for agent programs extends any ordinary programming language by adding primitives for unspecified choices that must be filled in by learning. (Here, we use pseudocode for the programming language.) The partial program can be arbitrarily complicated, as long as it terminates.

Partial program

It is easy to see that HRL includes ordinary RL as a special case. We simply provide the trivial partial program that allows the agent to keep choosing any action from

s

of actions that can be executed in the current state :

while true do choose(A(s)).

A(s), the set

The choose operator allows the agent to choose any element of the specified set. The learning process converts the partial agent program into a complete program by learning how each choice should be made. For example, the learning process might associate a Qfunction with each choice; once the Q-functions are learned, the program produces behavior by choosing the option with the highest Q-value each time it encounters a choice.

The agent programs for keepaway are more interesting. We’ll look at the partial program for a single player on the “keeper” team. The choice of what to do at the top level depends mainly on whether the player has the ball or not:

while not IS-TERMINAL(s) do if BALL-IN-MY-POSSESSION(s) then choose({PASS,HOLD,DRIBBLE}) else choose({STAY,MOVE, INTERCEPT-BALL}).

Each of these choices invokes a subroutine that may itself make further choices, all the way down to primitive actions that can be executed directly. For example, the high-level action PASS chooses a teammate to pass to, but also has the choice to do nothing and return control to the higher level if appropriate (e.g., if there is no one to pass to):

choose({PASS-TO(choose(TEAMMATES(s))), return}).

The PASS-TO routine then has to choose a speed and direction for the pass. While it is relatively easy for a human—even one with little expertise in soccer—to provide this kind of high-level advice to the learning agent, it would be difficult, if not impossible, to write down the rules for determining the speed and direction of the kick to maximize the probability of maintaining possession. Similarly, it is far from obvious how to choose the right teammate to receive the ball or where to move in order to make oneself available to receive the ball. The partial program provides general know-how—overall scaffolding and structural organization for complex behaviors—and the learning process works out all the details.

The theoretical foundations of HRL are based on the concept of the joint state space, in which each state (s,m) is composed of a physical state s and a machine state m. The machine state is defined by the current internal state of the agent program: the program counter for each subroutine on the current call stack, the values of the arguments, and the values of all local and global variables. For example, if the agent program has chosen to pass to teammate Ali and is in the middle of calculating the speed of the pass, then the fact that Ali is the argument of PASS-TO is part of the current machine state. A choice state

σ = (s,m)

is one in which the program counter for m is at a choice point in the agent program. Between two choice states, any number of computational transitions and physical actions may occur, but they are all preordained, so to speak: by definition, the agent isn’t making any choices in between choice states. Essentially, the hierarchical RL agent is solving a Markovian decision problem with the following elements: The states are the choice states σ of the joint state space. The actions at σ are the choices c available in σ according to the partial program. The reward function ρ(σ,c,σ ′ ) is the expected sum of rewards for all physical transitions occurring between the choice states σ and σ ′ . The transition model τ (σ,c,σ ′ ) is defined in the obvious way: if c invokes a physical action a, then τ borrows from the physical model

P (s′ |s,a); if c invokes a computational

transition, such as calling a subroutine, then the transition deterministically modifies the computational state m according to the rules of the programming language.6 6 Because more than one physical action may be executed before the next choice state is reached, the problem is technically a semi-Markov decision process, which allows actions to have different durations, including stochastic durations. If the discount factor γ < 1, then the action duration affects the discounting applied to the reward obtained during the action, which means that some extra discount bookkeeping has to be done and the transition model includes the duration distribution.

Joint state space

Choice state

By solving this decision problem, the agent finds the optimal policy that is consistent with original partial program.

Hierarchical RL can be a very effective method for learning complex behaviors. In keepaway, an HRL agent based on the partial program sketched above learns a policy that keeps possession forever against the standard taker policy—a significant improvement on the previous record of about 10 seconds. One important characteristic is that the lower-level skills are not fixed subroutines in the usual sense; their choices are sensitive to the entire internal state of the agent program, so they behave differently depending on where they are invoked within that program and what is going on at the time. If necessary, the Q-functions for the lower-level choices can be initialized by a separate training process with its own reward function, and then integrated into the overall system so they can be adapted to function well in the context of the whole agent.

In the preceding section we saw that shaping rewards can be helpful for learning complex behaviors. In HRL, the fact that learning takes place in the joint state space provides additional opportunities for shaping. For example, to help with learning the Q-function for accurate passing within the PASS-TO routine, we can provide a shaping reward that depends on the location of the intended recipient and the proximity of opponents to that player: the ball should be close to the recipient and far from the opponents. That seems entirely obvious; but the identity of the intended recipient for a pass is not part of the physical state of the world. The physical state consists only of the positions, orientations, and velocities of the players and the ball. There is no “passing” and no “recipient” in the physical world; these are entirely internal constructs. This means that there is no way to provide such sensible advice to a standard RL system.

The hierarchical structure of behavior also provides a natural additive decomposition of the overall utility function. Remember that utility is the sum of rewards over time, and consider a sequence of, say, ten time steps with rewards [r1 ,r2 , … ,r10 ]. Suppose that for the first five time steps the agent is doing PASS-TO(Ali) and for the remaining five steps it is doing MOVEINTO-SPACE. Then the utility for the initial state is the sum of the total reward during PASS-TO and the total reward during MOVE-INTO-SPACE. The former depends only on whether the ball gets to Ali with enough time and space for Ali to retain possession, and the latter depends only on whether the agent reaches a good location to receive the ball. In other words, the overall utility decomposes into several terms, each of which depends on only a few

variables. This, in turns, means that learning occurs much more quickly than if we try to learn a single utility function that depends on all the variables. This is somewhat analogous to the representation theorems underlying the conciseness of Bayes nets (Chapter 13 ).

Additive decomposition

22.5 Policy Search The final approach we will consider for reinforcement learning problems is called policy search. In some ways, policy search is the simplest of all the methods in this chapter: the idea is to keep twiddling the policy as long as its performance improves, then stop.

Policy search

π is a function that maps states to actions. We are interested primarily in parameterized representations of π that have Let us begin with the policies themselves. Remember that a policy

far fewer parameters than there are states in the state space (just as in the preceding section). For example, we could represent

π by a collection of parameterized Q-functions,

one for each action, and take the action with the highest predicted value:

(22.13) ˆ (s,a). π(s) = argmax  Q θ a

Each Q-function could be a linear function, as in Equation (22.9) , or it could be a nonlinear function such as a deep neural network. Policy search will then adjust the parameters

θ to improve the policy. Notice that if the policy is represented by Q-functions,

then policy search results in a process that learns Q-functions. This process is not the same as Q-learning!

θ such that Qˆ θ is “close” to Q , the optimal Q-function. Policy search, on the other hand, finds a value of θ In Q-learning with function approximation, the algorithm finds a value of ∗

that results in good performance; the values found by the two methods may differ very

Q sa

substantially. (For example, the approximate Q-function defined by ˆ θ ( , ) = gives optimal performance, even though it is not at all close to

πs

Q (s,a)/100 ∗

Q .) Another clear instance ∗

of the difference is the case where ( ) is calculated using, say, depth-10 look-ahead search

U

with an approximate utility function ˆ θ . A value of

U

θ that gives good results may be a long

way from making ˆ θ resemble the true utility function. One problem with policy representations of the kind given in Equation (22.13)  is that the policy is a discontinuous function of the parameters when the actions are discrete. That is, there will be values of

θ such that an infinitesimal change in θ causes the policy to switch

from one action to another. This means that the value of the policy may also change discontinuously, which makes gradient-based search difficult. For this reason, policy search methods often use a stochastic policy representation of selecting action

πθ (s,a), which specifies the probability

a in state s. One popular representation is the softmax function:

(22.14)

β Qθ s a πθ ( s,a ) = e β Q s a ∑a e θ ˆ ( , )

′

ˆ ( , ′)

.

Stochastic policy

The parameter

β > 0 modulates the softness of the softmax: for values of β that are large

compared to the separations between Q-values, the softmax approaches a hard max,

β close to zero the softmax approaches a uniform random choice among the actions. For all finite values of β, the softmax provides a differentiable function of θ; hence, the value of the policy (which depends continuously on the action-selection probabilities) is a differentiable function of θ. whereas for values of

Now let us look at methods for improving the policy. We start with the simplest case: a

ρθ

deterministic policy and a deterministic environment. Let ( ) be the policy value, that is, the expected reward-to-go when

πθ is executed. If we can derive an expression for ρ(θ) in

closed form, then we have a standard optimization problem, as described in Chapter 4 .

ρθ

ρθ

We can follow the policy gradient vector ∇θ ( ), provided ( ) is differentiable.

ρθ

Alternatively, if ( ) is not available in closed form, we can evaluate

πθ simply by executing

it and observing the accumulated reward. We can follow the empirical gradient by hill climbing—that is, evaluating the change in policy value for small increments in each

parameter. With the usual caveats, this process will converge to a local optimum in policy space.

Policy value

Policy gradient

When the environment (or the policy) is nondeterministic, things get more difficult.

ρθ

ρθ

θ

Suppose we are trying to do hill climbing, which requires comparing ( ) and ( + Δ ) for

θ

some small Δ . The problem is that the total reward for each trial may vary widely, so estimates of the policy value from a small number of trials will be quite unreliable; trying to compare two such estimates will be even more unreliable. One solution is simply to run lots of trials, measuring the sample variance and using it to determine that enough trials have

ρθ

been run to get a reliable indication of the direction of improvement for ( ). Unfortunately, this is impractical for many real problems in which trials may be expensive, timeconsuming, and perhaps even dangerous.

πθ (s,a), it is possible to obtain an unbiased estimate of the gradient at θ, ∇θ ρ(θ), directly from the results of trials executed at θ. For For the case of a nondeterministic policy

simplicity, we will derive this estimate for the simple case of an episodic environment in which each action

a obtains reward R(s ,a,s ) and the environment restarts in s . In this 0

0

0

case, the policy value is just the expected value of the reward, and we have

ρθ

∇θ ( ) = ∇θ

∑R s as π s a ∑R s as a

(

0,

,

0)

θ(

0,

)=

a

(

0,

,

0 )∇

θ πθ (s0 ,a).

Now we perform a simple trick so that this summation can be approximated by samples generated from the probability distribution defined by

j

trials in all, and the action taken on the th trial is

πθ (s ,a). Suppose that we have N

aj . Then

0

ρθ

∇θ ( ) =

∑ π s a R s aπs s aπ s a Rs a s π s a ∑ N π s a θ(

a

=≈

1

0,

(

)⋅

N

(

0,

,

0 )∇

θ(

0,

j=1

j,

θ θ(

0,

θ θ( θ( 0, j) 0 )∇

0,

)

)

0,

j)

.

Thus, the true gradient of the policy value is approximated by a sum of terms involving the gradient of the action-selection probability in each trial. For the sequential case, this generalizes to

ρθ

∇θ ( ) ≈

u s π sa N ∑ π sa 1

N

j=1

j(

)∇θ

θ( , j) θ( , j)

s visited, where aj is executed in s on the jth trial and uj (s) is the total reward received from state s onward in the jth trial. The resulting algorithm, called R , is due for each state

EINFORCE

to Ron Williams (1992); it is usually much more effective than hill climbing using lots of

θ

trials at each value of . However, it is still much slower than necessary.

Consider the following task: given two blackjack policies, determine which is best. The policies might have true net returns per hand of, say, −0.21% and +0.06%, so finding out which is better is very important. One way to do this is to have each policy play against a standard “dealer” for a certain number of hands and then to measure their respective winnings. The problem with this, as we have seen, is that the winnings of each policy fluctuate wildly depending on whether it receives good or bad cards. One would need several million hands to have a reliable indication of which policy is better. The same issue arises when using random sampling to compare two adjacent policies in a hill-climbing algorithm.

A better solution for blackjack is to generate a certain number of hands in advance and have each program play the same set of hands. In this way, we eliminate the measurement error due to differences in the cards received. Only a few thousand hands are needed to determine which of the two blackjack policies is better.

This idea, called correlated sampling, can be applied to policy search in general, given an environment simulator in which the random-number sequences can be repeated. It was implemented in a policy-search algorithm called PEGASUS (Ng and Jordan, 2000), which was one of the first algorithms to achieve completely stable autonomous helicopter flight (see

Figure 22.9(b) ). It can be shown that the number of random sequences required to ensure that the value of every policy is well estimated depends only on the complexity of the policy space, and not at all on the complexity of the underlying domain.

Figure 22.9

(a) Setup for the problem of balancing a long pole on top of a moving cart. The cart can be jerked left or right by a controller that observes the cart’s position x and velocity x ˙, as well as the pole’s angle θ and rate of change of angle θ˙. (b) Six superimposed time-lapse images of a single autonomous helicopter

performing a very difficult “nose-in circle” maneuver. The helicopter is under the control of a policy developed by the PEGASUS policy-search algorithm (Ng et al., 2003). A simulator model was developed by observing the effects of various control manipulations on the real helicopter; then the algorithm was run on the simulator model overnight. A variety of controllers were developed for different maneuvers. In all cases, performance far exceeded that of an expert human pilot using remote control. (Image courtesy of Andrew Ng.)

Correlated sampling

22.6 Apprenticeship and Inverse Reinforcement Learning Some domains are so complex that it is difficult to define a reward function for use in reinforcement learning. Exactly what do we want our self-driving car to do? Certainly it should not take too long to get to the destination, but it should not drive so fast as to incur undue risk or to get speeding tickets. It should conserve fuel/energy. It should avoid jostling or accelerating the passengers too much, but it can slam on the brakes in an emergency. And so on. Deciding how much weight to give to each of these factors is a difficult task. Worse still, there are almost certainly important factors we have forgotten, such as the obligation to behave with consideration for other drivers. Omitting a factor usually leads to behavior that assigns an extreme value to the omitted factor—in this case, extremely inconsiderate driving—in order to maximize the remaining factors.

One approach is to do extensive testing in simulation, notice problematic behaviors, and try to modify the reward function to eliminate those behaviors. Another approach is to seek additional sources of information about the appropriate reward function. One such source is the behavior of agents who are already optimizing (or, let’s say, nearly optimizing) that reward function—in this case, expert human drivers.

The general field of apprenticeship learning studies the process of learning how to behave well given observations of expert behavior. We show the algorithm examples of expert driving and tell it to “do it like that.” There are (at least) two ways to approach the apprenticeship learning problem. The first is the one we discussed briefly at the beginning of the chapter: assuming the environment is observable, we apply supervised learning to the observed state–action pairs to learn a policy π(s). This is called imitation learning. It has had some success in robotics (see page 966) but suffers from the the problem of brittleness: even small deviations from the training set lead to errors that grow over time and eventually to failure. Moreover, imitation learning will at best duplicate the teacher’s performance, not exceed it. When humans learn by imitation, we sometimes use the pejorative term “aping” to describe what they are doing. (It’s quite possible that apes use the term “humaning” amongst themselves, perhaps in an even more pejorative sense.) The implication is that the imitation learner doesn’t understand why it should perform any given action.

Apprenticeship learning

Imitation learning

The second approach to apprenticeship learning is to understand why: to observe the expert’s actions (and resulting states) and try to work out what reward function the expert is maximizing. Then we could derive an optimal policy with respect to that reward function. One expects that this approach will produce robust policies from relatively few examples of expert behavior; after all, the field of reinforcement learning is predicated on the idea that the reward function, rather than the policy or the value function, is the most succinct, robust, and transferable definition of the task. Furthermore, if the learner makes appropriate allowances for possible suboptimality on the part of the expert, then it may be able to do better than the expert by optimizing an accurate approximation to the true reward function. We call this approach inverse reinforcement learning (IRL): learning rewards by observing a policy, rather than learning a policy by observing rewards.

Inverse reinforcement learning

How do we find the expert’s reward function, given the expert’s actions? Let us begin by assuming that the expert was acting rationally. In that case, it seems we should be looking for a reward function

R

∗

such that the total expected discounted reward under the expert’s

policy is higher than (or at least the same as) under any other possible policy.

Unfortunately, there will be many reward functions that satisfy this constraint; one of them is

R (s,a,s ) = 0, because any policy is rational when there are no rewards at all.7 Another ∗

′

problem with this approach is that the assumption of a rational expert is unrealistic. It means, for example, that a robot observing Lee Sedol making what eventually turns out to

be a losing move against ALPHAGO would have to assume that Lee Sedol was trying to lose the game. 7 According to Equation (17.9)  on page 569, a reward function R′ (s,a,s′ ) =

R(s,a,s′ ) + γΦ(s′ ) − Φ(s) has exactly the same optimal policies as R(s,a,s ), so we can recover the reward function only up to the possible addition of any shaping function Φ(s). This is not such a serious problem, because a robot using R′ will behave just like a robot using the “correct” R. ′

To avoid the problem that R∗ (s,a,s′ ) = 0 explains any observed behavior, it helps to think in

a Bayesian way. (See Section 20.1  for a reminder of what this means.) Suppose we observe data d and let hR be the hypothesis that R is the true reward function. Then according to Bayes’ rule, we have

P (hR |d) = αP (d|hR )P (hR ). Now, if the prior P (hR ) is based on simplicity, then the hypothesis that R = 0 scores fairly well, because 0 is certainly simple. On the other hand, the term P (d|hR ) is infinitesimal for the hypothesis that R = 0, because it doesn’t explain why the expert chose that particular behavior out of the vast space of behaviors that would be optimal if the hypothesis were true. On the other hand, for a reward function R that has a unique optimal policy or a relatively small equivalence class of optimal policies, P (d|hR ) will be far higher. To allow for the occasional mistake by the expert, we simply allow P (d|hR ) to be nonzero even when d comes from behavior that is a little bit suboptimal according to R. A typical assumption—made, it must be said, more for mathematical convenience than faithfulness to actual human data—is that an agent whose true Q-function is Q(s,a) chooses not according to the deterministic policy π(s) = arg maxa Q(s,a) but instead according to a stochastic

policy defined by the softmax distribution from Equation (22.14) . This is sometimes called Boltzmann rationality because, in statistical mechanics, the state occupation probabilities in a Boltzmann distribution depend exponentially on their energy levels.

Boltzmann rationality

There are dozens of inverse RL algorithms in the literature. One of the simplest is called feature matching. It assumes that the reward function can be written as a weighted linear

combination of features:

R θ ( s ,a , s ) = ′

∑θ f sas n

, , ′) =

i i(

i=1

θ ⋅ f.

Feature matching

For example, the features in the driving domain might include speed, speed in excess of the speed limit, acceleration, proximity to nearest obstacle, etc. Recall from Equation (17.2)  on page 567 that the utility of executing a policy , starting in state

π

s , is defined to be 0

U π ( s) = E [

∑γ R S π S ∞

t

(

t=0

t,

(

t ),St+1 )] ,

E is with respect to the probability distribution over state sequences determined by s and π. Because R is assumed to be a linear combination of feature values, where the expectation

we can rewrite this as follows:

U π ( s)

=

∑γ ∑θ f S π S S ∑ θ E [∑ γ f S π S S ∑θ μ π θ μ π E[ n

=

=

Feature expectation

i=1 n

i=1

∞

t=0

t

n

i=1

∞

i

i i(

i i( t,

(

t ), t+1 )]

t

(

t ), t+1 )]

i( t,

t=0

)=

⋅

( )

μ π

where we have defined the feature expectation i ( ) as the expected discounted value of

f

π is executed. For example, if fi is the excess speed of the vehicle (above the speed limit), then μi (π) is the (time-discounted) average excess speed over the entire trajectory. The key point about feature expectations is the following: if a policy π produces feature expectations μi (π) that match those of the expert’s policy πE , then π is as good as the feature i when policy

the expert’s policy according to the expert’s own reward function. Now, we cannot measure the exact values for the feature expectations of the expert’s policy, but we can approximate them using the average values on the observed trajectories. Thus, we need to find values for the

θ

parameters i such that the feature expectations of the policy induced by the parameter values match those of the expert policy on the observed trajectories. The following algorithm achieves this with any desired error bound.

Pick an initial default policy

π

(0)

.

j = 1,2, … until convergence: – Find parameters θ j such that the expert’s policy maximally outperforms the policies π , … ,π j according to the expected utility θ j ⋅ μ(π). – Let π j be the optimal policy for the reward function R j = θ j ⋅ f. For

( )

(0)

( −1)

( )

( )

( )

( )

This algorithm converges to a policy that is close in value to the expert’s, according to the expert’s own reward function. It requires only demonstrations, where

O(n log n) iterations and O(n log n) expert

n is the number of features.

A robot can use inverse reinforcement learning to learn a good policy for itself, by understanding the actions of an expert. In addition, the robot can learn the policies used by other agents in a multiagent domain, whether they be adversarial or cooperative. And finally, inverse reinforcement learning can be used for scientific inquiry (without any thought of agent design), to better understand the behavior of humans and other animals.

A key assumption in inverse RL is that the “expert” is behaving optimally, or nearly optimally, with respect to some reward function in a single-agent MDP. This is a reasonable assumption if the learner is watching the expert through a one-way mirror while the expert goes about his or her business unawares. It is not a reasonable assumption if the expert is aware of the learner. For example, suppose a robot is in medical school, learning to be a surgeon by watching a human expert. An inverse RL algorithm would assume that the human performs the surgery in the usual optimal way, as if the robot were not there. But

that’s not what would happen: the human surgeon is motivated to have the robot (like any other medical student) learn quickly and well, and so she will modify her behavior considerably. She might explain what she is doing as she goes along; she might point out mistakes to avoid, such as making the incision too deep or the stitches too tight; she might describe the contingency plans in case something goes wrong during surgery. None of these behaviors make sense when performing surgery in isolation, so inverse RL algorithms will not be able to interpret the underlying reward function. Instead, we need to understand this kind of situation as a two-person assistance game, as described in Section 18.2.5 .

22.7 Applications of Reinforcement Learning We now turn to applications of reinforcement learning. These include game playing, where the transition model is known and the goal is to learn the utility function, and robotics, where the model is initially unknown.

22.7.1 Applications in game playing In Chapter 1  we described Arthur Samuel’s early work on reinforcement learning for checkers, which began in 1952. A few decades passed before the challenge was taken up again, this time by Gerry Tesauro in his work on backgammon. Tesauro’s first attempt (1990) was a system called NEUROGAMMON. The approach was an interesting variant on imitation learning. The input was a set of 400 games played by Tesauro against himself. Rather than learn a policy, NEUROGAMMON converted each move (s,a,s′ ) into a set of training examples, each of which labeled s′ as a better position than some other position s′′ reachable from s by a different move. The network had two separate halves, one for s′ and one for s′′ , and was constrained to choose which was better by comparing the outputs of the two halves. In this way, each half was forced to learn an evaluation function Uˆθ . NEUROGAMMON won the 1989 Computer Olympiad—the first learning program ever to win a computer game tournament—but never progressed past Tesauro’s own intermediate level of play.

Tesauro’s next system, TD-GAMMON (1992), adopted Sutton’s recently published TD learning method—essentially returning to the approach explored by Samuel, but with much greater technical understanding of how to do it right. The evaluation function Uˆθ was represented by a fully connected neural network with a single hidden layer containing 80 nodes. (It also used some manually designed input features borrowed from NEUROGAMMON.) After 300,000 training games, it reached a standard of play comparable to the top three human players in the world. Kit Woolsey, a top-ten player, said, “There is no question in my mind that its positional judgment is far better than mine.”

The next challenge was to learn from raw perceptual inputs—something closer to the real world—rather than discrete game board representations. Beginning in 2012, a team at DeepMind developed the deep Q-network (DQN) system, the first modern deep RL system. DQN uses a deep neural network to represent the Q-function; otherwise it is a typical

reinforcement learning system. DQN was trained separately on each of 49 different Atari video games. It learned to drive simulated race cars, shoot alien spaceships, and bounce balls with paddles. In each case, the agent learned a Q-function from raw image data with the reward signal being the game score. Overall, the system performed at roughly human expert level, although a few games gave it trouble. One game in particular, Montezuma’s Revenge, proved far too difficult, because it required extended planning strategies, and the rewards were too sparse. Subsequent work produced deep RL systems that generated more extensive exploratory behaviors and were able to conquer Montezuma’s Revenge and other difficult games.

Deep Q-network (DQN)

DeepMind’s ALPHAGO system also used deep reinforcement learning to beat the best human players at the game of Go (see Chapter 5 ). Whereas a Q-function with no look-ahead suffices for Atari games, which are primarily reactive in nature, Go requires substantial lookahead. For this reason, ALPHAGO learned both a value function and a Q-function that guided its search by predicting which moves are worth exploring. The Q-function, implemented as a convolutional neural network, is accurate enough by itself to beat most amateur human players without any search at all.

22.7.2 Application to robot control The setup for the famous cart–pole balancing problem, also known as the inverted pendulum, is shown in Figure 22.9(a) . The problem is to keep the pole roughly upright (

θ ≈ 90∘ ) by applying forces to move the cart right or left, while keeping the position x within the limits of the track. Several thousand papers in reinforcement learning and control theory have been published on this seemingly simple problem. One difficulty is that the state

˙, and θ˙ are continuous. The actions, however, are defined to be discrete: jerk variables x, θ, x left or jerk right, the so-called bang-bang control regime.

Cart–pole

Inverted pendulum

Bang-bang control

The earliest work on learning for this problem was carried out by Michie and Chambers (1968), using a real cart and pole, not a simulation. Their BOXES algorithm was able to balance the pole for over an hour after 30 trials. The algorithm first discretized the fourdimensional state space into boxes—hence the name. It then ran trials until the pole fell over. Negative reinforcement was associated with the final action in the final box and then propagated back through the sequence. Improved generalization and faster learning can be obtained using an algorithm that adaptively partitions the state space according to the observed variation in the reward, or by using a continuous-state, nonlinear function approximator such as a neural network. Nowadays, balancing a triple inverted pendulum (three poles joined together end to end) is a common exercise—a feat far beyond the capabilities of most humans, but achievable using reinforcement learning.

Still more impressive is the application of reinforcement learning to radio-controlled helicopter flight (Figure 22.9(b) ). This work has generally used policy search over large MDPs (Bagnell and Schneider, 2001; Ng et al., 2003), often combined with imitation learning and inverse RL given observations of a human expert pilot (Coates et al., 2009).

Inverse RL has also been applied successfully to interpret human behavior, including destination prediction and route selection by taxi drivers based on 100,000 miles of GPS data (Ziebart et al., 2008) and detailed physical movements by pedestrians in complex environments based on hours of video observation (Kitani et al., 2012). In the area of robotics, a single expert demonstration was enough for the LittleDog quadruped to learn a 25-feature reward function and nimbly traverse a previously unseen area of rocky terrain (Kolter et al., 2008). For more on how RL and inverse RL are used in robotics, see Sections 26.7  and 26.8 .

Summary This chapter has examined the reinforcement learning problem: how an agent can become proficient in an unknown environment, given only its percepts and occasional rewards. Reinforcement learning is a very broadly applicable paradigm for creating intelligent systems. The major points of the chapter are as follows.

The overall agent design dictates the kind of information that must be learned: – A model-based reinforcement learning agent acquires (or is equipped with) a transition model P (s′ |s,a) for the environment and learns a utility function U (s). – A model-free reinforcement learning agent may learn an action-utility function Q(s,a) or a policy

π (s ).

Utilities can be learned using several different approaches: – Direct utility estimation uses the total observed reward-to-go for a given state as direct evidence for learning its utility. – Adaptive dynamic programming (ADP) learns a model and a reward function from observations and then uses value or policy iteration to obtain the utilities or an optimal policy. ADP makes optimal use of the local constraints on utilities of states imposed through the neighborhood structure of the environment. – Temporal-difference (TD) methods adjust utility estimates to be more consistent with those of successor states. They can be viewed as simple approximations of the ADP approach that can learn without requiring a transition model. Using a learned model to generate pseudoexperiences can, however, result in faster learning. Action-utility functions, or Q-functions, can be learned by an ADP approach or a TD approach. With TD, Q-learning requires no model in either the learning or actionselection phase. This simplifies the learning problem but potentially restricts the ability to learn in complex environments, because the agent cannot simulate the results of possible courses of action. When the learning agent is responsible for selecting actions while it learns, it must trade off the estimated value of those actions against the potential for learning useful new information. An exact solution for the exploration problem is infeasible, but some simple heuristics do a reasonable job. An exploring agent must also take care to avoid premature death.

In large state spaces, reinforcement learning algorithms must use an approximate functional representation of U (s) or

Q(s,a) in order to generalize over states. Deep

reinforcement learning—using deep neural networks as function approximators—has achieved considerable success on hard problems. Reward shaping and hierarchical reinforcement learning are helpful for learning complex behaviors, particularly when rewards are sparse and long action sequences are required to obtain them. Policy-search methods operate directly on a representation of the policy, attempting to improve it based on observed performance. The variation in the performance in a stochastic domain is a serious problem; for simulated domains this can be overcome by fixing the randomness in advance. Apprenticeship learning through observation of expert behavior can be an effective solution when a correct reward function is hard to specify. Imitation learning formulates the problem as supervised learning of a policy from the expert’s state–action pairs. Inverse reinforcement learning infers reward information from the expert’s behavior.

Reinforcement learning continues to be one of the most active areas of machine learning research. It frees us from manual construction of behaviors and from labeling the vast data sets required for supervised learning, or having to hand-code control strategies. Applications in robotics promise to be particularly valuable; these will require methods for handling continuous, high-dimensional, partially observable environments in which successful behaviors may consist of thousands or even millions of primitive actions.

We have presented a variety of approaches to reinforcement learning because there is (at least so far) no single best approach. The question of model-based versus model-free methods is, at its heart, a question about the best way to represent the agent function. This is an issue at the foundations of artificial intelligence. As we stated in Chapter 1 , one of the key historical characteristics of much AI research is its (often unstated) adherence to the knowledge-based approach. This amounts to an assumption that the best way to represent the agent function is to build a representation of some aspects of the environment in which the agent is situated. Some argue that with access to sufficient data, model-free methods can succeed in any domain. Perhaps this is true in theory, but of course, the universe may not contain enough data to make it true in practice. (For example, it is not easy to imagine how a model-free approach would enable one to design and build, say, the LIGO gravity-wave

detector.) Our intuition, for what it’s worth, is that as the environment becomes more complex, the advantages of a model-based approach become more apparent.

Bibliographical and Historical Notes It seems likely that the key idea of reinforcement learning—that animals do more of what they are rewarded for and less of what they are punished for—played a significant role in the domestication of dogs at least 15,000 years ago. The early foundations of our scientific understanding of reinforcement learning include the work of the Russian physiologist Ivan Pavlov, who won the Nobel Prize in 1904, and that of the American psychologist Edward Thorndike—particularly his book Animal Intelligence (1911). Hilgard and Bower (1975) provide a good survey.

Alan Turing (1948, 1950) proposed reinforcement learning as an approach for teaching computers; he considered it a partial solution, writing, “The use of punishments and rewards can at best be a part of the teaching process.” Arthur Samuel’s checkers program (1959, 1967) was the first successful use of machine learning of any kind. Samuel suggested most of the modern ideas in reinforcement learning, including temporal-difference learning and function approximation. He experimented with multilayer representations of value functions, similar to today’s deep RL. In the end, he found that a simple linear evaluation function over handcrafted features worked best. This may have been a consequence of working with a computer roughly 100 billion times less powerful than a modern tensor processing unit.

Around the same time, researchers in adaptive control theory (Widrow and Hoff, 1960), building on work by Hebb (1949), were training simple networks using the delta rule. Thus, reinforcement learning combines influences from animal psychology, neuroscience, operations research, and optimal control theory.

The connection between reinforcement learning and Markov decision processes was first made by Werbos (1977). (Work by Ian Witten (1977) described a TD-like process in the language of control theory.) The development of reinforcement learning in AI stems primarily from work at the University of Massachusetts in the early 1980s (Barto et al., 1981). An influential paper by Rich Sutton (1988) provided a mathematical understanding of temporal-difference methods. The combination of temporal-difference learning with the model-based generation of simulated experiences was proposed in Sutton’s DYNA architecture (Sutton, 1990). Q-learning was developed in Chris Watkins’s Ph.D. thesis

(1989), while SARSA appeared in a technical report by Rummery and Niranjan (1994). Prioritized sweeping was introduced independently by Moore and Atkeson (1993) and Peng and Williams (1993).

Function approximation in reinforcement learning goes back to Arthur Samuel’s checkers program (1959). The use of neural networks to represent value functions was common in the 1980s and came to the fore in Gerry Tesauro’s TD-Gammon program (Tesauro, 1992, 1995). Deep neural networks are currently the most popular choice for function approximators in reinforcement learning. Arulkumaran et al. (2017) and Francois-Lavet et al. (2018) give overviews of deep RL. The DQN system (Mnih et al., 2015) uses a deep network to learn a Q-function, while ALPHAZERO (Silver et al., 2018) learns both a value function for use with a known model and a Q-function for use in metalevel decisions that guide search. Irpan (2018) warns that deep RL systems can perform poorly if the actual environment is even slightly different from the training environment.

Weighted linear combinations of features and neural networks are factored representations for function approximation. It is also possible to apply reinforcement learning to structured representations; this is called relational reinforcement learning (Tadepalli et al., 2004). The use of relational descriptions allows for generalization across complex behaviors involving different objects.

Analysis of the convergence properties of reinforcement learning algorithms using function approximation is an extremely technical subject. Results for TD learning have been progressively strengthened for the case of linear function approximators (Sutton, 1988; Dayan, 1992; Tsitsiklis and Van Roy, 1997), but several examples of divergence have been presented for nonlinear functions (see Tsitsiklis and Van Roy, 1997, for a discussion). Papavassiliou and Russell (1999) describe a type of reinforcement learning that converges with any form of function approximator, provided that the problem of fitting the hypothesis to the data is solvable. Liu et al. (2018) describe the family of gradient TD algorithms and provide extensive theoretical analysis of convergence and sample complexity.

A variety of exploration methods for sequential decision problems are discussed by Barto et al. (1995). Kearns and Singh (1998) and Brafman and Tennenholtz (2000) describe algorithms that explore unknown environments and are guaranteed to converge on nearoptimal policies with a sample complexity that is polynomial in the number of states.

Bayesian reinforcement learning (Dearden et al., 1998, 1999) provides another angle on both model uncertainty and exploration.

The basic idea underlying imitation learning is to apply supervised learning to a training set of expert actions. This is an old idea in adaptive control, but first came to prominence in AI with the work of Sammut et al. (1992) on “Learning to Fly” in a flight simulator. They called their method behavioral cloning. A few years later, the same research group reported that the method was much more fragile than had been reported initially (Camacho and Michie, 1995): even very small perturbations caused the learned policy to deviate from the desired trajectory, leading to compounding errors as the agent strayed further and further from the training set. (See also the discussion on page 966.) Work on apprenticeship learning aims to make the approach more robust, in part by including information about the desired outcomes rather than just the expert policy. Ng et al. (2003) and Coates et al. (2009) show how apprenticeship learning works for learning to fly an actual helicopter, as illustrated in Figure 22.9(b)  on page 817.

Inverse reinforcement learning (IRL) was introduced by Russell (1998), and the first algorithms were developed by Ng and Russell (2000). (A similar problem has been studied in economics for much longer, under the heading of structural estimation of MDPs (Sargent, 1978).) The algorithm given in the chapter is due to Abbeel and Ng (2004). Baker et al. (2009) describe how the understanding of another agent’s actions can be seen as inverse planning. Ho et al. (2017) show that agents can learn better from behaviors that are instructive rather than optimal. Hadfield-Menell et al. (2017a) extend IRL into a gametheoretic formulation that encompasses both observer and demonstrator, showing how teaching and learning behaviors emerge as solutions of the game.

García and Fernández (2015) give a comprehensive survey on safe reinforcement learning. Munos et al. (2017) describe an algorithm for safe off-policy (e.g., Q-learning) exploration. Hans et al. (2008) break the problem of safe exploration into two parts: defining a safety function to indicate which states to avoid, and defining a backup policy to lead the agent back to safety when it might otherwise enter an unsafe state. You et al. (2017) show how to train a deep reinforcement learning model to drive a car in simulation, and then use transfer learning to drive safely in the real world.

Thomas et al. (2017) offer an approach to learning that is guaranteed, with high probability, to do no worse than the current policy. Akametalu et al. (2014) describe a reachability-based approach, in which the learning process operates under the guidance of a control policy that ensures the agent never reaches an unsafe state. Saunders et al. (2018) demonstrate that a system can use human intervention to stop it from wandering out of the safe region, and can learn over time to need less intervention.

Policy search methods were brought to the fore by Williams (1992), who developed the REINFORCE family of algorithms, which stands for “REward

Increment = Nonnegative Factor × Offset Reinforcement × Characteristic Eligibility.” Later work by Marbach and Tsitsiklis (1998), Sutton et al. (2000), and Baxter and Bartlett (2000) strengthened and generalized the convergence results for policy search. Schulman et al. (2015b) describe trust region policy optimization, a theoretically well-founded and also practical policy search algorithm that has spawned many variants. The method of correlated sampling to reduce variance in Monte Carlo comparisons is due to Kahn and Marshall (1953); it is also one of a number of variance reduction methods explored by Hammersley and Handscomb (1964).

Early approaches to hierarchical reinforcement learning (HRL) attempted to construct hierarchies using state abstraction—that is, grouping states together into abstract states and then doing RL in the abstract state space (Dayan and Hinton, 1993). Unfortunately, the transition model for abstract states is typically non-Markovian, leading to divergent behavior of standard RL algorithms. The temporal abstraction approach in this chapter was developed in the late 1990s (Parr and Russell, 1998; Andre and Russell, 2002; Sutton et al., 2000) and extended to handle concurrent behaviors by Marthi et al. (2005). Dietterich (2000) introduced the notion of an additive decomposition of Q-functions induced by the subroutine hierarchy. Temporal abstraction is based on a much earlier result due to Forestier and Varaiya (1978), who showed that a large MDP can be decomposed into a twolayer system in which a supervisory layer chooses among low-level controllers, each of which returns control to the supervisor on completion. The problem of learning the abstraction hierarchy itself has been studied at least since the work of Peter Andreae (1985); for a recent exploration into learning robot motion primitives, see Frans et al. (2018). The keepaway game was introduced by Stone et al. (2005); the HRL solution given here is due to Bai and Russell (2017).

Neuroscience has often inspired reinforcement learning and confirmed the value of the approach. Research using single-cell recording suggests that the dopamine system in primate brains implements something resembling value-function learning (Schultz et al., 1997). The neuroscience text by Dayan and Abbott (2001) describes possible neural implementations of temporal-difference learning; related research describes other neuroscientific and behavioral experiments (Dayan and Niv, 2008; Niv, 2009; Lee et al., 2012).

Work in reinforcement learning has been accelerated by the availability of open-source simulation environments for developing and testing learning agents. The University of Alberta’s Arcade Learning Environment (ALE) (Bellemare et al., 2013) provided such a framework for 55 classic Atari video games. The pixels on the screen are provided to the agent as percepts, along with a hardwired score of the game so far. ALE was used by the DeepMind team to implement DQN learning and verify the generality of their system on a wide variety of games (Mnih et al., 2015).

DeepMind in turn open-sourced several agent platforms, including the DeepMind Lab (Beattie et al., 2016), the AI Safety Gridworlds (Leike et al., 2017), the Unity game platform (Juliani et al., 2018), and the DM Control Suite (Tassa et al., 2018). Blizzard released the StarCraft II Learning Environment (SC2LE), to which DeepMind added the PySC2 component for machine learning in Python (Vinyals et al., 2017a).

Facebook’s AI Habitat simulation (Savva et al., 2019) provides a photo-realistic virtual environment for indoor robotic tasks, and their HORIZON platform (Gauci et al., 2018) enables reinforcement learning in large-scale production systems. The SYNTHIA system (Ros et al., 2016) is a simulation environment designed for improving the computer vision capabilities of self-driving cars. The OpenAI Gym (Brockman et al., 2016) provides several environments for reinforcement learning agents, and is compatible with other simulations such as the Google Football simulator.

Littman (2015) surveys reinforcement learning for a general scientific audience. The canonical text by Sutton and Barto (2018), two of the field’s pioneers, shows how reinforcement learning weaves together the ideas of learning, planning, and acting. Kochenderfer (2015) takes a slightly less mathematical approach, with plenty of real-world examples. A short book by Szepesvari (2010) gives an overview of reinforcement learning

algorithms. Bertsekas and Tsitsiklis (1996) provide a rigorous grounding in the theory of dynamic programming and stochastic convergence. Reinforcement learning papers are published frequently in the journals Machine Learning and Journal of Machine Learning Research, and in the the proceedings of the International Conference on Machine Learning (ICML) and the Neural Information Processing Systems (NeurIPS) conferences.

VI Communicating, perceiving, and acting

Chapter 23 Natural Language Processing In which we see how a computer can use natural language to communicate with humans and learn from what they have written.

About 100,000 years ago, humans learned how to speak, and about 5,000 years ago they learned to write. The complexity and diversity of human language sets Homo sapiens apart from all other species. Of course there are other attributes that are uniquely human: no other species wears clothes, creates art, or spends two hours a day on social media in the way that humans do. But when Alan Turing proposed his test for intelligence, he based it on language, not art or haberdashery, perhaps because of its universal scope and because language captures so much of intelligent behavior: a speaker (or writer) has the goal of communicating some knowledge, then plans some language that represents the knowledge, and acts to achieve the goal. The listener (or reader) perceives the language, and infers the intended meaning. This type of communication via language has allowed civilization to grow; it is our main means of passing along cultural, legal, scientific, and technological knowledge. There are three primary reasons for computers to do natural language processing (NLP):

To communicate with humans. In many situations it is convenient for humans to use speech to interact with computers, and in most situations it is more convenient to use natural language rather than a formal language such as first-order predicate calculus. To learn. Humans have written down a lot of knowledge using natural language. Wikipedia alone has 30 million pages of facts such as “Bush babies are small nocturnal primates,” whereas there are hardly any sources of facts like this written in formal logic. If we want our system to know a lot, it had better understand natural language. To advance the scientific understanding of languages and language use, using the tools of AI in conjunction with linguistics, cognitive psychology, and neuroscience.

Natural language processing (NLP)

In this chapter we examine various mathematical models for language, and discuss the tasks that can be achieved using them.

23.1 Language Models Formal languages, such as first-order logic, are precisely defined, as we saw in Chapter 8 . A grammar defines the syntax of legal sentences and semantic rules define the meaning.

Natural languages, such as English or Chinese, cannot be so neatly characterized:

Language judgments vary from person to person and time to time. Everyone agrees that “Not to be invited is sad” is a grammatical sentence of English, but people disagree on the grammaticality of “To be not invited is sad.” Natural language is ambiguous (“He saw her duck” can mean either that she owns a waterfowl, or that she made a downwards evasive move) and vague (“That’s great!” does not specify precisely how great it is, nor what it is). The mapping from symbols to objects is not formally defined. In first-order logic, two uses of the symbol “Richard” must refer to the same person, but in natural language two occurrences of the same word or phrase may refer to different things in the world.

If we can’t make a definitive Boolean distinction between grammatical and ungrammatical strings, we can at least say how likely or unlikely each one is.

We define a language model as a probability distribution describing the likelihood of any string. Such a model should say that “Do I dare disturb the universe?” has a reasonable probability as a string of English, but “Universe dare the I disturb do?” is extremely unlikely.

Language model

With a language model, we can predict what words are likely to come next in a text, and thereby suggest completions for an email or text message. We can compute which alterations to a text would make it more probable, and thereby suggest spelling or grammar corrections. With a pair of models, we can compute the most probable translation of a

sentence. With some example question/answer pairs as training data, we can compute the most likely answer to a question. So language models are at the heart of a broad range of natural language tasks. The language modeling task itself also serves as a common benchmark to measure progress in language understanding.

Natural languages are complex, so any language model will be, at best, an approximation. The linguist Edward Sapir said “No language is tyrannically consistent. All grammars leak” (Sapir, 1921). The philosopher Donald Davidson said “there is no such thing as language, not if a language is ... a clearly defined shared structure” (Davidson, 1986), by which he meant there is no one definitive language model for English in the way that there is for Python 3.8; we all have different models, but we still somehow manage to muddle through and communicate. In this section we cover simplistic language models that are clearly wrong, but still manage to be useful for certain tasks.

23.1.1 The bag-of-words model Section 12.6.1  explained how a naive Bayes model based on the presence of specific words could reliably classify sentences into categories; for example sentence (1) below is categorized as

business, and (2) as weather.

1. Stocks rallied on Monday, with major indexes gaining 1% as optimism persisted over the first quarter earnings season. 2. Heavy rain continued to pound much of the east coast on Monday, with flood warnings issued in New York City and other locations.

This section revisits the naive Bayes model, casting it as a full language model. That means we don’t just want to know what category is most likely for each sentence; we want a joint probability distribution over all sentences and categories. That suggests we should consider

w , w , … wN (which we will write as w N , as in Chapter 14 ), the naive Bayes formula (Equation (12.21) ) gives all the words in the sentence. Given a sentence consisting of the words 1:

us

P(Class|w

1:

N ) = α P(Class)

∏ P(wj∣∣∣Class). j

1

2

Bag-of-words model

The application of naive Bayes to strings of words is called the bag-of-words model. It is a generative model that describes a process for generating a sentence: Imagine that for each category (business, weather, etc.) we have a bag full of words (you can imagine each word written on a slip of paper inside the bag; the more common the word, the more slips it is duplicated on). To generate text, first select one of the bags and discard the others. Reach into that bag and pull out a word at random; this will be the first word of the sentence. Then put the word back and draw a second word. Repeat until an end-of-sentence indicator (e.g., a period) is drawn.

This model is clearly wrong: it falsely assumes that each word is independent of the others, and therefore it does not generate coherent English sentences. But it does allow us to do classification with good accuracy using the naive Bayes formula: the words “stocks” and “earnings” are clear evidence for the business section, while “rain” and “cloudy” suggest the weather section.

We can learn the prior probabilities needed for this model via supervised training on a body or corpus of text, where each segment of text is labeled with a class. A corpus typically consists of at least a million words of text, and at least tens of thousands of distinct vocabulary words. Recently we are seeing even larger corpuses being used, such as the 2.5 billion words in Wikipedia or the 14 billion word iWeb corpus scraped from 22 million web pages.

Corpus

From a corpus we can estimate the prior probability of each category,

P(Class), by counting

how common each category is. We can also use counts to estimate the conditional probability of each word given the category, texts and 300 of them were classified as

P(wj ∣Class). For example, if we’ve seen 3000

business, then we can estimate

P (Class = business) ≈ 300/3000 = 0.1. And if within the business category we have seen 100,000 words and the word “stocks” appeared 700 times, then we can estimate

P (stocks|Class = business) ≈ 700/100, 000 = 0.007. Estimation by counting works well when we have high counts (and low variance), but we will see in Section 23.1.4  a better way to estimate probabilities when the counts are low.

Sometimes a different machine learning approach, such as logistic regression, neural networks, or support vector machines, can work even better than naive Bayes. The features of the machine learning model are the words in the vocabulary: “a,” “aardvark,” …, “zyzzyva,” and the values are the number of times each word appears in the text (or sometimes just a Boolean value indicating whether the word appears or not). That makes the feature vector large and sparse—we might have 100,000 words in the language model, and thus a feature vector of length 100,000, but for a short text almost all the features will be zero.

As we have seen, some machine learning models work better when we do feature selection, limiting ourselves to a subset of the words as features. We could drop words that are very rare (and thus have high variance in their predictive powers), as well as words that are common to all classes (such as “the”) but don’t discriminate between classes. We can also mix other features in with our word-based features; for example if we are classifying email messages we could add features for the sender, the time the message was sent, the words in the subject header, the presence of nonstandard punctuation, the percentage of uppercase letters, whether there is an attachment, and so on.

Note it is not trivial to decide what a word is. Is “aren’t” one word, or should it be broken up as “aren/’/t” or “are/n’t,” or something else? The process of dividing a text into a sequence of words is called tokenization.

Tokenization

23.1.2 N-gram word models

The bag-of-words model has limitations. For example, the word “quarter” is common in

business and sports categories. But the four-word sequence “first quarter earnings report” is common only in business and “fourth quarter touchdown passes” is common only in sports. We’d like our model to make that distinction. We could tweak the bag-of-words both the

model by treating special phrases like “first-quarter earnings report” as if they were single words, but a more principled approach is to introduce a new model, where each word is dependent on previous words. We can start by making a word dependent on all previous words in a sentence:

P (w N ) = ∏ P (wj ∣w N

1:

j=1

j ).

1: −1

This model is in a sense perfectly “correct” in that it captures all possible interactions between words, but it is not practical: with a vocabulary of 100,000 words and a sentence length of 40, this model would have 10

200

parameters to estimate. We can compromise with

a Markov chain model that considers only the dependence between

n adjacent words. This

is known as an n-gram model (from the Greek root gramma meaning “written thing”): a

n is called an n-gram, with special cases “unigram” for 1-gram, “bigram” for 2-gram, and “trigram” for 3-gram. In an n-gram model, the probability of each word is dependent only on the n − 1 previous words; that is: sequence of written symbols of length

P (w j ∣ w

j )

1: −1

P (w

1:

N)

= =

P (wj ∣wj

n j )

− +1: −1

∏jN P wj wj n (

∣

j ).

− +1: −1

=1

N-gram model

N -gram models work well for classifying newspaper sections, as well as for other classification tasks such as spam detection (distinguishing spam email from non-spam), sentiment analysis (classifying a movie or product review as positive or negative) and author attribution (Hemingway has a different style and vocabulary than Faulkner or Shakespeare).

Spam detection

Sentiment analysis

Author attribution

23.1.3 Other n-gram models An alternative to an n-gram word model is a character-level model in which the probability of each character is determined by the n − 1 previous characters. This approach is helpful for dealing with unknown words, and for languages that tend to run words together, as in the Danish word “Speciallægepraksisplanlægningsstabiliseringsperiode.”

Character-level model

Character-level models are well suited for the task of language identification: given a text, determine what language it is written in. Even with very short texts such as “Hello, world” or “Wie geht’s dir,” n-gram letter models can identify the first as English and the second as German, generally achieving accuracy greater than 99%. (Closely related languages such as Swedish and Norwegian are more difficult to distinguish and require longer samples; there, accuracy is in the 95% range.) Character models are good at certain classification tasks, such as deciding that “dextroamphetamine” is a drug name, “Kallenberger” is a person name, and “Plattsburg” is a city name, even if we have never seen these words before.

Language identification

Skip-gram

Another possibility is the skip-gram model, in which we count words that are near each other, but skip a word (or more) between them. For example, given the French text “je ne comprends pas” the 1-skip-bigrams are “je comprends,” and “ne pas.” Gathering these helps create a better model of French, because it tells us about conjugation (“je” goes with “comprends,” not “comprend”) and negation (“ne” goes with “pas”); we wouldn’t get that from regular bigrams alone.

23.1.4 Smoothing n-gram models

n

High-frequency -grams like “of the” have high counts in the training corpus, so their probability estimate is likely to be accurate: with a different training corpus we would get a

n

similar estimate. Low-frequency -grams have low counts that are subject to random noise —they have high variance. Our models will perform better if we can smooth out that variance.

Furthermore, there is always a chance that we will be asked to evaluate a text containing an unknown or out-of-vocabulary word: one that never appeared in the training corpus. But it would be a mistake to assign such a word a probability of zero, because then the probability of the whole sentence,

P (w

1:

N ), would be zero.

Out-of-vocabulary

One way to model unknown words is to modify the training corpus by replacing infrequent words with a special symbol, traditionally . We could decide in advance to keep only, say, the 50,000 most common words, or all words with frequency greater than 0.0001%, and

n

replace the others with . Then compute -gram counts for the corpus as usual, treating just like any other word. When an unknown word appears in a test set, we look up its probability under . Sometimes different unknown-word symbols are used for different types. For example, a string of digits might be replaced with , or an email

address with . (We note that it is also advisable to have a special symbol, such as

, to mark the start (and stop) of a text. That way, when the formula for bigram probabilities asks for the word before the first word, the answer is , not an error.)

n

Even after we’ve handled unknown words, we have the problem of unseen -grams. For example, a test text might contain the phrase “colorless aquamarine ideas,” three words that we may have seen individually in the training corpus, but never in that exact order. The

n

problem is that some low-probability -grams appear in the training corpus, while other

n

equally low-probability -grams happen to not appear at all. We don’t want some of them to have a zero probability while others have a small positive probability; we want to apply

n

smoothing to all the similar -grams—reserving some of the probability mass of the model

n

for never-seen -grams, to reduce the variance of the model.

Smoothing

The simplest type of smoothing was suggested by Pierre-Simon Laplace in the 18th century to estimate the probability of rare events, such as the sun failing to rise tomorrow. Laplace’s (incorrect) theory of the solar system suggested it was about

N = 2 million days old. Going

by the data, there were zero out of two million days when the sun failed to rise, yet we don’t want to say that the probability is exactly zero. Laplace showed that if we adopt a uniform prior, and combine that with the evidence so far, we get a best estimate of 1/(

N + 2) for the

probability of the sun’s failure to rise tomorrow—either it will or it won’t (that’s the 2 in the denominator) and a uniform prior says it is as likely as not (that’s the 1 in the numerator). Laplace smoothing (also called add-one smoothing) is a step in the right direction, but for many natural language applications it performs poorly.

Backoff model

Linear interpolation smoothing

n

Another choice is a backoff model, in which we start by estimating -gram counts, but for

n

any particular sequence that has a low (or zero) count, we back off to ( − 1)-grams. Linear interpolation smoothing is a backoff model that combines trigram, bigram, and unigram models by linear interpolation. It defines the probability estimate as

Pˆ(ci |ci

λi can be fixed, or they can be trained with an expectation–maximization algorithm. It is also possible to have the values of λi depend on where

λ

3

+

λ

2

+

λ

i ) = λ3 P (ci |ci−2:i−1 ) + λ2 P (ci |ci−1 ) + λ1 P (ci ),

−2: −1

1

= 1. The parameter values

the counts: if we have a high count of trigrams, then we weigh them relatively more; if only a low count, then we put more weight on the bigram and unigram models.

One camp of researchers has developed ever more sophisticated smoothing techniques (such as Witten-Bell and Kneser-Ney), while another camp suggests gathering a larger corpus so that even simple smoothing techniques work well (one such approach is called “stupid backoff”). Both are getting at the same goal: reducing the variance in the language model.

23.1.5 Word representations

N -grams can give us a model that accurately predicts the probability of word sequences, telling us that, for example, “a black cat” is a more likely English phrase than “cat black a” because “a black cat” appears in about 0.000014% of the trigrams in a training corpus, while

n

“cat black a” does not appear at all. Everything that the -gram word model knows, it learned from counts of specific word sequences.

But a native speaker of English would tell a different story: “a black cat” is valid because it follows a familiar pattern (article-adjective-noun), while “cat black a” does not.

Now consider the phrase “the fulvous kitten.” An English speaker could recognize this as also following the article-adjective-noun pattern (even a speaker who does not know that “fulvous” means “brownish yellow” could recognize that almost all words that end in “-ous” are adjectives). Furthermore, the speaker would recognize the close syntactic connection

between “a” and “the,” as well as the close semantic relation between “cat” and “kitten.” Thus, the appearance of “a black cat” in the data is evidence, through generalization, that “the fulvous kitten” is also valid English. The n-gram model misses this generalization because it is an atomic model: each word is an atom, distinct from every other word, with no internal structure. We have seen throughout this book that factored or structured models allow for more expressive power and better generalization. We will see in Section 24.1  that a factored model called word embeddings gives a better ability to generalize.

One type of structured word model is a dictionary, usually constructed through manual labor. For example, WordNet is an open-source, hand-curated dictionary in machinereadable format that has proven useful for many natural language applications1 Below is the WordNet entry for “kitten:”

1 And even computer vision applications: WordNet provides the set of categories used by ImageNet.

"kitten" ("young domestic cat") IS A: young_mammal "kitten" ("give birth to kittens") EXAMPLE: "our cat kittened again this year"

Dictionary

WordNet

WordNet will help you separate the nouns from the verbs, and get the basic categories (a kitten is a young mammal, which is a mammal, which is an animal), but it won’t tell you the

details of what a kitten looks like or acts like. WordNet will tell you that two subclasses of cat are Siamese cat and Manx cat, but won’t tell you any more about the breeds.

23.1.6 Part-of-speech (POS) tagging One basic way to categorize words is by their part of speech (POS), also called lexical category or tag: noun, verb, adjective, and so on. Parts of speech allow language models to capture generalizations such as “adjectives generally come before nouns in English.” (In other languages, such as French, it is the other way around (generally)).

Part of speech (POS)

Everyone agrees that “noun” and “verb” are parts of speech, but when we get into the details there is no one definitive list. Figure 23.1  shows the 45 tags used in the Penn Treebank, a corpus of over three million words of text annotated with part-of-speech tags. As we will see later, the Penn Treebank also annotates many sentences with syntactic parse trees, from which the corpus gets its name. Here is an excerpt saying that “from” is tagged as a preposition (IN), “the” as a determiner (DT), and so on:

Figure 23.1

Part-of-speech tags (with an example word for each tag) for the Penn Treebank corpus (Marcus et al., 1993). Here “3rd-sing” is an abbreviation for “third person singular present tense.”

Penn Treebank

The task of assigning a part of speech to each word in a sentence is called part-of-speech tagging. Although not very interesting in its own right, it is a useful first step in many other NLP tasks, such as question answering or translation. Even for a simple task like text-tospeech synthesis, it is important to know that the noun “record” is pronounced differently from the verb “record.” In this section we will see how two familiar models can be applied to the tagging task, and in Chapter 24  we will consider a third model.

Part-of-speech tagging

One common model for POS tagging is the hidden Markov model (HMM). Recall from Section 14.3  that a hidden Markov model takes in a temporal sequence of evidence observations and predicts the most likely hidden states that could have produced that sequence. In the HMM example on page 473, the evidence consisted of observations of a person carrying an umbrella (or not), and the hidden state was rain (or not) in the outside world. For POS tagging, the evidence is the sequence of words, are the lexical categories,

C

1:

W

1:

N , and the hidden states

N.

The HMM is a generative model that says that the way to produce language is to start in one state, such as IN, the state for prepositions, and then make two choices: what word (such as from) should be emitted, and what state (such as DT) should come next. The model does not consider any context other than the current part-of-speech state, nor does it have any idea of what the sentence is actually trying to convey. And yet it is a useful model—if we apply the Viterbi algorithm (Section 14.2.3 ) to find the most probable sequence of hidden states (tags), we find that the tagging achieves very high accuracy; usually around 97%.

Viterbi algorithm

To create a HMM for POS tagging, we need the transition model, which gives the probability of one part of speech following another,

P(Wt |Ct ). For example, P (Ct = V B|Ct

−1

=

P ( C t |C t

−1 ),

and the sensor model,

MD) = 0.8 means that given a modal verb (such

as would), we can expect the following word to be a verb (such as think) with probability

n

0.8. Where does the 0.8 number come from? Just as with -gram models, from counts in the corpus, with appropriate smoothing. It turns out that there are 13124 instances of Penn Treebank, and 10471 of them are followed by a

For the sensor model,

V B.

MD in the

P (Wt = would|Ct = MD) = 0.1 means that when we are choosing a

modal verb, we will choose would 10% of the time. These numbers also come from corpus

counts, with smoothing.

A weakness of HMM models is that everything we know about language has to be expressed in terms of the transition and sensor models. The part of speech for the current word is determined solely by the probabilities in these two models and by the part of speech of the previous word. There is no easy way for a system developer to say, for example, that any word that ends in “ous” is likely an adjective, nor that in the phrase “attorney general,” attorney is a noun, not an adjective.

Fortunately, logistic regression does have the ability to represent information like this. Recall from Section 19.6.5  that in a logistic regression model, the input is a vector, feature values. We then take the dot product, vector of weights

w

,

w x ⋅

,

x

,

of

of those features with a pretrained

and transform that sum into a number between 0 and 1 that can be

interpreted as the probability that the input is a positive example of a category.

The weights in the logistic regression model correspond to how predictive each feature is for each category; the weight values are learned by gradient descent. For POS tagging we would build 45 different logistic regression models, one for each part of speech, and ask each model how probable it is that the example word is a member of that category, given the feature values for that word in its particular context.

The question then is what should the features be? POS taggers typically use binary-valued features that encode information about the word being tagged, wi (and perhaps other nearby words), as well as the category that was assigned to the previous word, ci

−1

(and

perhaps the category of earlier words). Features can depend on the exact identity of a word, some aspects of the way it is spelled, or some attribute from a dictionary entry. A set of POS tagging features might include: wi−1

= “I”

wi−1

= “you”

 

wi+1 ci−1

= “for” = IN

wi  ends with “ous”

wi  contains a hyphen

wi  ends with “ly”

wi  contains a digit

wi  starts with “un”

wi  is all uppercase

wi−2

=  “to” and 

wi−1

=  “I” and 

ci−1

wi+1

= VB = “to”

wi−2  has attribute PRESENT wi−2  has attribute PAST

For example, the word “walk” can be a noun or a verb, but in “I walk to school,” the feature in the last row, left column could be used to classify “walk” as a verb (VBP). As another example, the word “cut” can be either a noun (NN), past tense verb (VBD), or present tense verb (VBP). Given the sentence “Yesterday I cut the rope,” the feature in the last row, right column could help tag “cut” as VBD, while in the sentence “Now I cut the rope,” the feature above that one could help tag “cut” as VBP.

All together, there might be a million features, but for any given word, only a few dozen will be nonzero. The features are usually hand-crafted by a human system designer who thinks up interesting feature templates.

Logistic regression does not have the notion of a sequence of inputs—you give it a single feature vector (information about a single word) and it produces an output (a tag). But we can force logistic regression to handle a sequence with a greedy search: start by choosing the most likely category for the first word, and proceed to the rest of the words in left-to-

c

right order. At each step the category i is assigned according to

ci =

argmax

c ∈Categories ′

P (c ∣w ′

1:

N,

c

i ).

1: −1

That is, the classifier is allowed to look at any of the non-category features for any of the words anywhere in the sentence (because these features are all fixed), as well as any previously assigned categories.

Note that the greedy search makes a definitive category choice for each word, and then moves on to the next word; if that choice is contradicted by evidence later in the sentence, there is no possibility to go back and reverse the choice. That makes the algorithm fast. The Viterbi algorithm, in contrast, keeps a table of all possible category choices at each step, and always has the option of changing. That makes the algorithm more accurate, but slower. For both algorithms, a compromise is a beam search, in which we consider every possible category at each time step, but then only keep the less-likely tags. Changing

b most likely tags, dropping the other

b trades off speed versus accuracy.

Naive Bayes and Hidden Markov models are generative models (see Section 20.2.3 ). That is, they learn a joint probability distribution,

P(W , C), and we can generate a random

sentence by sampling from that probability distribution to get a first word (with category) of the sentence, and then adding words one at a time.

Logistic regression on the other hand is a discriminative model. It learns a conditional probability distribution

P(C|W ), meaning that it can assign categories given a sequence of

words, but it can’t generate random sentences. Generally, researchers have found that discriminative models have a lower error rate, perhaps because they model the intended output directly, and perhaps because they make it easier for an analyst to create additional features. However, generative models tend to converge more quickly, and so may be preferred when the available training time is short, or when there is limited training data.

23.1.7 Comparing language models

n

To get a feeling for what different -gram models are like, we built unigram (i.e., bag-ofwords), bigram, trigram, and 4-gram models over the words in this book and then randomly sampled word sequences from each of the four models:

n = 1: logical are as are confusion a may right tries agent goal the was n = 2: systems are very similar computational approach would be represented n = 3: planning and scheduling are integrated the success of naive Bayes model is n = 4: taking advantage of the structure of Bayesian networks and developed various languages for writing “templates” with logical variables, from which large networks could be constructed automatically for each problem instance

From this small sample it should be clear that the unigram model is a very poor approximation of either English in general or an AI textbook in particular, and that the 4gram model is imperfect but much better. Next, to demonstrate how samples switch between training sources (and definitely not just for fun) we added the text of the King James Bible to the 4-gram model yielding these random samples:

Prove that any 3-SAT problem can be reduced to simpler ones using the laws of thy God. Masters, give unto your servants that which is true iff both

P and Q in any model m by a simple

experiment: put your hand unto, ye and your households for it is pleasant. Many will intreat the LORD your God, Saying, No; but we will ignore this issue for now; Chapters 7  and 8  suggest methods for compactly representing very large belief states. And it came to pass, as if it had no successors.

The direct utility estimation is just an instance of the general or algorithm in which new function symbols are constructed “on the fly.” For example, the first child of the Holy Ghost. There is a limit to n-gram models—as n increases, they will produce language that is more fluent, but they tend to reproduce long passages from their training data verbatim, rather than generate novel text. Language models with more complex representations of words and context can do better. The rest of this chapter demonstrates how grammar can improve a language model, and Chapter 24  shows how deep learning methods have recently produced impressive language models. One such deep learning model, GPT-2, can produce fluent English samples when given a prompt. We gave GPT-2 the first two sentences of this paragraph as a prompt; it produced the following two samples:

Numerous studies have shown that the best language models that can generate language are those that use neural networks, and in particular that use deep learning to model speech. I have used one of these models to generate hundreds of sentences in multiple languages, and it has the ability to generate almost perfect translations of my training sentences. It’s a great model to have, and will save you a lot of time in the future when you need to translate between many languages. These models can also do better if their samples are chosen at random, rather than by an expert in the domain. However, as a general rule, n-gram and language models that include a large number of features tend to produce text better, since these features contribute the most to language learning. To measure the performance of a model, I’ve made some sample sentences that I’d like to train to better understand speech and language models. You can read them to get a grasp of how the model predicts.

We see that these passages are diverse and grammatically fluent; moreover, they stick to topics that are relevant to the prompt sentences. But the sentences do not build on each other to advance a coherent thesis. The GPT-2 language model is known as a transformer model, which will be covered in Section 24.4 ; further examples from GPT-2 are in Figure 24.14 . Another transformer model is the Conditional Transformer Language, CTRL. It can be controlled more flexibly; in the following samples CTRL was asked to generate text in the category product reviews, with a rating of 1 and of 4 (out of 5): specified rating (out of 5):

1.0: I bought this for my son who is a huge fan of the show. He was so excited to get it and when he opened it, we were all very disappointed. The quality of the product is terrible. It looks like

something you would buy at a dollar store. 4.0: I bought this for my husband and he loves it. He has a small wrist so it is hard to find watches that fit him well. This one fits perfectly.

23.2 Grammar In Chapter 7  we used Backus–Naur Form (BNF) to write down a grammar for the language of first-order logic. A grammar is a set of rules that defines the tree structure of allowable phrases, and a language is the set of sentences that follow those rules.

Natural languages do not work exactly like the formal language of first-order logic—they do not have a hard boundary between allowable and unallowable sentences, nor do they have a single definitive tree structure for each sentence. However, hierarchical structure is important in natural language. The word “Stocks” in “Stocks rallied on Monday” is not just a word, nor is it just a noun; in this sentence it also comprises a noun phrase, which is the subject of the following verb phrase. Syntactic categories such as noun phrase or verb phrase help to constrain the probable words at each point within a sentence, and the phrase structure provides a framework for the meaning or semantics of the sentence.

Syntactic category

Phrase structure

There are many competing language models based on the idea of hierarchical syntactic structure; in this section we will describe a popular model called the probabilistic contextfree grammar, or PCFG. A probabilistic grammar assigns a probability to each string, and “context-free” means that any rule can be used in any context: the rules for a noun phrase at the beginning of a sentence are the same as for another noun phrase later in the sentence, and if the same phrase occurs in two locations, it must have the same probability each time. We will define a PCFG grammar for a tiny fragment of English that is suitable for communication between agents exploring the wumpus world. We call this language Figure 23.2 ). A grammar rule such as

E0 (see

Adjs

→ |

Adjective Adjective Adjs

[0.80] [0.20]

Figure 23.2

E

S

The grammar for 0 , with example phrases for each rule. The syntactic categories are sentence ( ), noun phrase ( ), verb phrase ( ), list of adjectives ( ), prepositional phrase ( ), and relative clause

NP (RelClause).

VP

Probabilistic context-free grammar

Adjs

PP

Adjs can consist of either a single Adjective with probability 0.80, or of an Adjective followed by a string that constitutes an Adjs, with means that the syntactic category

,

probability 0.20.

Unfortunately, the grammar overgenerates: that is, it generates sentences that are not grammatical, such as “Me go I.” It also undergenerates: there are many sentences of English that it rejects, such as “I think the wumpus is smelly.” We will see how to learn a better grammar later; for now we concentrate on what we can do with this very simple grammar.

Overgeneration

Undergeneration

23.2.1 The lexicon of

E

0

The lexicon, or list of allowable words, is defined in Figure 23.3 . Each of the lexical categories ends in … to indicate that there are other words in the category. For nouns, names, verbs, adjectives, and adverbs, it is infeasible even in principle to list all the words. Not only are there tens of thousands of members in each class, but new ones—like humblebrag or microbiome—are being added constantly. These five categories are called open classes. Pronouns, relative pronouns, articles, prepositions, and conjunctions are called closed classes; they have a small number of words (a dozen or so), and change over the course of centuries, not months. For example, “thee” and “thou” were commonly used pronouns in the 17th century, were on the decline in the 19th century, and are seen today only in poetry and some regional dialects.

Figure 23.3

E RelPro

The lexicon for 0 . is short for relative pronoun, The sum of the probabilities for each category is 1.

Lexicon

Open class

Closed class

Prep for preposition, and Conj for conjunction.

23.3 Parsing Parsing is the process of analyzing a string of words to uncover its phrase structure, according to the rules of a grammar. We can think of it as a search for a valid parse tree whose leaves are the words of the string. Figure 23.4  shows that we can start with the

S

symbol and search top down, or we can start with the words and search bottom up. Pure top-down or bottom-up parsing strategies can be inefficient, however, because they can end up repeating effort in areas of the search space that lead to dead ends. Consider the following two sentences:

Have the students in section 2 of Computer Science 101 take the exam.

Have the students in section 2 of Computer Science 101 taken the exam?

Figure 23.4

Parsing the string “The wumpus is dead” as a sentence, according to the grammar

S X

X

E . Viewed as a top0

down parse, we start with , and on each step match one nonterminal with a rule of the form ( → …) and replace in the list of items with … ; for example replacing with the sequence

X Y Y S NP  V P . Viewed as a bottom-up parse, we start with the words “the wumpus is dead”, and on each step match a string of tokens such as (Y … ) against a rule of the form (X → Y …) and replace the tokens with X; for example replacing “the” with Article or Article Noun with NP .

Parsing

Even though they share the first 10 words, these sentences have very different parses, because the first is a command and the second is a question. A left-to-right parsing algorithm would have to guess whether the first word is part of a command or a question and will not be able to tell if the guess is correct until at least the eleventh word, take or taken. If the algorithm guesses wrong, it will have to backtrack all the way to the first word and reanalyze the whole sentence under the other interpretation.

To avoid this source of inefficiency we can use dynamic programming: every time we analyze a substring, store the results so we won’t have to reanalyze it later. For example, once we discover that “the students in section 2 of Computer Science 101” is an

NP , we can

record that result in a data structure known as a chart. An algorithm that does this is called a chart parser. Because we are dealing with context-free grammars, any phrase that was found in the context of one branch of the search tree can work just as well in any other branch of the search tree. There are many types of chart parsers; we describe a probabilistic version of a bottom-up chart parsing algorithm called the CYK algorithm, after its inventors, Ali Cocke, Daniel Younger, and Tadeo Kasami.2

2 Sometimes the authors are credited in the order CKY.

Chart parser

CYK algorithm

The CYK algorithm is shown in Figure 23.5 . It requires a grammar with all rules in one of two very specific formats: lexical rules of the form form

X

→

YZ p  

 [ ],

X

→

word [p] , and syntactic rules of the

with exactly two categories on the right-hand side. This grammar

format, called Chomsky Normal Form, may seem restrictive, but it is not: any context-free grammar can be automatically transformed into Chomsky Normal Form. Exercise 23.CNFX leads you through the process.

Figure 23.5

The CYK algorithm for parsing. Given a sequence of words, it finds the most probable parse tree for the sequence and its subsequences. The table P [X, i, k] gives the probability of the most probable tree of

category X spanning wordsi:k . The output table T [X,  i, k] contains the most probable tree of category X spanning positions i to k inclusive. The function SUBSPANS returns all tuples (i,  j,  k) covering a span of wordsi:k , with i ≤ j < k, listing the tuples by increasing length of the i : k span, so that when we go to combine two shorter spans into a longer one, the shorter spans are already in the table. LEXICALRULES(word) returns a collection of ( X, p) pairs, one for each rule of the form X → GRAMMARRULES gives (X,  Y ,  Z ,  p) tuples, one for each grammar rule of the form X →

Chomsky Normal Form

word [p] , and Y   Z   [p] .

The CYK algorithm uses space of words in the sentence, and takes time

O(n m) for the P and T tables, where n is the number of 2

m is the number of nonterminal symbols in the grammar, and

O(n m). If we want an algorithm that is guaranteed to work for all possible 3

context-free grammars, then we can’t do any better than that. But actually we only want to parse natural languages, not all possible grammars. Natural languages have evolved to be easy to understand in real time, not to be as tricky as possible, so it seems that they should be amenable to a faster parsing algorithm.

To try to get to

O(n), we can apply A

∗

search in a fairly straightforward way: each state is a

list of items (words or categories), as shown in Figure 23.4 . The start state is a list of words, and a goal state is the single item S. The cost of a state is the inverse of its probability as defined by the rules applied so far, and there are various heuristics to estimate the remaining distance to the goal; the best heuristics in current use come from machine learning applied to a corpus of sentences.

With the

A

∗

algorithm we don’t have to search the entire state space, and we are guaranteed

that the first parse found will be the most probable (assuming an admissible heuristic). This will usually be faster than CYK, but (depending on the details of the grammar) still slower than

O(n). An example result of a parse is shown in Figure 23.6 .

Figure 23.6

E . Each interior node of

Parse tree for the sentence “Every wumpus smells” according to the grammar the tree is labeled with its probability. The probability of the tree as a whole is

0

0.9 × 0.25 × 0.05 × 0.15 × 0.40 × 0.10 = 0.0000675. The tree can also be written in linear form as

S NP [Article every][Noun wumpus]][V P [V erb smells]]].

[ [

Just as with part-of-speech tagging, we can use a beam search for parsing, where at any time we consider only the

b most probable alternative parses. This means we are not

guaranteed to find the parse with highest probability, but (with a careful implementation) the parser can operate in

O(n) time and still finds the best parse most of the time.

A beam search parser with

b = 1 is called a deterministic parser. One popular deterministic

approach is shift-reduce parsing, in which we go through the sentence word by word, choosing at each point whether to shift the word onto a stack of constituents, or to reduce the top constituent(s) on the stack according to a grammar rule. Each style of parsing has its adherents within the NLP community. Even though it is possible to transform a shift-reduce system into a PCFG (and vice versa), when you apply machine learning to the problem of inducing a grammar, the inductive bias and hence the generalizations that each system will make will be different (Abney et al., 1999).

Deterministic parser

Shift-reduce parsing

23.3.1 Dependency parsing There is a widely used alternative syntactic approach called dependency grammar, which assumes that syntactic structure is formed by binary relations between lexical items, without a need for syntactic constituents. Figure 23.7  shows a sentence with a dependency parse and a phrase structure parse.

Figure 23.7

A dependency-style parse (top) and the corresponding phrase structure parse (bottom) for the sentence I detect the smelly wumpus near me.

Dependency grammar

In one sense, dependency grammar and phrase structure grammar are just notational variants. If the phrase structure tree is annotated with the head of each phrase, you can recover the dependency tree from it. In the other direction, we can convert a dependency tree into a phrase structure tree by introducing arbitrary categories (although we might not always get a natural-looking tree this way).

Therefore we wouldn’t prefer one notation over the other because one is more powerful; rather we would prefer one because it is more natural—either more familiar for the human developers of a system, or more natural for a machine learning system which will have to learn the structures. In general, phrase structure trees are natural for languages (like English) with mostly fixed word order; dependency trees are natural for languages (such as Latin) with mostly free word order, where the order of words is determined more by pragmatics than by syntactic categories.

The popularity of dependency grammar today stems in large part from the Universal Dependencies project (Nivre et al., 2016), an open-source treebank project that defines a set of relations and provides millions of parsed sentences in over 70 languages.

23.3.2 Learning a parser from examples Building a grammar for a significant portion of English is laborious and error prone. This suggests that it would be better to learn the grammar rules (and probabilities) rather than writing them down by hand. To apply supervised learning, we need input/output pairs of sentences and their parse trees. The Penn Treebank is the best known source of such data, with over 100 thousand sentences annotated with parse-tree structure. Figure 23.8  shows an annotated tree from the Penn Treebank.

Figure 23.8

Annotated tree for the sentence “Her eyes were glazed as if she didn’t hear or even see him.” from the Penn Treebank. Note a grammatical phenomenon we have not covered yet: the movement of a phrase from one part of the tree to another. This tree analyzes the phrase “hear or even see him” as consisting of

VP VP

NP

V P  [ADV P  even] see [NP *-1]], both of which have a NP labeled elsewhere in the tree as [NP -1 him].

two constituent s, [  hear [  *-1]] and [ missing object, denoted *-1, which refers to the

NP *-2] refers to the [NP -2 Her eyes].

Similarly, the [

Given a treebank, we can create a PCFG just by counting the number of times each nodetype appears in a tree (with the usual caveats about smoothing low counts). In Figure 23.8 ,

S NP …][V P …]]. We would count these, and all the other subtrees with root S in the corpus. If there are 1000 S nodes of which 600 are of this form, there are two nodes of the form [ [

then we create the rule:

S

→

NP V P [0.6] .

All together, the Penn Treebank has over 10,000 different node types. This reflects the fact that English is a complex language, but it also indicates that the annotators who created the treebank favored flat trees, perhaps flatter than we would like. For example, the phrase “the good and the bad” is parsed as a single noun phrase rather than as two conjoined noun phrases, giving us the rule:

NP

→

Article Noun Conjunction Article Noun .

There are hundreds of similar rules that define a noun phrase as a string of categories with a conjunction somewhere in the middle; a more concise grammar could capture all the conjoined noun phrase rules with the single rule

NP

→

NP Conjunction NP .

Bod et al. (2003) and Bod (2008) show how to automatically recover generalized rules like this, greatly reducing the number of rules that come out of the treebank, and creating a grammar that ends up generalizing better for previously unseen sentences. They call their approach data-oriented parsing.

We have seen that treebanks are not perfect—they contain errors, and have idiosyncratic parses. It is also clear that creating a treebank requires a lot of hard work; that means that treebanks will remain relatively small in size, compared to all the text that has not been annotated with trees. An alternative approach is unsupervised parsing, in which we learn a new grammar (or improve an existing grammar) using a corpus of sentences without trees.

Unsupervised parsing

The inside–outside algorithm (Dodd, 1988), which we will not cover here, learns to estimate the probabilities in a PCFG from example sentences without trees, similar to the way the forward-backward algorithm (Figure 14.4 ) estimates probabilities. Spitkovsky et al. (2010a) describe an unsupervised learning approach that uses curriculum learning: start with the easy part of the curriculum—short unambiguous 2-word sentences like “He left” can be easily parsed based on prior knowledge or annotations. Each new parse of a short sentence extends the system’s knowledge so that it can eventually tackle 3-word, then 4word, and eventually 40-word sentences.

Curriculum learning

We can also use semisupervised parsing, in which we start with a small number of trees as data to build an initial grammar, then add a large number of unparsed sentences to improve the grammar. The semisupervised approach can make use of partial bracketing: we can use widely available text that has been marked up by the authors, not by linguistic experts, with a partial tree-like structure, in the form of HTML or similar annotations. In HTML text most

brackets correspond to a syntactic component, so partial bracketing can help learn a grammar (Pereira and Schabes, 1992; Spitkovsky et al., 2010b). Consider this HTML text from a newspaper article:

In 1998, however, as I established in The New Republic and Bill Clinton just confirmed in his memoirs, Netanyahu changed his mind

Semisupervised parsing

Partial bracketing

The words surrounded by tags form a noun phrase, and the two strings of words surrounded by tags each form verb phrases.

23.4 Augmented Grammars So far we have dealt with context-free grammars. But not every

NP can appear in every

context with equal probability. The sentence “I ate a banana” is fine, but “Me ate a banana” is ungrammatical, and “I ate a bandanna” is unlikely.

The issue is that our grammar is focused on lexical categories, like Pronoun, but while “I” and “me” are both pronouns, only “I” can be the subject of a sentence. Similarly, “banana” and “bandanna” are both nouns, but the former is much more likely to be object of “ate”. Linguists say that the pronoun “I” is in the subjective case (i.e., is the subject of a verb) and “me” is in the objective case3 (i.e., is the object of a verb). They also say that “I” is in the first person (“you” is second person, and “she” is third person) and is singular (“we” is plural). A category like Pronoun that has been augmented with features like “subjective case, first person singular” is called a subcategory.

3 The subjective case is also sometimes called the nominative case and the objective case is sometimes called the accusative case. Many languages also make another distinction with a dative case for words in the indirect object position.

Subcategory

In this section we show how a grammar can represent this kind of knowledge to make finergrained distinctions about which sentences are more likely. We will also show how to construct a representation of the semantics of a phrase, in a compositional way. All of this will be accomplished with an augmented grammar in which the nonterminals are not just

NP , but are structured representations. For example, the noun phrase “I” could be represented as NP (Sbj, 1S , Speaker), which means “a noun phrase atomic symbols like Pronoun or

that is in the subjective case, first person singular, and whose meaning is the speaker of the sentence.” In contrast, “me” would be represented as that it is in the objective case.

NP (Obj, 1S, Speaker), marking the fact

Augmented grammar

Consider the sequence “Noun and Noun or Noun,” which can be parsed either as “[Noun and Noun] or Noun,” or as “Noun and [Noun or Noun].” Our context-free grammar has no way to express a preference for one parse over the other, because the rule for conjoined

NP s, NP → NP  Conjunction NP [0.05], will give the same probability to each parse. We would like a grammar that prefers the parses “[[spaghetti and meatballs] or lasagna]” and “[spaghetti and [pie or cake]]” over the alternative bracketing for each of these phrases.

A lexicalized PCFG is a type of augmented grammar that allows us to assign probabilities based on properties of the words in a phrase other than just the syntactic categories. The data would be very sparse indeed if the probability of, say, a 40-word sentence depended on all 40 words—this is the same problem we noted with n-grams. To simplify, we introduce the notion of the head of a phrase—the most important word. Thus, “banana” is the head of

NP “a banana” and “ate” is the head of the V P “ate a banana.” The notation V P (v) denotes a phrase with category V P whose head word is v. Here is a lexicalized PCFG: the

V P (v) → V erb(v) NP (n) V P (v) → V erb(v) NP (n) → Article(a) Adjs(j) Noun(n) NP (n) → NP (n) Conjunction(c) NP (m) V erb(ate) → ate Noun(banana) → banana

Lexicalized PCFG

Head

P (v, n)] [P (v)] [P (n, a)] [P (n, c, m)]

[

1

2

3

4

[0.002] [0.0007]

P (v, n) means the probability of a V P headed by v joining with an NP headed by n to form a V P . We can specify that “ate a banana” is more probable than “ate a bandanna” by ensuring that P (ate, banana) > P (ate, bandanna). Note that since we are considering only Here

1

1

1

phrase heads, the distinction between “ate a banana” and “ate a rancid banana” will not be caught by

P . Conceptually, P 1

1

is a huge table of probabilities: if there are 5,000 verbs and

10,000 nouns in the vocabulary, then

P

1

requires 50 million entries, but most of them will

not be stored explicitly; rather they will be derived from smoothing and backoff. For example, we can back off from

P (v, n) to a model that depends only on v. Such a model 1

would require 10,000 times fewer parameters, but can still capture important regularities, such as the fact that a transitive verb like “ate” is more likely to be followed by an

NP

(regardless of the head) than an intransitive verb like “sleep.” We saw in Section 23.2  that the simple grammar for

E

0

overgenerates, producing non-

sentences such as “I saw she” or “I sees her.” To avoid this problem, our grammar would have to know that “her,” not “she,” is a valid object of “saw” (or of any other verb) and that “see,” not “sees,” is the form of the verb that accompanies the subject “I.”

We could encode these facts completely in the probability entries, for example making

P (v, she) be a very small number, for all verbs v. But it is more concise and modular to augment the category NP with additional variables: NP (c, pn, n) is used to represent a noun phrase with case c (subjective or objective), person and number pn (e.g., third person singular), and head noun n. Figure 23.9  shows an augmented lexicalized grammar that 1

handles these additional variables. Let’s consider one grammar rule in detail:

S(v) → NP (Sbj, pn, n) V P (pn, v)   [P (n, v)]. 5

Figure 23.9

Part of an augmented grammar that handles case agreement, subject–verb agreement, and head words. Capitalized names are constants: ,, and , for subjective and objective case; 1 for first person

P

P

Sbj

Obj

S

singular; 1 and 3 for first and third person plural. As usual, lowercase names are variables. For simplicity, the probabilities have been omitted.

This rule says that when an NP is followed by a VP they can form an S, but only if the NP

Sbj) case and the person and number (pn) of the NP and V P are identical. (We say that they are in agreement.) If that holds, then we have an S whose head is the verb from the V P . Here is an example lexical rule, has the subjective (

Pronoun (Sbj, 1S, I ) →  I   [0.005] which says that “I” is a Pronoun in the subjective case, first-person singular, with head “I.”

23.4.1 Semantic interpretation To show how to add semantics to a grammar, we start with an example that is simpler than English: the semantics of arithmetic expressions. Figure 23.10  shows a grammar for arithmetic expressions, where each rule is augmented with a single argument indicating the semantic interpretation of the phrase. The semantics of a digit such as “3” is the digit itself. The semantics of the expression “3 + 4” is the operator “+” applied to the semantics of the phrases “3” and “4.” The grammar rules obey the principle of compositional semantics—the semantics of a phrase is a function of the semantics of the subphrases. Figure 23.11  shows the parse tree for 3 + (4 ÷ 2) according to this grammar. The root of the parse tree is an expression whose semantic interpretation is 5.

Figure 23.10

Exp(5),

A grammar for arithmetic expressions, augmented with semantics. Each variable xi represents the semantics of a constituent.

Figure 23.11

Parse tree with semantic interpretations for the string “3 + (4 ÷ 2)”.

Compositional semantics

Now let’s move on to the semantics of English, or at least a tiny portion of it. We will use first-order logic for our semantic representation. So the simple sentence “Ali loves Bo”

Loves(Ali, Bo). But what about the constituent phrases? We can represent the NP “Ali” with the logical term Ali. But the V P “loves Bo” is should get the semantic representation

neither a logical term nor a complete logical sentence. Intuitively, “loves Bo” is a description that might or might not apply to a particular person. (In this case, it applies to Ali.) This means that “loves Bo” is a predicate that, when combined with a term that represents a person, yields a complete logical sentence.

λ

Using the -notation (see page 259), we can represent “loves Bo” as the predicate

λx Loves(x, Bo). Now we need a rule that says “an

NP with semantics n followed by a V P with semantics

pred yields a sentence whose semantics is the result of applying pred to n:”

S(pred(n))

→

NP (n) V P (pred) .

The rule tells us that the semantic interpretation of “Ali loves Bo” is (

which is equivalent to

λx Loves(x, Bo))(Ali),

Loves(Ali, Bo). Technically, we say that this is a β-reduction of the

lambda function application.

The rest of the semantics follows in a straightforward way from the choices we have made

V P s are represented as predicates, verbs should be predicates as well. The verb “loves” is represented as λy λxLoves(x, y), the predicate that, when given the argument Bo, returns the predicate λx Loves(x, Bo). We end up with the grammar and parse tree so far. Because

shown in Figure 23.12 . In a more complete grammar, we would put all the augmentations (semantics, case, person-number, and head) together into one set of rules. Here we show only the semantic augmentation to make it clearer how the rules work.

Figure 23.12

(a) A grammar that can derive a parse tree and semantic interpretation for “Ali loves Bo” (and three other sentences). Each category is augmented with a single argument representing the semantics. (b) A parse tree with semantic interpretations for the string “Ali loves Bo.”

23.4.2 Learning semantic grammars Unfortunately, the Penn Treebank does not include semantic representations of its sentences, just syntactic trees. So if we are going to learn a semantic grammar, we will need a different source of examples. Zettlemoyer and Collins (2005) describe a system that learns a grammar for a question-answering system from examples that consist of a sentence paired with the semantic form for the sentence:

SENTENCE: What states border Texas? LOGICAL FORM:

λx. state(x) ∧ λx. borders(x, T exas)

Given a large collection of pairs like this and a little bit of hand-coded knowledge for each new domain, the system generates plausible lexical entries (for example, that “Texas” and “state” are nouns such that state(T exas) is true), and simultaneously learns parameters for a grammar that allows the system to parse sentences into semantic representations. Zettlemoyer and Collins’s system achieved 79% accuracy on two different test sets of unseen sentences. Zhao and Huang (2015) demonstrate a shift-reduce parser that runs faster, and achieves 85% to 89% accuracy.

A limitation of these systems is that the training data includes logical forms. These are expensive to create, requiring human annotators with specialized expertise—not everyone

understands the subtleties of lambda calculus and predicate logic. It is much easier to gather examples of question/answer pairs:

QUESTION: What states border Texas? ANSWER: Louisiana, Arkansas, Oklahoma, New Mexico.

QUESTION: How many times would Rhode Island fit into California? ANSWER: 135

Such question/answer pairs are quite common on the Web, so a large database can be put together without human experts. Using this large source of data it is possible to build parsers that outperform those that use a small database of annotated logical forms (Liang et al., 2011; Liang and Potts, 2015). The key approach described in these papers is to invent an internal logical form that is compositional but does not allow an exponentially large search space.

23.5 Complications of Real Natural Language The grammar of real English is endlessly complex (and other languages are equally complex). We will briefly mention some of the topics that contribute to this complexity.

QUANTIFICATION: Consider the sentence “Every agent feels a breeze.” The sentence has only one syntactic parse under

E , but it is semantically ambiguous: is there one breeze that 0

is felt by all the agents, or does each agent feel a separate personal breeze? The two interpretations can be represented as

a a ∈ Agents ⇒ ∃b b ∈ Breezes ∧ Feel (a, b) ; ∃b b ∈ Breezes ∧ ∀a a ∈ Agents ⇒ Feel (a, b) . ∀

Quantification

One standard approach to quantification is for the grammar to define not an actual logical semantic sentence, but rather a quasi-logical form that is then turned into a logical sentence by algorithms outside of the parsing process. Those algorithms can have preference rules for choosing one quantifier scope over another—preferences that need not be reflected directly in the grammar.

Quasi-logical form

PRAGMATICS: We have shown how an agent can perceive a string of words and use a grammar to derive a set of possible semantic interpretations. Now we address the problem of completing the interpretation by adding context-dependent information about the current

situation. The most obvious need for pragmatic information is in resolving the meaning of indexicals, which are phrases that refer directly to the current situation. For example, in the sentence “I am in Boston today,” both “I” and “today” are indexicals. The word “I” would be represented by

Speaker, a fluent that refers to different objects at different times, and it

would be up to the hearer to resolve the referent of the fluent—that is not considered part of the grammar but rather an issue of pragmatics.

Pragmatics

Indexical

Another part of pragmatics is interpreting the speaker’s intent. The speaker’s utterance is considered a speech act, and it is up to the hearer to decipher what type of action it is—a question, a statement, a promise, a warning, a command, and so on. A command such as “go to 2 2” implicitly refers to the hearer. So far, our grammar for

S covers only declarative

sentences. We can extend it to cover commands—a command is a verb phrase where the subject is implicitly the hearer of the command:

S(Command(pred(Hearer))) → V P (pred).  

Speech act

LONG-DISTANCE DEPENDENCIES: In Figure 23.8  we saw that “she didn’t hear or even see him” was parsed with two gaps where an

NP is missing, but refers to the NP “him.” We

can use the symbol ⊔ to represent the gaps: “she didn’t [hear ⊔ or even see ⊔] him.” In general, the distance between the gap and the

NP it refers to can be arbitrarily long: in

“Who did the agent tell you to give the gold to ⊔?” the gap refers to “Who,” which is 11 words away.

Long-distance dependencies

A complex system of augmented rules can be used to make sure that the missing

NP s match

up properly. The rules are complex; for example, you can’t have a gap in one branch of an

NP conjunction: “What did she play [NP Dungeons and ⊔]?” is ungrammatical. But you can have the same gap in both branches of a V P conjunction, as in the sentence “What did you [ V P [V P smell ⊔] and [V P shoot an arrow at ⊔]]?” TIME AND TENSE: Suppose we want to represent the difference between “Ali loves Bo” and “Ali loved Bo.” English uses verb tenses (past, present, and future) to indicate the relative time of an event. One good choice to represent the time of events is the event calculus notation of Section 10.3 . In event calculus we have

E ∈ Loves(Ali, Bo) ∧ During(Now, Extent(E )) Ali loved Bo: E ∈ Loves(Ali, Bo) ∧ After(Now, Extent(E )).

Ali loves Bo:

1

2

1

2

Time and tense

This suggests that our two lexical rules for the words “loves” and “loved” should be these:

V erb (λy λx e ∈ Loves (x, y) ∧ During (Now, e)) → loves V erb (λy λx e ∈ Loves (x, y) ∧ After (Now, e)) → loved. Other than this change, everything else about the grammar remains the same, which is encouraging news; it suggests we are on the right track if we can so easily add a complication like the tense of verbs (although we have just scratched the surface of a complete grammar for time and tense).

Ambiguity: We tend to think of ambiguity as a failure in communication; when a listener is consciously aware of an ambiguity in an utterance, it means that the utterance is unclear or confusing. Here are some examples taken from newspaper headlines:

Squad helps dog bite victim.

Police begin campaign to run down jaywalkers.

Helicopter powered by human flies.

Once-sagging cloth diaper industry saved by full dumps.

Include your children when baking cookies.

Portable toilet bombed; police have nothing to go on.

Milk drinkers are turning to powder.

Two sisters reunited after 18 years in checkout counter.

Such confusions are the exception; most of the time the language we hear seems unambiguous. Thus, when researchers first began to use computers to analyze language in the 1960s, they were quite surprised to learn that almost every sentence is ambiguous, with multiple possible parses (sometimes hundreds), even when the single preferred parse is the only one that native speakers notice. For example, we understand the phrase “brown rice and black beans” as “[brown rice] and [black beans],” and never consider the lowprobability interpretation “brown [rice and black beans],” where the adjective “brown” is modifying the whole phrase, not just the “rice.” When we hear “Outside of a dog, a book is a person’s best friend,” we interpret “outside of” as meaning “except for,” and find it funny when the next sentence of the Groucho Marx joke is “Inside of a dog it’s too dark to read.”

Lexical ambiguity is when a word has more than one meaning: “back” can be an adverb (go back), an adjective (back door), a noun (the back of the room), a verb (back a candidate), or a proper noun (a river in Nunavut, Canada). “Jack” can be a proper name, a noun (a playing card, a six-pointed metal game piece, a nautical flag, a fish, a bird, a cheese, a socket, etc.), or a verb (to jack up a car, to hunt with a light, or to hit a baseball hard). Syntactic

ambiguity refers to a phrase that has multiple parses: “I smelled a wumpus in 2,2” has two parses: one where the prepositional phrase “in 2,2” modifies the noun and one where it modifies the verb. The syntactic ambiguity leads to a semantic ambiguity, because one parse means that the wumpus is in 2,2 and the other means that a stench is in 2,2. In this case, getting the wrong interpretation could be a deadly mistake.

Lexical ambiguity

Syntactic ambiguity

Semantic ambiguity

There can also be ambiguity between literal and figurative meanings. Figures of speech are important in poetry, and are common in everyday speech as well. A metonymy is a figure of speech in which one object is used to stand for another. When we hear “Chrysler announced a new model,” we do not interpret it as saying that companies can talk; rather we understand that a spokesperson for the company made the announcement. Metonymy is common and is often interpreted unconsciously by human hearers.

Metonymy

Unfortunately, our grammar as it is written is not so facile. To handle the semantics of metonymy properly, we need to introduce a whole new level of ambiguity. We could do this by providing two objects for the semantic interpretation of every phrase in the sentence: one for the object that the phrase literally refers to (Chrysler) and one for the metonymic

reference (the spokesperson). We then have to say that there is a relation between the two. In our current grammar, “Chrysler announced” gets interpreted as

x = Chrysler ∧ e ∈ Announce(x) ∧ After(Now, Extent(e)). We need to change that to

x = Chrysler ∧ e ∈ Announce(m) ∧ After(Now, Extent(e)) ∧Metonymy (m, x) . This says that there is one entity

x that is equal to Chrysler, and another entity m that did

the announcing, and that the two are in a metonymy relation. The next step is to define what kinds of metonymy relations can occur. The simplest case is when there is no metonymy at all—the literal object ∀

x and the metonymic object m are identical:

m, x (m = x) ⇒ Metonymy(m, x).

For the Chrysler example, a reasonable generalization is that an organization can be used to stand for a spokesperson of that organization: ∀

m, x x ∈ Organizations ∧ Spokesperson(m, x) ⇒ Metonymy(m, x).

Other metonymies include the author for the works (I read Shakespeare) or more generally the producer for the product (I drive a Honda) and the part for the whole (The Red Sox need a strong arm). Some examples of metonymy, such as “The ham sandwich on Table 4 wants another beer,” are more novel and are interpreted with respect to a situation (such as waiting on tables and not knowing a customer’s name).

A metaphor is another figure of speech, in which a phrase with one literal meaning is used to suggest a different meaning by way of an analogy. Thus, metaphor can be seen as a kind of metonymy where the relation is one of similarity.

Metaphor

Disambiguation

Disambiguation is the process of recovering the most probable intended meaning of an utterance. In one sense we already have a framework for solving this problem: each rule has a probability associated with it, so the probability of an interpretation is the product of the probabilities of the rules that led to the interpretation. Unfortunately, the probabilities reflect how common the phrases are in the corpus from which the grammar was learned, and thus reflect general knowledge, not specific knowledge of the current situation. To do disambiguation properly, we need to combine four models:

1. The world model: the likelihood that a proposition occurs in the world. Given what we know about the world, it is more likely that a speaker who says “I’m dead” means “I am in big trouble” or “I lost this video game” rather than “My life ended, and yet I can still talk.” 2. The mental model: the likelihood that the speaker forms the intention of communicating a certain fact to the hearer. This approach combines models of what the speaker believes, what the speaker believes the hearer believes, and so on. For example, when a politician says, “I am not a crook,” the world model might assign a probability of only 50% to the proposition that the politician is not a criminal, and 99.999% to the proposition that he is not a hooked shepherd’s staff. Nevertheless, we select the former interpretation because it is a more likely thing to say. 3. The language model: the likelihood that a certain string of words will be chosen, given that the speaker has the intention of communicating a certain fact. 4. The acoustic model: for spoken communication, the likelihood that a particular sequence of sounds will be generated, given that the speaker has chosen a given string of words. (For handwritten or typed communication, we have the problem of optical character recognition.)

23.6 Natural Language Tasks Natural language processing is a big field, deserving an entire textbook or two of its own (Goldberg, 2017; Jurafsky and Martin, 2020). In this section we briefly describe some of the main tasks; you can use the references to get more details.

Speech recognition is the task of transforming spoken sound into text. We can then perform further tasks (such as question answering) on the resulting text. Current systems have a word error rate of about 3% to 5% (depending on details of the test set), similar to human transcribers. The challenge for a system using speech recognition is to respond appropriately even when there are errors on individual words.

Speech recognition

Top systems today use a combination of recurrent neural networks and hidden Markov models (Hinton et al., 2012; Yu and Deng, 2016; Deng, 2016; Chiu et al., 2017; Zhang et al., 2017). The introduction of deep neural nets for speech in 2011 led to an immediate and dramatic improvement of about 30% in error rate—this from a field that seemed to be mature and was previously progressing at only a few percent per year. Deep neural networks are a good fit because the problem of speech recognition has a natural compositional breakdown: waveforms to phonemes to words to sentences. They will be covered in the next chapter.

Text-to-speech synthesis is the inverse process—going from text to sound. Taylor (2009) gives a book-length overview. The challenge is to pronounce each word correctly, and to make the flow of each sentence seem natural, with the right pauses and emphasis.

Text-to-speech

Another area of development is in synthesizing different voices—starting with a choice between a generic male or female voice, then allowing for regional dialects, and even imitating celebrity voices. As with speech recognition, the introduction of deep recurrent neural networks led to a large improvement, with about 2/3 of listeners saying that the neural WaveNet system (van den Oord et al., 2016a) sounded more natural than the previous non-neural system.

Machine translation transforms text in one language to another. Systems are usually trained using a bilingual corpus: a set of paired documents, where one member of the pair is in, say, English, and the other is in, say, French. The documents do not need to be annotated in any way; the machine translation system learns to align sentences and phrases and then when presented with a novel sentence in one language, can generate a translation to the other.

Systems in the early 2000s used n-gram models, and achieved results that were usually good enough to get across the meaning of a text, but contained syntactic errors in most sentences. One problem was the limit on the length of the n-grams: even with a large limit of 7, it was difficult for information to flow from one end of the sentence to the other. Another problem was that all the information in an n-gram model is at the level of individual words. Such a system could learn that “black cat” translates to “chat noir,” but it could not learn the rule that adjectives generally come before the noun in English and after the noun in French.

Recurrent neural sequence-to-sequence models (Sutskever et al., 2015) got around the problem. They could generalize better (because they could use word embeddings rather than n-gram counts of specific words) and could form compositional models throughout the various levels of the deep network to effectively pass information along. Subsequent work using the attention-focusing mechanism of the transformer model (Vaswani et al., 2018) increased performance further, and a hybrid model incorporating aspects of both these models does better still, approaching human-level performance on some language pairs (Wu et al., 2016b; Chen et al., 2018).

Information extraction is the process of acquiring knowledge by skimming a text and looking for occurrences of particular classes of objects and for relationships among them. A typical task is to extract instances of addresses from Web pages, with database fields for street, city, state, and zip code; or instances of storms from weather reports, with fields for

temperature, wind speed, and precipitation. If the source text is well structured (for example, in the form of a table), then simple techniques such as regular expressions can extract the information (Cafarella et al., 2008). It gets harder if we are trying to extract all facts, rather than a specific type (such as weather reports); Banko et al. (2007) describe the TEXTRUNNER system that performs extraction over an open, expanding set of relations. For free-form text, techniques include hidden Markov models and rule-based learning systems (as used in TEXTRUNNER and NELL (Never-Ending Language Learning) (Mitchell et al., 2018)). More recent systems use recurrent neural networks, taking advantage of the flexibility of word embeddings. You can find an overview in Kumar (2017).

Information extraction

Information retrieval is the task of finding documents that are relevant and important for a given query. Internet search engines such as Google and Baidu perform this task billions of times a day. Three good textbooks on the subject are Manning et al. (2008), Croft et al. (2010), and Baeza-Yates and Ribeiro-Neto (2011).

Information retrieval

Question Answering is a different task, in which the query really is a question, such as “Who founded the U.S. Coast Guard?” and the response is not a ranked list of documents but rather an actual answer: “Alexander Hamilton.” There have been question-answering systems since the 1960s that rely on syntactic parsing as discussed in this chapter, but only since 2001 have such systems used Web information retrieval to radically increase their breadth of coverage. Katz (1997) describes the START parser and question answerer. Banko et al. (2002) describe ASKMSR, which was less sophisticated in terms of its syntactic parsing ability, but more aggressive in using Web search and sorting through the results. For example, to answer “Who founded the U.S. Coast Guard?” it would search for queries such as [* founded the U.S. Coast Guard] and [the U.S. Coast Guard was founded by *], and

then examine the multiple resulting Web pages to pick out a likely response, knowing that the query word “who” suggests that the answer should be a person. The Text REtrieval Conference (TREC) gathers research on this topic and has hosted competitions on an annual basis since 1991 (Allan et al., 2017). Recently we have seen other test sets, such as the AI2 ARC test set of basic science questions (Clark et al., 2018).

Question Answering

Summary The main points of this chapter are as follows: Probabilistic language models based on n-grams recover a surprising amount of information about a language. They can perform well on such diverse tasks as language identification, spelling correction, sentiment analysis, genre classification, and namedentity recognition. These language models can have millions of features, so preprocessing and smoothing the data to reduce noise is important. In building a statistical language system, it is best to devise a model that can make good use of available data, even if the model seems overly simplistic. Word embeddings can give a richer representation of words and their similarities. To capture the hierarchical structure of language, phrase structure grammars (and in particular, context-free grammars) are useful. The probabilistic context-free grammar (PCFG) formalism is widely used, as is the dependency grammar formalism. Sentences in a context-free language can be parsed in O(n3 ) time by a chart parser such as the CYK algorithm, which requires grammar rules to be in Chomsky Normal Form. With a small loss in accuracy, natural languages can be parsed in O(n) time, using a beam search or a shift-reduce parser. A treebank can be a resource for learning a PCFG grammar with parameters. It is convenient to augment a grammar to handle issues such as subject–verb agreement and pronoun case, and to represent information at the level of words rather than just at the level of categories. Semantic interpretation can also be handled by an augmented grammar. We can learn a semantic grammar from a corpus of questions paired either with the logical form of the question, or with the answer. Natural language is complex and difficult to capture in a formal grammar.

Bibliographical and Historical Notes N -gram letter models for language modeling were proposed by Markov (1913). Claude Shannon (Shannon and Weaver, 1949) was the first to generate n-gram word models of English. The bag-of-words model gets its name from a passage from linguist Zellig Harris (1954), “language is not merely a bag of words but a tool with particular properties.” Norvig

n

(2009) gives some examples of tasks that can be accomplished with -gram models.

Chomsky (1956, 1957) pointed out the limitations of finite-state models compared with context-free models, concluding, “Probabilistic models give no particular insight into some of the basic problems of syntactic structure.” This is true, but probabilistic models do provide insight into some other basic problems—problems that context-free models ignore. Chomsky’s remarks had the unfortunate effect of scaring many people away from statistical models for two decades, until these models reemerged for use in the field of speech recognition (Jelinek, 1976), and in cognitive science, where optimality theory (Smolensky and Prince, 1993; Kager, 1999) posited that language works by finding the most probable candidate that optimally satisfies competing constraints.

Add-one smoothing, first suggested by Pierre-Simon Laplace (1816), was formalized by Jeffreys (1948). Other smoothing techniques include interpolation smoothing (Jelinek and Mercer, 1980), Witten–Bell smoothing (1991), Good–Turing smoothing (Church and Gale, 1991), Kneser–Ney smoothing (1995, 2004), and stupid backoff (Brants et al., 2007). Chen and Goodman (1996) and Goodman (2001) survey smoothing techniques.

n

Simple -gram letter and word models are not the only possible probabilistic models. The latent Dirichlet allocation model (Blei et al., 2002; Hoffman et al., 2011) is a probabilistic text model that views a document as a mixture of topics, each with its own distribution of words. This model can be seen as an extension and rationalization of the latent semantic indexing model of Deerwester et al. (1990) and is also related to the multiple-cause mixture model of (Sahami et al., 1996). And of course there is great interest in non-probabilistic language models, such as the deep learning models covered in Chapter 24 .

Joulin et al. (2016) give a bag of tricks for efficient text classification. Joachims (2001) uses statistical learning theory and support vector machines to give a theoretical analysis of when

classification will be successful. Apté et al. (1994) report an accuracy of 96% in classifying Reuters news articles into the “Earnings” category. Koller and Sahami (1997) report accuracy up to 95% with a naive Bayes classifier, and up to 98.6% with a Bayes classifier.

Schapire and Singer (2000) show that simple linear classifiers can often achieve accuracy almost as good as more complex models, and run faster. Zhang et al. (2016) describe a character-level (rather than word-level) text classifier. Witten et al. (1999) describe compression algorithms for classification, and show the deep connection between the LZW compression algorithm and maximum-entropy language models.

Wordnet (Fellbaum, 2001) is a publicly available dictionary of about 100,000 words and phrases, categorized into parts of speech and linked by semantic relations such as synonym, antonym, and part-of. Charniak (1996) and Klein and Manning (2001) discuss parsing with treebank grammars. The British National Corpus (Leech et al., 2001) contains 100 million words, and the World Wide Web contains several trillion words; Franz and Brants (2006) describe the publicly available Google n-gram corpus of 13 million unique words from a trillion words of Web text. Buck et al. (2014) describe a similar data set from the Common Crawl project. The Penn Treebank (Marcus et al., 1993; Bies et al., 2015) provides parse trees for a 3-million-word corpus of English.

Many of the n-gram model techniques are also used in bioinformatics problems. Biostatistics and probabilistic NLP are coming closer together, as each deals with long, structured sequences chosen from an alphabet.

Early part-of-speech (POS) taggers used a variety of techniques, including rule sets (Brill, 1992), n-grams (Church, 1988), decision trees (Màrquez and Rodrí guez, 1998), HMMs (Brants, 2000), and logistic regression (Ratnaparkhi, 1996). Historically, a logistic regression model was also called a “maximum entropy Markov model” or MEMM, so some work is under that name. Jurafsky and Martin (2020) have a good chapter on POS tagging. Ng and Jordan (2002) compare discriminative and generative models for classification tasks.

Like semantic networks, context-free grammars were first discovered by ancient Indian grammarians (especially Panini, ca. 350

BCE)

studying Shastric Sanskrit (Ingerman, 1967).

They were reinvented by Noam Chomsky (1956) for the analysis of English and independently by John Backus (1959) and Peter Naur for the analysis of Algol-58.

Probabilistic context-free grammars were first investigated by Booth (1969) and Salomaa (1969). Algorithms for PCFGs are presented in the excellent short monograph by Charniak (1993) and the excellent long textbooks by Manning and Schütze (1999) and Jurafsky and Martin (2020). Baker (1979) introduces the inside–outside algorithm for learning a PCFG.

n

Lexicalized PCFGs (Charniak, 1997; Hwa, 1998) combine the best aspects of PCFGs and gram models. Collins (1999) describes PCFG parsing that is lexicalized with head features, and Johnson (1998) shows how the accuracy of a PCFG depends on the structure of the treebank from which its probabilities were learned.

There have been many attempts to write formal grammars of natural languages, both in “pure” linguistics and in computational linguistics. There are several comprehensive but informal grammars of English (Quirk et al., 1985; McCawley, 1988; Huddleston and Pullum, 2002). Since the 1980s, there has been a trend toward lexicalization: putting more information in the lexicon and less in the grammar.

Lexical-functional grammar, or LFG (Bresnan, 1982) was the first major grammar formalism to be highly lexicalized. If we carry lexicalization to an extreme, we end up with categorial grammar (Clark and Curran, 2004), in which there can be as few as two grammar rules, or with dependency grammar (Smith and Eisner, 2008; Kübler et al., 2009) in which there are no syntactic categories, only relations between words.

Computerized parsing was first demonstrated by Yngve (1955). Efficient algorithms were developed in the 1960s, with a few twists since then (Kasami, 1965; Younger, 1967; Earley, 1970; Graham et al., 1980). Church and Patil (1982) describe syntactic ambiguity and address ways to resolve it.

Klein and Manning (2003) describe A* parsing, and Pauls and Klein (2009) extend that to Kbest A* parsing, in which the result is not a single parse but the

K best. Goldberg et al.

(2013) describe the necessary implementation tricks to make sure that a beam search parser is

O(n) and not O(n ). Zhu et al. (2013) describe a fast deterministic shift-reduce parser for 2

natural languages, and Sagae and Lavie (2006) show how adding search to a shift-reduce parser can make it more accurate, at the cost of some speed.

Today, highly accurate open-source parsers include Google’s Parsey McParseface (Andor et al., 2016), the Stanford Parser (Chen and Manning, 2014), the Berkeley Parser (Kitaev and

Klein, 2018), and the SPACY parser. They all do generalization through neural networks and achieve roughly 95% accuracy on Wall Street Journal or Penn Treebank test sets. There is some criticism of the field that it is focusing too narrowly on measuring performance on a few select corpora, and perhaps overfitting on them.

Formal semantic interpretation of natural languages originates within philosophy and formal logic, particularly Alfred Tarski’s (1935) work on the semantics of formal languages. BarHillel (1954) was the first to consider the problems of pragmatics (such as indexicals) and propose that they could be handled by formal logic. Richard Montague’s essay “English as a formal language” (1970) is a kind of manifesto for the logical analysis of language, but there are other books that are more readable (Dowty et al., 1991; Portner and Partee, 2002; Cruse, 2011). While semantic interpretation programs are designed to pick the most likely interpretation, literary critics (Empson, 1953; Hobbs, 1990) have been ambiguous about whether ambiguity is something to be resolved or cherished. Norvig (1988) discusses the problems of considering multiple simultaneous interpretations, rather than settling for a single maximum-likelihood interpretation. Lakoff and Johnson (1980) give an engaging analysis and catalog of common metaphors in English. Martin (1990) and Gibbs (2006) offer computational models of metaphor interpretation.

The first NLP system to solve an actual task was the BASEBALL question answering system (Green et al., 1961), which handled questions about a database of baseball statistics. Close after that was Winograd’s (1972) SHRDLU, which handled questions and commands about a blocks-world scene, and Woods’s (1973) LUNAR, which answered questions about the rocks brought back from the moon by the Apollo program.

Banko et al. (2002) present the ASKMSR question-answering system; a similar system is due to Kwok et al. (2001). Pasca and Harabagiu (2001) discuss a contest-winning questionanswering system.

Modern approaches to semantic interpretation usually assume that the mapping from syntax to semantics will be learned from examples (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; Zhao and Huang, 2015). The first important result on grammar induction was a negative one: Gold (1967) showed that it is not possible to reliably learn an exactly correct context-free grammar, given a set of strings from that grammar. Prominent linguists, such as Chomsky (1957) and Pinker (2003), have used Gold’s result to argue that there must

be an innate universal grammar that all children have from birth. The so-called Poverty of the Stimulus argument says that children aren’t given enough input to learn a CFG, so they must already “know” the grammar and be merely tuning some of its parameters.

Universal grammar

While this argument continues to hold sway throughout much of Chomskyan linguistics, it has been dismissed by other linguists (Pullum, 1996; Elman et al., 1997) and most computer scientists. As early as 1969, Horning showed that it is possible to learn, in the sense of PAC learning, a probabilistic context-free grammar. Since then, there have been many convincing empirical demonstrations of language learning from positive examples alone, such as learning semantic grammars with inductive logic programming (Muggleton and De Raedt, 1994; Mooney, 1999), the Ph.D. theses of Schütze (1995) and de Marcken (1996), and the entire line of modern language processing systems based on the transformer model (Section 24). There is an annual International Conference on Grammatical Inference (ICGI).

James Baker’s DRAGON system (Baker, 1975) could be considered the first succesful speech recognition system. It was the first to use HMMs for speech. After several decades of systems based on probabilistic language models, the field began to switch to deep neural networks (Hinton et al., 2012). Deng (2016) describes how the introduction of deep learning enabled rapid improvement in speech recognition, and reflects on the implications for other NLP tasks. Today deep learning is the dominant approach for all large-scale speech recognition systems. Speech recognition can be seen as the first application area that highlighted the success of deep learning, with computer vision following shortly thereafter.

Interest in the field of information retrieval was spurred by widespread usage of Internet searching. Croft et al. (2010) and Manning et al. (2008) provide textbooks that cover the basics. The TREC conference hosts an annual competition for IR systems and publishes proceedings with results.

Brin and Page (1998) describe the PageRank algorithm, which takes into account the links between pages, and give an overview of the implementation of a Web search engine.

Silverstein et al. (1998) investigate a log of a billion Web searches. The journal Information Retrieval and the proceedings of the annual flagship SIGIR conference cover recent developments in the field.

Information extraction has been pushed forward by the annual Message Understanding Conferences (MUC), sponsored by the U.S. government. Surveys of template-based systems are given by Roche and Schabes (1997), Appelt (1999), and Muslea (1999). Large databases of facts were extracted by Craven et al. (2000), Pasca et al. (2006), Mitchell (2007), and Durme and Pasca (2008). Freitag and McCallum (2000) discuss HMMs for Information Extraction. Conditional random fields have also been used for this task (Lafferty et al., 2001; McCallum, 2003); a tutorial with practical guidance is given by Sutton and McCallum (2007). Sarawagi (2007) gives a comprehensive survey.

Two early influential approaches to automated knowledge engineering for NLP were by Riloff (1993), who showed that an automatically constructed dictionary performed almost as well as a carefully handcrafted domain-specific dictionary, and by Yarowsky (1995), who showed that the task of word sense classification could be accomplished through unsupervised training on a corpus of unlabeled text with accuracy as good as supervised methods.

The idea of simultaneously extracting templates and examples from a handful of labeled examples was developed independently and simultaneously by Blum and Mitchell (1998), who called it cotraining, and by Brin (1998), who called it DIPRE (Dual Iterative Pattern Relation Extraction). You can see why the term cotraining has stuck. Similar early work, under the name of bootstrapping, was done by Jones et al. (1999). The method was advanced by the QXTRACT (Agichtein and Gravano, 2003) and KNOWITALL (Etzioni et al., 2005) systems. Machine reading was introduced by Mitchell (2005) and Etzioni et al. (2006) and is the focus of the TEXTRUNNER project (Banko et al., 2007; Banko and Etzioni, 2008).

This chapter has focused on natural language sentences, but it is also possible to do information extraction based on the physical structure or geometric layout of text rather than on the linguistic structure. Lists, tables, charts, graphs, diagrams, etc., whether encoded in HTML or accessed through the visual analysis of pdf documents, are home to data that can be extracted and consolidated (Hurst, 2000; Pinto et al., 2003; Cafarella et al., 2008).

Ken Church (2004) shows that natural language research has cycled between concentrating on the data (empiricism) and concentrating on theories (rationalism); he describes the advantages of having good language resources and evaluation schemes, but wonders if we have gone too far (Church and Hestness, 2019). Early linguists concentrated on actual language usage data, including frequency counts. Noam Chomsky (1956) demonstrated the limitations of finite-state models, leading to an emphasis on theoretical studies of syntax, disregarding actual language performance. This approach dominated for twenty years, until empiricism made a comeback based on the success of work in statistical speech recognition (Jelinek, 1976). Today, the emphasis on empirical language data continues, and there is heightened interest in models that consider higher-level constructs, such as syntactic and semantic relations, not just sequences of words. There is also a strong emphasis on deep learning neural network models of language, which we will cover in Chapter 24 .

Work on applications of language processing is presented at the biennial Applied Natural Language Processing conference (ANLP), the conference on Empirical Methods in Natural Language Processing (EMNLP), and the journal Natural Language Engineering. A broad range of NLP work appears in the journal Computational Linguistics and its conference, ACL, and in the International Computational Linguistics (COLING) conference. Jurafsky and Martin (2020) give a comprehensive introduction to speech and NLP.

Chapter 24 Deep Learning for Natural Language Processing In which deep neural networks perform a variety of language tasks, capturing the structure of natural language as well as its fluidity. Chapter 23  explained the key elements of natural language, including grammar and semantics. Systems based on parsing and semantic analysis have demonstrated success on many tasks, but their performance is limited by the endless complexity of linguistic phenomena in real text. Given the vast amount of text available in machine-readable form, it makes sense to consider whether approaches based on data-driven machine learning can be more effective. We explore this hypothesis using the tools provided by deep learning systems (Chapter 21 ). We begin in Section 24.1  by showing how learning can be improved by representing words as points in a high-dimensional space, rather than as atomic values. Section 24.2  covers the use of recurrent neural networks to capture meaning and long-distance context as text is processed sequentially. Section 24.3  focuses primarily on machine translation, one of the major successes of deep learning applied to NLP. Sections 24.4  and 24.5  cover models that can be trained from large amounts of unlabeled text and then applied to specific tasks, often achieving state-of-the-art performance. Finally, Section 24.6  takes stock of where we are and how the field may progress.

24.1 Word Embeddings We would like a representation of words that does not require manual feature engineering, but allows for generalization between related words—words that are related syntactically (“colorless” and “ideal” are both adjectives), semantically (“cat” and “kitten” are both felines), topically (“sunny” and “sleet” are both weather terms), in terms of sentiment (“awesome” has opposite sentiment to “cringeworthy”), or otherwise.

How should we encode a word into an input vector x for use in a neural network? As

explained in Section 21.2.1  (page 756), we could use a one-hot vector—that is, we encode the ith word in the dictionary with a 1 bit in the ith input position and a 0 in all the other positions. But such a representation would not capture the similarity between words.

Following the linguist John R. Firth’s (1957) maxim, “You shall know a word by the company it keeps,” we could represent each word with a vector of n-gram counts of all the phrases that the word appears in. However, raw n-gram counts are cumbersome. With a 100,000-word vocabulary, there are

1025 5-grams to keep track of (although vectors in this

1025 -dimensional space would be quite sparse—most of the counts would be zero). We

would get better generalization if we reduced this to a smaller-size vector, perhaps with just a few hundred dimensions. We call this smaller, dense vector a word embedding: a lowdimensional vector representing a word. Word embeddings are learned automatically from the data. (We will see later how this is done.) What are these learned word embeddings like? On the one hand, each one is just a vector of numbers, where the individual dimensions and their numeric values do not have discernible meanings:

“aardvark” = [−0.7, +0.2, −3.2, …] “abacus” = [+0.5, +0.9, −1.3, …] … “zyzzyva” = [−0.1, +0.8, −0.4, …].

Word embedding

On the other hand, the feature space has the property that similar words end up having similar vectors. We can see that in Figure 24.1 , where there are separate clusters for country, kinship, transportation, and food words.

Figure 24.1

Word embedding vectors computed by the GloVe algorithm trained on 6 billion words of text. 100dimensional word vectors are projected down onto two dimensions in this visualization. Similar words appear near each other.

It turns out, for reasons we do not completely understand, that the word embedding vectors have additional properties beyond mere proximity for similar words. For example, suppose we look at the vectors

A for Athens and B for Greece. For these words the vector difference

B − A seems to encode the country/capital relationship. Other pairs—France and Paris, Russia and Moscow, Zambia and Lusaka—have essentially the same vector difference.

We can use this property to solve word analogy problems such as “Athens is to Greece as

C for the Oslo vector and D for the unknown, we hypothesize that B − A = D − C, giving us D = C + (B − A). And when we compute this new vector D, Oslo is to [what]?” Writing

we find that it is closer to “Norway” than to any other word. Figure 24.2  shows that this type of vector arithmetic works for many relationships.

Figure 24.2

A is to B as C is to [what]?” with vector AB C D

A word embedding model can sometimes answer the question “

arithmetic: given the word embedding vectors for the words , , and , compute the vector = + ( − ) and look up the word that is closest to . (The answers in column were computed

D C B A

D

automatically by the model. The descriptions in the “Relationship” column were added by hand.) Adapted from Mikolov et al. (2013, 2014).

However, there is no guarantee that a particular word embedding algorithm run on a particular corpus will capture a particular semantic relationship. Word embeddings are popular because they have proven to be a good representation for downstream language tasks (such as question answering or translation or summarization), not because they are guaranteed to answer analogy questions on their own.

Using word embedding vectors rather than one-hot encodings of words turns out to be helpful for essentially all applications of deep learning to NLP tasks. Indeed, in many cases it is possible to use generic pretrained vectors, obtained from any of several suppliers, for one’s particular NLP task. At the time of writing, the commonly used vector dictionaries include WORD2VEC, GloVe (Global Vectors), and FASTTEXT, which has embeddings for 157 languages. Using a pretrained model can save a great deal of time and effort. For more on these resources, see Section 24.5.1 .

It is also possible to train your own word vectors; this is usually done at the same time as training a network for a particular task. Unlike generic pretrained embeddings, word embeddings produced for a specific task can be trained on a carefully selected corpus and

will tend to emphasize aspects of words that are useful for the task. Suppose, for example, that the task is part-of-speech (POS) tagging (see Section 23.1.6 ). Recall that this involves predicting the correct part of speech for each word in a sentence. Although this is a simple task, it is nontrivial because many words can be tagged in multiple ways—for example, the word cut can be a present-tense verb (transitive or intransitive), a past-tense verb, an infinitive verb, a past participle, an adjective, or a noun. If a nearby temporal adverb refers to the past, that suggests that this particular occurrence of cut is a past-tense verb; and we might hope, then, that the embedding will capture the past-referring aspect of adverbs.

POS tagging serves as a good introduction to the application of deep learning to NLP, without the complications of more complex tasks like question answering (see Section 24.5.3 ). Given a corpus of sentences with POS tags, we learn the parameters for the word embeddings and the POS tagger simultaneously. The process works as follows:

1. Choose the width w (an odd number of words) for the prediction window to be used to tag each word. A typical value is w = 5, meaning that the tag is predicted based on the word plus the two words to the left and the two words to the right. Split every sentence in your corpus into overlapping windows of length w. Each window produces one training example consisting of the w words as input and the POS category of the middle word as output. 2. Create a vocabulary of all of the unique word tokens that occur more than, say, 5 times in the training data. Denote the total number of words in the vocabulary as v. 3. Sort this vocabulary in any arbitrary order (perhaps alphabetically). 4. Choose a value d as the size of each word embedding vector. 5. Create a new v-by-d weight matrix called E. This is the word embedding matrix. Row i of E is the word embedding of the ith word in the vocabulary. Initialize E randomly (or from pretrained vectors). 6. Set up a neural network that outputs a part of speech label, as shown in Figure 24.3 . The first layer will consist of w copies of the embedding matrix. We might use two additional hidden layers, z1 and z2 (with weight matrices W1 and W2 , respectively), followed by a softmax layer yielding an output probability distribution

yˆ over the possible part-of-speech categories for the middle word: z = σ ( W x) z = σ (W z ) yˆ = softmax(Wout z ) . 1

1

2

2

1

2

Figure 24.3

Feedforward part-of-speech tagging model. This model takes a 5-word window as input and predicts the tag of the word in the middle—here, cut. The model is able to account for word position because each of the 5 input embeddings is multiplied by a different part of the first hidden layer. The parameter values for the word embeddings and for the three layers are all learned simultaneously during training.

7. To encode a sequence of w words into an input vector, simply look up the embedding for each word and concatenate the embedding vectors. The result is a real-valued input vector x of length wd. Even though a given word will have the same embedding vector whether it occurs in the first position, the last, or somewhere in between, each embedding will be multiplied by a different part of the first hidden layer; therefore we are implicitly encoding the relative position of each word. 8. Train the weights E and the other weight matrices W1 , W2 , and Wout using gradient descent. If all goes well, the middle word, cut, will be labeled as a past-tense verb, based on the evidence in the window, which includes the temporal past word “yesterday,” the third-person subject pronoun “they” immediately before cut, and so on.

An alternative to word embeddings is a character-level model in which the input is a sequence of characters, each encoded as a one-hot vector. Such a model has to learn how

characters come together to form words. The majority of work in NLP sticks with word-level rather than character-level encodings.

24.2 Recurrent Neural Networks for NLP We now have a good representation for single words in isolation, but language consists of an ordered sequence of words in which the context of surrounding words is important. For simple tasks like part of speech tagging, a small, fixed-size window of perhaps five words usually provides enough context.

More complex tasks such as question answering or reference resolution may require dozens of words as context. For example, in the sentence “Eduardo told me that Miguel was very sick so I took him to the hospital,” knowing that him refers to Miguel and not Eduardo requires context that spans from the first to the last word of the 14-word sentence.

24.2.1 Language models with recurrent neural networks We’ll start with the problem of creating a language model with sufficient context. Recall that a language model is a probability distribution over sequences of words. It allows us to predict the next word in a text given all the previous words, and is often used as a building block for more complex tasks. Building a language model with either an n-gram model (as in Section 23.1 ) or a feedforward network with a fixed window of n words can run into difficulty due to the problem of context: either the required context will exceed the fixed window size or the model will have too many parameters, or both.

In addition, a feedforward network has the problem of asymmetry: whatever it learns about, say, the appearance of the word him as the 12th word of the sentence it will have to relearn for the appearance of him at other positions in the sentence, because the weights are different for each word position. In Section 21.6 , we introduced the recurrent neural network or RNN, which is designed to process time-series data, one datum at a time. This suggests that RNNs might be useful for processing language, one word at a time. We repeat Figure 21.8  here as Figure 24.4 .

Figure 24.4

(a) Schematic diagram of an RNN where the hidden layer z has recurrent connections; the Δ symbol indicates a delay. Each input x is the word embedding vector of the next word in the sentence. Each

output y is the output for that time step. (b) The same network unrolled over three timesteps to create a feedforward network. Note that the weights are shared across all timesteps.

In an RNN language model each input word is encoded as a word embedding vector, xi . There is a hidden layer zt which gets passed as input from one time step to the next. We are interested in doing multiclass classification: the classes are the words of the vocabulary. Thus the output yt will be a softmax probability distribution over the possible values of the next word in the sentence.

The RNN architecture solves the problem of too many parameters. The number of parameters in the weight matrixes wz,z , wx,z , and wz,y stays constant, regardless of the number of words—it is O(1). This is in contrast to feedforward networks, which have O(n) parameters, and n-gram models, which have

O(vn ) parameters, where v is the size of the

vocabulary.

The RNN architecture also solves the problem of asymmetry, because the weights are the same for every word position.

The RNN architecture can sometimes solve the limited context problem as well. In theory there is no limit to how far back in the input the model can look. Each update of the hidden layer zt has access to both the current input word xt and the previous hidden layer zt−1 , which means that information about any word in the input can be kept in the hidden layer indefinitely, copied over (or modified as appropriate) from one time step to the next. Of course, there is a limited amount of storage in z, so it can’t remember everything about all the previous words.

In practice RNN models perform well on a variety of tasks, but not on all tasks. It can be hard to predict whether they will be succesful for a given problem. One factor that contributes to success is that the training process encourages the network to allocate storage space in z to the aspects of the input that will actually prove to be useful. To train an RNN language model, we use the training process described in Section 21.6.1 . The inputs, xt , are the words in a training corpus of text, and the observed outputs are the same words offset by 1. That is, for the training text “hello world,” the first input x1 is the word embedding for “hello” and the first output y1 is the word embedding for “world.” We are training the model to predict the next word, and expecting that in order to do so it will use the hidden layer to represent useful information. As explained in Section 21.6.1  we compute the difference between the observed output and the actual output computed by the network, and back-propagate through time, taking care to keep the weights the same for all time steps.

Once the model has been trained, we can use it to generate random text. We give the model an initial input word x1 , from which it will produce an output y1 which is a softmax probability distribution over words. We sample a single word from the distribution, record the word as the output for time t, and feed it back in as the next input word x2 . We repeat for as long as desired. In sampling from y1 we have a choice: we could always take the most likely word; we could sample according to the probability of each word; or we could oversample the less-likely words, in order to inject more variety into the generated output. The sampling weight is a hyperparameter of the model.

Here is an example of random text generated by an RNN model trained on Shakespeare’s works (Karpathy, 2015):

Marry, and will, my lord, to weep in such a one were prettiest; Yet now I was adopted heir Of the world’s lamentable day, To watch the next way with his father with his face?

24.2.2 Classification with recurrent neural networks It is also possible to use RNNs for other language tasks, such as part of speech tagging or coreference resolution. In both cases the input and hidden layers will be the same, but for a

POS tagger the output will be a softmax distribution over POS tags, and for coreference resolution it will be a softmax distribution over the posible antecedents. For example, when the network gets to the input him in “Eduardo told me that Miguel was very sick so I took him to the hospital” it should output a high probability for “Miguel.”

Training an RNN to do classification like this is done the same way as with the language model. The only difference is that the training data will require labels—part of speech tags or reference indications. That makes it much harder to collect the data than for the case of a language model, where unlabelled text is all we need.

In a language model we want to predict the nth word given the previous words. But for classification, there is no reason we should limit ourselves to looking at only the previous words. It can be very helpful to look ahead in the sentence. In our coreference example, the referent him would be different if the sentence concluded “to see Miguel” rather than “to the hospital,” so looking ahead is crucial. We know from eye-tracking experiments that human readers do not go strictly left-to-right.

To capture the context on the right, we can use a bidirectional RNN, which concatenates a separate right-to-left model onto the left-to-right model. An example of using a bidirectional RNN for POS tagging is shown in Figure 24.5 .

Figure 24.5

A bidirectional RNN network for POS tagging.

Bidirectional RNN

In the case of a multilayer RNN, zt will be the hidden vector of the last layer. For a bidirectional RNN, zt is usually taken to be the concatenation of vectors from the left-toright and right-to-left models.

RNNs can also be used for sentence-level (or document-level) classification tasks, in which a single output comes at the end, rather than having a stream of outputs, one per time step. For example in sentiment analysis the goal is to classify a text as having either Positive or Negative sentiment. For example, “This movie was poorly written and poorly acted” should be classified as Negative. (Some sentiment analysis schemes use more than two categories, or use a numeric scalar value.)

Using RNNs for a sentence-level task is a bit more complex, since we need to obtain an aggregate whole-sentence representation, y from the per-word outputs yt of the RNN. The simplest way to do this is to use the RNN hidden state corresponding to the last word of the input, since the RNN will have read the entire sentence at that timestep. However, this can implicitly bias the model towards paying more attention to the end of the sentence. Another common technique is to pool all of the hidden vectors. For instance, average pooling computes the element-wise average over all of the hidden vectors:

z˜ =

1

s

∑z s

t=1

t.

Average pooling

The pooled d-dimensional vector z ˜ can then be fed into one or more feedforward layers before being fed into the output layer.

24.2.3 LSTMs for NLP tasks We said that RNNs sometimes solve the limited context problem. In theory, any information could be passed along from one hidden layer to the next for any number of time steps. But in practice the information can get lost or distorted, just as in playing the game of telephone, in which players stand in line and the first player whispers a message to the second, who repeats it to the third, and so on down the line. Usually, the message that comes out at the end is quite corrupted from the original message. This problem for RNNs is similar to the vanishing gradient problem we described on page 756, except that we are dealing now with layers over time rather than with deep layers. In Section 21.6.2  we introduced the long short-term memory (LSTM) model. This is a kind of RNN with gating units that don’t suffer from the problem of imperfectly reproducing a message from one time step to the next. Rather, an LSTM can choose to remember some parts of the input, copying it over to the next timestep, and to forget other parts. Consider a language model handling a text such as

The athletes, who all won their local qualifiers and advanced to the finals in Tokyo, now ...

At this point if we asked the model which next word was more probable, “compete” or “competes,” we would expect it to pick “compete” because it agrees with the subject “The athletes.” An LSTM can learn to create a latent feature for the subject person and number and copy that feature forward without alteration until it is needed to make a choice like this. A regular RNN (or an n-gram model for that matter) often gets confused in long sentences with many intervening words between the subject and verb.

24.3 Sequence-to-Sequence Models One of the most widely studied tasks in NLP is machine translation (MT), where the goal is to translate a sentence from a source language to a target language—for example, from Spanish to English. We train an MT model with a large corpus of source/target sentence pairs. The goal is to then accurately translate new sentences that are not in our training data.

Machine translation (MT)

Source language

Target language

Can we use RNNs to create an MT system? We can certainly encode the source sentence with an RNN. If there were a one-to-one correspondence between source words and target words, then we could treat MT as a simple tagging task—given the source word “perro” in Spanish, we tag it as the corresponding English word “dog.” But in fact, words are not oneto-one: in Spanish the three words “caballo de mar” corresponds to the single English word “seahorse,” and the two words “perro grande” translate to “big dog,” with the word order reversed. Word reordering can be even more extreme; in English the subject is usually at the start of a sentence, but in Fijian the subject is usually at the end. So how do we generate a sentence in the target language?

It seems like we should generate one word at a time, but keep track of the context so that we can remember parts of the source that haven’t been translated yet, and keep track of

what has been translated so that we don’t repeat ourselves. It also seems that for some sentences we have to process the entire source sentence before starting to generate the target. In other words, the generation of each target word is conditional on the entire source sentence and on all previously generated target words.

This gives text generation for MT a close connection to a standard RNN language model, as described in Section 24.2 . Certainly, if we had trained an RNN on English text, it would be more likely to generate “big dog” than “dog big.” However, we don’t want to generate just any random target language sentence; we want to generate a target language sentence that corresponds to the source language sentence. The simplest way to do that is to use two RNNs, one for the source and one for the target. We run the source RNN over the source sentence and then use the final hidden state from the source RNN as the initial hidden state for the target RNN. This way, each target word is implicitly conditioned on both the entire source sentence and the previous target words.

This neural network architecture is called a basic sequence-to-sequence model, an example of which is shown in Figure 24.6 . Sequence-to-sequence models are most commonly used for machine translation, but can also be used for a number of other tasks, like automatically generating a text caption from an image, or summarization: rewriting a long text into a shorter one that maintains the same meaning.

Figure 24.6

Basic sequence-to-sequence model. Each block represents one LSTM timestep. (For simplicity, the embedding and output layers are not shown.) On successive steps we feed the network the words of the source sentence “The man is tall,” followed by the tag to indicate that the network should start

producing the target sentence. The final hidden state at the end of the source sentence is used as the hidden state for the start of the target sentence. After that, each target sentence word at time t is used as input at time t + 1, until the network produces the tag to indicate that sentence generation is finished.

Sequence-to-sequence model

Basic sequence-to-sequence models were a significant breakthrough in NLP and MT specifically. According to Wu et al. (2016b) the approach led to a 60% error reduction over the previous MT methods. But these models suffer from three major shortcomings:

NEARBY CONTEXT BIAS: whatever RNNs want to remember about the past, they have to fit into their hidden state. For example, let’s say an RNN is processing word (or timestep) 57 in a 70-word sequence. The hidden state will likely contain more information about the word at timestep 56 than the word at timestep 5, because each time the hidden vector is updated it has to replace some amount of existing information with new information. This behavior is part of the intentional design of the model, and often makes sense for NLP, since nearby context is typically more important. However, far-away context can be crucial as well, and can get lost in an RNN model; even LSTMs have difficulty with this task. FIXED CONTEXT SIZE LIMIT: In an RNN translation model the entire source sentence is compressed into a single fixed-dimensional hidden state vector. An LSTM used in a state-of-the-art NLP model typically has around 1024 dimensions, and if we have to represent, say, a 64-word sentence in 1024 dimensions, this only gives us 16 dimensions per word—not enough for complex sentences. Increasing the hidden state vector size can lead to slow training and overfitting. SLOWER SEQUENTIAL PROCESSING: As discussed in Section 21.3 , neural networks realize considerable efficiency gains by processing the training data in batches so as to take advantage of efficient hardware support for matrix arithmetic. RNNs, on the other hand, seem to be constrained to operate on the training data one word at a time.

24.3.1 Attention What if the target RNN were conditioned on all of the hidden vectors from the source RNN, rather than just the last one? This would mitigate the shortcomings of nearby context bias and fixed context size limits, allowing the model to access any previous word equally well. One way to achieve this access is to concatenate all of the source RNN hidden vectors. However, this would cause a huge increase in the number of weights, with a concomitant

increase in computation time and potentially overfitting as well. Instead, we can take advantage of the fact that when the target RNN is generating the target one word at a time, it is likely that only a small part of the source is actually relevant to each target word.

Crucially, the target RNN must pay attention to different parts of the source for every word. Suppose a network is trained to translate English to Spanish. It is given the words “The front door is red” followed by an end of sentence marker, which means it is time to start outputting Spanish words. So ideally it should first pay attention to “The” and generate “La,” then pay attention to “door” and output “puerta,” and so on.

We can formalize this concept with a neural network component called attention, which can be used to create a “context-based summarization” of the source sentence into a fixeddimensional representation. The context vector ci contains the most relevant information for generating the next target word, and will be used as an additional input to the target RNN. A sequence-to-sequence model that uses attention is called an attentional sequence-tosequence model. If the standard target RNN is written as:

hi

=

RNN (hi

−1 ,

xi ) ,

Attention

Attentional sequence-to-sequence model

the target RNN for attentional sequence-to-sequence models can be written as:

hi

=

RNN (hi

−1 ,

[xi ; ci ])

where [xi ; ci ] is the concatenation of the input and context vectors, ci , defined as:

rij = hi−1 ⋅ sj aij = erij /( ci =

∑

∑ k

erik )

aij ⋅ sj

j

where hi−1 is the target RNN vector that is going to be used for predicting the word at timestep i, and sj is the output of the source RNN vector for the source word (or timestep) j. Both hi−1 and sj are d-dimensional vectors, where d is the hidden size. The value of rij is therefore the raw “attention score” between the current target state and the source word j. These scores are then normalized into a probability aij using a softmax over all source words. Finally, these probabilities are used to generate a weighted average of the source RNN vectors, ci (another d-dimensional vector). An example of an attentional sequence-to-sequence model is given in Figure 24.7(a) . There are a few important details to understand. First, the attention component itself has no learned weights and supports variable-length sequences on both the source and target side. Second, like most of the other neural network modeling techniques we’ve learned about, attention is entirely latent. The programmer does not dictate what information gets used when; the model learns what to use. Attention can also be combined with multilayer RNNs. Typically attention is applied at each layer in that case.

Figure 24.7

(a) Attentional sequence-to-sequence model for English-to-Spanish translation. The dashed lines represent attention. (b) Example of attention probability matrix for a bilingual sentence pair, with darker boxes representing higher values of aij . The attention probabilities sum to one over each column.

The probabilistic softmax formulation for attention serves three purposes. First, it makes attention differentiable, which is necessary for it to be used with back-propagation. Even though attention itself has no learned weights, the gradients still flow back through attention to the source and target RNNs. Second, the probabilistic formulation allows the model to capture certain types of long-distance contextualization that may have not been captured by the source RNN, since attention can consider the entire source sequence at once, and learn to keep what is important and ignore the rest. Third, probabilistic attention allows the network to represent uncertainty—if the network does not know exactly what source word to translate next, it can distribute the attention probabilities over several options, and then actually choose the word using the target RNN.

Unlike most components of neural networks, attention probabilities are often interpretable by humans and intuitively meaningful. For example, in the case of machine translation, the attention probabilities often correspond to the word-to-word alignments that a human would generate. This is shown in Figure 24.7(b) .

Sequence-to-sequence models are a natural for machine translation, but almost any natural language task can be encoded as a sequence-to-sequence problem. For example, a questionanswering system can be trained on input consisting of a question followed by a delimiter followed by the answer.

24.3.2 Decoding At training time, a sequence-to-sequence model attempts to maximize the probability of each word in the target training sentence, conditioned on the source and all of the previous target words. Once training is complete, we are given a source sentence, and our goal is to generate the corresponding target sentence. As shown in Figure 24.7 , we can generate the target one word at a time, and then feed back in the word that we generated at the next timestep. This procedure is called decoding.

Decoding

Greedy decoding

The simplest form of decoding is to select the highest probability word at each timestep and then feed this word as input to the next timestep. This is called greedy decoding because after each target word is generated, the system has fully committed to the hypothesis that it has produced so far. The problem is that the goal of decoding is to maximize the probability of the entire target sequence, which greedy decoding may not achieve. For example, consider using a greedy decoder to translate into Spanish the English sentence we saw before, The front door is red.

The correct translation is “La puerta de entrada es roja”—literally “The door of entry is red.” Suppose the target RNN correctly generates the first word La for The. Next, a greedy decoder might propose entrada for front. But this is an error—Spanish word order should put the noun puerta before the modifier. Greedy decoding is fast—it only considers one choice at each timestep and can do so quickly—but the model has no mechanism to correct mistakes.

We could try to improve the attention mechanism so that it always attends to the right word and guesses correctly every time. But for many sentences it is infeasible to guess correctly all the words at the start of the sentence until you have seen what’s at the end.

A better approach is to search for an optimal decoding (or at least a good one) using one of the search algorithms from Chapter 3 . A common choice is a beam search (see Section 4.1.3 ). In the context of MT decoding, beam search typically keeps the top k hypotheses at each stage, extending each by one word using the top k choices of words, then chooses the best k of the resulting k2 new hypotheses. When all hypotheses in the beam generate the special token, the algorithm outputs the highest scoring hypothesis. A visualization of beam search is given in Figure 24.8 . As deep learning models become more accurate, we can usually afford to use a smaller beam size. Current state-of-the-art neural MT models use a beam size of 4 to 8, whereas the older generation of statistical MT models would use a beam size of 100 or more.

Figure 24.8

Beam search with beam size of b = 2. The score of each word is the log-probability generated by the target RNN softmax, and the score of each hypothesis is the sum of the word scores. At timestep 3, the highest scoring hypothesis La entrada can only generate low-probability continuations, so it “falls off the beam.”

24.4 The Transformer Architecture The influential article “Attention is all you need” (Vaswani et al., 2018) introduced the transformer architecture, which uses a self-attention mechanism that can model longdistance context without a sequential dependency.

Transformer

Self-attention

24.4.1 Self-attention Previously, in sequence-to-sequence models, attention was applied from the target RNN to the source RNN. Self-attention extends this mechanism so that each sequence of hidden states also attends to itself—the source to the source, and the target to the target. This allows the model to additionally capture long-distance (and nearby) context within each sequence.

Self-attention

The most straightforward way of applying self-attention is where the attention matrix is directly formed by the dot product of the input vectors. However, this is problematic. The dot product between a vector and itself will always be high, so each hidden state will be biased towards attending to itself. The transformer solves this by first projecting the input into three different representations using three different weight matrices:

The query vector

q W x is the one being attended from, like the target in the standard i =

q i

attention mechanism.

Query vector

The key vector

k W x is the one being attended to, like the source in the basic i =

k i

attention mechanism.

Key vector

The value vector

v W x is the context that is being generated. i =

v i

Value vector

In the standard attention mechanism, the key and value networks are identical, but intuitively it makes sense for these to be separate representations. The encoding results of the ith word,

c , can be calculated by applying an attention mechanism to the projected i

vectors:

=( i⋅

aij

= erij /(

c

Multiheaded attention

q k

rij

j )/√d

∑

erik )

∑ v k

i =

aij ⋅

j

j,

where d is the dimension of k and q. Note that i and j are indexes in the same sentence, since we are encoding the context using self-attention. In each transformer layer, selfattention uses the hidden vectors from the previous layer, which initially is the embedding layer.

There are several details worth mentioning here. First of all, the self-attention mechanism is asymmetric, as rij is different from rji . Second, the scale factor

√d was added to improve

numerical stability. Third, the encoding for all words in a sentence can be calculated simultaneously, as the above equations can be expressed using matrix operations that can be computed efficiently in parallel on modern specialized hardware.

The choice of which context to use is completely learned from training examples, not prespecified. The context-based summarization, ci , is a sum over all previous positions in the sentence. In theory, and information from the sentence could appear in ci , but in practice, sometimes important information gets lost, because it is essentially averaged out over the whole sentence. One way to address that is called multiheaded attention. We divide the sentence up into m equal pieces and apply the attention model to each of the m pieces. Each piece has its own set of weights. Then the results are concatenated together to form ci . By concatenating rather than summing, we make it easier for an important subpiece to stand out.

Multiheaded attention

24.4.2 From self-attention to transformer Self-attention is only one component of the transformer model. Each transformer layer consists of several sub-layers. At each transformer layer, self-attention is applied first. The output of the attention module is fed through feedforward layers, where the same feedforward weight matrices are applied independently at each position. A nonlinear activation function, typically ReLU, is applied after the first feedforward layer. In order to address the potential vanishing gradient problem, two residual connections (are added into

the transformer layer. A single-layer transformer in shown in Figure 24.9 . In practice, transformer models usually have six or more layers. As with the other models that we’ve learned about, the output of layer i is used as the input to layer i + 1.

Figure 24.9

A single-layer transformer consists of self-attention, a feedforward network, and residual connections.

Positional embedding

The transformer architecture does not explicitly capture the order of words in the sequence, since context is modeled only through self-attention, which is agnostic to word order. To capture the ordering of the words, the transformer uses a technique called positional embedding. If our input sequence has a maximum length of n, then we learn n new embedding vectors—one for each word position. The input to the first transformer layer is the sum of the word embedding at position t plus the positional embedding corresponding to position t.

Figure 24.10  illustrates the transformer architecture for POS tagging, applied to the same sentence used in Figure 24.3 . At the bottom, the word embedding and the positional embeddings are summed to form the input for a three-layer transformer. The transformer produces one vector per word, as in RNN-based POS tagging. Each vector is fed into a final output layer and softmax layer to produce a probability distribution over the tags.

Figure 24.10

Using the transformer architecture for POS tagging.

In this section, we have actually only told half the transformer story: the model we described here is called the transformer encoder. It is useful for text classification tasks. The full transformer architecture was originally designed as a sequence-to-sequence model for machine translation. Therefore, in addition to the encoder, it also includes a transformer decoder. The encoder and decoder are nearly identical, except that the decoder uses a version of self-attention where each word can only attend to the words before it, since text is generated left-to-right. The decoder also has a second attention module in each transformer layer that attends to the output of the transformer encoder.

Transformer encoder

Transformer decoder

24.5 Pretraining and Transfer Learning Getting enough data to build a robust model can be a challenge. In computer vision (see Chapter 25 ), that challenge was addressed by assembling large collections of images (such as ImageNet) and hand-labeling them.

For natural language, it is more common to work with text that is unlabeled. The difference is in part due to the difficulty of labeling: an unskilled worker can easily label an image as “cat” or “sunset,” but it requires extensive training to annotate a sentence with part-ofspeech tags or parse trees. The difference is also due to the abundance of text: the Internet adds over 100 billion words of text each day, including digitized books, curated resources such as Wikipedia, and uncurated social media posts.

Projects such as Common Crawl provide easy access to this data. Any running text can be used to build n-gram or word embedding models, and some text comes with structure that can be helpful for a variety of tasks—for example, there are many FAQ sites with questionanswer pairs that can be used to train a question-answering system. Similarly, many Web sites publish side-by-side translations of texts, which can be used to train machine translation systems. Some text even comes with labels of a sort, such as review sites where users annotate their text reviews with a 5-star rating system.

We would prefer not to have to go to the trouble of creating a new data set every time we want a new NLP model. In this section, we introduce the idea of pretraining: a form of transfer learning (see Section 21.7.2 ) in which we use a large amount of shared generaldomain language data to train an initial version of an NLP model. From there, we can use a smaller amount of domain-specific data (perhaps including some labeled data) to refine the model. The refined model can learn the vocabulary, idioms, syntactic structures, and other linguistic phenomena that are specific to the new domain.

Pretraining

24.5.1 Pretrained word embeddings In Section 24.1 , we briefly introduced word embeddings. We saw that how similar words like banana and apple end up with similar vectors, and we saw that we can solve analogy problems with vector subtraction. This indicates that the word embeddings are capturing substantial information about the words.

In this section we will dive into the details of how word embeddings are created using an entirely unsupervised process over a large corpus of text. That is in contrast to the embeddings from Section 24.1 , which were built during the process of supervised part of speech tagging, and thus required POS tags that come from expensive hand annotation.

We will concentrate on one specific model for word embeddings, the GloVe (Global Vectors) model. The model starts by gathering counts of how many times each word appears within a window of another word, similar to the skip-gram model. First choose window size (perhaps 5 words) and let Xij be the number of times that words i and j cooccur within a window, and let Xi be the number of times word i co-occurs with any other word. Let Pij

=

Xij /Xi be the probability that word j appears in the context of word i. As

before, let Ei be the word embedding for word i. Part of the intuition of the GloVe model is that the relationship between two words can best be captured by comparing them both to other words. Consider the words ice and steam. Now consider the ratio of their probabilities of co-occurrence with another word, w, that is:

Pw ice /Pw steam . ,

,

When w is the word solid the ratio will be high (meaning solid applies more to ice) and when

w is the word gas it will be low (meaning gas applies more to steam). And when w is a noncontent word like the, a word like water that is equally relevant to both, or an equally irrelevant word like fashion, the ratio will be close to 1.

The GloVe model starts with this intuition and goes through some mathematical reasoning (Pennington et al., 2014) that converts ratios of probabilities into vector differences and dot products, eventually arriving at the constraint

E i Ek ⋅

′

= log

(Pij ) .

In other words, the dot product of two word vectors is equal to the log probability of their co-occurrence. That makes intuitive sense: two nearly-orthogonal vectors have a dot product close to 0, and two nearly-identical normalized vectors have a dot product close to 1. There is a technical complication wherein the GloVe model creates two word embedding vectors for each word, Ei and Ei′ ; computing the two and then adding them together at the end helps limit overfitting.

Training a model like GloVe is typically much less expensive than training a standard neural network: a new model can be trained from billions of words of text in a few hours using a standard desktop CPU.

It is possible to train word embeddings on a specific domain, and recover knowledge in that domain. For example, Tshitoyan et al. (2019) used 3.3 million scientific abstracts on the subject of material science to train a word embedding model. They found that, just as we saw that a generic word embedding model can answer “Athens is to Greece as Oslo is to what?” with “Norway,” their material science model can answer “NiFe is to ferromagnetic as IrMn is to what?” with “antiferromagnetic.”

Their model does not rely solely on co-occurrence of words; it seems to be capturing more complex scientific knowledge. When asked what chemical compounds can be classified as “thermoelectric” or “topological insulator,” their model is able to answer correctly. For example, CsAgGa2 Se4 never appears near “thermoelectric” in the corpus, but it does appear near “chalcogenide,” “band gap,” and “optoelectric,” which are all clues enabling it to be classified as similar to “thermoelectric.” Furthermore, when trained only on abstracts up to the year 2008 and asked to pick compounds that are “thermoelectric” but have not yet appeared in abstracts, three of the model’s top five picks were discovered to be thermoelectric in papers published between 2009 and 2019.

24.5.2 Pretrained contextual representations Word embeddings are better representations than atomic word tokens, but there is an important issue with polysemous words. For example, the word rose can refer to a flower or the past tense of rise. Thus, we expect to find at least two entirely distinct clusters of word contexts for rose: one similar to flower names such as dahlia, and one similar to upsurge. No single embedding vector can capture both of these simultaneously. Rose is a clear example of a word with (at least) two distinct meanings, but other words have subtle shades of

meaning that depend on context, such as the word need in you need to see this movie versus humans need oxygen to survive. And some idiomatic phrases like break the bank are better analyzed as a whole rather than as component words.

Therefore, instead of just learning a word-to-embedding table, we want to train a model to generate contextual representations of each word in a sentence. A contextual representation maps both a word and the surrounding context of words into a word embedding vector. In other words, if we feed this model the word rose and the context the gardener planted a rose bush, it should produce a contextual embedding that is similar (but not necessarily identical) to the representation we get with the context the cabbage rose had an unusual fragrance, and very different from the representation of rose in the context the river rose five feet.

Contextual representations

Figure 24.11  shows a recurrent network that creates contextual word embeddings—the boxes that are unlabeled in the figure. We assume we have already built a collection of noncontextual word embeddings. We feed in one word at a time, and ask the model to predict the next word. So for example in the figure at the point where we have reached the word “car,” the the RNN node at that time step will receive two inputs: the noncontextual word embedding for “car” and the context, which encodes information from the previous words “The red.” The RNN node will then output a contextual representation for “car.” The network as a whole then outputs a prediction for the next word, “is.” We then update the network’s weights to minimize the error between the prediction and the actual next word.

Figure 24.11

Training contextual representations using a left-to-right language model.

This model is similar to the one for POS tagging in Figure 24.5 , with two important differences. First, this model is unidirectional (left-to-right), whereas the POS model is bidirectional. Second, instead of predicting the POS tags for the current word, this model predicts the next word using the prior context. Once the model is built, we can use it to retrieve representations for words and pass them on to some other task; we need not continue to predict the next word. Note that computing a contextual representations always requires two inputs, the current word and the context.

24.5.3 Masked language models A weakness of standard language models such as n-gram models is that the contextualization of each word is based only on the previous words of the sentence. Predictions are made from left to right. But sometimes context from later in a sentence—for example, feet in the phrase rose five feet—helps to clarify earlier words.

One straightforward workaround is to train a separate right-to-left language model that contextualizes each word based on subsequent words in the sentence, and then concatenate the left-to-right and right-to-left representations. However, such a model fails to combine evidence from both directions.

Instead, we can use a masked language model (MLM). MLMs are trained by masking (hiding) individual words in the input and asking the model to predict the masked words.

For this task, one can use a deep bidirectional RNN or transformer on top of the masked sentence. For example, given the input sentence “The river__rose five feet” we can mask the middle word to get “The river five feet” and ask the model to fill in the blank.

Masked language model (MLM)

The final hidden vectors that correspond to the masked tokens are then used to predict the words that were masked—in this example, rose. During training a single sentence can be used multiple times with different words masked out. The beauty of this approach is that it requires no labeled data; the sentence provides its own label for the masked word. If this model is trained on a large corpus of text, it generates pretrained representations that perform well across a wide variety of NLP tasks (machine translation, question answering, summarization, grammaticality judgments, and others).

24.6 State of the art Deep learning and transfer learning have markedly advanced the state of the art for NLP—so much so that one commentator in 2018 declared that “NLP’s ImageNet moment has arrived” (Ruder, 2018). The implication is that just as a turning point occurred in 2012 for computer vision when deep learning systems produced surprising good results in the ImageNet competition, a turning point occurred in 2018 for NLP. The principal impetus for this turning point was the finding that transfer learning works well for natural language problems: a general language model can be downloaded and fine-tuned for a specific task.

It started with simple word embeddings from systems such WORD2VEC in 2013 and GloVe in 2014. Researchers can download such a model or train their own relatively quickly without access to supercomputers. Pretrained contextual representations, on the other hand, are orders of magnitude more expensive to train.

These models became feasible only after hardware advances (GPUs and TPUs) became widespread, and in this case researchers were grateful to be able to download models rather than having to spend the resources to train their own. The transformer model allowed for efficient training of much larger and deeper neural networks than was previously possible (this time due to software advances, not hardware). Since 2018, new NLP projects typically start with a pretrained transformer model.

Although these transformer models were trained to predict the next word in a text, they do a surprisingly good job at other language tasks. A ROBERTA model with some fine-tuning achieves state-of-the-art results in question answering and reading comprehension tests (Liu et al., 2019b). GPT-2, a transformer-like language model with 1.5 billion parameters trained on 40GB of Internet text, achieves good results on such diverse tasks as translation between French and English, finding referents of long-distance dependencies, and generalknowledge question answering, all without fine-tuning for the particular task. As Figure 24.14  illustrates, GPT-2 can generate fairly convincing text given just a few words as a prompt.

Figure 24.12

Masked language modeling: pretrain a bidirectional model—for example, a multilayer RNN—by masking input words and predicting only those masked words.

Figure 24.13

Questions from an 8th grade science exam that the ARISTO system can answer correctly using an ensemble of methods, with the most influential being a ROBERTA language model. Answering these questions requires knowledge about natural language, the structure of multiple-choice tests, commonsense, and science.

Figure 24.14

Example completion texts generated by the GPT-2 language model, given the prompts in bold. Most of the texts are quite fluent English, at least locally. The final example demonstrates that sometimes the model just breaks down.

As an example state-of-the-art NLP system, ARISTO (Clark et al., 2019) achieved a score of 91.6% on an 8th grade multiple-choice science exam (see Figure 24.13 ). ARISTO consists of an ensemble of solvers: some use information retrieval (similar to a web search engine), some do textual entailment and qualitative reasoning, and some use large transformer language models. It turns out that ROBERTA, by itself, scores 88.2% on the test. ARISTO also scores 83% on the more advanced 12th grade exam. (A score of 65% is considered “meeting the standards” and 85% is “meeting the standards with distinction”.)

There are limitations of ARISTO. It deals only with multiple-choice questions, not essay questions, and it can neither read nor generate diagrams.1

1 It has been pointed out that in some multiple-choice exams, it is possible to get a good score even without looking at the questions, because there are tell-tale signs in the incorrect answers (Gururangan et al., 2018). That seems to be true for visual question answering as well (Chao et al., 2018).

T5 (the Text-to-Text Transfer Transformer) is designed to produce textual responses to various kinds of textual input. It includes a standard encoder–decoder transformer model, pretrained on 35 billion words from the 750 GB Colossal Clean Crawled Corpus (C4). This unlabeled training is designed to give the model generalizable linguistic knowledge that will be useful for multiple specific tasks. T5 is then trained for each task with input consisting of the task name, followed by a colon and some content. For example, when given “translate English to German: That is good,” it produces as output “Das ist gut.” For some tasks, the input is marked up; for example in the Winograd Schema Challenge, the input highlights a pronoun with an ambiguous referent. Given the input “referent: The city councilmen refused the demonstrators a permit because they feared violence,” the correct response is “The city councilmen” (not “the demonstrators”).

Much work remains to be done to improve NLP systems. One issue is that transformer models rely on only a narrow context, limited to a few hundred words. Some experimental approaches are trying to extend that context; the Reformer system (Kitaev et al., 2020) can handle context of up to a million words.

Recent results have shown that using more training data results in better models—for example, ROBERTA achieved state-of-the-art results after training on 2.2 trillion words. If using more textual data is better, what would happen if we included other types of data: structured databases, numerical data, images, and video? We would need a breakthrough in hardware processing speeds to train on a large corpus of video, and we may need several breakthroughs in AI as well.

The curious reader may wonder why we learned about grammars, parsing, and semantic interpretation in the previous chapter, only to discard those notions in favor of purely datadriven models in this chapter? At present, the answer is simply that the data-driven models are easier to develop and maintain, and score better on standard benchmarks, compared to the hand-built systems that can be constructed using a reasonable amount of human effort

with the approaches described in Chapter 23 . It may be that transformer models and their relatives are learning latent representations that capture the same basic ideas as grammars and semantic information, or it may be that something entirely different is happening within these enormous models; we simply don’t know. We do know that a system that is trained with textual data is easier to maintain and to adapt to new domains and new natural languages than a system that relies on hand-crafted features.

It may also be the case that future breakthroughs in explicit grammatical and semantic modeling will cause the pendulum to swing back. Perhaps more likely is the emergence of hybrid approaches that combine the best concepts from both chapters. For example, Kitaev and Klein (2018) used an attention mechanism to improve a traditional constituency parser, achieving the best result ever recorded on the Penn Treebank test set. Similarly, Ringgaard et al. (2017) demonstrate how a dependency parser can be improved with word embeddings and a recurrent neural network. Their system, SLING, parses directly into a semantic frame representation, mitigating the problem of errors building up in a traditional pipeline system.

There is certainly room for improvement: not only do NLP systems still lag human performance on many tasks, but they do so after processing thousands of times more text than any human could read in a lifetime. This suggests that there is plenty of scope for new insights from linguists, psychologists, and NLP researchers.

Summary The key points of this chapter are as follows:

Continuous representations of words with word embeddings are more robust than discrete atomic representations, and can be pretrained using unlabeled text data. Recurrent neural networks can effectively model local and long-distance context by retaining relevant information in their hidden-state vectors. Sequence-to-sequence models can be used for machine translation and text generation problems. Transformer models use self-attention and can model long-distance context as well as local context. They can make effective use of hardware matrix multiplication. Transfer learning that includes pretrained contextual word embeddings allows models to be developed from very large unlabeled corpora and applied to a range of tasks. Models that are pretrained to predict missing words can handle other tasks such as question answering and textual entailment, after fine-tuning for the target domain.

Bibliographical and Historical Notes The distribution of words and phrases in natural language follow Zipf’s Law (Zipf, 1935,

n

n

1949): the frequency of the th most popular word is roughly inversely proportional to . That means we have a data sparsity problem: even with billions of words of training data, we are constantly running into novel words and phrases that were not seen before.

Generalization to novel words and phrases is aided by representations that capture the basic insight that words with similar meanings appear in similar contexts. Deerwester et al. (1990) projected words into low-dimensional vectors by decomposing the co-occurrence matrix formed by words and the documents the words appear in. Another possibility is to treat the surrounding words—say, a 5-word window—as context. Brown et al. (1992) grouped words into hierarchical clusters according to the bigram context of words; this has proven to be effective for tasks such as named entity recognition (Turian et al., 2010). The WORD2VEC system (Mikolov et al., 2013) was the first significant demonstration of the advantages of word embeddings obtained from training neural networks. The GloVe word embedding vectors (Pennington et al., 2014) were obtained by operating directly on a word cooccurrence matrix obtained from billions of words of text. Levy and Goldberg (2014) explain why and how these word embeddings are able to capture linguistic regularities.

Bengio et al. (2003) pioneered the use of neural networks for language models, proposing to combine “(1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations.” Mikolov et al. (2010) demonstrated the use of RNNs for modeling local context in language models. Jozefowicz et al., (2016) showed how an RNN trained on a billion words can outperform

n

carefully hand-crafted -gram models. Contextual representations for words were emphasized by Peters et al. (2018), who called them ELMO (Embeddings from Language Models) representations.

Note that some authors compare language models by measuring their perplexity. The perplexity of a probability distribution is 2H , where

H is the entropy of the distribution (see

Section 19.3.3 ). A language model with lower perplexity is, all other things being equal, a better model. But in practice, all other things are rarely equal. Therefore it is more informative to measure performance on a real task rather than relying on perplexity.

Perplexity

Howard and Ruder (2018) describe the ULMFIT (Universal Language Model Fine-tuning) framework, which makes it easier to fine-tune a pretrained language model without requiring a vast corpus of target-domain documents. Ruder et al. (2019) give a tutorial on transfer learning for NLP.

Mikolov et al. (2010) introduced the idea of using RNNs for NLP, and Sutskever et al. (2015) introduced the idea of sequence to sequence learning with deep networks. Zhu et al. (2017) and (Liu et al. 2018b) showed that an unsupervised approach works, and makes data collection much easier. It was soon found that these kinds of models could perform surprisingly well at a variety of tasks, for example, image captioning (Karpathy and Fei-Fei, 2015; Vinyals et al., 2017b).

Devlin et al. (2018) showed that transformer models pretrained with the masked language modeling objective can be directly used for multiple tasks. The model was called BERT (Bidirectional Encoder Representations from Transformers). Pretrained BERT models can be fine-tuned for particular domains and particular tasks, including question answering, named entity recognition, text classification, sentiment analysis, and natural language inference.

The XLNET system (Yang et al., 2019) improves on BERT by eliminating a discrepancy between the pretraining and fine-tuning. The ERNIE 2.0 framework (Sun et al., 2019) extracts more from the training data by considering sentence order and the presence of named entities, rather than just co-occurrence of words, and was shown to outperform BERT and XLNET. In response, researchers revisited and improved on BERT: the ROBERTA system (Liu et al., 2019b) used more data and different hyperparameters and training procedures, and found that it could match XLNET. The Reformer system (Kitaev et al., 2020) extends the range of the context that can be considered all the way up to a million words. Meanwhile, ALBERT (A Lite BERT) went in the other direction, reducing the number of parameters from 108 million to 12 million (so as to fit on mobile devices) while maintaining high accuracy.

The XLM system (Lample and Conneau, 2019) is a transformer model with training data from multiple languages. This is useful for machine translation, but also provides more

robust representations for monolingual tasks. Two other important systems, GPT-2 (Radford et al., 2019) and T5 (Raffel et al., 2019), were described in the chapter. The later paper also introduced the 35 billion word Colossal Clean Crawled Corpus (C4).

Various promising improvements on pretraining algorithms have been proposed (Yang et al., 2019; Liu et al., 2019b). Pretrained contextual models are described by Peters et al. (2018) and Dai and Le (2016).

The GLUE (General Language Understanding Evaluation) benchmark, a collection of tasks and tools for evaluating NLP systems, was introduced by Wang et al. (2018a). Tasks include question answering, sentiment analysis, textual entailment, translation, and parsing. Transformer models have so dominated the leaderboard (the human baseline is way down at ninth place) that a new version, SUPERGLUE (Wang et al., 2019), was introduced with tasks that are designed to be harder for computers, but still easy for humans.

At the end of 2019, T5 was the overall leader with a score of 89.3, just half a point below the human baseline of 89.8. On three of the ten tasks, T5 actually exceeds human performance: yes/no question answering (such as “Is France the same time zone as the UK?”) and two reading comprehension tasks involving answering questions after reading either a paragraph or a news article.

Machine translation is a major application of language models. In 1933, Petr Troyanskii received a patent for a “translating machine,” but there were no computers available to implement his ideas. In 1947, Warren Weaver, drawing on work in cryptography and information theory, wrote to Norbert Wiener: “When I look at an article in Russian, I say: ‘This is really written in English, but it has been coded in strange symbols. I will now proceed to decode.”’ The community proceeded to try to decode in this way, but they didn’t have sufficient data and computing resources to make the approach practical.

In the 1970s that began to change, and the SYSTRAN system (Toma, 1977) was the first commercially successful machine translation system. SYSTRAN relied on lexical and grammatical rules hand-crafted by linguists as well as on training data. In the 1980s, the community embraced purely statistical models based on frequency of words and phrases (Brown et al., 1988; Koehn, 2009). Once training sets reached billions or trillions of tokens (Brants et al., 2007), this yielded systems that produced comprehensible but not fluent

results (Och and Ney, 2004; Zollmann et al., 2008). Och and Ney (2002) show how discriminative training led to an advance in machine translation in the early 2000s.

Sutskever et al. (2015) first showed that it is possible to learn an end-to-end sequence-tosequence neural model for machine translation. Bahdanau et al. (2015) demonstrated the advantage of a model that jointly learns to align sentences in the source and target language and to translate between the languages. Vaswani et al. (2018) showed that neural machine translation systems can further be improved by replacing LSTMs with transformer architectures, which use the attention mechanism to capture context. These neural translation systems quickly overtook statistical phrase-based methods, and the transformer architecture soon spread to other NLP tasks.

Research on question answering was facilitated by the creation of SQUAD, the first largescale data set for training and testing question-answering systems (Rajpurkar et al., 2016). Since then, a number of deep learning models have been developed for this task (Seo et al., 2017; Keskar et al., 2019). The ARISTO system (Clark et al., 2019) uses deep learning in conjunction with an ensemble of other tactics. Since 2018, the majority of questionanswering models use pretrained language representations, leading to a noticeable improvement over earlier systems.

Natural language inference is the task of judging whether a hypothesis (dogs need to eat) is entailed by a premise (all animals need to eat). This task was popularized by the PASCAL Challenge (Dagan et al., 2005). Large-scale data sets are now available (Bowman et al., 2015; Williams et al., 2018). Systems based on pretrained models such as ELMO and BERT currently provide the best performance on language inference tasks.

The Conference on Computational Natural Language Learning (CoNLL) focuses on learning for NLP. All the conferences and journals mentioned in Chapter 23  now include papers on deep learning, which now has a dominant position in the field of NLP.

Chapter 25 Computer Vision In which we connect the computer to the raw, unwashed world through the eyes of a camera.

Most animals have eyes, often at significant cost: eyes take up a lot of space; use energy; and are quite fragile. This cost is justified by the immense value that eyes provide. An agent that can see can predict the future—it can tell what it might bump into; it can tell whether to attack or to flee or to court; it can guess whether the ground ahead is swampy or firm; and it can tell how far away the fruit is. In this chapter, we describe how to recover information from the flood of data that comes from eyes or cameras.

25.1 Introduction Vision is a perceptual channel that accepts a stimulus and reports some representation of the world. Most agents that use vision use passive sensing—they do not need to send out light to see. In contrast, active sensing involves sending out a signal such as radar or ultrasound, and sensing a reflection. Examples of agents that use active sensing include bats (ultrasound), dolphins (sound), abyssal fishes (light), and some robots (light, sound, radar). To understand a perceptual channel, one must study both the physical and statistical phenomena that occur in sensing and what the perceptual process should produce. We concentrate on vision in this chapter, but robots in the real world use a variety of sensors to perceive sound, touch, distance, temperature, global position, and acceleration.

A feature is a number obtained by applying simple computations to an image. Very useful information can be obtained directly from features. The wumpus agent had five sensors, each of which extracted a single bit of information. These bits, which are features, could be interpreted directly by the program. As another example, many flying animals compute a simple feature that gives a good estimate of time to contact with a nearby object; this feature can be passed directly to muscles that control steering or wings, allowing very fast changes of direction. This feature extraction approach emphasizes simple, direct computations applied to sensor responses.

Feature

The model-based approach to vision uses two kinds of models. An object model could be the kind of precise geometric model produced by computer aided design systems. It could also be a vague statement about general properties of objects, for example, the claim that all faces viewed in low resolution look approximately the same. A rendering model describes the physical, geometric, and statistical processes that produce the stimulus from the world. While rendering models are now sophisticated and exact, the stimulus is usually ambiguous. A white object under low light may look like a black object under intense light. A small,

nearby object may look the same as a large, distant object. Without additional evidence, we cannot tell if what we see is a toy Godzilla tearing up a toy building, or a real monster destroying a real building.

There are two main ways to manage these ambiguities. First, some interpretations are more likely than others. For example, we can be confident that the picture doesn’t show a real Godzilla destroying a real building, because there are no real Godzillas. Second, some ambiguities are insignificant. For example, distant scenery may be trees or may be a flat painted surface. For most applications, the difference is unimportant, because the objects are far away and so we will not bump into them or interact with them soon.

The two core problems of computer vision are reconstruction, where an agent builds a model of the world from an image or a set of images, and recognition, where an agent draws distinctions among the objects it encounters based on visual and other information. Both problems should be interpreted very broadly. Building a geometric model from images is obviously reconstruction (and solutions are very valuable), but sometimes we need to build a map of the different textures on a surface, and this is reconstruction, too. Attaching names to objects that appear in an image is clearly recognition. Sometimes we need to answer questions like: Is it asleep? Does it eat meat? Which end has teeth? Answering these questions is recognition, too.

Reconstruction

Recognition

The last thirty years of research have produced powerful tools and methods for addressing these core problems. Understanding these methods requires an understanding of the processes by which images are formed.

25.2 Image Formation Imaging distorts the appearance of objects. A picture taken looking down a long straight set of railway tracks will suggest that the rails converge and meet. If you hold your hand in front of your eye, you can block out the moon, even though the moon is larger than your hand (this works with the sun too, but you could damage your eyes checking it). If you hold a book flat in front of your face and tilt it backward and forward, it will seem to shrink and grow in the image. This effect is known as foreshortening (Figure 25.1 ). Models of these effects are essential for building competent object recognition systems and also yield powerful cues for reconstructing geometry.

Figure 25.1

Geometry in the scene appears distorted in images. Parallel lines appear to meet, like the railway tracks in a desolate town. Buildings that have right angles in the real world scene have distorted angles in the image.

25.2.1 Images without lenses: The pinhole camera Image sensors gather light scattered from objects in a scene and create a two-dimensional (2D) image. In the eye, these sensors consist of two types of cell: There are about 100 million rods, which are sensitive to light at a wide range of wavelengths, and 5 million cones. Cones, which are essential for color vision, are of three main types, each of which is sensitive to a different set of wavelengths. In cameras, the image is formed on an image plane. In film cameras the image plane is coated with silver halides. In digital cameras, the image plane is subdivided into a grid of a few million pixels.

Scene

Image

Pixels

Sensor

We refer to the whole image plane as a sensor, but each pixel is an individual tiny sensor— usually a charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS). Each photon arriving at the sensor produces an electrical effect, whose strength depends on the wavelength of the photon. The output of the sensor is the sum of all these effects in some time window, meaning that image sensors report a weighted average of the intensity of light arriving at the sensor. The average is over wavelength, direction from which photons can arrive, time, and the area of the sensor.

To see a focused image, we must ensure that all the photons arriving at a sensor come from approximately the same spot on the object in the world. The simplest way to form a focused image is to view stationary objects with a pinhole camera, which consists of a pinhole opening,

O, at the front of a box, and an image plane at the back of the box (Figure 25.2 ).

The opening is called the aperture. If the pinhole is small enough, each tiny sensor in the image plane will see only photons that come from approximately the same spot on the object, and so the image is focused. We can form focused images of moving objects with a pinhole camera, too, as long as the object moves only a short distance in the sensors’ time window. Otherwise, the image of the moving object is defocused, an effect known as motion blur. One way to manipulate the time window is to open and close the pinhole.

Figure 25.2

Each light sensitive element at the back of a pinhole camera receives light that passes through the pinhole from a small range of directions. If the pinhole is small enough, the result is a focused image behind the pinhole. The process of projection means that large, distant objects look the same as smaller, nearby objects—the point ' in the image plane could have come from a nearby toy tower at point or from a

P

distant real tower at point

P

Q.

Pinhole camera

Aperture

Motion blur

Pinhole cameras make it easy to understand the geometric model of camera behavior (which is more complicated—but similar—with most other imaging devices). We will use a

O, and will consider a point P in the scene, with coordinates (X, Y , Z ). P gets projected to the point P in the image plane with coordinates (x, y, z). If f is the focal length—the distance from the pinhole to the image three-dimensional (3D) coordinate system with the origin at

'

plane—then by similar triangles, we can derive the following equations:

x X −y Y f = Z, f = Z

−

⇒

x=

fX Z ,

−

y=

fY Z .

−

Focal length

These equations define an image formation process known as perspective projection. Note that the Z in the denominator means that the farther away an object is, the smaller its image will be. Also, note that the minus signs mean that the image is inverted, both left–right and up–down, compared with the scene.

Perspective projection

Perspective imaging has a number of geometric effects. Distant objects look small. Parallel lines converge to a point on the horizon. (Think of railway tracks, Figure 25.1 .) A line in the scene in the direction (U , V , W ) and passing through the point (X0 , Y0 , Z0 ) can be described as the set of points (X0 + λU , Y0 + λV , Z0 + λW ), with λ varying between −∞ and +∞. Different choices of (X0 , Y0 , Z0 ) yield different lines parallel to one another. The projection of a point Pλ from this line onto the image plane is given by

Pλ = ( f As λ → ∞ or

X0 + λU Y0 + λV ,f ). Z0 + λW Z0 + λW

λ → −∞, this becomes P∞ = (fU /W , fV /W ) if W ≠ 0. This means that two parallel lines leaving different points in space will converge in the image—for large λ, the image points are nearly the same, whatever the value of (X0 , Y0 , Z0 ) (again, think railway tracks, Figure 25.1 ). We call P∞ the vanishing point associated with the family of straight lines with direction (U , V , W ). Lines with the same direction share the same vanishing point.

Vanishing point

25.2.2 Lens systems Pinhole cameras can focus light well, but because the pinhole is small, only a little light will get in, and the image will be dark. Over a short period of time, only a few photons will hit each point on the sensor, so the signal at each point will be dominated by random fluctuations; we say that a dark film image is grainy and a dark digital image is noisy; either way, the image is of low quality.

Enlarging the hole (the aperture) will make the image brighter by collecting more light from a wider range of directions. However, with a larger aperture the light that hits a particular point in the image plane will have come from multiple points in the real world scene, so the image will be defocused. We need some way to refocus the image.

Vertebrate eyes and modern cameras use a lens system—a single piece of transparent tissue in the eye and a system of multiple glass lens elements in a camera. In Figure 25.3  we see that light from the tip of the candle spreads out in all directions. A camera (or an eye) with a lens captures all the light that hits anywhere on the lens—a much larger area than a pinhole —and focuses all that light to a single point on the image plane. Light from other parts of the candle would similarly be gathered and focused to other points on the image plane. The result is a brighter, less noisy, focused image.

Figure 25.3

Lenses collect the light leaving a point in the scene (here, the tip of the candle flame) in a range of directions, and steer all the light to arrive at a single point on the image plane. Points in the scene near the focal plane—within the depth of field—will be focused properly. In cameras, elements of the lens system move to change the focal plane, whereas in the eye, the shape of the lens is changed by specialized muscles.

Lens

Lens systems do not focus all the light from everywhere in the real world; the lens design restricts them to focusing light only from points that lie within a range of Z depths from the lens. The center of this range—where focus is sharpest—is called the focal plane, and the range of depths for which focus remains sharp enough is called the depth of field. The larger the lens aperture (opening), the smaller the depth of field.

Focal plane

Depth of field

What if you want to focus on something at a different distance? To move the focal plane, the lens elements in a camera can move back and forth, and the lens in the eye can change shape—but with age the eye lens tends to harden, making it less able to adjust focal distances, and requiring many humans to augment their vision with external lens— eyeglasses.

25.2.3 Scaled orthographic projection The geometric effects of perspective imaging aren’t always pronounced. For example, windows on a building across the street look much smaller than ones right nearby, but two windows that are next to each other will have about the same size even though one is slightly farther away. We have the option to handle the windows with a simplified model called scaled orthographic projection, rather than perspective projection. If the depth

Z ≪ Z , then the perspective scaling factor f /Z can be approximated by a constant s = f /Z . The equations for projection from the scene coordinates (X, Y , Z ) to the image plane become x = sX and y = sY . all points on an object fall within the range

Z

0

Z

Z of

± Δ , with Δ

0

0

Foreshortening still occurs in the scaled orthographic projection model, because it is caused by the object tilting away from the view.

Scaled orthographic projection

25.2.4 Light and shading The brightness of a pixel in the image is a function of the brightness of the surface patch in the scene that projects to the pixel. For modern cameras, this function is linear for middling intensities of light, but has pronounced nonlinearities for darker and brighter illumination. We will use a linear model. Image brightness is a strong, if ambiguous, cue to both the shape and the identity of objects. The ambiguity occurs because there are three factors that contribute to the amount of light that comes from a point on an object to the image: the overall intensity of ambient light); whether the point is facing the light or is in shadow); and the amount of light reflected from the point.

Ambient light

Reflection

People are surprisingly good at disambiguating brightness—they usually can tell the difference between a black object in bright light and a white object in shadow, even if both have the same overall brightness. However, people sometimes get shading and markings mixed up—-a streak of dark makeup under a cheekbone will often look like a shading effect, making the face look thinner.

Most surfaces reflect light by a process of diffuse reflection. Diffuse reflection scatters light evenly across the directions leaving a surface, so the brightness of a diffuse surface doesn’t depend on the viewing direction. Most cloth has this property, as do most paints, rough wooden surfaces, most vegetation, and rough stone or concrete.

Diffuse reflection

Specular reflection causes incoming light to leave a surface in a lobe of directions that is determined by the direction the light arrived from. A mirror is one example. What you see depends on the direction in which you look at the mirror. In this case, the lobe of directions is very narrow, which is why you can resolve different objects in a mirror.

Specular reflection

For many surfaces, the lobe is broader. These surfaces display small bright patches, usually called specularities. As the surface or the light moves, the specularities move, too. Away from these patches, the surface behaves as if it is diffuse. Specularities are often seen on metal surfaces, painted surfaces, plastic surfaces, and wet surfaces. These are easy to identify, because they are small and bright (Figure 25.4 ). For almost all purposes, it is enough to model all surfaces as being diffuse with specularities.

Figure 25.4

This photograph illustrates a variety of illumination effects. There are specularities on the stainless steel cruet. The onions and carrots are bright diffuse surfaces because they face the light direction. The shadows appear at surface points that cannot see the light source at all. Inside the pot are some dark diffuse surfaces where the light strikes at a tangential angle. (There are also some shadows inside the pot.)

Photo by Ryman Cabannes/Image Professionals GmbH/Alamy Stock Photo.

Specularities

The main source of illumination outside is the sun, whose rays all travel parallel to one another in a known direction because it is so far away. We model this behavior with a

distant point light source. This is the most important model of lighting, and is quite effective for indoor scenes as well as outdoor scenes. The amount of light collected by a surface patch in this model depends on the angle θ between the illumination direction and the normal (perpendicular) to the surfaces (Figure 25.5 ).

Figure 25.5

Two surface patches are illuminated by a distant point source, whose rays are shown as light arrows. Patch A is tilted away from the source (θ is close to 90∘ ) and collects less energy, because it cuts fewer light rays per unit surface area. Patch B, facing the source (θ is close to 0∘ ), collects more energy.

Distant point light source

A diffuse surface patch illuminated by this model will reflect some fraction of the light it collects, given by the diffuse albedo. For practical surfaces, this lies in the range 0.05-0.95. Lambert’s cosine law states the brightness of a diffuse patch is given by I

Diffuse albedo

=

ρI0 cos θ,

Lambert’s cosine law

Shadow

where I0 is the intensity of the light source, θ is the angle between the light source direction and the surface normal, and ρ is the diffuse albedo. This law predicts that bright image pixels come from surface patches that face the light directly and dark pixels come from patches that see the light only tangentially, so that the shading on a surface provides some shape information. If the surface cannot see the source, then it is in shadow. Shadows are very seldom a uniform black, because the shadowed surface usually receives some light from other sources. Outdoors, the most important source other than the sun is the sky, which is quite bright. Indoors, light reflected from other surfaces illuminates shadowed patches. These interreflections can have a significant effect on the brightness of other surfaces, too. These effects are sometimes modeled by adding a constant ambient illumination term to the predicted intensity.

Interreflections

Ambient illumination

25.2.5 Color Fruit is a bribe that a tree offers to animals to carry its seeds around. Trees that can signal when this bribe is ready have an advantage, as do animals that can read these signals. As a result, most fruits start green, and turn red or yellow when ripe, and most fruit-eating animals can see these color changes. Generally, light arriving at the eye has different amounts of energy at different wavelengths, and is represented by a spectral energy density.

Cameras and the human vision system respond to light at wavelengths ranging from about 380nm (violet) to about 750nm (red). In color imaging systems, there are different types of receptor that respond more or less strongly to different wavelengths. In humans, the sensation of color occurs when the vision system compares the responses of receptors near each other on the retina. Animal color vision systems typically have relatively few types of receptor, and so represent relatively little of the detail in the spectral energy density function (some animals have only one type of receptor; some have as many as six types). Human color vision is produced by three types of receptor. Most color camera systems use only three types of receptor, too, because the images are produced for humans, but some specialized systems can produce very detailed measurements of the spectral energy density.

Because most humans have three types of color-sensitive receptors, the principle of trichromacy applies. This idea, first proposed by Thomas Young in 1802, states that a human observer can match the visual appearance of any spectral energy density, however complex, by mixing appropriate amounts of just three primaries. Primaries are colored light sources, chosen so that no mixture of any two will match the third. A common choice is to have one red primary, one green, and one blue, abbreviated as RGB. Although a given colored object may have many component frequencies of light, we can match the color by mixing just the three primaries, and most people will agree on the proportions of the mixture. That means we can represent color images with just three numbers per pixel—the RGB values.

Principle of trichromacy

Primaries

RGB

For most computer vision applications, it is accurate enough to model a surface as having three different (RGB) diffuse albedos and to model light sources as having three (RGB) intensities. We then apply Lambert’s cosine law to each to get red, green, and blue pixel values. This model predicts, correctly, that the same surface will produce different colored image patches under different colored lights. In fact, human observers are quite good at ignoring the effects of different colored lights and appear to estimate the color the surface would have under white light, an effect known as color constancy.

Color constancy

25.3 Simple Image Features Light reflects off objects in the scene to form an image consisting of, say, twelve million three-byte pixels. As with all sensors there will be noise in the image, and in any case there is a lot of data to deal with. The way to get started analyzing this data is to produce simplified representations that expose what’s important, but reduce detail. Much current practice learns these representations from data. But there are four properties of images and video that are particularly general: edges, texture, optical flow and segmentation into regions.

An edge occurs where there is a big difference in pixel intensity across part of an image. Building representations of edges involves local operations on an image—you need to compare a pixel value to some values nearby—and doesn’t require any knowledge about what is in the image. Thus, edge detection can come early in the pipeline of operations and we call it an “early” or “low-level” operation.

The other operations require handling a larger area of the image. For example, a texture description applies to a pool of pixels—to say “stripey,” you need to see some stripes. Optical flow represents where pixels move to from one image in a sequence to the next, and this can cover a larger area. Segmentation cuts an image into regions of pixels that naturally belong together, and doing so requires looking at the whole region. Operations like this are sometimes referred to as “mid-level” operations.

25.3.1 Edges Edges are straight lines or curves in the image plane across which there is a “significant” change in image brightness. The goal of edge detection is to abstract away from the messy, multi-megabyte image and towards a more compact, abstract representation, as in Figure 25.6 . Effects in the scene very often result in large changes in image intensity, and so produce edges in the image. Depth discontinuities (labeled 1 in the figure) can cause edges because when you cross the discontinuity, the color typically changes. When the surface normal changes (labeled 2 in the figure), the image intensity often changes. When the surface reflectance changes (labeled 3), the image intensity often changes. Finally, a shadow (labeled 4) is a discontinuity in illumination that causes an edge in the image, even though

there is not an edge in the object. Edge detectors can’t disentangle the cause of the discontinuity, which is left to later processing.

Figure 25.6

Different kinds of edges: (1) depth discontinuities; (2) surface orientation discontinuities; (3) reflectance discontinuities; (4) illumination discontinuities (shadows).

Edges

Finding edges requires care. Figure 25.7  (top) shows a one-dimensional crosssection of an image perpendicular to an edge, with an edge at x = 50.

Figure 25.7

Top: Intensity profile I (x) along a one-dimensional section across a step edge. Middle: The derivative of intensity, I '(x). Large values of this function correspond to edges, but the function is noisy. Bottom: The derivative of a smoothed version of the intensity. The noisy candidate edge at x = 75 has disappeared.

Noise

You might differentiate the image and look for places where the magnitude of the derivative I '(x) is large. This almost works, but in Figure 25.7  (middle), we see that although there is

a peak at x = 50, there are also subsidiary peaks at other locations (e.g., x = 75) that could be mistaken for true edges. These arise because of the presence of “noise” in the image. Noise here means changes to the value of a pixel that don’t have to do with an edge. For example, there could be thermal noise in the camera; there could be scratches on the object surface that change the surface normal at the finest scale; there could be minor variations in the surface albedo; and so on. Each of these effects make the gradient look big, but don’t

mean that an edge is present. If we “smooth” the image first, the spurious peaks are diminished, as we see in Figure 25.7  (bottom).

Smoothing involves using surrounding pixels to suppress noise. We will predict the “true” value of our pixel as a weighted sum of nearby pixels, with more weight for the closest pixels. A natural choice of weights is a Gaussian filter. Recall that the zero-mean Gaussian function with standard deviation

G σ ( x) =

σ is

1 √2

πσ

e

−

1 Gσ (x, y) = 2πσ e 2

x /2σ 2

−(

2

in one dimension, or

x +y )/2σ 2

2

2

in two dimensions.

Gaussian filter

I x , y ) with the sum, over all (x, y) pixels, of I (x, y)Gσ (d), where d is the distance from (x , y ) to (x, y). This kind of

Applying a Gaussian filter means replacing the intensity (

0

0

0

0

weighted sum is so common that there is a special name and notation for it. We say that the function

h is the convolution of two functions f and g (denoted as h = f ∗ g) if we have h ( x) =

∑ fugx u ∑ ∑ fuvgx +∞

u=−∞

h ( x, y ) =

+∞

( ) (

−

)

in one dimension, or

+∞

u=−∞ v=−∞

( , ) (

−

u, y − v )

in two dimensions.

Convolution

So the smoothing function is achieved by convolving the image with the Gaussian,

I ∗ Gσ . A

σ of 1 pixel is enough to smooth over a small amount of noise, whereas 2 pixels will smooth a larger amount, but at the loss of some detail. Because the Gaussian’s influence fades

rapidly with distance, in practice we can replace the ±∞ in the sums with something like

σ

±3 .

We have a chance to make an optimization here: we can combine the smoothing and the

f and g, the derivative of the convolution, (f ∗ g) , is equal to the convolution with the derivative, f ∗ (g) edge finding into a single operation. It is a theorem that for any functions '

. So rather than smoothing the image and then differentiating, we can just convolve the image with the derivative of the Gaussian smoothing function,

Gσ . We then mark as edges ′

those peaks in the response that are above some threshold, chosen to eliminate spurious peaks due to noise.

There is a natural generalization of this algorithm from one-dimensional crosssections to

θ

general 2D images. In two dimensions edges may be at any angle . Considering the image

xy

brightness as a scalar function of the variables , , its gradient is a vector

∇

I=

⎛ xI ⎞ ⎝ Iy ⎠ ∂

∂

∂

∂

Edges correspond to locations in images where the brightness undergoes a sharp change,

I

and thus the magnitude of the gradient, ∥∇ ∥ should be large at an edge point. When the image gets brighter or darker, the gradient vector at each point gets longer or shorter, but the direction of the gradient

I = ( cos θ ) sin θ ∥∇I ∥ ∇

does not change. This gives us a

θ = θ(x, y) at every pixel, which defines the edge

orientation at that pixel. This feature is often useful, because it does not depend on image intensity.

Orientation

'

As you might expect from the discussion on detecting edges in one-dimensional signals, to form the gradient, we don’t actually compute

∇I , but rather I ∗ Gσ , after smoothing the

image by convolving it with a Gaussian. A property of convolutions is that this is equivalent to convolving the image with the partial derivatives of the Gaussian. Once we have computed the gradient, we can obtain edges by finding edge points and linking them together. To tell whether a point is an edge point, we must look at other points a small distance forward and back along the direction of the gradient. If the gradient magnitude at one of these points is larger, then we could get a better edge point by shifting the edge curve very slightly. Furthermore, if the gradient magnitude is too small, the point cannot be an edge point. So at an edge point, the gradient magnitude is a local maximum along the direction of the gradient, and the gradient magnitude is above a suitable threshold.

Once we have marked edge pixels by this algorithm, the next stage is to link those pixels that belong to the same edge curves. This can be done by assuming that any two neighboring pixels that are both edge pixels with consistent orientations belong to the same edge curve. Edge detection isn’t perfect. Figure 25.8(a)  shows an image of a scene containing a stapler resting on a desk, and Figure 25.8(b)  shows the output of an edge detection algorithm on this image. As you can see, the output is not perfect: there are gaps where no edge appears, and there are “noise” edges that do not correspond to anything of significance in the scene. Later stages of processing will have to correct for these errors.

Figure 25.8

(a) Photograph of a stapler. (b) Edges computed from (a).

25.3.2 Texture

In everyday language, the texture of surfaces hints at what they feel like when you run a finger over them (the words “texture,” “textile,” and “text” have the same Latin root, a word for weaving). In computational vision, texture refers to a pattern on a surface that can be sensed visually. Usually, these patterns are roughly regular. Examples include the pattern of windows on a building, the stitches on a sweater, the spots on a leopard’s skin, blades of grass on a lawn, pebbles on a beach, and a crowd of people in a stadium.

Texture

Sometimes the arrangement is quite periodic, as in the stitches on a sweater; in other instances, such as pebbles on a beach, the regularity is only in a statistical sense: the density of pebbles is roughly the same on different parts of the beach. A usual rough model of texture is a repetitive pattern of elements, sometimes called texels. This model is quite useful because it is surprisingly hard to make or find real textures that never repeat.

Texels

Texture is a property of an image patch, rather than a pixel in isolation. A good description of a patch’s texture should summarize what the patch looks like. The description should not change when the lighting changes. This rules out using edge points; if a texture is brightly lit, many locations within the patch will have high contrast and will generate edge points; but if the same texture is viewed under less bright light, many of these edges will not be above the threshold. The description should change in a sensible way when the patch rotates. It is important to preserve the difference between vertical stripes and horizontal stripes but not if the vertical stripes are rotated to the horizontal.

Texture representations with these properties have been shown to be useful for two key tasks. The first is identifying objects—a zebra and horse have similar shape, but different

textures. The second is matching patches in one image to patches in another image, a key step in recovering 3D information from multiple images (Section 25.6.1 ).

Here is a basic construction for a texture representation. Given an image patch, compute the gradient orientation at each pixel in the patch, and then characterize the patch by a histogram of orientations. Gradient orientations are largely invariant to changes in illumination (the gradient will get longer, but it will not change direction). The histogram of orientations seems to capture important aspects of the texture. For example, vertical stripes will have two peaks in the histogram (one for the left side of each stripe and one for the right); leopard spots will have more uniformly distributed orientations.

But we do not know how big a patch to describe. There are two strategies. In specialized applications, image information reveals how big the patch should be (for example, one might grow a patch full of stripes until it covers the zebra). An alternative is to describe a patch centered at each pixel for a range of scales. This range usually runs from a few pixels to the extent of the image. Now divide the patch into bins, and in each bin construct an orientation histogram, then summarize the pattern of histograms across bins. It is no longer usual to construct these descriptions by hand. Instead, convolutional neural networks are used to produce texture representations. But the representations constructed by the networks seem to mirror this construction very roughly.

25.3.3 Optical flow Next, let us consider what happens when we have a video sequence, instead of just a single static image. Whenever there is relative movement between the camera and one or more objects in the scene, the resulting apparent motion in the image is called optical flow. This describes the direction and speed of motion of features in the image as a result of relative motion between the viewer and the scene. For example, distant objects viewed from a moving car have much slower apparent motion than nearby objects, so the rate of apparent motion can tell us something about distance.

Optical flow

In Figure 25.9  we show two frames from a video of a tennis player. On the right we display the optical flow vectors computed from these images. The optical flow encodes useful information about scene structure—the tennis player is moving and the background (largely) isn’t. Furthermore, the flow vectors reveal something about what the player is doing—one arm and one leg are moving fast, and the other body parts aren’t.

Figure 25.9

Two frames of a video sequence and the optical flow field corresponding to the displacement from one frame to the other. Note how the movement of the tennis racket and the front leg is captured by the directions of the arrows.

(Images courtesy of Thomas Brox.)

The optical flow vector field can be represented by its components and

vx (x, y) in the x direction

vy (x, y) in the y direction. To measure optical flow, we need to find corresponding

points between one time frame and the next. A very simple-minded technique is based on the fact that image patches around corresponding points have similar intensity patterns.

p x , y ), at time t. This block of pixels is to be compared with pixel blocks centered at various candidate pixels qi at (x + Dx , y + Dy )  at time t + D . One possible measure of similarity is the sum of squared differences (SSD): Consider a block of pixels centered at pixel , (

0

0

0

0

t

SSD(

Dx , Dy ) =

∑x y I x y t

I x + Dx , y + Dy , t + D )) .

( ( , , )− (

( , )

Sum of squared differences (SSD)

t

2

xy

x , y ). We find the (Dx , Dy ) that minimizes the SSD. The optical flow at (x , y ) is then (vx , vy )=(Dx /Dt , Dy /Dt ,). Note that Here, ( , ) ranges over pixels in the block centered at ( 0

0

0

0

for this to work, there should be some texture in the scene, resulting in windows containing a significant variation in brightness among the pixels. If one is looking at a uniform white

q

wall, then the SSD is going to be nearly the same for the different candidate matches , and the algorithm is reduced to making a blind guess. The best-performing algorithms for measuring optical flow rely on a variety of additional constraints to deal with situations in which the scene is only partially textured.

25.3.4 Segmentation of natural images Segmentation is the process of breaking an image into groups of similar pixels. The basic idea is that each image pixel can be associated with certain visual properties, such as brightness, color, and texture. Within an object, or a single part of an object, these attributes vary relatively little, whereas across an inter-object boundary there is typically a large change in one or more of these attributes. We need to find a partition of the image into sets of pixels such that these constraints are satisfied as well as possible. Notice that it isn’t enough just to find edges, because many edges are not object boundaries. So, for example, a tiger in grass may generate an edge on each side of each stripe and each blade of grass. In all the confusing edge data, we may miss the tiger for the stripes.

Segmentation

There are two ways of studying the problem, one focusing on detecting the boundaries of these groups, and the other on detecting the groups themselves, called regions. We illustrate this in Figure 25.10 , showing boundary detection in (b) and region extraction in (c) and (d).

Figure 25.10

(a) Original image. (b) Boundary contours, where the higher the Pb value, the darker the contour. (c) Segmentation into regions, corresponding to a fine partition of the image. Regions are rendered in their mean colors. (d) Segmentation into regions, corresponding to a coarser partition of the image, resulting in fewer regions.

(Images courtesy of Pablo Arbelaez, Michael Maire, Charless Fowlkes and Jitendra Malik.)

Regions

One way to formalize the problem of detecting boundary curves is as a classification problem, amenable to the techniques of machine learning. A boundary curve at pixel location (x, y) will have an orientation θ. An image neighborhood centered at (x, y) looks roughly like a disk, cut into two halves by a diameter oriented at θ. We can compute the probability Pb (x, y, θ) that there is a boundary curve at that pixel along that orientation by comparing features in the two halves. The natural way to predict this probability is to train a machine learning classifier using a data set of natural images in which humans have marked the ground truth boundaries—the goal of the classifier is to mark exactly those boundaries marked by humans and no others.

Boundaries detected by this technique are better than those found using the simple edge detection technique described previously. But there are still two limitations: (1) the boundary pixels formed by thresholding Pb (x, y, θ) are not guaranteed to form closed curves, so this approach doesn’t deliver regions, and (2) the decision making exploits only local context, and does not use global consistency constraints.

The alternative approach is based on trying to “cluster” the pixels into regions based on their brightness, color and texture properties. There are a number of different ways in which this intuition can be formalized mathematically. For instance, Shi and Malik (2000) set this up as a graph partitioning problem. The nodes of the graph correspond to pixels, and edges to connections between pixels. The weight

Wij on the edge connecting a pair of pixels i and

j is based on how similar the two pixels are in brightness, color, texture, etc. They then find

partitions that minimize a normalized cut criterion. Roughly speaking, the criterion for partitioning the graph is to minimize the sum of weights of connections across the groups and maximize the sum of weights of connections within the groups.

It turns out that the approaches based on finding boundaries and on finding regions can be coupled, but we will not explore these possibilities here. Segmentation based purely on lowlevel, local attributes such as brightness and color can not be expected to deliver the final correct boundaries of all the objects in the scene. To reliably find boundaries associated with objects, it is also necessary to incorporate high-level knowledge of the kinds of objects one may expect to encounter in a scene. At this time, a popular strategy is to produce an over-segmentation of an image, where one is guaranteed not to have missed marking any of the true boundaries but may have marked many extra false boundaries as well. The resulting regions, called superpixels, provide a significant reduction in computational complexity for various algorithms, as the number of superpixels may be in the hundreds, compared to millions of raw pixels. Exploiting high-level knowledge of objects is the subject of the next section, and actually detecting the objects in images is the subject of Section 25.5 .

25.4 Classifying Images Image classification applies to two main cases. In one, the images are of objects, taken from a given taxonomy of classes, and there’s not much else of significance in the picture—for example, a catalog of clothing or furniture images, where the background doesn’t matter, and the output of the classifier is “cashmere sweater” or “desk chair.”

In the other case, each image shows a scene containing multiple objects. So in grassland you might see a giraffe and a lion, and in the living room you might see a couch and lamp, but you don’t expect a giraffe or a submarine in a living room. We now have methods for largescale image classification that can accurately output “grassland” or “living room.”

Appearance

Modern systems classify images using appearance (i.e., color and texture, as opposed to geometry). There are two difficulties. First, different instances of the same class could look different—some cats are black and others are orange. Second, the same cat could look different at different times depending on several effects, (as illustrated in Figure 25.11 ):

LIGHTING, which changes the brightness and color of the image. FORESHORTENING, which causes a pattern viewed at a glancing angle to be distorted. ASPECT, which causes objects to look different when seen from different directions. A doughnut seen from the side looks like a flattened oval, but from above it is an annulus. OCCLUSION, where some parts of the object are hidden. Objects can occlude one another, or parts of an object can occlude other parts, an effect known as self-occlusion. DEFORMATION, where the object changes its shape. For example, the tennis player moves her arms and legs.

Figure 25.11

Important sources of appearance variation that can make different images of the same object look different. First, elements can foreshorten, like the circular patch on the top left. This patch is viewed at a glancing angle, and so is elliptical in the image. Second, objects viewed from different directions can change shape quite dramatically, a phenomenon known as aspect. On the top right are three different aspects of a doughnut. Occlusion causes the handle of the mug on the bottom left to disappear when the mug is rotated. In this case, because the body and handle belong to the same mug, we have selfocclusion. Finally, on the bottom right, some objects can deform dramatically.

Modern methods deal with these problems by learning representations and classifiers from very large quantities of training data using a convolutional neural network. With a sufficiently rich training set the classifier will have seen any effect of importance many times in training, and so can adjust for the effect.

25.4.1 Image classification with convolutional neural networks Convolutional neural networks (CNNs) are spectacularly successful image classifiers. With enough training data and enough training ingenuity, CNNs produce very successful classification systems, much better than anyone has been able to produce with other methods.

The ImageNet data set played a historic role in the development of image classification systems by providing them with over 14 million training images, classified into over 30,000 fine-grained categories. ImageNet also spurred progress with an annual competition.

Systems are evaluated by both the classification accuracy of their single best guess and by top-5 accuracy, in which systems are allowed to submit five guesses—for example, malamute, husky, akita, samoyed, eskimo dog. ImageNet has 189 subcategories of dog, so even dog-loving humans find it hard to label images correctly with a single guess.

In the first ImageNet competition in 2010, systems could do no better than 70% top-5 accuracy. The introduction of convolutional neural networks in 2012 and their subsequent refinement led to an accuracy of 98% in top-5 (surpassing human performance) and 87% in top-1 accuracy by 2019. The primary reason for this success seems to be that the features that are being used by CNN classifiers are learned from data, not hand-crafted by a researcher; this ensures that the features are actually useful for classification.

Progress in image classification has been rapid because of the availability of large, challenging data sets such as ImageNet; because of competitions based on these data sets that are fair and open; and because of the widespread dissemination of successful models. The winners of competitions publish the code and often the pretrained parameters of their models, making it easy for others to fiddle with successful architectures and try to make them better.

25.4.2 Why convolutional neural networks classify images well Image classification is best understood by looking at data sets, but ImageNet is much too large to look at in detail. The MNIST data set is a collection of 70,000 images of handwritten digits, 0–9, which is often used as a standard warmup data set. Looking at this data set (some examples appear in Figure 25.12 ) exposes some important, quite general, properties. You can take an image of a digit and make a number of small alterations without changing the identity of the digit: you can shift it, rotate it, make it brighter or darker, smaller or larger. This means that individual pixel values are not particularly informative— we know that an 8 should have some dark pixels in the center and a 0 should not, but those dark pixels will be in slightly different pixel locations in each instance of an 8.

Figure 25.12

On the far left, some images from the MNIST data set. Three kernels appear on the center left. They are shown at actual size (tiny blocks) and magnified to reveal their content: mid-grey is zero, light is positive, and dark is negative. Center right shows the results of applying these kernels to the images. Right shows pixels where the response is bigger than a threshold (green) or smaller than a threshold (red). You should notice that this gives (from top to bottom): a horizontal bar detector; a vertical bar detector; and (harder to note) a line ending detector. These detectors pay attention to the contrast of the bar, so (for example) a horizontal bar that is light on top and dark below produces a positive (green) response, and one that is dark on top and light below gets a negative (red) response. These detectors are moderately effective, but not perfect.

Another important property of images is that local patterns can be quite informative: The digits 0, 6, 8 and 9 have loops; the digits 4 and 8 have crossings; the digits 1, 2, 3, 5 and 7 have line endings, but no loops or crossings; the digits 6 and 9 have loops and line endings. Furthermore, spatial relations between local patterns are informative. A 1 has two line endings above one another; a 6 has a line ending above a loop. These observations suggest a strategy that is a central tenet of modern computer vision: you construct features that respond to patterns in small, localized neighborhoods; then other features look at patterns of those features; then others look at patterns of those, and so on.

This is what convolutional neural networks do well. You should think of a layer—a convolution followed by a ReLU activation function—as a local pattern detector (Figure 25.12 ). The convolution measures how much each local window of the image looks like the kernel pattern; the ReLU sets low-scoring windows to zero, and emphasizes highscoring windows. So convolution with multiple kernels finds multiple patterns; furthermore, composite patterns can be detected by applying another layer to the output of the first layer.

Think about the output of the first convolutional layer. Each location receives inputs from pixels in a window about that location. The output of the ReLU, as we have seen, forms a simple pattern detector. Now if we put a second layer on top of this, each location in the second layer receives inputs from first-layer values in a window about that location. This means that locations in the second layer are affected by a larger window of pixels than those in the first layer. You should think of these as representing “patterns of patterns.” If we place a third layer on top of the second layer, locations in that third layer will depend on an even larger window of pixels; a fourth layer will depend on a yet larger window, and so on. The network is creating patterns at multiple levels, and is doing that by learning from the data rather than having the patterns given to it by a programmer.

While training a CNN “out of the box” does sometimes work, it helps to know a few practical techniques. One of the most important is data set augmentation, in which training examples are copied and modified slightly. For example, one might randomly shift, rotate, or stretch an image by a small amount, or randomly shift the hue of the pixels by a small amount. Introducing this simulated variation in viewpoint or lighting to the data set helps to increase the size of the data set, though of course the new examples are highly correlated with the originals. It is also possible to use augmentation at test time rather than training time. In this approach, the image is replicated and modified several times (e.g., with random cropping) and the classifier is run on each of the modified images. The outputs of the classifier from each copy are then used to vote for a final decision on the overall class.

Data set augmentation

When you are classifying images of scenes, every pixel could be helpful. But when you are classifying images of objects, some pixels aren’t part of the object, and so might be a distraction. For example, if a cat is lying on a dog bed, we want a classifier to concentrate on the pixels of the cat, not the bed. Modern image classifiers handle this well, classifying an image as “cat” accurately even if few pixels actually lie on the cat. There are two reasons for this. First, CNN-based classifiers are good at ignoring patterns that aren’t discriminative. Second, patterns that lie off the object might be discriminative (e.g., a cat toy, a collar with a little bell, or a dish of cat food might actually help tell that we are looking at a cat). This

effect is known as context. Context can help or can hurt, depending quite strongly on the particular data set and application.

Context

25.5 Detecting Objects Image classifiers predict what is in the image—they classify the whole image as belonging to one class. Object detectors find multiple objects in an image, report what class each object is, and also report where each object is by giving a bounding box around the object.1 The set of classes is fixed in advance. So we might try to detect all faces, all cars, or all cats.

1 We will use the term “box” to mean any axis-aligned rectangular region of the image, and the term “window” mostly as a synonym for “box,” but with the connotation that we have a window onto the input where we are hoping to see something, and a bounding box in the output when we have found it.

Bounding box

Sliding window

We can build an object detector by looking at a small sliding window onto the larger image —a rectangle. At each spot, we classify what we see in the window, using a CNN classifier. We then take the high-scoring classifications—a cat over here and a dog over there—and ignore the other windows. After some work resolving conflicts, we have a final set of objects with their locations. There are still some details to work out:

DECIDE ON A WINDOW SHAPE: The easiest choice by far is to use axis-aligned rectangles. (The alternative—some form of mask that cuts the object out of the image—is hardly ever used, because it is hard to represent and to compute with.) We still need to choose the width and height of the rectangles. BUILD A CLASSIFIER FOR WINDOWS: We already know how to do this with a CNN. DECIDE WHICH WINDOWS TO LOOK AT: Out of all possible windows, we want to select ones that are likely to have interesting objects in them.

CHOOSE WHICH WINDOWS TO REPORT: Windows will overlap, and we don’t want to report the same object multiple times in slightly different windows. Some objects are not worth mentioning; think about the number of chairs and people in a picture of a large packed lecture hall. Should they all be reported as individual objects? Perhaps only the objects that appear large in the image—the front row—should be reported. The choice depends on the intended use of the object detector. REPORT PRECISE LOCATIONS OF OBJECTS USING THESE WINDOWS: Once we know that the object is somewhere in the window, we can afford to do more computation to figure out a more precise location within the window.

Let’s look more carefully at the problem of deciding which windows to look at. Searching all possible windows isn’t efficient—in an n × n pixel image there are O(n4 ) possible rectangular windows. But we know that windows that contain objects tend to have quite coherent color and texture. On the other hand, windows that cut an object in half have regions or edges that cross the side of the window. So it makes sense to have a mechanism that scores “objectness”—whether a box has an object in it, independent of what that object is. We can find the boxes that look like they have an object in them, and then classify the object for just those boxes that pass the objectness test.

A network that finds regions with objects is called a regional proposal network (RPN). The object detector known as Faster RCNN encodes a large collection of bounding boxes as a map of fixed size. Then it builds a network that can predict a score for each box, and trains this network so the score is large when the box contains an object, and small otherwise. Encoding boxes as a map is straightforward. We consider boxes centered on points throughout the image; we don’t need to consider every possible point (because moving by one pixel is not likely to change the classification); a good choice is a stride (the offset between center points) of 16 pixels. For each center point we consider several possible boxes, called anchor boxes. Faster RCNN uses nine boxes: small, medium, and large sizes; and tall, wide, and square aspect ratios.

Regional proposal network (RPN)

In terms of the neural network architecture, construct a 3D block where each spatial location in the block has two dimensions for the center point and one dimension for the type of box. Now any box with a good enough objectness score is called a region of interest (ROI), and must be checked by a classifier. But CNN classifiers prefer images of fixed size, and the boxes that pass the objectness test will differ in size and shape. We can’t make the boxes have the same number of pixels, but we can make them have the same number of features by sampling the pixels to extract features, a process called ROI pooling. This fixed-size feature map is then passed to the classifier.

Now for the problem of deciding which windows to report. Assume we look at windows of size 32 × 32 with a stride of 1: each window is offset by just one pixel from the one before. There will be many windows that are similar, and should have similar scores. If they all have a score above threshold we don’t want to report all of them, because they very likely all refer to slightly different views of the same object. On the other hand if the stride is too large, it might be that an object is not contained within any one window, and will be missed. Instead, we can use a greedy algorithm called non-maximum suppression. First, build a sorted list of all windows with scores over a threshold. Then, while there are windows in the list, choose the window with the highest score and accept it as containing an object; discard from the list all other largely overlapping windows.

Non-maximum suppression

Finally, we have the problem of reporting the precise location of objects. Assume we have a window that has a high score, and has passed through non-maximum suppression. This window is unlikely to be in exactly the right place (remember, we looked at a relatively small number of windows with a small number of possible sizes). We use the feature representation computed by the classifier to predict improvements that will trim the window down to a proper bounding box, a step known as bounding box regression.

Bounding box regression

Evaluating object detectors takes care. First we need a test set: a collection of images with each object in the image marked by a ground truth category label and bounding box. Usually, the boxes and labels are supplied by humans. Then we feed each image to the object detector and compare its output to the ground truth. We should be willing to accept boxes that are off by a few pixels, because the ground truth boxes won’t be perfect. The evaluation score should balance recall (finding all the objects that are there) and precision (not finding objects that are not there).

Figure 25.13

Faster RCNN uses two networks. A picture of a young Nelson Mandela is fed into the object detector. One network computes “objectness” scores of candidate image boxes, called “anchor boxes,” centered at a grid point. There are nine anchor boxes (three scales, three aspect ratios) at each grid point. For the example image, an inner green box and an outer blue box have passed the objectness test. The second network is a feature stack that computes a representation of the image suitable for classification. The boxes with highest objectness score are cut from the feature map, standardized in size with ROI pooling, and passed to a classifier. The blue box has a higher score than the green box and overlaps it, so the green box is rejected by non-maximum suppression. Finally, bounding box regression the blue box so that it fits the face. This means that the relatively coarse sampling of locations, scales, and aspect ratios does not weaken accuracy.

Photo by Sipa/Shutterstock.

25.6 The 3D World Images show a 2D picture of a 3D world. But this 2D picture is rich with cues about the 3D world. One kind of cue occurs when we have multiple pictures of the same world, and can match points between pictures. Another kind of cue is available within a single picture.

25.6.1 3D cues from multiple views Two pictures of objects in a 3D world are better than one for several reasons:

If you have two images of the same scene taken from different viewpoints and you know enough about the two cameras, you can construct a 3D model—a collection of points with their coordinates in 3 dimensions—by figuring out which point in the first view corresponds to which point in the second view and applying some geometry. This is true for almost all pairs of viewing directions and almost all kinds of camera. If you have two views of enough points, and you know which point in the first view corresponds to which point in the second view, you do not need to know much about the cameras to construct a 3D model. Two views of two points gives you four x, y coordinates, and you only need three coordinates to specify a point in 3D space; the extra coordinate comes in helpful to figure out what you need to know about the cameras. This is true for almost all pairs of viewing directions and almost all kinds of camera.

The key problem is to establish which point in the first view corresponds to which in the second view. Detailed descriptions of the local appearance of a point using simple texture features (like those in section 25.3.2 ) are often enough to match points. For example, in a scene of traffic on a street, there might be only one green light visible in two images taken of the scene; we can then hypothesize that these correspond to each other. The geometry of multiple camera views is very well understood (but sadly too complicated to expound here). The theory produces geometric constraints on which point in one image can match with which point in the other. Other constraints can be obtained by reasoning about the smoothness of the reconstructed surfaces.

There are two ways of getting multiple views of a scene. One is to have two cameras or two eyes (section 25.6.2 ). Another is to move (section 25.6.3 ). If you have more than two views, you can recover both the geometry of the world and the details of the view very accurately. Section 25.7.3  discusses some applications for this technology.

25.6.2 Binocular stereopsis Most vertebrates have two eyes. This is useful for redundancy in case of a lost eye, but it helps in other ways too. Most prey have eyes on the side of the head to enable a wider field of vision. Predators have the eyes in the front, enabling them to use binocular stereopsis. Hold both index fingers up in front of your face, with one eye closed, and adjust them so the front finger occludes the other finger in the open eye’s view. Now swap eyes; you should notice that the fingers have shifted position with respect to one another. This shifting of position from left view to right view is known as disparity. In the right choice of coordinate system, if we superimpose left and right images of an object at some depth, the object shifts horizontally in the superimposed image, and the size of the shift is the reciprocal of the depth. You can see this in Figure 25.14 , where the nearest point of the pyramid is shifted to the left in the right image and to the right in the left image.

Figure 25.14

Translating a camera parallel to the image plane causes image features to move in the camera plane. The disparity in positions that results is a cue to depth. If we superimpose left and right images, as in (b), we see the disparity.

Binocular stereopsis

Disparity

To measure disparity we need to solve the correspondence problem—to determine for a point in the left image, its “partner” in the right image which results from the projection of the same scene point. This is analogous to what is done in measuring optical flow, and the most simple-minded approaches are somewhat similar. These methods search for blocks of left and right pixels that match, using the sum of squared differences (as in Section 25.3.3 ). More sophisticated methods use more detailed texture representations of blocks of pixels (as in Section 25.3.2 ). In practice, we use much more sophisticated algorithms, which exploit additional constraints.

Baseline

Assuming that we can measure disparity, how does this yield information about depth in the scene? We will need to work out the geometrical relationship between disparity and depth. We will consider first the case when both the eyes (or cameras) are looking forward with their optical axes parallel. The relationship of the right camera to the left camera is then just

x

b

a displacement along the -axis by an amount , the baseline. We can use the optical flow equations from Section 25.3.3 , if we think of this as resulting from a translation vector acting for time

δt, with Tx b δt and Ty Tz =

/

=

= 0.

The horizontal and vertical disparity are

given by the optical flow components, multiplied by the time step Carrying out the substitutions, we get the result that

H bZ V =

/

,

δt H vx δt V vy δt ,

= 0.

=

,

=

.

In other words, the

horizontal disparity is equal to the ratio of the baseline to the depth, and the vertical disparity is zero. We can recover the depth

T

Z given that we know b, and can measure H .

Under normal viewing conditions, humans fixate; that is, there is some point in the scene at which the optical axes of the two eyes intersect. Figure 25.15  shows two eyes fixated at a point P0 , which is at a distance Z from the midpoint of the eyes. For convenience, we will

compute the angular disparity, measured in radians. The disparity at the point of fixation P0 is zero. For some other point P in the scene that is δZ farther away, we can compute the angular displacements of the left and right images of P , which we will call PL and PR , respectively. If each of these is displaced by an angle δθ/2 relative to P0 , then the

displacement between PL and PR , which is the disparity of P , is just δθ. From Figure 25.15 , tan

θ=

b/2 b/2 Z and tan(θ − δθ/2) = Z+δZ , but for small angles, tan θ ≈ θ, so

b/2 δθ/2 = Z

−

b/2 Z + δZ

≈

bδZ 2Z 2

Figure 25.15

The relation between disparity and depth in stereopsis. The centers of projection of the two eyes are

distance b apart, and the optical axes intersect at the fixation point P0 . The point P in the scene projects to points PL and PR in the two eyes. In angular terms, the disparity between these is δθ (the diagram shows two angles of δθ/2).

Fixate

and, since the actual disparity is δθ, we have

disparity =

bδZ Z2

In humans, the baseline b is about 6 cm. Suppose that Z is about 100 cm and that the smallest detectable δθ (corresponding to the size of a single pixel) is about 5 seconds of arc, giving a δZ of 0.4 mm. For Z = 30 cm, we get the impressively small value δZ = 0.036mm. That is, at a distance of 30 cm, humans can discriminate depths that differ by as little as 0.036 mm, enabling us to thread needles and the like.

25.6.3 3D cues from a moving camera Assume we have a camera moving in a scene. Take Figure 25.14  and label the left image “Time t” and the right image “Time t + 1”. The geometry has not changed, so all the material from the discussion of stereopsis also applies when a camera moves. What we called disparity in that section is now thought of as apparent motion in the image, and called optical flow. This is a source of information for both the movement of the camera and the geometry of the scene. To understand this, we state (without proof) an equation that relates the optical flow to the viewer’s translational velocity

T and the depth in the scene.

The optical flow field is a vector field of velocities in the image, (vx (x, y), vy (x, y)). Expressions for these components, in a coordinate frame centered on the camera and assuming a focal length of f = 1, are

v x ( x, y ) = where

−Tx + xTz

Z ( x, y )

and

v y ( x, y ) =

−Ty +yTz

Z ( x, y )

Z (x, y) is the z-coordinate (that is, depth) of the point in the scene corresponding to

the point in the image at (x, y). Note that both components of the optical flow, vx (x, y) and vy (x, y), are zero at the point

x = Tx /Tz , y = Ty /Tz .. This point is called the focus of expansion of the flow field. Suppose we change the origin in the x–y plane to lie at the focus of expansion; then the expressions for optical flow take on a particularly simple form. Let (x', y') be the new coordinates defined by x' =

x − Tx /Tz , y' = y − Ty /Tz . Then v x ( x ', y ' ) =

x' T z Z ( x' , y ' )

,

v y ( x' , y ' ) =

y ' Tz Z ( x' , y ' )

Focus of expansion

Note that there is a scale factor ambiguity here (which is why assuming a focal length of

f = 1 is harmless). If the camera was moving twice as fast, and every object in the scene was twice as big and at twice the distance to the camera, the optical flow field would be exactly the same. But we can still extract quite useful information.

1. Suppose you are a fly trying to land on a wall and you want useful information from the optical flow field. The optical flow field cannot tell you the distance to the wall or the velocity to the wall, because of the scale ambiguity. But if you divide the distance by the velocity, the scale ambiguity cancels. The result is the time to contact, given by Z /Tz , and is very useful indeed to control the landing approach. There is considerable experimental evidence that many different animal species exploit this cue. 2. Consider two points at depths Z1 , Z2 respectively. We may not know the absolute value of either of these, but by considering the inverse of the ratio of the optical flow magnitudes at these points, we can determine the depth ratio Z1 /Z2 . This is the cue of motion parallax, one we use when we look out of the side window of a moving car or train and infer that the slower-moving parts of the landscape are farther away.

25.6.4 3D cues from one view Even a single image provides a rich collection of information about the 3D world. This is true even if the image is just a line drawing. Line drawings have fascinated vision scientists, because people have a sense of 3D shape and layout even though the drawing seems to contain very little information to choose from the vast collection of scenes that could produce the same drawing. Occlusion is one key source of information: if there is evidence in the picture that one object occludes another, then the occluding object is closer to the eye. In images of real scenes, texture is a strong cue to 3D structure. Section 25.3.2  stated that texture is a repetitive pattern of texels. Although the distribution of texels may be uniform on objects in the scene—for example, pebbles on a beach—it may not be uniform in image— the farther pebbles appear smaller than the nearer pebbles. As another example, think about

a piece of polka-dot fabric. All the dots are the same size and shape on the fabric, but in a perspective view some dots are ellipses due to foreshortening. Modern methods exploit these cues by learning a mapping from images to 3D structure (Section 25.7.4 ), rather than reasoning directly about the underlying mathematics of texture.

Shading—variation in the intensity of light received from different portions of a surface in a scene—is determined by the geometry of the scene and by the reflectance properties of the surfaces. There is very good evidence that shading is a cue to 3D shape. The physical argument is easy. From the physical model of section 25.2.4 , we know that if a surface normal points toward the light source, the surface is brighter, and if it points away, the surface is darker. This argument gets more complicated if the reflectance of the surface isn’t known, and the illumination field isn’t even, but humans seem to be able to get a useful perception of shape from shading. We know frustratingly little about algorithms to do this.

If there is a familiar object in the picture, what it looks like depends very strongly on its pose, that is, its position and orientation with respect to the viewer. There are straightforward algorithms for recovering pose from correspondences between points on an object and points on a model of the object. Recovering the pose of a known object has many applications. For instance, in an industrial manipulation task, the robot arm cannot pick up an object until the pose is known. Robotic surgery applications depend on exactly computing the transformations between the camera’s position and the positions of the surgical tool and the patient (to yield the transformation from the tool’s position to the patient’s position).

Pose

Spatial relations between objects are another important cue. Here is an example. All pedestrians are about the same height, and they tend to stand on a ground plane. If we know where the horizon is in an image, we can rank pedestrians by distance to the camera. This works because we know where their feet are, and pedestrians whose feet are closer to the horizon in the image are farther away from the camera, and so must be smaller in the image. This means we can rule out some detector responses—if a detector finds a pedestrian

who is large in the image and whose feet are close to the horizon, it has found an enormous pedestrian; these don’t exist, so the detector is wrong. In turn, a reasonably reliable pedestrian detector is capable of producing estimates of the horizon, if there are several pedestrians in the scene at different distances from the camera. This is because the relative scaling of the pedestrians is a cue to where the horizon is. So we can extract a horizon estimate from the detector, then use this estimate to prune the pedestrian detector’s mistakes.

25.7 Using Computer Vision Here we survey a range of computer vision applications. There are now many reliable computer vision tools and toolkits, so the range of applications that are successful and useful is extraordinary. Many are developed at home by enthusiasts for special purposes, which is testimony to how usable the methods are and how much impact they have. (For example, an enthusiast created a great object-detection-based pet door that refuses entry to a cat if it is bringing in a dead mouse–a Web search will find it for you).

25.7.1 Understanding what people are doing If we could build systems that understood what people are doing by analyzing video, we could build human-computer interfaces that watch people and react to their behavior. With these interfaces, we could: design buildings and public places better, by collecting and using data about what people do in public; build more accurate and less intrusive security surveillance systems; build automated sports commentators; make construction sites and workplaces safer by generating warnings when people and machines get dangerously close; build computer games that make a player get up and move around; and save energy by managing heat and light in a building to match where the occupants are and what they are doing.

The state of the art for some problems is now extremely strong. There are methods that can predict the locations of a person’s joints in an image very accurately. Quite good estimates of the 3D configuration of that person’s body follow (see Figure 25.16 ). This works because pictures of the body tend to have weak perspective effects, and body segments don’t vary much in length, so the foreshortening of a body segment in an image is a good cue to the angle between it and the camera plane. With a depth sensor, these estimates can be made fast enough to build them into computer game interfaces.

Figure 25.16

Reconstructing humans from a single image is now practical. Each row shows a reconstruction of 3D body shape obtained using a single image. These reconstructions are possible because methods can estimate the location of joints, the joint angles in 3D, the shape of the body, and the pose of the body with respect to an image. Each row shows the following: far left a picture; center left the picture with the reconstructed body superimposed; center right another view of the reconstructed body; and far right yet another view of the reconstructed body. The different views of the body make it much harder to conceal errors in reconstruction. Figure courtesy of Angjoo Kanazawa, produced by a system described in Kanazawa et al. (2018a).

Classifying what people are doing is harder. Video that shows rather structured behaviors, like ballet, gymnastics, or tai chi, where there are quite specific vocabularies that refer to very precisely delineated activities on simple backgrounds, is quite easy to deal with. Good results can be obtained with a lot of labeled data and an appropriate convolutional neural network. However, it can be difficult to prove that the methods actually work, because they rely so strongly on context. For example, a classifier that labels “swimming” sequences very well might just be a swimming pool detector, which wouldn’t work for (say) swimmers in rivers.

More general problems remain open—for example, how to link observations of the body and the objects nearby to the goals and intentions of the moving people. One source of difficulty is that similar behaviors look different, and different behaviors look similar, as Figure 25.17  shows.

Figure 25.17

The same action can look very different; and different actions can look similar. These examples show actions taken from a data set of natural behaviors; the labels are chosen by the curators of the data set, rather than predicted by an algorithm. Top: examples of the label “opening fridge,” some shown in closeup and some from afar. Bottom: examples of the label “take something out of fridge.” Notice how in both rows the subject’s hand is close to the fridge door—telling the difference between the cases requires quite subtle judgment about where the hand is and where the door is. Figure courtesy of David Fouhey, taken from a data set described in Fouhey et al. (2018).

Another difficulty is caused by time scale. What someone is doing depends quite strongly on the time scale, as Figure 25.18  illustrates. Another important effect shown in that figure is that behavior composes—several recognized behaviors may be combined to form a single higher-level behavior such as fixing a snack.

Figure 25.18

What you call an action depends on the time scale. The single frame at the top is best described as opening the fridge (you don’t gaze at the contents when you close a fridge). But if you look at a short clip of video (indicated by the frames in the center row), the action is best described as getting milk from the

fridge. If you look at a long clip (the frames in the bottom row), the action is best described as fixing a snack. Notice that this illustrates one way in which behavior composes: getting milk from the fridge is sometimes part of fixing a snack, and opening the fridge is usually part of getting milk from the fridge. Figure courtesy of David Fouhey, taken from a data set described in Fouhey et al. (2018).

It may also be that unrelated behaviors are going on at the same time, such as singing a song while fixing a snack. A challenge is that we don’t have a common vocabulary for the pieces of behavior. People tend to think they know a lot of behavior names but can’t produce long lists of such words on demand. That makes it harder to get data sets of consistently labeled behaviors.

Learned classifiers are guaranteed to behave well only if the training and test data come from the same distribution. We have no way of checking that this constraint applies to images, but empirically we observe that image classifiers and object detectors work very well. But for activity data, the relationship between training and test data is more untrustworthy because people do so many things in so many contexts. For example, suppose we have a pedestrian detector that performs well on a large data set. There will be rare phenomena (for example, people mounting unicycles) that do not appear in the training set, so we can’t say for sure how the detector will work in such cases. The challenge is to prove that the detector is safe whatever pedestrians do, which is difficult for current theories of learning.

25.7.2 Linking pictures and words Many people create and share pictures and videos on the Internet. The difficulty is finding what you want. Typically, people want to search using words (rather than, say, example sketches). Because most pictures don’t come with words attached, it is natural to try and build tagging systems that tag images with relevant words. The underlying machinery is straightforward—we apply image classification and object detection methods and tag the image with the output words. But tags aren’t a comprehensive description of what is happening in an image. It matters who is doing what, and tags don’t capture this. For example, tagging a picture of a cat in the street with the object categories “cat”, “street”, “trash can” and “fish bones” leaves out the information that the cat is pulling the fish bones out of an open trash can on the street.

Tagging system

As an alternative to tagging, we might build captioning systems—systems that write a caption of one or more sentences describing the image. The underlying machinery is again straightforward—couple a convolutional network (to represent the image) to a recurrent neural network or transformer network (to generate sentences), and train the result with a data set of captioned images. There are many images with captions available on the Internet; curated data sets use human labor to augment each image with additional captions to capture the variation in natural language. For example, the COCO (Common Objects in Context) data set is a comprehensive collection of over 200,000 images labeled with five captions per image.

Captioning systems

Current methods for captioning use detectors to find a set of words that describe the image, and provide those words to a sequence model that is trained to generate a sentence. The most accurate methods search through the sentences that the model can generate to find the best, and strong methods appear to require a slow search. Sentences are evaluated with a set of scores that check whether the generated sentence (a) uses phrases common in the ground truth annotations and (b) doesn’t use other phrases. These scores are hard to use directly as a loss function, but reinforcement learning methods can be used to train networks that get very good scores. Often there will be an image in the training set whose description has the same set of words as an image in the test set; in that case a captioning system can just retrieve a valid caption rather than having to generate a new one. Caption writing systems produce a mix of excellent results and embarrassing errors (see Figure 25.19 ).

Figure 25.19

Automated image captioning systems produce some good results and some failures. The two captions at left describe the respective images well, although “eating ... in his mouth” is a disfluency that is fairly typical of the recurrent neural network language models used by early captioning systems. For the two captions on the right, the captioning system seems not to know about squirrels, and so guesses the animal from context; it also fails to recognize that the two squirrels are eating. Image credits: geraine/Shutterstock; ESB Professional/Shutterstock; BushAlex/Shutterstock; Maria.Tem/Shutterstock. The images shown are similar but not identical to the original images from which the captions were generated. For the original images see Aneja et al. (2018).

Captioning systems can hide their ignorance by omitting to mention details they can’t get right or by using contextual cues to guess. For example, captioning systems tend to be poor at identifying the gender of people in images, and often guess based on training data statistics. That can lead to errors—men also like shopping and women also snowboard. One way to establish whether a system has a good representation of what is happening in an image is to force it to answer questions about the image. This is a visual question answering or VQA system. An alternative is a visual dialog system, which is given a picture, its caption, and a dialog. The system must then answer the last question in the dialog. As Figure 25.20  shows, vision remains extremely hard and VQA systems often make errors.

Figure 25.20

Visual question-answering systems produce answers (typically chosen from a multiple-choice set) to natural-language questions about images. Top: the system is producing quite sensible answers to rather difficult questions about the image. Bottom: less satisfactory answers. For example, the system is guessing about the number of holes in a pizza, because it doesn’t understand what counts as a hole, and it has real difficulty counting. Similarly, the system selects brown for the cat’s leg because the background is brown and it can’t localize the leg properly.

Image credits: (Top) Tobyanna/Shutterstock; 679411/Shutterstock; ESB Professional/Shutterstock; Africa Studio/Shutterstock; (Bottom) Stuart Russell; Maxisport/Shutterstock; Chendongshan/Shutterstock; Scott Biales DitchTheMap/Shutterstock. The images shown are similar but not identical to the original images to which the question-answering system was applied. For the original images see Goyal et al. (2017).

Visual question answering (VQA)

Visual dialog

25.7.3 Reconstruction from many views Reconstructing a set of points from many views—which could come from video or from an aggregation of tourist photographs—is similar to reconstructing the points from two views, but there are some important differences. There is far more work to be done to establish correspondence between points in different views, and points can go in and out of view, making the matching and reconstruction process messier. But more views means more constraints on the reconstruction and on the recovered viewing parameters, so it is usually possible to produce extremely accurate estimates of both the position of the points and of the viewing parameters. Rather roughly, reconstruction proceeds by matching points over pairs of images, extending these matches to groups of images, coming up with a rough solution for both geometry and viewing parameters, then polishing that solution. Polishing means minimizing the error between points predicted by the model (of geometry and viewing parameters) and the locations of image features. The detailed procedures are too complex to cover fully, but are now very well understood and quite reliable.

All the geometric constraints on correspondences are known for any conceivably useful form of camera. The procedures can be generalized to deal with views that are not orthographic; to deal with points that are observed in only some views; to deal with

unknown camera parameters (like focal length); and to exploit various sophisticated searches for appropriate correspondences. It is practical to accurately reconstruct a model of an entire city from images. Some applications are:

MODEL BUILDING: For example, one might build a modeling system that takes many views depicting an object and produces a very detailed 3D mesh of textured polygons for use in computer graphics and virtual reality applications. It is routine to build models like this from video, but such models can now be built from apparently random sets of pictures. For example, you can build a 3D model of the Statue of Liberty from pictures found on the Internet. MIX ANIMATION WITH LIVE ACTORS IN VIDEO: To place computer graphics characters into real video, we need to know how the camera moved for the real video, so we can render the character correctly, changing the view as the camera moves. PATH RECONSTRUCTION: Mobile robots need to know where they have been. If the robot has a camera, we can build a model of the camera’s path through the world; that will serve as a representation of the robot’s path. CONSTRUCTION MANAGEMENT: Buildings are enormously complicated artifacts, and keeping track of what is happening during construction is difficult and expensive. One way to keep track is to fly drones through the construction site once a week, filming the current state. Then build a 3D model of the current state and explore the difference between the plans and the reconstruction using visualization techniques. Figure 25.21  illustrates this application. Figure 25.21

3D models of construction sites are produced from images by structure-from-motion and multiview stereo algorithms. They help construction companies to coordinate work on large buildings by comparing a 3D model of the actual construction to date with the building plans. Left: A

visualization of a geometric model captured by drones. The reconstructed 3D points are rendered in color, so the result looks like progress to date (note the partially completed building with crane). The small pyramids show the pose of a drone when it captured an image, to allow visualization of the flight path. Right: These systems are actually used by construction teams; this team views the model of the as-built site, and compares it with building plans as part of the coordination meeting. Figure courtesy of Derek Hoiem, Mani Golparvar-Fard and Reconstruct, produced by a commercial system described in a blog post at medium.com/reconstruct-inc.

25.7.4 Geometry from a single view Geometric representations are particularly useful if you want tomove, because they can reveal where you are, where you can go, and what you are likely bump into. But it is not always convenient to use multiple views to produce a geometric model. For example, when you open the door and step into a room, your eyes are too close together to recover a good representation of the depth to distant objects across the room. You could move your head back and forth, but that is time-consuming and inconvenient.

An alternative is to predict a depth map—an array giving the depth to each pixel in the image, nominally from the camera—from a single image. For many kinds of scenes, this is surprisingly easy to do accurately, because the depth map has quite a simple structure. This is particularly true of rooms and indoor scenes in general. The mechanics are straightforward. One obtains a data set of images and depth maps, then trains a network to predict depth maps from images. A variety of interesting variations of the problem can be solved. The problem with a depth map is that it doesn’t tell you anything about the backs of objects, or the space behind the objects. But there are methods that can predict what voxels (3D pixels) are occupied by known objects (the object geometry is known) and what a depth map would look like if an object were removed (and so where you could hide objects). These methods work because object shapes are quite strongly stylized.

Depth map

As we saw in Section 25.6.4 , recovering the pose of a known object using a 3D model is straightforward. Now imagine you see a single image of, say, a sparrow. If you have seen many images of sparrow-like birds in the past, you can reconstruct a reasonable estimate of both the pose of the sparrow and its geometric model from that single image. Using the past

images you build a small, parametric family of geometric models for sparrow-like birds; then an optimization procedure is used to find the best set of parameters and viewpoints to explain the image that you see. This argument works to supply texture for that model, too, even for the parts you cannot see (Figure 25.22 ).

Figure 25.22

If you have seen many pictures of some category—say, birds (top)—you can use them to produce a 3D reconstruction from a single new view (bottom). You need to be sure that all objects have a fairly similar geometry (so a picture of an ostrich won’t help if you’re looking at a sparrow), but classification methods can sort this out. From the many images you can estimate how texture values in the image are distributed across the object, and thus complete the texture for parts of the bird you haven’t seen yet (bottom). Figure courtesy of Angjoo Kanazawa, produced by a system described in Kanazawa et al. (2018b).

Top photo credit: Satori/123RF; Bottom left credit: Four Oaks/Shutterstock.

25.7.5 Making pictures It is now common to insert computer graphics models into photographs in a convincing fashion, as in Figure 25.23 , where a statue has been placed into a photo of a room. First estimate a depth map and albedo for the picture. Then estimate the lighting in the image by matching it to other images with known lighting. Place the object in the image’s depth map, and render the resulting world with a physical rendering program—a standard tool in computer graphics. Finally, blend the modified image with the original image.

Figure 25.23

On the left, an image of a real scene. On the right, a computer graphics object has been inserted into the scene. You can see that the light appears to be coming from the right direction, and that the object seems to cast appropriate shadows. The generated image is convincing even if there are small errors in the lighting and shadows, because people are not expert at identifying these errors. Figure courtesy of Kevin Karsch, produced by a system described in Karsch et al. (2011)

Neural networks can also be trained to do image transformation: mapping images from type X—for example, a blurry image; an aerial image of a town; or a drawing of a new product—to images of type Y—for example, a deblurred version of the image; a road map; or a product photograph. This is easiest when the training data consists of (X, Y) pairs of images—in Figure 25.24  each example pair has an aerial image and the corresponding road map section. The training loss compares the output of the network with the desired output, and also has a loss component from a generative adversarial network (GAN) that ensures that the output has the right kinds of features for images of type Y. As we see in the test portion of Figure 25.24 , systems of this kind perform very well.

Figure 25.24

Paired image translation where the input consists of aerial images and the corresponding map tiles, and the goal is to train a network to produce a map tile from an aerial image. (The system can also learn to generate aerial images from map tiles.) The network is trained by comparing ˆi (the output for example

y xi of type X) to the right output yi of type Y . Then at test time, the network must make new images of type Y from new inputs of type X. Figure courtesy of Phillip Isola, Jun-Yan Zhu and Alexei A. Efros, produced by a system described in Isola et al. (2017). Map data © 2019 Google.

Image transformation

Sometimes we don’t have images that are paired with each other, but we do have a big collection of images of type X (say, pictures of horses) and a separate collection of type Y (say, pictures of zebras). Imagine an artist who is tasked with creating an image of a zebra running in a field. The artist would appreciate being able to select just the right image of a horse, and then having the computer automatically transform the horse into a zebra (Figure 25.25 ). To achieve this we can train two transformation networks, with an additional constraint called a cycle constraint. The first network maps horses to zebras; the second network maps zebras to horses; and the cycle constraint requires that when you map X to Y to X (or Y to X to Y), you get what you started with. Again, GAN losses ensure that the horse (or zebra) pictures that the networks output are “like” real horse (or zebra) pictures.

Figure 25.25

Unpaired image translation: given two populations of images (here type X is horses and type Y is zebras), but no corresponding pairs, learn to translate a horse into a zebra. The method trains two predictors: one

x x

that maps type X to type Y, and another that maps type Y to type X. If the first network maps a horse i to a zebra ˆi , the second network should map ˆi back to the original i . The difference between i and ˆi is what trains the two networks. The cycle from Y to X and back must be closed. Such networks can

y

y

x

x

successfully impose rich transformations on images.

Figure courtesy of Alexei A. Efros; see Zhu et al. (2017). Running horse photo by Justyna Furmanczyk Gibaszek/Shutterstock.

Another artistic effect is called style transfer: the input consists of two images—the content (for example, a photograph of a cat); and the style (for example, an abstract painting). The output is a version of the cat rendered in the abstract style (see Figure 25.26 ). The key insight to solving this problem is that if we examine a deep convolutional neural network (CNN) that has been trained to do object recognition (say, on ImageNet), we find that the early layers tend to represent the style of a picture, and the late layers represent the content.

p be the content image and s be the style image, and let E(x) be the vector of activations of an early layer on image x and L(x) be the vector of activations of a late layer on image x. Then we want to generate some image x that has similar content to the house photo, that is, minimizes |L(x) − L(p)|, and also has similar style to the impressionist painting, that is, minimizes |E(x) − E(s)|. We use gradient descent with a loss function that is a linear combination of these two factors to find an image x that minimizes the loss. Let

Figure 25.26

Style transfer: The content of a photo of a cat is combined with the style of an abstract painting to yield a new image of the cat rendered in the abstract style (right). The painting is Wassily Kandinsky’s Lyrisches or The Lyrical (public domain); the cat is Cosmo.

Style transfer

Generative adversarial networks (GANs) can create novel photorealistic images, fooling most people most of the time. One kind of image is the deepfake—an image or video that looks like a particular person, but is generated from amodel. For example, when Carrie Fisher was 60, a generated replica of her 19-year-old face was superimposed on another actor’s body for the making of Rogue One. The movie industry creates ever-better deepfakes

for artistic purposes, and researchers work on countermeasures for detecting deepfakes, to mitigate the destructive effects of fake news.

Deepfake

Generated images can also be used to maintain privacy. For example, there are image data sets in radiological practices that would be useful for researchers, but can’t be published because of patient confidentiality. Generative image models can take a private data set of images and produce a synthetic data set that can be shared with researchers. This data set should be (a) like the training data set; (b) different; and (c) controllable. Consider chest Xrays. The synthetic data set should be like the training data set in the sense that each image individually would fool a radiologist and the frequencies of each effect should be right, so a radiologist would not be surprised by how often (say) pneumonia appears. The new data set should be different, in the sense that it does not reveal personally identifiable information. The new data set should be controllable, so that the frequencies of effects can be adjusted to reflect the communities of interest. For example, pneumonias are more common in the elderly than in young adults. Each of these goals is technically difficult to reach, but image data sets have been created that fool practicing radiologists some of the time (Figure 25.27 ).

Figure 25.27

GAN generated images of lung X-rays. On the left, a pair consisting of a real X-ray and a GAN-generated X-ray. On the right, results of a test asking radiologists, given a pair of X-rays as seen on the left, to tell which is the real X-ray. On average, they chose correctly 61% of the time, somewhat better than chance. But they differed in their accuracy—the chart on the right shows the error rate for 12 different radiologists; one of them had an error rate near 0% and another had 80% errors. The size of each dot indicates the number of images each radiologist viewed. Figure courtesy of Alex Schwing, produced by a system described in Deshpande et al. (2019).

25.7.6 Controlling movement with vision One of the principal uses of vision is to provide information both for manipulating objects— picking them up, grasping them, twirling them, and so on—and for navigating while avoiding obstacles. The ability to use vision for these purposes is present in the most primitive of animal visual systems. In many cases, the visual system is minimal, in the sense that it extracts from the available light field just the information the animal needs to inform its behavior. Quite probably, modern vision systems evolved from early, primitive organisms that used a photosensitive spot at one end in order to orient themselves toward (or away from) the light. We saw in Section 25.6  that flies use a very simple optical flow detection system to land on walls.

Suppose that, rather than landing on walls, we want to build a self-driving car. This is a project that places much greater demands on the perceptual system. Perception in a selfdriving car has to support the following tasks:

LATERAL CONTROL: Ensure that the vehicle remains securely within its lane or changes lanes smoothly when required. LONGITUDINAL CONTROL: Ensure that there is a safe distance to the vehicle in front. OBSTACLE AVOIDANCE: Monitor vehicles in neighboring lanes and be prepared for evasive maneuvers. Detect pedestrians and allow them to cross safely. OBEY TRAFFIC SIGNALS: These include traffic lights, stop signs, speed limit signs, and police hand signals.

The problem for a driver (human or computer) is to generate appropriate steering, acceleration, and braking actions to best accomplish these tasks.

To make good decisions, the driver should construct a model of the world and the objects in it. Figure 25.28  shows some of the visual inferences that are necessary to build this model.

For lateral control, the driver needs tomaintain a representation of the position and orientation of the car relative to the lane. For longitudinal control, the driver needs to keep a safe distance from the vehicle in front (which may not be easy to identify on, say, curving multilane roads). Obstacle avoidance and following traffic signals require additional inferences.

Figure 25.28

Mobileye’s camera-based sensing for autonomous vehicles. Top row: Two images from a front-facing camera, taken a few seconds apart. The green area is the free space—the area to which the vehicle could physically move in the immediate future. Objects are displayed with 3D bounding boxes defining their sides (red for the rear, blue for the right side, yellow for the left side, and green for the front). Objects include vehicles, pedestrians, the inner edge of the self-lane marks (necessary for lateral control), other painted road and crosswalk marks, traffic signs, and traffic lights. Not shown are animals, poles and cones, sidewalks, railings, and general objects (e.g., a couch that fell from the back of a truck). Each object is then marked with a 3D position and velocity. Bottom row: A full physical model of the environment, rendered from the detected objects.

(Images show Mobileye’s vision-only system results). Images courtesy of Mobileye.

Roads were designed for humans who navigate using vision, so it should in principle be possible to drive using vision alone. However, in practice, commercial self-driving cars use a variety of sensors, including cameras, lidars, radars, and microphones. A lidar or radar

enables direct measurement of depth, which can be more accurate than the vision-only methods of Section 25.6 . Having multiple sensors increases performance in general, and is particularly important in conditions of poor visibility; for example, radar can cut through fog that blocks cameras and lidars. Microphones can detect approaching vehicles (especially ones with sirens) before they become visible.

There has also been much research on mobile robots navigating in indoor and outdoor environments. Applications abound, such as the last mile of package or pizza delivery. Traditional approaches break this task up into two stages as shown in Figure 25.29 :

MAP BUILDING: Simultaneous Localization and Mapping or SLAM (see page 935) is the task of constructing a 3D model of the world, including the location of the robot in the world (or more specifically, the location of each of the robot’s cameras). This model (typically represented as a point cloud of obstacles) can be built from a series of images from different camera positions. PATH PLANNING: Once the robot has access to this 3D map and can localize itself in it, the objective becomes one of finding a collision-free trajectory from the current position to the goal location (see Section 26.6 ).

Figure 25.29

Navigation is tackled by decomposition into two problems: mapping and planning. With each successive time step, information from sensors is used to incrementally build an uncertain model of the world. This model along with the goal specification is passed to a planner that outputs the next action that the robot should take in order to achieve the goal. Models of the world can be purely geometric (as in classical SLAM), or semantic (as obtained via learning), or even topological (based on landmarks). The actual robot appears on the right.

Figures courtesy of Saurabh Gupta.

Many variants of this general approach have been explored. For instance, in the cognitive mapping and planning approach, the two stages of map building and path planning are two modules in a neural network that is trained end-to-end to minimize a loss function. Such a system does not have to build a complete map—which is often redundant and unnecessary— if all you need is enough information to navigate from point A to point B without colliding with obstacles.

Summary Although perception appears to be an effortless activity for humans, it requires a significant amount of sophisticated computation. The goal of vision is to extract information needed for tasks such as manipulation, navigation, and object recognition.

The geometry and optics of image formation is well understood. Given a description of a 3D scene, we can easily produce a picture of it from some arbitrary camera position— this is the graphics problem. The inverse problem, the computer vision problem—taking a picture and turning it into a 3D description—is more difficult. Representations of images capture edges, texture, optical flow, and regions. These yield cues to the boundaries of objects and to correspondence between images. Convolutional neural networks produce accurate image classifiers that use learned features. Rather roughly, the features are patterns of patterns of patterns. . . . It is hard to predict when these classifiers will work well, because the test data may be unlike the training data in some important way. Experience teaches that they are often accurate enough to use in practice. Image classifiers can be turned into object detectors. One classifier scores boxes in an image for objectness; another then decides whether an object is in the box, and what object it is. Object detection methods aren’t perfect, but are usable for a wide variety of applications. With more than one view of a scene, it is possible to recover the 3D structure of the scene and the relationship between views. In many cases, it is possible to recover 3D geometry from a single view. The methods of computer vision are being very widely applied.

Bibliographical and Historical Notes This chapter has concentrated on vision, but other perceptual channels have been studied and put to use in robotics. For auditory perception (hearing), we have already covered speech recognition, and there has also been considerable work on music perception (Koelsch and Siebel, 2005) and machine learning of music (Engel et al., 2017) as well as on machine learning for sounds in general (Sharan and Moir, 2016).

Tactile perception or touch (Luo et al., 2017) is important in robotics and is discussed in Chapter 26 . Automated olfactory perception (smell) has seen less work, but it has been shown that deep learning models can learn to predict smells based on the structure of molecules (Sanchez-Lengeling et al., 2019).

Systematic attempts to understand human vision can be traced back to ancient times. Euclid (ca. 300

BCE)

wrote about natural perspective—the mapping that associates, with each point

P in the three-dimensional world, the direction of the ray OP joining the center of projection O to the point P . He was well aware of the notion of motion parallax. Ancient Roman paintings, such as the ones perserved by the eruption of Vesuvius in 79 CE, used an informal kind of perspective, with more than one horizon line.

The mathematical understanding of perspective projection, this time in the context of projection onto planar surfaces, had its next significant advance in the 15th century in Renaissance Italy. Brunelleschi is usually credited with creating the first paintings based on geometrically correct projection of a three-dimensional scene in about 1413. In 1435, Alberti codified the rules and inspired generations of artists. Particularly notable in their development of the science of perspective, as it was called in those days, were Leonardo da Vinci and Albrecht Dürer. Leonardo’s late 15th-century descriptions of the interplay of light and shade (chiaroscuro), umbra and penumbra regions of shadows, and aerial perspective are still worth reading in translation (Kemp, 1989).

Although perspective was known to the Greeks, they were curiously confused by the role of the eyes in vision. Aristotle thought of the eyes as devices emitting rays, rather in the manner of modern laser range finders. This mistaken view was laid to rest by the work of Arab scientists, such as Alhazen, in the 10th century.

The development of various kinds of cameras followed. These consisted of rooms (camera is Latin for “chamber”) where light would be let in through a small hole in one wall to cast an image of the scene outside on the opposite wall. Of course, in all these cameras, the image was inverted, which caused no end of confusion. If the eye was to be thought of as such an imaging device, how do we see right side up? This enigma exercised the greatest minds of the era (including Leonardo). It took the work of Kepler and Descartes to settle the question. Descartes placed an eye from which the opaque cuticle had been removed in a hole in a window shutter. The result was an inverted image formed on a piece of paper laid out on the retina. Although the retinal image is indeed inverted, this does not cause a problem because the brain interprets the image the right way. In modern jargon, one just has to access the data structure appropriately.

The next major advances in the understanding of vision took place in the 19th century. The work of Helmholtz and Wundt, described in Chapter 1 , established psychophysical experimentation as a rigorous scientific discipline. Through the work of Young, Maxwell, and Helmholtz, a trichromatic theory of color vision was established. The fact that humans can see depth if the images presented to the left and right eyes are slightly different was demonstrated by Wheatstone’s (1838) invention of the stereoscope. The device immediately became popular in parlors and salons throughout Europe.

The essential concept of binocular stereopsis—that two images of a scene taken from slightly different viewpoints carry information sufficient to obtain a three-dimensional reconstruction of the scene—was exploited in the field of photogrammetry. Key mathematical results were obtained; for example, Kruppa (1913) proved that, given two views of five distinct points in a scene, one could reconstruct the rotation and translation between the two camera positions as well as the depth of the scene (up to a scale factor).

Although the geometry of stereopsis had been understood for a long time, the correspondence problem in photogrammetry used to be solved by humans trying to match up corresponding points. The amazing ability of humans in solving the correspondence problem was illustrated by Julesz’s (1971) random dot stereograms. The field of computer vision has devoted much effort towards an automatic solution of the correspondence problem.

In the first half of the 20th century, the most significant research results in vision were obtained by the Gestalt school of psychology, led by Max Wertheimer. They pointed out the importance of perceptual organization: for a human, the image is not a collection of pointillist photoreceptor outputs (pixels), rather it is organized into coherent groups. The computer vision task of finding regions and curves traces back to this insight. The Gestaltists also drew attention to the “figure-ground” phenomenon—a contour separating two image regions that in the world are at different depths appears to belong only to the nearer region, the “figure,” and not to the farther region, the “ground.”

The gestalt work was carried on by J. J. Gibson (1950, 1979), who pointed out the importance of optical flow and texture gradients in the estimation of environmental variables such as surface slant and tilt. He reemphasized the importance of the stimulus and how rich it was. Gibson, Olum, and Rosenblatt (1955) pointed out that the optical flow field contained enough information to determine the motion of the observer relative to the environment. Gibson particularly emphasized the role of the active observer, whose selfdirected movement facilitates the pickup of information about the external environment.

Computer vision dates back to the 1960s. Roberts’s (1963) thesis at MIT on perceiving cubes and other blocks-world objects was one of the earliest publications in the field. Roberts introduced several key ideas, including edge detection and model-based matching.

In the 1960s and 1970s progress was slow, hampered by the lack of computational and storage resources. Low-level visual processing received a lot of attention, with techniques drawn from related fields such as signal processing, pattern recognition, and data clustering.

Edge detection was treated as an essential first step in image processing, as it reduced the amount of data to be processed. The widely used Canny edge detection technique was introduced by John Canny (1986). Martin, Fowlkes, and Malik (2004) showed how to combine multiple clues, such as brightness, texture and color, in a machine learning framework to better find boundary curves.

The closely related problem of finding regions of coherent brightness, color, and texture naturally lends itself to formulations where finding the best partition becomes an optimization problem. Three leading examples are based on Markov Random Fields due to

Geman and Geman (1984), the variational formulation of Mumford and Shah (1989), and normalized cuts by Shi and Malik (2000).

Through much of the 1960s, 1970s, and 1980s, there were two distinct paradigms in which visual recognition was pursued, dictated by different perspectives on what was perceived to be the primary problem. Computer vision research on object recognition largely focused on issues arising from the projection of three-dimensional objects onto two-dimensional images. The idea of alignment, also first introduced by Roberts, resurfaced in the 1980s in the work of Lowe (1987) and Huttenlocher and Ullman (1990).

The pattern recognition community took a different approach, viewing the 3D–to–2D aspects of the problem as insignificant. Their motivating examples were in domains such as optical character recognition and handwritten zip code recognition, in which the primary concern is that of learning the typical variations characteristic of a class of objects and separating them from other classes. We can trace neural net architectures for image analysis back to Hubel and Wiesel’s (1962, 1968) studies of the visual cortex in cats and monkeys. They developed a hierarchical model of the visual pathway with neurons in lower areas of the brain (especially the area called V1) responding to features such as oriented edges and bars, and neurons in higher areas responding to more specific stimuli (“grandmother cells” in the cartoon version).

Fukushima (1980) proposed a neural network architecture for pattern recognition explicitly motivated by Hubel and Wiesel’s hierarchy. His model had alternating layers of simple cells and complex cells, thus incorporating downsampling, and also had shift invariance, thus incorporating convolutional structure. LeCun et al. (1989) took the additional step of using back-propagation to train the weights of this network, and what we today call convolutional neural networks were born. See LeCun et al. (1995) for a comparison of approaches.

Starting in the late 1990s, accompanying a much greater role of probabilistic modeling and statistical machine learning in the field of artificial intelligence in general, there was a rapprochement between these two traditions. Two lines of work contributed significantly. One was research on face detection (Rowley et al., 1998; Viola and Jones, 2004) that demonstrated the power of pattern recognition techniques on clearly important and useful tasks.

The other was the development of point descriptors, which enable the construction of feature vectors from parts of objects (Schmid and Mohr, 1996). There are three key strategies to build a good local point descriptor: one uses orientations to get illumination invariance; one needs to describe image structure close to a point in detail, and further away only roughly; and one needs to use spatial histograms to suppress variations caused by small errors in locating the point. Lowe’s (2004) SIFT descriptor exploited these ideas very effectively; another popular variant was the HOG descriptor due to Dalal and Triggs (2005).

The 1990s and 2000s saw a continuing debate between the devotees of clever feature design such as SIFT and HOG and the aficionados of neural networks who believed that good features should emerge automatically from end-to-end training. The way to settle such a debate is through benchmarks on standard data sets, and in the 2000s results on a standard object detection data set, PASCAL VOC, argued in favor of hand-designed features. This changed when Krizhevsky et al. (2013) showed that on the task of image classification on the ImageNet data set, their neural network (called AlexNet) gave significantly lower error rates than the mainstream computer vision techniques.

What was the secret sauce behind the success of AlexNet? Besides the technical innovations (such as the use of ReLU activation units) we must give a lot of credit to big data and big computation. By big data we mean the availability of large data sets with category labels, such as ImageNet, which provided the training data for these large, deep networks with millions of parameters. Previous data sets like Caltech-101 or PASCAL VOC didn’t have enough training data, and MNIST and CIFAR were regarded as “toy data sets” by the computer vision community. This strand of labeling data sets for benchmarking and for extracting image statistics itself was enabled by the desire of people to upload their photo collections to the Internet on sites such as Flickr. The way big computation proved most helpful was through GPUs, a hardware development initially driven by the needs of the video game industry.

Within a year or two, the evidence was quite clear. For example, the region-based convolutional neural network (RCNN) work of Girshick et al. (2016) showed that the AlexNet architecture could be modified, by making use of computer vision ideas such as region proposals, to make possible state-of-the-art object detection on PASCAL VOC.We have realized that generally deeper networks work better and that overfitting fears are

overblown. We have new techniques such as batch normalization to deal with regularization.

The reconstruction of three-dimensional structure from multiple views has its roots in the photogrammetry literature. In the computer vision era, Ullman (1979), and Longuet-Higgins (1981) are influential early works. Concerns about the stability of structure from motion were significantly allayed by the work of Tomasi and Kanade (1992) who showed that with the use of multiple frames, and the resulting wide baseline, shape could be recovered quite accurately.

A conceptual innovation introduced in the 1990s was the study of projective structure from motion. Here camera calibration is not necessary, as was shown by Faugeras (1992). This discovery is related to the introduction of the use of geometrical invariants in object recognition, as surveyed by Mundy and Zisserman (1992), and the development of affine structure from motion by Koenderink and Van Doorn (1991).

In the 1990s, with great increase in computer speed and storage and the widespread availability of digital video, motion analysis found many new applications. Building geometrical models of real-world scenes for rendering by computer graphics techniques proved particularly popular, led by reconstruction algorithms such as the one developed by Debevec et al. (1996). The books by Hartley and Zisserman (2000) and Faugeras et al. (2001) provide a comprehensive treatment of the geometry of multiple views.

Humans can perceive shape and spatial layout from a single image, and modeling this has proved to be quite a challenge for computer vision researchers. Inferring shape from shading was first studied by Berthold Horn (1970), and Horn and Brooks (1989) present an extensive survey of the main papers from a period when this was a much studied problem. Gibson (1950) was the first to propose texture gradients as a cue to shape. The mathematics of occluding contours, and more generally understanding the visual events in the projection of smooth curved objects, owes much to the work of Koenderink and van Doorn, which finds an extensive treatment in Koenderink’s (1990) Solid Shape.

More recently, attention has turned to treating the problem of shape and surface recovery from a single image as a probabilistic inference problem, where geometrical cues are not

modeled explicitly, but used implicitly in a learning framework. A good example is the work of Hoiem et al. (2007); recently this has been reworked using deep neural networks.

Turning now to the applications of computer vision for guiding action, Dickmanns and Zapp (1987) first demonstrated a self-driving car driving on freeways at high speeds; Pomerleau (1993) achieved similar performance using a neural network approach. Today building selfdriving cars is a big business, with the established car companies competing with new entrants such as Baidu, Cruise, Didi, Google Waymo, Lyft, Mobileye, Nuro, Nvidia, Samsung, Tata, Tesla, Uber, and Voyage to market systems that provide capabilities ranging from driver assistance to full autonomy.

For the reader interested in human vision, Vision Science: Photons to Phenomenology by Stephen Palmer (1999) provides the best comprehensive treatment; Visual Perception: Physiology, Psychology and Ecology by Vicki Bruce, Patrick Green, and Mark Georgeson (2003) is a shorter textbook. The books Eye, Brain and Vision by David Hubel (1988) and Perception by Irvin Rock (1984) are friendly introductions centered on neurophysiology and perception respectively. David Marr’s book Vision (Marr, 1982) played a historical role in connecting computer vision to the traditional areas of biological vision—psychophysics and neurobiology. While many of his specific models for tasks such as edge detection and object recognition haven’t stood the test of time, the theoretical perspective where each task is analyzed at an informational, computational, and implementation level is still illuminating.

For the field of computer vision, the most comprehensive textbooks available today are Computer Vision: A Modern Approach (Forsyth and Ponce, 2002) and Computer Vision: Algorithms and Applications (Szeliski, 2011). Geometrical problems in computer vision are treated thoroughly in Multiple View Geometry in Computer Vision (Hartley and Zisserman, 2000). These books were written before the deep learning revolution, so for the latest results, consult the primary literature.

Two of the main journals for computer vision are the IEEE Transactions on Pattern Analysis and Machine Intelligence and the International Journal of Computer Vision. Computer vision conferences include ICCV (International Conference on Computer Vision), CVPR (Computer Vision and Pattern Recognition), and ECCV (European Conference on Computer Vision). Research with a significant machine learning component is also published at NeurIPS (Neural Information Processing Systems), and work on the interface with computer

graphics often appears at the ACM SIGGRAPH (Special Interest Group in Graphics) conference. Many vision papers appear as preprints on the arXiv server, and early reports of new results appear in blogs from the major research labs.

Chapter 26 Robotics In which agents are endowed with sensors and physical effectors with which to move about and make mischief in the real world.

26.1 Robots Robots are physical agents that perform tasks by manipulating the physical world. To do so, they are equipped with effectors such as legs, wheels, joints, and grippers. Effectors are designed to assert physical forces on the environment. When they do this, a few things may happen: the robot’s state might change (e.g., a car spins its wheels and makes progress on the road as a result), the state of the environment might change (e.g., a robot arm uses its gripper to push a mug across the counter), and even the state of the people around the robot might change (e.g., an exoskeleton moves and that changes the configuration of a person’s leg; or a mobile robot makes progress toward the elevator doors, and a person notices and is nice enough to move out of the way, or even push the button for the robot).

Robot

Effector

Robots are also equipped with sensors, which enable them to perceive their environment. Present-day robotics employs a diverse set of sensors, including cameras, radars, lasers, and microphones to measure the state of the environment and of the people around it; and gyroscopes, strain and torque sensors, and accelerometers to measure the robot’s own state.

Sensor

Maximizing expected utility for a robot means choosing how to actuate its effectors to assert the right physical forces—the ones that will lead to changes in state that accumulate as much

expected reward as possible. Ultimately, robots are trying to accomplish some task in the physical world.

Robots operate in environments that are partially observable and stochastic: cameras cannot see around corners, and gears can slip. Moreover, the people acting in that same environment are unpredictable, so the robot needs to make predictions about them.

Robots usually model their environment with a continuous state space (the robot’s position has continuous coordinates) and a continuous action space (the amount of current a robot sends to its motor is also measured in continuous units). Some robots operate in highdimensional spaces: cars need to know the position, orientation, and velocity of themselves and the nearby agents; robot arms have six or seven joints that can each be independently moved; and robots that mimic the human body have hundreds of joints.

Robotic learning is constrained because the real world stubbornly refuses to operate faster than real time. In a simulated environment, it is possible to use learning algorithms (such as the Q-learning algorithm described in Chapter 22 ) to learn in a few hours from millions of trials. In a real environment, it might take years to run these trials, and the robot cannot risk (and thus cannot learn from) a trial that might cause harm. Thus, transferring what has been learned in simulation to a real robot in the real world—the sim-to-real problem—is an active area of research. Practical robotic systems need to embody prior knowledge about the robot, the physical environment, and the tasks to be performed so that the robot can learn quickly and perform safely.

Robotics brings together many of the concepts we have seen in this book, including probabilistic state estimation, perception, planning, unsupervised learning, reinforcement learning, and game theory. For some of these concepts robotics serves as a challenging example application. For other concepts this chapter breaks new ground, for instance in introducing the continuous version of techniques that we previously saw only in the discrete case.

26.2 Robot Hardware So far in this book, we have taken the agent architecture—sensors, effectors, and processors —as given, and have concentrated on the agent program. But the success of real robots depends at least as much on the design of sensors and effectors that are appropriate for the task.

26.2 Types of robots from the hardware perspective When you think of a robot, you might imagine something with a head and two arms, moving around on legs or wheels. Such anthropomorphic robots have been popularized in fiction such as the movie The Terminator and the cartoon The Jetsons. But real robots come in many shapes and sizes.

Anthropomorphic robot

Manipulator

Manipulators are just robot arms. They do not necessarily have to be attached to a robot body; they might simply be bolted onto a table or a floor, as they are in factories (Figure 26.1 (a) ). Some have a large payload, like those assembling cars, while others, like wheelchair-mountable arms that assist people with motor impairments (Figure 26.1 (b) ), can carry less but are safer in human environments.

Figure 26.1

(a) An industrial robotic arm with a custom end-effector. Image credit: Macor/123RF. (b) A Kinova® JACO® Assistive Robot arm mounted on a wheelchair. Kinova and JACO are trademarks of Kinova, Inc.

Mobile robots are those that use wheels, legs, or rotors to move about the environment. Quadcopter drones are a type of unmanned aerial vehicle (UAV); autonomous underwater vehicles (AUVs) roam the oceans. But many mobile robots stay indoors and move on wheels, like a vacuum cleaner or a towel delivery robot in a hotel. Their outdoor counterparts include autonomous cars or rovers that explore new terrain, even on the surface of Mars (Figure 26.2 ). Finally, legged robots are meant to traverse rough terrain that is inaccessible with wheels. The downside is that controlling legs to do the right thing is more challenging than spinning wheels.

Figure 26.2

(a) NASA’s Curiosity rover taking a selfie on Mars. Image courtesy of NASA. (b) A Skydio drone accompanying a family on a bike ride. Image courtesy of Skydio.

Mobile robot

Quadcopter drone

UAV

AUV

Autonomous car

Rover

Legged robot

Other kinds of robots include prostheses, exoskeletons, robots with wings, swarms, and intelligent environments in which the robot is the entire room.

26.2.2 Sensing the world Sensors are the perceptual interface between robot and environment. Passive sensors, such as cameras, are true observers of the environment: they capture signals that are generated by other sources in the environment. Active sensors, such as sonar, send energy into the environment. They rely on the fact that this energy is reflected back to the sensor. Active sensors tend to provide more information than passive sensors, but at the expense of increased power consumption and with a danger of interference when multiple active sensors are used at the same time. We also distinguish whether a sensor is directed at sensing the environment, the robot’s location, or the robot’s internal configuration.

Passive sensor

Active sensor

Range finder

Sonar

Stereo vision

Range finders are sensors that measure the distance to nearby objects. Sonar sensors are active range finders that emit directional sound waves, which are reflected by objects, with some of the sound making it back to the sensor. The time and intensity of the returning signal indicates the distance to nearby objects. Sonar is the technology of choice for autonomous underwater vehicles, and was popular in the early days of indoor robotics. Stereo vision (see Section 25.6 ) relies on multiple cameras to image the environment from slightly different viewpoints, analyzing the resulting parallax in these images to compute the range of surrounding objects.

For mobile ground robots, sonar and stereo vision are now rarely used, because they are not reliably accurate. The Kinect is a popular low-cost sensor that combines a camera and a structured light projector, which projects a pattern of grid lines onto a scene. The camera sees how the grid lines bend, giving the robot information about the shape of the objects in the scene. If desired, the projection can be infrared light, so as not to interfere with other sensors (such as human eyes).

Structured light

Most ground robots are now equipped with active optical range finders. Just like sonar sensors, optical range sensors emit active signals (light) and measure the time until a reflection of this signal arrives back at the sensor. Figure 26.3(a)  shows a time-of-flight camera. This camera acquires range images like the one shown in Figure 26.3(b)  at up to 60 frames per second. Autonomous cars often use scanning lidars (short for light detection and ranging)—active sensors that emit laser beams and sense the reflected beam, giving

range measurements accurate to within a centimeter at a range of 100 meters. They use complex arrangements of mirrors or rotating elements to sweep the beam across the environment and build a map. Scanning lidars tend to work better than time-of-flight cameras at longer ranges, and tend to perform better in bright daylight.

Figure 26.3

(a) Time-of-flight camera; image courtesy of Mesa Imaging GmbH. (b) 3D range image obtained with this camera. The range image makes it possible to detect obstacles and objects in a robot’s vicinity. Image courtesy of Willow Garage, LLC.

Time-of-fligh camera

Scanning lidar

Radar is often the range finding sensor of choice for air vehicles (autonomous or not). Radar sensors can measure distances up to kilometers, and have an advantage over optical sensors in that they can see through fog. On the close end of range sensing are tactile sensors such

as whiskers, bump panels, and touch-sensitive skin. These sensors measure range based on physical contact, and can be deployed only for sensing objects very close to the robot.

Radar

Tactile sensor

Location sensor

Global Positioning System

A second important class is location sensors. Most location sensors use range sensing as a primary component to determine location. Outdoors, the Global Positioning System (GPS) is the most common solution to the localization problem. GPS measures the distance to satellites that emit pulsed signals. At present, there are 31 operational GPS satellites in orbit, and 24 GLONASS satellites, the Russian counterpart. GPS receivers can recover the distance to a satellite by analyzing phase shifts. By triangulating signals from multiple satellites, GPS receivers can determine their absolute location on Earth to within a few meters. Differential GPS involves a second ground receiver with known location, providing millimeter accuracy under ideal conditions.

Differential GPS

Unfortunately, GPS does not work indoors or underwater. Indoors, localization is often achieved by attaching beacons in the environment at known locations. Many indoor environments are full of wireless base stations, which can help robots localize through the analysis of the wireless signal. Underwater, active sonar beacons can provide a sense of location, using sound to inform AUVs of their relative distances to those beacons.

The third important class is proprioceptive sensors, which inform the robot of its own motion. To measure the exact configuration of a robotic joint, motors are often equipped with shaft decoders that accurately measure the angular motion of a shaft. On robot arms, shaft decoders help track the position of joints. On mobile robots, shaft decoders report wheel revolutions for odometry—the measurement of distance traveled. Unfortunately, wheels tend to drift and slip, so odometry is accurate only over short distances. External forces, such as wind and ocean currents, increase positional uncertainty. Inertial sensors, such as gyroscopes, reduce uncertainty by relying on the resistance of mass to the change of velocity.

Proprioceptive sensor

Shaft decoder

Odometry

Inertial sensor

Other important aspects of robot state are measured by force sensors and torque sensors. These are indispensable when robots handle fragile objects or objects whose exact size and

shape are unknown. Imagine a one-ton robotic manipulator screwing in a light bulb. It would be all too easy to apply too much force and break the bulb. Force sensors allow the robot to sense how hard it is gripping the bulb, and torque sensors allow it to sense how hard it is turning. High-quality sensors can measure forces in all three translational and three rotational directions. They do this at a frequency of several hundred times a second so that a robot can quickly detect unexpected forces and correct its actions before it breaks a light bulb. However, it can be a challenge to outfit a robot with high-end sensors and the computational power to monitor them.

Force sensor

Torque sensor

26.2.3 Producing motion The mechanism that initiates the motion of an effector is called an actuator; examples include transmissions, gears, cables, and linkages. The most common type of actuator is the electric actuator, which uses electricity to spin up a motor. These are predominantly used in systems that need rotational motion, like joints on a robot arm. Hydraulic actuators use pressurized hydraulic fluid (like oil or water) and pneumatic actuators use compressed air to generate mechanical motion.

Actuator

Hydraulic actuator

Pneumatic actuator

Actuators are often used to move joints, which connect rigid bodies (links). Arms and legs have such joints. In revolute joints, one link rotates with respect to the other. In prismatic joints, one link slides along the other. Both of these are single-axis joints (one axis of motion). Other kinds of joints include spherical, cylindrical, and planar joints, which are multi-axis joints.

Revolute joint

Prismatic joint

Parallel jaw gripper

To interact with objects in the environment, robots use grippers. The most basic type of gripper is the parallel jaw gripper, with two fingers and a single actuator that moves the fingers together to grasp objects. This effector is both loved and hated for its simplicity. Three-fingered grippers offer slightly more flexibility while maintaining simplicity. At the other end of the spectrum are humanoid (anthropomorphic) hands. For instance, the Shadow Dexterous Hand has a total of 20 actuators. This offers a lot more flexibility for complex manipulation, including in-hand manipulator maneuvers (think of picking up your cell phone and rotating it in-hand to orient it right-side up), but this flexibility comes at a price—learning to control these complex grippers is more challenging.

26.3 What kind of problem is robotics solving? Now that we know what the robot hardware might be, we’re ready to consider the agent software that drives the hardware to achieve our goals. We first need to decide the computational framework for this agent. We have talked about search in deterministic environments, MDPs for stochastic but fully observable environments, POMDPs for partial observability, and games for situations in which the agent is not acting in isolation. Given a computational framework, we need to instantiate its ingredients: reward or utility functions, states, actions, observation spaces, etc.

We have already noted that robotics problems are nondeterministic, partially observable, and multiagent. Using the game-theoretic notions from Chapter 18 , we can see that sometimes the agents are cooperative and sometimes they are competitive. In a narrow corridor where only one agent can go first, a robot and a person collaborate because they both want to make sure they don’t bump into each other. But in some cases they might compete a bit to reach their destination quickly. If the robot is too polite and always makes room, it might get stuck in crowded situations and never reach its goal.

Therefore, when robots act in isolation and know their environment, the problem they are solving can be formulated as an MDP; when they are missing information it becomes a POMDP; and when they act around people it can often be formulated as a game.

What is the robot’s reward function in this formulation? Usually the robot is acting in service of a human—for example delivering a meal to a hospital patient for the patient’s reward, not its own. For most robotics settings, even though robot designers might try to specify a good enough proxy reward function, the true reward function lies with the user whom the robot is supposed to help. The robot will either need to decipher the user’s desires, or rely on an engineer to specify an approximation of the user’s desires.

As for the robot’s action, state, and observation spaces, the most general form is that observations are raw sensor feeds (e.g., the images coming in from cameras, or the laser hits coming in from lidar); actions are raw electric currents being sent to the motors; and state is what the robot needs to know for its decision making. This means there is a huge gap

between the low-level percepts and motor controls, and the high-level plans the robot needs to make. To bridge the gap, roboticists decouple aspects of the problem to simplify it.

For instance, we know that when we solve POMDPs properly, perception and action interact: perception informs which actions make sense, but action also informs perception, with agents taking actions to gather information when that information has value in later time steps. However, robots often separate perception from action, consuming the outputs of perception and pretending they will not get any more information in the future. Further, hierarchical planning is called for, because a high-level goal like “get to the cafeteria” is far removed from a motor command like “rotate the main axle 1 ,” ∘

In robotics we often use a three-level hierarchy. The task planning level decides a plan or policy for high-level actions, sometimes called action primitives or subgoals: move to the door, open it, go to the elevator, press the button, etc. Then motion planning is in charge of finding a path that gets the robot from one point to another, achieving each subgoal. Finally, control is used to achieve the planned motion using the robot’s actuators. Since the task planning level is typically defined over discrete states and actions, in this chapter we will focus primarily on motion planning and control.

Task planning

Control

Separately, preference learning is in charge of estimating an end user’s objective, and people prediction is used to forecast the actions of other people in the robot’s environment. All these combine to determine the robot’s behavior.

Preference learning

People prediction

Whenever we split a problem into separate pieces we reduce complexity, but we give up opportunities for the pieces to help each other. Action can help improve perception, and also determine what kind of perception is useful. Similarly, decisions at the motion level might not be the best when accounting for how that motion will be tracked; or decisions at the task level might render the task plan uninstantiatable at the motion level. So, with progress in these separate areas comes the push to reintegrate them: to do motion planning and control together, to do task and motion planning together, and to reintegrate perception, prediction, and action—closing the feedback loop. Robotics today is about continuing progress in each area while also building on this progress to achieve better integration.

26.4 Robotic Perception Perception is the process by which robots map sensor measurements into internal representations of the environment. Much of it uses the computer vision techniques from the previous chapter. But perception for robotics must deal with additional sensors like lidar and tactile sensors.

Perception is difficult because sensors are noisy and the environment is partially observable, unpredictable, and often dynamic. In other words, robots have all the problems of state estimation (or filtering) that we discussed in Section 14.2 . As a rule of thumb, good internal representations for robots have three properties:

1. They contain enough information for the robot to make good decisions. 2. They are structured so that they can be updated efficiently. 3. They are natural in the sense that internal variables correspond to natural state variables in the physical world. In Chapter 14 , we saw that Kalman filters, HMMs, and dynamic Bayes nets can represent the transition and sensor models of a partially observable environment, and we described both exact and approximate algorithms for updating the belief state—the posterior probability distribution over the environment state variables. Several dynamic Bayes net models for this process were shown in Chapter 14 . For robotics problems, we include the robot’s own past actions as observed variables in the model. Figure 26.4  shows the notation used in this chapter: time

Xt is the state of the environment (including the robot) at

t Zt is the observation received at time t ,

observation is received.

Figure 26.4

,

and

At is the action taken after the

Robot perception can be viewed as temporal inference from sequences of actions and measurements, as illustrated by this dynamic decision network.

P(Xt ∣ z t , a t ), from the current belief ), and the new observation zt . We did this in Section 14.2 , but

We would like to compute the new belief state, state,

P ( X t |z t , a 1:

t

1: −1

+1

1: +1

1:

+1

there are two differences here: we condition on the actions as well as the observations, and we deal with continuous rather than discrete variables. Thus, we modify the recursive filtering equation (14.5  on page 467) to use integration rather than summation: (26.1)

P ( X t |z t , a = α P ( z t |Xt +1

1: +1 +1

1:

t)

+1 )

P ( X t | x t , a t ) P ( x t |z t , a

∫

+1

1:

This equation states that the posterior over the state variables

1: −1 )

t

d xt .

X at time t + 1 is calculated

recursively from the corresponding estimate one time step earlier. This calculation involves

z

the previous action at and the current sensor measurement t+1 . For example, if our goal is

Xt might include the location of the soccer ball relative to the robot. The posterior P(Xt | z t , a t ) is a probability distribution over all states that to develop a soccer-playing robot,

+1

1:

1: −1

captures what we know from past sensor measurements and controls. Equation (26.1)  tells us how to recursively estimate this location, by incrementally folding in sensor measurements (e.g., camera images) and robot motion commands. The probability

P ( Xt

+1

∣

xt , at ) is called the transition model or motion model, and P(zt

sensor model.

+1

∣

Xt

+1 )

is the

Motion model

26.4.1 Localization and mapping Localization is the problem of finding out where things are—including the robot itself. To keep things simple, let us consider a mobile robot that moves slowly in a flat twodimensional world. Let us also assume the robot is given an exact map of the environment. (An example of such a map appears in Figure 26.7 .) The pose of such a mobile robot is defined by its two Cartesian coordinates with values x and y and its heading with value θ, as illustrated in Figure 26.5 (a) . If we arrange those three values in a vector, then any particular state is given by

X

t

= (xt , yt , θt )⊤ . So far so good.

Figure 26.5

(a) A simplified kinematic model of a mobile robot. The robot is shown as a circle with an interior radius

x

line marking the forward direction. The state t consists of the (xt , yt ) position (shown implicitly) and the orientation θt . The new state t+1 is obtained by an update in position of vt Δt and in orientation of ωt Δt . Also shown is a landmark at (xi , yi ) observed at time t. (b) The range-scan sensor model. Two possible

x

robot poses are shown for a given range scan (z1 , z2 , z3 , z4 ). It is much more likely that the pose on the left generated the range scan than the pose on the right.

Localization

In the kinematic approximation, each action consists of the “instantaneous” specification of two velocities—a translational velocity

t

vt and a rotational velocity ωt . For small time

intervals Δ , a crude deterministic model of the motion of such robots is given by

Xt ^

+1+

=

f ( X t vt ω t ) = X t + ,

⎛

,

⎜

at

⎝



vt △t cos θt vt △t sin θt ωt △t

⎞ ⎟

.

⎠

X

The notation ˆ refers to a deterministic state prediction. Of course, physical robots are somewhat unpredictable. This is commonly modeled by a Gaussian distribution with mean

f (Xt , vt , ωt ) and covariance Σx . (See Appendix A  for a mathematical definition.) P ( Xt

+1

∣

Xt , vt , ωt ) = N (Xˆ t

+1 ,

Σx ).

This probability distribution is the robot’s motion model. It models the effects of the motion

at on the location of the robot.

Next, we need a sensor model. We will consider two kinds of sensor models. The first assumes that the sensors detect stable, recognizable features of the environment called landmarks. For each landmark, the range and bearing are reported. Suppose the robot’s state is

xt

= (

xt , yt , θt )

⊤

x yi )

and it senses a landmark whose location is known to be ( i ,

⊤

.

Without noise, a prediction of the range and bearing can be calculated by simple geometry (see Figure 26.5 (a) ):

zˆt = h(xt ) =

⎛ ⎜ ⎜

√ x t xi

⎝

yt − yi ) yi − yt − θ arctan x −x t

(

−

2

) +(

i

t

2

⎞ ⎟ ⎟

.

⎠

Landmark

Again, noise distorts our measurements. To keep things simple, assume Gaussian noise with covariance Σz , giving us the sensor model

P (zt |xt ) = N (zˆt ,Σz ).

A somewhat different sensor model is used for a sensor array of range sensors, each of which has a fixed bearing relative to the robot. Such sensors produce a vector of range values

zt = (z , … , zM ) 1

⊤

.

Sensor array

Given a pose

xt , let zˆj be the computed range along the jth beam direction from xt to the

nearest obstacle. As before, this will be corrupted by Gaussian noise. Typically, we assume that the errors for the different beam directions are independent and identically distributed, so we have

M

P (zt ∣ xt ) = α ∏ e

−(

j=1

zj−ˆz j)/2σ . 2

Figure 26.5 (b)  shows an example of a four-beam range scan and two possible robot poses, one of which is reasonably likely to have produced the observed scan and one of which is not. Comparing the range-scan model to the landmark model, we see that the range-scan model has the advantage that there is no need to identify a landmark before the range scan can be interpreted; indeed, in Figure 26.5 (b) , the robot faces a featureless wall. On the other hand, if there are visible, identifiable landmarks, they may provide instant localization. Section 14.4  described the Kalman filter, which represents the belief state as a single multivariate Gaussian, and the particle filter, which represents the belief state by a collection of particles that correspond to states. Most modern localization algorithms use one of these two representations of the robot’s belief

Monte Carlo localization

P( Xt

+1

∣

z t, a t 1:

1: −1 ).

Localization using particle filtering is called Monte Carlo localization, or MCL. The MCL algorithm is an instance of the particle-filtering algorithm of Figure 14.17  (page 492). All we need to do is supply the appropriate motion model and sensor model. Figure 26.6  shows one version using the range-scan sensor model. The operation of the algorithm is illustrated in Figure 26.7  as the robot finds out where it is inside an office building. In the first image, the particles are uniformly distributed based on the prior, indicating global uncertainty about the robot’s position. In the second image, the first set of measurements arrives and the particles form clusters in the areas of high posterior belief. In the third, enough measurements are available to push all the particles to a single location.

Figure 26.6

A Monte Carlo localization algorithm using a range-scan sensor model with independent noise.

Figure 26.7

Monte Carlo localization, a particle filtering algorithmformobile robot localization. (a) Initial, global uncertainty. (b) Approximately bimodal uncertainty after navigating in the (symmetric) corridor. (c) Unimodal uncertainty after entering a room and finding it to be distinctive.

The Kalman filter is the other major way to localize. A Kalman filter represents the posterior

P ( Xt

+1

∣

z

1:t ,

a1:t−1 ) by a Gaussian. The mean of this Gaussian will be denoted μt and its

covariance Σt . The main problem with Gaussian beliefs is that they are closed only under linear motion models f and linear measurement models h. For nonlinear f or h, the result of updating a filter is in general not Gaussian. Thus, localization algorithms using the Kalman filter linearize the motion and sensor models. Linearization is a local approximation of a nonlinear function by a linear function. Figure 26.8  illustrates the concept of linearization for a (one-dimensional) robot motion model. On the left, it depicts a nonlinear motion

model f (xt , at ) (the control at is omitted in this graph since it plays no role in the linearization). On the right, this function is approximated by a linear function f˜(xt , at ). This linear function is tangent to f at the point μt , the mean of our state estimate at time t. Such a linearization is called first degree Taylor expansion. A Kalman filter that linearizes f and h

via Taylor expansion is called an extended Kalman filter (or EKF). Figure 26.9  shows a

sequence of estimates of a robot running an extended Kalman filter localization algorithm.

Figure 26.8

One-dimensional illustration of a linearized motion model: (a) The function f , and the projection of a

mean μt and a covariance interval (based on Σt ) into time t + 1. (b) The linearized version is the tangent ˜ t+1 differs from of f at μt . The projection of the mean μt is correct. However, the projected covariance Σ Σt+1 .

Figure 26.9

Localization using the extended Kalman filter. The robot moves on a straight line. As it progresses, its uncertainty in its location estimate increases, as illustrated by the error ellipses. When it observes a landmark with known position, the uncertainty is reduced.

Linearization

Taylor expansion

As the robot moves, the uncertainty in its location estimate increases, as shown by the error ellipses. Its error decreases as it senses the range and bearing to a landmark with known location and increases again as the robot loses sight of the landmark. EKF algorithms work well if landmarks are easily identified. Otherwise, the posterior distribution may be multimodal, as in Figure 26.7(b) . The problem of needing to know the identity of landmarks is an instance of the data association problem discussed in Figure 15.3 .

In some situations, no map of the environment is available. Then the robot will have to acquire a map. This is a bit of a chicken-and-egg problem: the navigating robot will have to determine its location relative to a map it doesn’t quite know, at the same time building this map while it doesn’t quite know its actual location. This problem is important for many robot applications, and it has been studied extensively under the name simultaneous localization and mapping, abbreviated as SLAM.

Simultaneous localization and mapping

SLAM problems are solved using many different probabilistic techniques, including the extended Kalman filter discussed above. Using the EKF is straightforward: just augment the state vector to include the locations of the landmarks in the environment. Luckily, the EKF update scales quadratically, so for small maps (e.g., a few hundred landmarks) the computation is quite feasible. Richer maps are often obtained using graph relaxation methods, similar to the Bayesian network inference techniques discussed in Chapter 13 . Expectation–maximization is also used for SLAM.

26.4.2 Other types of perception Not all of robot perception is about localization or mapping. Robots also perceive temperature, odors, sound, and so on. Many of these quantities can be estimated using variants of dynamic Bayes networks. All that is required for such estimators are conditional

probability distributions that characterize the evolution of state variables over time, and sensor models that describe the relation of measurements to state variables.

It is also possible to program a robot as a reactive agent, without explicitly reasoning about probability distributions over states. We cover that approach in Section 26.9.1 .

The trend in robotics is clearly towards representations with well-defined semantics. Probabilistic techniques outperform other approaches in many hard perceptual problems such as localization and mapping. However, statistical techniques are sometimes too cumbersome, and simpler solutions may be just as effective in practice. To help decide which approach to take, experience working with real physical robots is your best teacher.

26.4.3 Supervised and unsupervised learning in robot perception Machine learning plays an important role in robot perception. This is particularly the case when the best internal representation is not known. One common approach is to map highdimensional sensor streams into lower-dimensional spaces using unsupervised machine learning methods (see Chapter 19 ). Such an approach is called low-dimensional embedding. Machine learning makes it possible to learn sensor and motion models from data, while simultaneously discovering a suitable internal representation.

Low-dimensional embedding

Another machine learning technique enables robots to continuously adapt to big changes in sensor measurements. Picture yourself walking from a sunlit space into a dark room with neon lights. Clearly, things are darker inside. But the change of light source also affects all the colors: neon light has a stronger component of green light than sunlight has. Yet somehow we seem not to notice the change. If we walk together with people into a neon-lit room, we don’t think that their faces suddenly turned green. Our perception quickly adapts to the new lighting conditions, and our brain ignores the differences.

Adaptive perception techniques enable robots to adjust to such changes. One example is shown in Figure 26.10 , taken from the autonomous driving domain. Here an unmanned ground vehicle adapts its classifier of the concept “drivable surface.” How does this work? The robot uses a laser to provide classification for a small area immediately in front of the robot. When this area is found to be flat in the laser range scan, it is used as a positive training example for the concept “drivable surface.” A mixture-of-Gaussians technique similar to the EM algorithm discussed in Chapter 20  is then trained to recognize the specific color and texture coefficients of the small sample patch. The images in Figure 26.10  are the result of applying this classifier to the full image.

Figure 26.10

Sequence of “drivable surface” classifications using adaptive vision. (a) Only the road is classified as drivable (pink area). The V-shaped blue line shows where the vehicle is heading. (b) The vehicle is commanded to drive off the road, and the classifier is beginning to classify some of the grass as drivable. (c) The vehicle has updated its model of drivable surfaces to correspond to grass as well as road. Courtesy of Sebastian Thrun.

Methods that make robots collect their own training data (with labels!) are called selfsupervised. In this instance, the robot uses machine learning to leverage a short-range sensor that works well for terrain classification into a sensor that can see much farther. That allows the robot to drive faster, slowing down only when the sensor model says there is a change in the terrain that needs to be examined more carefully by the short-range sensors.

Self-supervised learning

26.5 Planning and Control The robot’s deliberations ultimately come down to deciding how to move, from the abstract task level all the way down to the currents that are sent to its motors. In this section, we simplify by assuming that perception (and, where needed, prediction) are given, so the world is observable. We further assume deterministic transitions (dynamics) of the world.

We start by separating motion from control. We define a path as a sequence of points in geometric space that a robot (or a robot part, such as an arm) will follow. This is related to the notion of path in Chapter 3 , but here we mean a sequence of points in space rather than a sequence of discrete actions. The task of finding a good path is called motion planning.

Path

Once we have a path, the task of executing a sequence of actions to follow the path is called trajectory tracking control. A trajectory is a path that has a time associated with each point on the path. A path just says “go from A to B to C, etc.” and a trajectory says “start at A, take 1 second to get to B, and another 1.5 seconds to get to C, etc.”

Trajectory tracking control

Trajectory

26.5.1 Configuration space

Imagine a simple robot,

R, in the shape of a right triangle as shown by the lavender triangle

in the lower left corner of Figure 26.11 . The robot needs to plan a path that avoids a

O. The physical space that a robot moves about in is called the workspace. This particular robot can move in any direction in the x − y plane, but cannot rectangular obstacle,

rotate. The figure shows five other possible positions of the robot with dashed outlines; these are each as close to the obstacle as the robot can get.

Figure 26.11

A simple triangular robot that can translate, and needs to avoid a rectangular obstacle. On the left is the workspace, on the right is the configuration space.

Workspace

xy

xyz

The body of the robot could be represented as a set of ( , ) points (or ( , , ) points for a three-dimensional robot), as could the obstacle. With this representation, avoiding the obstacle means that no point on the robot overlaps any point on the obstacle. Motion planning would require calculations on sets of points, which can be complicated and timeconsuming.

We can simplify the calculations by using a representation scheme in which all the points that comprise the robot are represented as a single point in an abstract multidimensional space, which we call the configuration space, or C-space. The idea is that the set of points that comprise the robot can be computed if we know (1) the basic measurements of the robot (for our triangle robot, the length of the three sides will do) and (2) the current pose of the robot—its position and orientation.

Configuration space

C-space

For our simple triangular robot, two dimensions suffice for the C-space: if we know the

xy

( , ) coordinates of a specific point on the robot—we’ll use the right-angle vertex—then we

can calculate where every other point of the triangle is (because we know the size and shape of the triangle and because the triangle cannot rotate). In the lower-left corner of Figure 26.11 , the lavender triangle can be represented by the configuration (0,0).

If we change the rules so that the robot can rotate, then we will need three dimensions,

xyθ

( , , ), to be able to calculate where every point is. Here

θ is the robot’s angle of rotation in

the plane. If the robot also had the ability to stretch itself, growing uniformly by a scaling

s

xyθs

factor , then the C-space would have four dimensions, ( , , , ).

For now we’ll stick with the simple two-dimensional C-space of the non-rotating triangle robot. The next task is to figure out where the points in the obstacle are in C-space. Consider the five dashed-line triangles on the left of Figure 26.11  and notice where the right-angle vertex is on each of these. Then imagine all the ways that the triangle could slide about. Obviously, the right-angle vertex can’t go inside the obstacle, and neither can it get any closer than it is on any of the five dashed-line triangles. So you can see that the area where the right-angle vertex can’t go—the C-space obstacle—is the five-sided polygon on the right of Figure 26.11  labeled

Cobs

C-space obstacle

In everyday language we speak of there being multiple obstacles for the robot—a table, a chair, some walls. But the math notation is a bit easier if we think of all of these as

combining into one “obstacle” that happens to have disconnected components. In general, the C-space obstacle is the set of all points

q in C such that, if the robot were placed in that

configuration, its workspace geometry would intersect the workspace obstacle.

Let the obstacles in the workspace be the set of points robot in configuration

O, and let the set of all points on the

q be A(q). Then the C-space obstacle is defined as Cobs = {q : q ∈ C  andA(q) ∩ O ≠ { }}

and the free space is

Cfree = C − Cobs .

Free space

The C-space becomes more interesting for robots with moving parts. Consider the two-link arm from Figure 26.12(a) . It is bolted to a table so the base does not move, but the arm has two joints that move independently—we call these degrees of freedom (DOF). Moving the

xy

joints alters the ( , ) coordinates of the elbow, the gripper, and every point on the arm. The

θ

θ

θ

arm’s configuration space is two-dimensional: ( shou , elb ) , where shou is the angle of the

θ

shoulder joint, and elb is the angle of the elbow joint. Figure 26.12

(a) Workspace representation of a robot arm with two degrees of freedom. The workspace is a box with a flat obstacle hanging from the ceiling. (b) Configuration space of the same robot. Only white regions in the space are configurations that are free of collisions. The dot in this diagram corresponds to the configuration of the robot shown on the left.

Degrees of freedom (DOF)

Knowing the configuration for our two-link arm means we can determine where each point on the arm is through simple trigonometry. In general, the forward kinematics mapping is a function

ϕb C :

→

W

Forward kinematics

that takes in a configuration and outputs the location of a particular point

b on the robot

when the robot is in that configuration. A particularly useful forward kinematics mapping is

ϕ configuration q is denoted by A(q) ⊂ W :

that for the robot’s end effector, EE . The set of all points on the robot in a particular

A(q) = ⋃ {ϕb (q)}. b

The inverse problem, of mapping a desired location for a point on the robot to the configuration(s) the robot needs to be in for that to happen, is known as inverse kinematics:

IKb : x ∈ W ↦ {q ∈ C  s. t.  ϕb (q) = x}.

Inverse kinematics

Sometimes the inverse kinematics mapping might take not just a position, but also a desired orientation as input. When we want a manipulator to grasp an object, for instance, we can compute a desired position and orientation for its gripper, and use inverse kinematics to determine a goal configuration for the robot. Then a planner needs to find a way to get the robot from its current configuration to the goal configuration without intersecting obstacles.

Workspace obstacles are often depicted as simple geometric forms—especially in robotics textbooks, which tend to focus on polygonal obstacles. But how do the obstacles look in configuration space?

For the two-link arm, simple obstacles in the workspace, like a vertical line, have very complex C-space counterparts, as shown in Figure 26.12(b) . The different shadings of the occupied space correspond to the different objects in the robot’s workspace: the dark region surrounding the entire free space corresponds to configurations in which the robot collides with itself. It is easy to see that extreme values of the shoulder or elbow angles cause such a violation. The two oval-shaped regions on both sides of the robot correspond to the table on which the robot is mounted. The third oval region corresponds to the left wall.

Finally, the most interesting object in configuration space is the vertical obstacle that hangs from the ceiling and impedes the robot’s motions. This object has a funny shape in configuration space: it is highly nonlinear and at places even concave. With a little bit of imagination the reader will recognize the shape of the gripper at the upper left end.

We encourage the reader to pause for a moment and study this diagram. The shape of this obstacle in C-space is not at all obvious! The dot inside Figure 26.12(b)  marks the configuration of the robot in Figure 26.12(a) . Figure 26.13  depicts three additional configurations, both in workspace and in configuration space. In configuration conf-1, the gripper is grasping the vertical obstacle.

Figure 26.13

Three robot configurations, shown in workspace and configuration space.

We see that even if the robot’s workspace is represented by flat polygons, the shape of the free space can be very complicated. In practice, therefore, one usually probes a configuration space instead of constructing it explicitly. A planner may generate a configuration and then test to see if it is in free space by applying the robot kinematics and then checking for collisions in workspace coordinates.

26.5.2 Motion planning

The motion planning problem is that of finding a plan that takes a robot from one configuration to another without colliding with an obstacle. It is a basic building block for movement and manipulation. In Section 26.5.4  we will discuss how to do this under complicated dynamics, like steering a car that may drift off the path if you take a curve too fast. For now, we will focus on the simple motion planning problem of finding a geometric path that is collision free. Motion planning is a quintessentially continuous-state search problem, but it is often possible to discretize the space and apply the search algorithms from Chapter 3 .

Motion planning

The motion planning problem is sometimes referred to as the piano mover’s problem. It gets its name from a mover’s struggles with getting a large, irregular-shaped piano from one room to another without hitting anything. We are given:

W in either R an obstacle region O ⊂ W , a workspace world

2

for the plane or

a robot with a configuration space

q

a starting configuration s ∈

q

a goal configuration g ∈

C.

R

3

for three dimensions,

C and set of points A(q) for q ∈ C ,

C , and

Piano mover’problem

The obstacle region induces a C-space obstacle

Cobs and its corresponding free space Cfree

defined as in the previous section. We need to find a continuous path through free space.

τt

τ

q

τ

q

We will use a parameterized curve, ( ), to represent the path, where (0) = s and (1) = g

τt

t between 0 and 1 is some point in Cfree . That is, t parameterizes how far we are along the path, from start to goal. Note that t acts somewhat like time in that as t and ( ) for every

increases the distance along the path increases, but

t is always a point on the interval [0,1]

and is not measured in seconds.

The motion planning problem can be made more complex in various ways: defining the goal as a set of possible configurations rather than a single configuration; defining the goal in the workspace rather than the C-space; defining a cost function (e.g., path length) to be minimized; satisfying constraints (e.g., if the path involves carrying a cup of coffee, making sure that the cup is always oriented upright so the coffee does not spill).

THE SPACES OF MOTION PLANNING: Let’s take a step back and make sure we understand the spaces involved in motion planning. First, there is the workspace or world

W . Points in W are points in the everyday three-dimensional world. Next, we have the space of configurations, C . Points q in C are d-dimensional, with d the robot’s number of degrees of freedom, and map to sets of points A(q) in W . Finally, there is the space of paths. The space of paths is a space of functions. Each point in this space maps to an entire curve through C-space. This space is ∞-dimensional! Intuitively, we need

d dimensions for each

configuration along the path, and there are as many configurations on a path as there are points in the number line interval [0,1]. Now let’s consider some ways of solving the motion planning problem.

Visibility graphs For the simplified case of two-dimensional configuration spaces and polygonal C-space obstacles, visibility graphs are a convenient way to solve the motion planning problem with

V

a guaranteed shortest-path solution. Let obs ⊂ making up

Cobs , and let V

=

Vobs ∪ {qs , qg }.

C be the set of vertices of the polygons

Visibility graph

We construct a graph

G = (V , E) on the vertex set V

e

with edges ij ∈

E connecting a vertex

vi to another vertex vj if the line connecting the two vertices is collision-free—that is, if {λvi + (1 − λ)vj : λ ∈ [0,1]} ∩ Cobs = { }. When this happens, we say the two vertices “can see each other,” which is where “visibility” graphs got their name.

To solve the motion planning problem, all we need to do is run a discrete graph search (e.g., best-first search) on the graph

G with starting state qs and goal qg

.

In Figure 26.14  we see a

visibility graph and an optimal three-step solution. An optimal search on visibility graphs will always give us the optimal path (if one exists), or report failure if no path exists.

Figure 26.14

A visibility graph. Lines connect every pair of vertices that can “see” each other—lines that don’t go through an obstacle. The shortest path must lie upon these lines.

Voronoi diagrams Visibility graphs encourage paths that run immediately adjacent to an obstacle—if you had to walk around a table to get to the door, the shortest path would be to stick as close to the table as possible. However, if motion or sensing is nondeterministic, that would put you at risk of bumping into the table. One way to address this is to pretend that the robot’s body is a bit larger than it actually is, providing a buffer zone. Another way is to accept that path length is not the only metric we want to optimize. Section 26.8.2  shows how to learn a good metric from human examples of behavior.

Figure 26.15

A Voronoi diagram showing the set of points (black lines) equidistant to two or more obstacles in configuration space.

A third way is to use a different technique, one that puts paths as far away from obstacles as possible rather than hugging close to them. A Voronoi diagram is a representation that allows us to do just that. To get an idea for what a Voronoi diagram does, consider a space where the obstacles are, say, a dozen small points scattered about a plane. Now surround each of the obstacle points with a region consisting of all the points in the plane that are closer to that obstacle point than to any other obstacle point. Thus, the regions partition the plane. The Voronoi diagram consists of the set of regions, and the Voronoi graph consists of the edges and vertices of the regions.

Voronoi diagram

Region

Voronoi graph

When obstacles are areas, not points, everything stays pretty much the same. Each region still contains all the points that are closer to one obstacle than to any other, where distance is measured to the closest point on an obstacle. The boundaries between regions still correspond to points that are equidistant between two obstacles, but now the boundary may be a curve rather than a straight line. Computing these boundaries can be prohibitively expensive in high-dimensional spaces.

q

To solve the motion planning problem, we connect the start point s to the closest point on

q

the Voronoi graph via a straight line, and the same for the goal point g . We then use discrete graph search to find the shortest path on the graph. For problems like navigating through corridors indoors, this gives a nice path that goes down the middle of the corridor. However, in outdoor settings it can come up with inefficient paths, for example suggesting an unnecessary 100 meter detour to stick to the middle of a wide-open 200-meter space.

Cell decomposition An alternative approach to motion planning is to discretize the

C -space. Cell

decomposition methods decompose the free space into a finite number of contiguous regions, called cells. These cells are designed so that the path-planning problem within a single cell can be solved by simple means (e.g., moving along a straight line). The pathplanning problem then becomes a discrete graph search problem (as with visibility graphs and Voronoi graphs) to find a path through a sequence of cells.

Cell decomposition

The simplest cell decomposition consists of a regularly spaced grid. Figure 26.16(a)  shows a square grid decomposition of the space and a solution path that is optimal for this grid size. Grayscale shading indicates the value of each free-space grid cell—the cost of the shortest path from that cell to the goal. (These values can be computed by a deterministic

form of the VALUE-ITERATION algorithm given in Figure 17.6  on page 573.) Figure 26.16(b)  shows the corresponding workspace trajectory for the arm. Of course, we could also use the A* algorithm to find a shortest path.

Figure 26.16

(a) Value function and path found for a discrete grid cell approximation of the configuration space. (b) The same path visualized in workspace coordinates. Notice how the robot bends its elbow to avoid a collision with the vertical obstacle.

This grid decomposition has the advantage that it is simple to implement, but it suffers from three limitations. First, it is workable only for low-dimensional configuration spaces, because the number of grid cells increases exponentially with d, the number of dimensions. (Sounds familiar? This is the curse of dimensionality.) Second, paths through discretized state space will not always be smooth. We see in Figure 26.16(a)  that the diagonal parts of the path are jagged and hence very difficult for the robot to follow accurately. The robot can attempt to smooth out the solution path, but this is far from straightforward.

Third, there is the problem of what to do with cells that are “mixed”—that is, neither entirely within free space nor entirely within occupied space. A solution path that includes such a cell may not be a real solution, because there may be no way to safely cross the cell. This would make the path planner unsound. On the other hand, if we insist that only completely free cells may be used, the planner will be incomplete, because it might be the case that the

only paths to the goal go through mixed cells—it might be that a corridor is actually wide enough for the robot to pass, but the corridor is covered only by mixed cells.

The first approach to this problem is further subdivision of the mixed cells—perhaps using cells of half the original size. This can be continued recursively until a path is found that lies entirely within free cells. This method works well and is complete if there is a way to decide if a given cell is a mixed cell, which is easy only if the configuration space boundaries have relatively simple mathematical descriptions.

It is important to note that cell decomposition does not necessarily require explicitly representing the obstacle space

Cobs . We can decide to include a cell or not by using a

collision checker. This is a crucial notion to motion planning. A collision checker is a

γq

function ( ) that maps to 1 if the configuration collides with an obstacle, and 0 otherwise. It is much easier to check whether a specific configuration is in collision than to explicitly construct the entire obstacle space

Cobs .

Collision checker

Examining the solution path shown in Figure 26.16(a) , we can see an additional difficulty that will have to be resolved. The path contains arbitrarily sharp corners, but a physical robot has momentum and cannot change direction instantaneously. This problem can be solved by storing, for each grid cell, the exact continuous state (position and velocity) that was attained when the cell was reached in the search. Assume further that when propagating information to nearby grid cells, we use this continuous state as a basis, and apply the continuous robot motion model for jumping to nearby cells. So we don’t make an ∘

instantaneous 90 turn; we make a rounded turn governed by the laws of motion. We can now guarantee that the resulting trajectory is smooth and can indeed be executed by the robot. One algorithm that implements this is hybrid A*.

Hybrid A*

Randomized motion planning Randomized motion planning does graph search on a random decomposition of the configuration space, rather than a regular cell decomposition. The key idea is to sample a random set of points and to create edges between them if there is a very simple way to get from one to the other (e.g., via a straight line) without colliding; then we can search on this graph.

A probabilistic roadmap (PRM) algorithm is one way to leverage this idea. We assume

γ (defined on page 946 ), and to a simple planner B(q , q ) that returns a path from q to q (or failure) but does so quickly. This simple planner is not

access to a collision checker

1

1

2

2

going to be complete—it might return failure even if a solution actually exists. Its job is to quickly try to connect

q

1

and

q

2

and let the main algorithm know if it succeeds. We will use

it to define whether an edge exists between two vertices.

Probabilistic roadmap (PRM)

Simple planner

Milestone

The algorithm starts by sampling

M milestones—points in Cfree —in addition to the points qs

qg . It uses rejection sampling, where configurations are sampled randomly and collisionchecked using γ until a total of M milestones are found. Next, the algorithm uses the simple and

planner to try to connect pairs of milestones. If the simple planner returns success, then an edge between the pair is added to the graph; otherwise, the graph remains as is. We try to connect each milestone either to its

r

k nearest neighbors (we call this k-PRM), or to all

milestones in a sphere of a radius . Finally, the algorithm searches for a path on this graph

from

qs to qg

.

If no path is found, then

M more milestones are sampled, added to the graph,

and the process is repeated. Figure 26.17  shows a roadmap with the path found between two configurations. PRMs are not complete, but they are what is called probabilistically complete—they will eventually find a path, if one exists. Intuitively, this is because they keep sampling more milestones. PRMs work well even in high-dimensional configuration spaces.

Figure 26.17

The probabilistic roadmap (PRM) algorithm. Top left: the start and goal configurations. Top right: sample collision-free milestones (here = 5). Bottom left: connect each milestone to its nearest

M

neighbors (here graph.

k

= 3).

M

k

Bottom right: find the shortest path from the start to the goal on the resulting

Probabilistically complete

PRMs are also popular for multi-query planning, in which we have multiple motion planning problems within the same C-space. Often, once the robot reaches a goal, it is called upon to reach another goal in the same workspace. PRMs are really useful, because

the robot can dedicate time up front to constructing a roadmap, and amortize the use of that roadmap over multiple queries.

Multi-query planning

Rapidly-exploring random trees An extension of PRMs called rapidly exploring random trees (RRTs) is popular for singlequery planning. We incrementally build two trees, one with qs as the root and one with qg as the root. Random milestones are chosen, and an attempt is made to connect each new milestone to the existing trees. If a milestone connects both trees, that means a solution has been found, as in Figure 26.18 . If not, the algorithm finds the closest point in each tree and adds to the tree a new edge that extends from the point by a distance δ towards the milestone. This tends to grow the tree towards previously unexplored sections of the space.

Figure 26.18

The bidirectional RRT algorithm constructs two trees (one from the start, the other from the goal) by incrementally connecting each sample to the closest node in each tree, if the connection is possible. When a sample connects to both trees, that means we have found a solution path.

Rapidly exploring random trees (RRTs)

Roboticists love RRTs for their ease of use. However, RRT solutions are typically nonoptimal and lack smoothness. Therefore, RRTs are often followed by a post-processing step. The most common one is “short-cutting,” in which we randomly select one of the vertices on the solution path and try to remove it by connecting its neighbors to each other (via the simple planner). We do this repeatedly for as many steps as we have compute time for. Even then, the trajectories might look a little unnatural due to the random positions of the milestone that were selected, as shown in Figure 26.19 .

Figure 26.19

Snapshots of a trajectory produced by an RRT and post-processed with shortcutting. Courtesy of Anca Dragan.

RRT* is a modification to RRT that makes the algorithm asymptotically optimal: the solution converges to the optimal solution as more and more milestones are sampled. The key idea is to pick the nearest neighbor based on a notion of cost to come rather than distance from the milestone only, and to rewire the tree, swapping parents of older vertices if it is cheaper to reach them via the new milestone.

RRT*

Trajectory optimization for kinematic planning

Randomized sampling algorithms tend to first construct a complex but feasible path and then optimize it. Trajectory optimization does the opposite: it starts with a simple but infeasible path, and then works to push it out of collision. The goal is to find a path that optimizes a cost function1 over paths. That is, we want to minimize the cost function J (τ ), where τ (0) = qs and τ (1) = qg . 1 Roboticists like to minimize a cost function J , whereas in other parts of AI we try to maximize a utility function U or a reward

J

R.

is called a functional because it is a function over functions. The argument to J is τ , which

is itself a function: τ (t) takes as input a point in the [0,1] interval and maps it to a configuration. A standard cost functional trades off between two important aspects of the robot’s motion: collision avoidance and efficiency,

J

=

Jobs + λJeff

where the efficiency Jeff measures the length of the path and may also measure smoothness. A convenient way to define efficiency is with a quadratic: it integrates the squared first derivative of τ (we will see in a bit why this does in fact incentivize short paths):

Jeff

=

∫

1 0

1 2

2 ∥ τ˙(s) ∥

ds.

For the obstacle term, assume we can compute the distance d(x) from any point x ∈

W

to

the nearest obstacle edge. This distance is positive outside of obstacles, 0 at the edge, and negative inside. This is called a signed distance field. We can now define a cost field in the workspace, call it c, that has high cost inside of obstacles, and a small cost right outside. With this cost, we can make points in the workspace really hate being inside obstacles, and dislike being right next to them (avoiding the visibility graph problem of their always hanging out by the edges of obstacles). Of course, our robot is not a point in the workspace, so we have some more work to do—we need to consider all points b on the robot’s body:

Jobs = ∫

1 0

∫ c(ϕb (τ (s)) ) ∥ b

 ∈W

d ϕb (τ (s)) ∥  db ds. ds  ∈W

Signed distance field

This is called a path integral—it does not just integrate c along the way for each body point, but it multiplies by the derivative to make the cost invariant to retiming of the path. Imagine a robot sweeping through the cost field, accumulating cost as is moves. Regardless of how fast or slow the arm moves through the field, it must accumulate the exact same cost.

Path integral

The simplest way to solve the optimization problem above and find a path is gradient descent. If you are wondering how to take gradients of functionals with respect to functions, something called the calculus of variations is here to help. It is especially easy for functionals of the form 1

J [τ ] = ∫ F (s, τ (s), τ˙(s))ds 0

which are integrals of functions that depend just on the parameter s, the value of the function at s, and the derivative of the function at s. In such a case, the Euler-Lagrange equation says that the gradient is

Js

∇τ ( ) =

d ∂F F ( s) − (s). ∂ τ ( s) dt ∂τ˙(s) ∂

Euler-Lagrange equation

If we look closely at Jeff and Jobs , they both follow this pattern. In particular for Jeff , we 2

have F (s, τ (s), τ˙(s)) =∥ τ˙(s)∥ ∥ . To get a bit more comfortable with this, let’s compute the

gradient for Jeff only. We see that F does not have a direct dependence on τ (s), so the first term in the formula is 0. We are left with

Js

∇τ ( ) = 0 −

d τ˙(s) dt

since the partial of F with respect to τ˙(s) is τ˙(s). Notice how we made things easier for ourselves when defining Jeff —it’s a nice quadratic of the derivative (and we even put a

1 2

in front so that the 2 nicely cancels out). In practice,

you will see this trick happen a lot for optimization—the art is not just in choosing how to optimize the cost function, but also in choosing a cost function that will play nicely with how you will optimize it. Simplifying our gradient, we get

Js

τ s

∇τ ( ) = − ¨( ).

Now, since Jeff is a quadratic, setting this gradient to 0 gives us the solution for τ if we didn’t have to deal with obstacles. Integrating once, we get that the first derivative needs to be constant; integrating again we get that τ (s) =

a ⋅ s + b, with a and b determined by the endpoint constraints for τ (0) and τ (1). The optimal path with respect to Jeff is thus the straight line from start to goal! It is indeed the most efficient way to go from one to the other if there are no obstacles to worry about. Of course, the addition of Jobs is what makes things difficult—and we will spare you deriving its gradient here. The robot would typically initialize its path to be a straight line, which would plow right through some obstacles. It would then calculate the gradient of the cost about the current path, and the gradient would serve to push the path away from the obstacles (Figure 26.20 ). Keep in mind that gradient descent will only find a locally optimal solution—just like hill climbing. Methods such as simulated annealing (Section 4.1.2 ) can be used for exploration, to make it more likely that the local optimum is a good one.

Figure 26.20

Trajectory optimization for motion planning. Two point-obstacles with circular bands of decreasing cost around them. The optimizer starts with the straight line trajectory, and lets the obstacles bend the line away from collisions, finding the minimum path through the cost field.

26.5.3 Trajectory tracking control

Control theory

We have covered how to plan motions, but not how to actually move—to apply current to motors, to produce torque, to move the robot. This is the realm of control theory, a field of increasing importance in AI. There are two main questions to deal with: how do we turn a mathematical description of a path into a sequence of actions in the real world (open-loop control), and how do we make sure that we are staying on track (closed-loop control)?

Figure 26.21

The task of reaching to grasp a bottle solved with a trajectory optimizer. Left: the initial trajectory, plotted for the end effector. Middle: the final trajectory after optimization. Right: the goal configuration. Courtesy of Anca Dragan. See Ratliff et al., (2009).

FROM CONFIGURATIONS TO TORQUES FOR OPEN-LOOP TRACKING: Our path τ (t) gives us configurations. The robot starts at rest at qs = τ (0). From there the robot’s motors will turn currents into torques, leading to motion. But what torques should the robot aim for, such that it ends up at qg = τ (1)? This is where the idea of a dynamics model (or transition model) comes in. We can give the robot a function f that computes the effects torques have on the configuration. Remember

F = ma from physics? Well, there is something like that for torques too, in the form u = f −1 (q,q˙,q¨), with u a torque, q˙ a velocity, and q¨ an acceleration.2 If the robot is at configuration q and velocity q˙, and applied torque u, that would lead to acceleration q¨ = f (q,q˙,u). The tuple (q,q˙) is a dynamic state, because it includes velocity, whereas q is the kinematic state and is not sufficient for computing exactly what torque to apply. f is a deterministic dynamics model in the MDP over dynamic states with torques as actions. f −1 is the inverse dynamics, telling us what torque to apply if we want a particular acceleration, which leads to a change in velocity and thus a change in dynamic state. 2 We omit the details of f −1 here, but they involve mass, inertia, gravity, and Coriolis and centrifugal forces.

Dynamics model

Dynamic state

Kinematic state

Inverse dynamics

Now, naively, we could think of

t ∈ [0,1] as “time” on a scale from 0 to 1 and select our

torque using inverse dynamics:

(26.2)

u( t ) = f τ

−1

τt τt τt

( ( ), ˙( ), ¨( ))

τ

assuming that the robot starts at ( (0), ˙(0)). In reality though, things are not that easy.

τ was created as a sequence of points, without taking velocities and accelerations into account. As such, the path may not satisfy τ˙(0) = 0 (the robot starts at 0 velocity), or even that τ is differentiable (let alone twice differentiable). Further, the meaning of the The path

endpoint “1” is unclear: how many seconds does that map to?

Retiming

In practice, before we even think of tracking a reference path, we usually retime it, that is,

ξt

T

transform it into a trajectory ( ) that maps the interval [0, ] for some time duration points in the configuration space

T into

C . (The symbol ξ is the Greek letter Xi.) Retiming is

trickier than you might think, but there are approximate ways to do it, for instance by picking a maximum velocity and acceleration, and using a profile that accelerates to that maximum velocity, stays there as long as it can, and then decelerates back to 0. Assuming we can do this, Equation (26.2)  above can be rewritten as (26.3)

u( t ) = f Even with the change from

−1

ξt ξ t ξ t

( ( ), ˙( ), ¨( )).

τ to ξ, an actual trajectory, the equation of applying torques from

above (called a control law) has a problem in practice. Thinking back to the reinforcement learning section, you might guess what it is. The equation works great in the situation where

f is exact, but pesky reality gets in the way as usual: in real systems, we can’t measure masses and inertias exactly, and f might not properly account for physical phenomena like stiction in the motors (the friction that tends to prevent stationary surfaces from being set in motion—to make them stick). So, when the robot arm starts applying those torques but

f is

wrong, the errors accumulate and you deviate further and further from the reference path.

Control law

Stiction

Rather than just letting those errors accumulate, a robot can use a control process that looks at where it thinks it is, compares that to where it wanted to be, and applies a torque to minimize the error.

A controller that provides force in negative proportion to the observed error is known as a proportional controller or P controller for short. The equation for the force is:

u( t ) = K P ( ξ ( t ) − q t )

P controller

qt is the current configuration, and KP is a constant representing the gain factor of the controller. KP regulates how strongly the controller corrects for deviations between the actual state qt and the desired state ξ(t).

where

Gain factor

Figure 26.22(a)  illustrates what can go wrong with proportional control. Whenever a deviation occurs—whether due to noise or to constraints on the forces the robot can apply— the robot provides an opposing force whose magnitude is proportional to this deviation. Intuitively, this might appear plausible, since deviations should be compensated by a counterforce to keep the robot on track. However, as Figure 26.22(a)  illustrates, a proportional controller can cause the robot to apply too much force, overshooting the desired path and zig-zagging back and forth. This is the result of the natural inertia of the robot: once driven back to its reference position the robot has a velocity that can’t instantaneously be stopped.

Figure 26.22

Robot arm control using (a) proportional control with gain factor 1.0, (b) proportional control with gain factor 0.1, and (c) PD (proportional derivative) control with gain factors 0.3 for the proportional component and 0.8 for the differential component. In all cases the robot arm tries to follow the smooth line path, but in (a) and (b) deviates substantially from the path.

In Figure 26.22(a) , the parameter smaller value for

KP

= 1.

At first glance, one might think that choosing a

KP would remedy the problem, giving the robot a gentler approach to the

desired path. Unfortunately, this is not the case. Figure 26.22(b)  shows a trajectory for

KP

= .1,

still exhibiting oscillatory behavior. The lower value of the gain parameter helps,

but does not solve the problem. In fact, in the absence of friction, the P controller is essentially a spring law; so it will oscillate indefinitely around a fixed target location.

There are a number of controllers that are superior to the simple proportional control law. A controller is said to be stable if small perturbations lead to a bounded error between the

robot and the reference signal. It is said to be strictly stable if it is able to return to and then stay on its reference path upon such perturbations. Our P controller appears to be stable but not strictly stable, since it fails to stay anywhere near its reference trajectory.

Stable

Strictly stable

The simplest controller that achieves strict stability in our domain is a PD controller. The letter ‘P’ stands again for proportional, and ‘D’ stands for derivative. PD controllers are described by the following equation:

(26.4)

u(t) = KP (ξ(t) − qt ) + KD (ξ˙(t) − q˙t ).

PD controller

As this equation suggests, PD controllers extend P controllers by a differential component,

ut

which adds to the value of ( ) a term that is proportional to the first derivative of the error

ξ(t) − qt over time. What is the effect of such a term? In general, a derivative term dampens the system that is being controlled. To see this, consider a situation where the error is changing rapidly over time, as is the case for our P controller above. The derivative of this error will then counteract the proportional term, which will reduce the overall response to the perturbation. However, if the same error persists and does not change, the derivative will vanish and the proportional term dominates the choice of control.

Figure 26.22(c)  shows the result of applying this PD controller to our robot arm, using as gain parameters

KP = .3 and KD = .8. Clearly, the resulting path is much smoother, and

does not exhibit any obvious oscillations.

PD controllers do have failure modes, however. In particular, PD controllers may fail to regulate an error down to zero, even in the absence of external perturbations. Often such a situation is the result of a systematic external force that is not part of the model. For example, an autonomous car driving on a banked surface may find itself systematically pulled to one side. Wear and tear in robot arms causes similar systematic errors. In such situations, an over-proportional feedback is required to drive the error closer to zero. The solution to this problem lies in adding a third term to the control law, based on the integrated error over time:

(26.5)

u( t ) Here

=

t

KP (ξ(t) − qt ) + KI ∫ (ξ(s) − qs )ds + KD (ξ˙(t) − q˙t ). 0

KI is a third gain parameter. The term ∫ t (ξ(s) calculates the integral of the error over 0

time. The effect of this term is that long-lasting deviations between the reference signal and the actual state are corrected. Integral terms, then, ensure that a controller does not exhibit systematic long-term error, although they do pose a danger of oscillatory behavior.

PID controller

A controller with all three terms is called a PID controller (for proportional integral derivative). PID controllers are widely used in industry, for a variety of control problems. Think of the three terms as follows—proportional: try harder the farther away you are from the path; derivative: try even harder if the error is increasing; integral: try harder if you haven’t made progress for a long time.

A middle ground between open-loop control based on inverse dynamics and closed-loop PID control is called computed torque control. We compute the torque our model thinks

we will need, but compensate for model inaccuracy with proportional error terms:

(26.6)

u (t) = f (ξ(t), ξ˙(t), ξ¨(t)) + m(ξ(t)) (KP (ξ(t) − qt ) + KD (ξ˙(t) − q˙t )). −1

feedforward



feedback

Computed torque control

The first term is called the feedforward component because it looks forward to where the robot needs to go and computes what torque might be required. The second is the feedback component because it feeds the current error in the dynamic state back into the control law.

m(q) is the inertia matrix at configuration q—unlike normal PD control, the gains change with the configuration of the system.

Feedforward component

Feedback component

Plans versus policies Let’s take a step back and make sure we understand the analogy between what happened so far in this chapter and what we learned in the search, MDP, and reinforcement learning chapters. With motion in robotics, we are really considering an underlying MDP where the states are dynamic states (configuration and velocity), and the actions are control inputs, usually in the form of torques. If you take another look at our control laws above, they are policies, not plans—they tell the robot what action to take from any state it might reach. However, they are usually far from optimal policies. Because the dynamic state is continuous

and high dimensional (as is the action space), optimal policies are computationally difficult to extract.

Instead, what we did here is to break up the problem. We come up with a plan first, in a simplified state and action space: we use only the kinematic state, and assume that states are reachable from one another without paying attention to the underlying dynamics. This is motion planning, and it gives us the reference path. If we knew the dynamics perfectly, we could turn this into a plan for the original state and action space with Equation (26.3) .

But because our dynamics model is typically erroneous, we turn it instead into a policy that tries to follow the plan—getting back to it when it drifts away. When doing this, we introduce suboptimality in two ways: first by planning without considering dynamics, and second by assuming that if we deviate from the plan, the optimal thing to do is to return to the original plan. In what follows, we describe techniques that compute policies directly over the dynamic state, avoiding the separation altogether.

26.5.4 Optimal control Rather than using a planner to create a kinematic path, and only worrying about the dynamics of the system after the fact, here we discuss how we might be able to do it all at once. We’ll take the trajectory optimization problem for kinematic paths, and turn it into true trajectory optimization with dynamics: we will optimize directly over the actions, taking the dynamics (or transitions) into account.

This brings us much closer to what we’ve seen in the search and MDP chapters. If we know the system’s dynamics, then we can find a sequence of actions to execute, as we did in Chapter 3 . If we’re not sure, then we might want a policy, as in Chapter 17 .

In this section, we are looking more directly at the underlying MDP the robot works in. We’re switching from the familiar discrete MDPs to continuous ones. We will denote our dynamic state of the world by x, as is common practice—the equivalent of s in discrete MDPs. Let xs and xg be the starting and goal states.

We want to find a sequence of actions that, when executed by the robot, result in stateaction pairs with low cumulative cost. The actions are torques which we denote with u(t) for

t starting at 0 and ending at T minimize a cumulative cost J

.

Formally, we want to find the sequence of torques

u that

:

(26.7) min

u

∫

T 0

J x t u t dt (

( ),

( ))

subject to the constraints

t xt fxt ut x xs x T xg

∀ ,  ˙( ) = (0) =

(

, 

( ),

(

) =

( )) .

How is this connected to motion planning and trajectory tracking control? Well, imagine we take the notion of efficiency and clearance away from the obstacles and put it into the cost function

J

,

just as we did before in trajectory optimization over kinematic state. The

dynamic state is the configuration and velocity, and torques

u change it via the dynamics f

from open-loop trajectory tracking. The difference is that now we’re thinking about the configurations and the torques at the same time. Sometimes, we might want to treat collision avoidance as a hard constraint as well, something we’ve also mentioned before when we looked at trajectory optimization for the kinematic state only.

J —not with respect to the sequence τ of configurations anymore, but directly with respect to the controls u It is sometimes helpful to include the state sequence x as a decision variable too, and use the dynamics constraints to ensure that x and u are consistent. There are various trajectory To solve this optimization problem, we can take gradients of

.

optimization techniques using this approach; two of them go by the names multiple shooting and direct collocation. None of these techniques will find the global optimal solution, but in practice they can effectively make humanoid robots walk and make autonomous cars drive.

Magic happens when in the problem above,

J is quadratic and f is linear in x and u

want to minimize

min

∞

∫

xT Qx uT Rudt +

0

subject to

t xt

∀ ,  ˙( ) =

Ax t

( ) +

Bu t

( ).

.

We

We can optimize over an infinite horizon rather than a finite one, and we obtain a policy from any state rather than just a sequence of controls.

Q and R need to be positive definite

matrices for this to work. This gives us the linear quadratic regulator (LQR). With LQR, the optimal value function (called cost to go) is quadratic, and the optimal policy is linear. The policy looks like

u

= −

Kx

,

where finding the matrix

K requires solving an algebraic Riccati

equation—no local optimization, no value iteration, no policy iteration are needed!

Linear quadratic regulator (LQR)

Riccati equation

Iterative LQR (ILQR)

Because of the ease of finding the optimal policy, LQR finds many uses in practice despite the fact that real problems seldom actually have quadratic costs and linear dynamics. A really useful method is called iterative LQR (ILQR), which works by starting with a solution and then iteratively computing a linear approximation of the dynamics and a quadratic approximation of the cost around it, then solving the resulting LQR system to arrive at a new solution. Variants of LQR are also often used for trajectory tracking.

26.6 Planning Uncertain Movements In robotics, uncertainty arises from partial observability of the environment and from the stochastic (or unmodeled) effects of the robot’s actions. Errors can also arise from the use of approximation algorithms such as particle filtering, which does not give the robot an exact belief state even if the environment is modeled perfectly.

The majority of today’s robots use deterministic algorithms for decision making, such as the path-planning algorithms of the previous section, or the search algorithms that were introduced in Chapter 3 . These deterministic algorithms are adapted in two ways: first, they deal with the continuous state space by turning it into a discrete space (for example with visibility graphs or cell decomposition). Second, they deal with uncertainty in the current state by choosing the most likely state from the probability distribution produced by the state estimation algorithm. That approach makes the computation faster and makes a better fit for the deterministic search algorithms. In this section we discuss methods for dealing with uncertainty that are analogous to the more complex search algorithms covered in Chapter 4 .

Most likely state

First, instead of deterministic plans, uncertainty calls for policies. We already discussed how trajectory tracking control turns a plan into a policy to compensate for errors in dynamics. Sometimes though, if the most likely hypothesis changes enough, tracking the plan designed for a different hypothesis is too suboptimal. This is where online replanning comes in: we can recompute a new plan based on the new belief. Many robots today use a technique called model predictive control (MPC), where they plan for a shorter time horizon, but replan at every time step. (MPC is therefore closely related to real-time search and gameplaying algorithms.) This effectively results in a policy: at every step, we run a planner and take the first action in the plan; if new information comes along, or we end up not where we

expected, that’s OK, because we are going to replan anyway and that will tell us what to do next.

Online replanning

Model predictive control (MPC)

Second, uncertainty calls for information gathering actions. When we consider only the information we have and make a plan based on it (this is called separating estimation from control), we are effectively solving (approximately) a new MDP at every step, corresponding to our current belief about where we are or how the world works. But in reality, uncertainty is better captured by the POMDP framework: there is something we don’t directly observe, be it the robot’s location or configuration, the location of objects in the world, or the parameters of the dynamics model itself—for example, where exactly is the center of mass of link two on this arm?

What we lose when we don’t solve the POMDP is the ability to reason about future information the robot will get: in MDPs we only plan with what we know, not with what we might eventually know. Remember the value of information? Well, robots that plan using their current belief as if they will never find out anything more fail to account for the value of information. They will never take actions that seem suboptimal right now according to what they know, but that will actually result in a lot of information and enable the robot to do well.

What does such an action look like for a navigation robot? The robot could get close to a landmark to get a better estimate of where it is, even if that landmark is out of the way according to what it currently knows. This action is optimal only if the robot considers the new observations it will get, as opposed to looking only at the information it already has.

Guarded movement

To get around this, robotics techniques sometimes define information gathering actions explicitly—such as moving a hand until it touches a surface (called guarded movements)— and make sure the robot does that before coming up with a plan for reaching its actual goal. Each guarded motion consists of (1) a motion command and (2) a termination condition, which is a predicate on the robot’s sensor values saying when to stop.

Sometimes, the goal itself could be reached via a sequence of guarded moves guaranteed to succeed regardless of uncertainty. As an example, Figure 26.23  shows a two-dimensional configuration space with a narrow vertical hole. It could be the configuration space for insertion of a rectangular peg into a hole or a car key into the ignition. The motion commands are constant velocities. The termination conditions are contact with a surface. To model uncertainty in control, we assume that instead of moving in the commanded direction, the robot’s actual motion lies in the cone Cv about it.

Figure 26.23

A two-dimensional environment, velocity uncertainty cone, and envelope of possible robot motions. The intended velocity is v, but with uncertainty the actual velocity could be anywhere in Cv , resulting in a final configuration somewhere in the motion envelope, which means we wouldn’t know if we hit the hole or not.

The figure shows what would happen if the robot attempted to move straight down from the initial configuration. Because of the uncertainty in velocity, the robot could move anywhere in the conical envelope, possibly going into the hole, but more likely landing to one side of

it. Because the robot would not then know which side of the hole it was on, it would not know which way to move. A more sensible strategy is shown in Figures 26.24  and 26.25 . In Figure 26.24 , the robot deliberately moves to one side of the hole. The motion command is shown in the figure, and the termination test is contact with any surface. In Figure 26.25 , a motion command is given that causes the robot to slide along the surface and into the hole. Because all possible velocities in the motion envelope are to the right, the robot will slide to the right whenever it is in contact with a horizontal surface.

Figure 26.24

The first motion command and the resulting envelope of possible robot motions. No matter what actual motion ensues, we know the final configuration will be to the left of the hole.

Figure 26.25

The second motion command and the envelope of possible motions. Even with error, we will eventually get into the hole.

It will slide down the right-hand vertical edge of the hole when it touches it, because all possible velocities are down relative to a vertical surface. It will keep moving until it reaches the bottom of the hole, because that is its termination condition. In spite of the control uncertainty, all possible trajectories of the robot terminate in contact with the bottom of the hole—that is, unless surface irregularities cause the robot to stick in one place.

Coastal navigation

Other techniques beyond guarded movements change the cost function to incentivize actions we know will lead to information—like the coastal navigation heuristic which requires the robot to stay near known landmarks. More generally, techniques can incorporate the expected information gain (reduction of entropy of the belief) as a term in the cost function, leading to the robot explicitly reasoning about how much information each action might bring when deciding what to do. While more difficult computationally, such approaches have the advantage that the robot invents its own information gathering actions rather than relying on human-provided heuristics and scripted strategies that often lack flexibility.

26.7 Reinforcement Learning in Robotics Thus far we have considered tasks in which the robot has access to the dynamics model of the world. In many tasks, it is very difficult to write down such a model, which puts us in the domain of reinforcement learning (RL).

One challenge of RL in robotics is the continuous nature of the state and action spaces, which we handle either through discretization, or, more commonly, through function approximation. Policies or value functions are represented as combinations of known useful features, or as deep neural networks. Neural nets can map from raw inputs directly to outputs, and thus largely avoid the need for feature engineering, but they do require more data.

A bigger challenge is that robots operate in the real world. We have seen how reinforcement learning can be used to learn to play chess or Go by playing simulated games. But when a real robot moves in the real world, we have to make sure that its actions are safe (things break!), and we have to accept that progress will be slower than in a simulation because the world refuses to move faster than one second per second. Much of what is interesting about using reinforcement learning in robotics boils down to how we might reduce the real world sample complexity—the number of interactions with the physical world that the robot needs before it has learned how to do the task.

26.7.1 Exploiting models A natural way to avoid the need for many real-world samples is to use as much knowledge of the world’s dynamics as possible. For instance, we might not know exactly what the coefficient of friction or the mass of an object is, but we might have equations that describe the dynamics as a function of these parameters. In such a case, model-based reinforcement learning (Chapter 22 ) is appealing, where the robot can alternate between fitting the dynamics parameters and computing a better policy. Even if the equations are incorrect because they fail to model every detail of physics, researchers have experimented with learning an error term, in addition to the parameters, that can compensate for the inaccuracy of the physical model. Or, we can abandon the

equations and instead fit locally linear models of the world that each approximate the dynamics in a region of the state space, an approach that has been successful in getting robots to master complex dynamic tasks like juggling.

Sim-to-real

A model of the world can also be useful in reducing the sample complexity of model-free reinforcement learning methods by doing sim-to-real transfer: transferring policies that work in simulation to the real world. The idea is to use the model as a simulator for a policy search (Section 22.5 ). To learn a policy that transfers well, we can add noise to the model during training, thereby making the policy more robust. Or, we can train policies that will work with a variety of models by sampling different parameters in the simulations— sometimes referred to as domain randomization. An example is in Figure 26.26 , where a dexterous manipulation task is trained in simulation by varying visual attributes, as well as physical attributes like friction or damping.

Figure 26.26

Training a robust policy. (a) Multiple simulations are run of a robot hand manipulating objects, with different randomized parameters for physics and lighting. Courtesy of Wojciech Zaremba. (b) The realworld environment, with a single robot hand in the center of a cage, surrounded by cameras and range

finders. (c) Simulation and real-world training yields multiple different policies for grasping objects; here a pinch grasp and a quadpod grasp. Courtesy of OpenAI. See Andrychowicz et al., (2018a).

Domain randomization

Finally, hybrid approaches that borrow ideas from both model-based and model-free algorithms are meant to give us the best of both. The hybrid approach originated with the Dyna architecture, where the idea was to iterate between acting and improving the policy, but the policy improvement would come in two complementary ways: 1) the standard model-free way of using the experience to directly update the policy, and 2) the modelbased way of using the experience to fit a model, then plan with it to generate a policy.

More recent techniques have experimented with fitting local models, planning with them to generate actions, and using these actions as supervision to fit a policy, then iterating to get better and better models around the areas that the policy needs. This has been successfully applied in end-to-end learning, where the policy takes pixels as input and directly outputs torques as actions—it enabled the first demonstration of deep RL on physical robots.

Models can also be exploited for the purpose of ensuring safe exploration. Learning slowly but safely may be better than learning quickly but crashing and burning half way through. So arguably, more important than reducing real-world samples is reducing real-world samples in dangerous states—we don’t want robots falling off cliffs, and we don’t want them breaking our favorite mugs or, even worse, colliding with objects and people. An approximate model, with uncertainty associated to it (for example by considering a range of values for its parameters), can guide exploration and impose constraints on the actions that the robot is allowed to take in order to avoid these dangerous states. This is an active area of research in robotics and control.

26.7.2 Exploiting other information Models are useful, but there is more we can do to further reduce sample complexity.

When setting up a reinforcement learning problem, we have to select the state and action spaces, the representation of the policy or value function, and the reward function we’re using. These decisions have a large impact on how easy or how hard we are making the problem.

One approach is to use higher-level motion primitives instead of low-level actions like torque commands. A motion primitive is a parameterized skill that the robot has. For example, a robotic soccer player might have the skill of “pass the ball to the player at (x, y).” All the policy needs to do is to figure out how to combine them and set their parameters, instead of reinventing them. This approach often learns much faster than low-level approaches, but does restrict the space of possible behaviors that the robot can learn.

Motion primitive

Another way to reduce the number of real-world samples required for learning is to reuse information from previous learning episodes on other tasks, rather than starting from scratch. This falls under the umbrella of metalearning or transfer learning.

Finally, people are a great source of information. In the next section, we talk about how to interact with people, and part of it is how to use their actions to guide the robot’s learning.

26.8 Humans and Robots Thus far, we’ve focused on a robot planning and learning how to act in isolation. This is useful for some robots, like the rovers we send out to explore distant planets on our behalf. But, for the most part, we do not build robots to work in isolation. We build them to help us, and to work in human environments, around and with us.

This raises two complementary challenges. First is optimizing reward when there are people acting in the same environment as the robot. We call this the coordination problem (see Section 18.1 ). When the robot’s reward depends on not just its own actions, but also the actions that people take, the robot has to choose its actions in a way that meshes well with theirs. When the human and the robot are on the same team, this turns into collaboration.

Second is the challenge of optimizing for what people actually want. If a robot is to help people, its reward function needs to incentivize the actions that people want the robot to execute. Figuring out the right reward function (or policy) for the robot is itself an interaction problem. We will explore these two challenges in turn.

26.8.1 Coordination Let’s assume for now, as we have been, that the robot has access to a clearly defined reward function. But, instead of needing to optimize it in isolation, now the robot needs to optimize it around a human who is also acting. For example, as an autonomous car merges on the highway, it needs to negotiate the maneuver with the human driver coming in the target lane—should it accelerate and merge in front, or slow down and merge behind? Later, as it pulls to a stop sign, preparing to take a right, it has to watch out for the cyclist in the bicycle lane, and for the pedestrian about to step onto the crosswalk.

Or, consider a mobile robot in a hallway. Someone heading straight toward the robot steps slightly to the right, indicating which side of the robot they want to pass on. The robot has to respond, clarifying its intentions.

Humans as approximately rational agents

One way to formulate coordination with a human is to model it as a game between the robot and the human (Section 18.2 ). With this approach, we explicitly make the assumption that people are agents incentivized by objectives. This does not automatically mean that they are perfectly rational agents (i.e., find optimal solutions in the game), but it does mean that the robot can structure the way it reasons about the human via the notion of possible objectives that the human might have. In this game:

the state of the environment captures the configurations of both the robot and human agents; call it

x = (xR , xH );

each agent can take actions,

uR and uH respectively;

each agent has an objective that can be represented as a cost,

JR and JH : each agent

wants to get to its goal safely and efficiently; and, as in any game, each objective depends on the state and on the actions of both agents:

JR (x, uR , uH ) and JH (x, uH , uR ). Think of the car-pedestrian interaction—the car

should stop if the pedestrian crosses, and should go forward if the pedestrian waits.

Incomplete information game

Three important aspects complicate this game. First is that the human and the robot don’t necessarily know each other’s objectives. This makes it an incomplete information game.

Second is that the state and action spaces are continuous, as they’ve been throughout this chapter. We learned in Chapter 5  how to do tree search to tackle discrete games, but how do we tackle continuous spaces?

Third, even though at the high level the game model makes sense—humans do move, and they do have objectives—a human’s behavior might not always be well-characterized as a solution to the game. The game comes with a computational challenge not only for the robot, but for us humans too. It requires thinking about what the robot will do in response to what the person does, which depends on what the robot thinks the person will do, and pretty soon we get to “what do you think I think you think I think”— it’s turtles all the way

down! Humans can’t deal with all of that, and exhibit certain suboptimalities. This means that the robot should account for these suboptimalities.

So, then, what is an autonomous car to do when the coordination problem is this hard? We will do something similar to what we’ve done before in this chapter. For motion planning and control, we took an MDP and broke it up into planning a trajectory and then tracking it with a controller. Here too, we will take the game, and break it up into making predictions about human actions, and deciding what the robot should do given these predictions.

Predicting human action Predicting human actions is hard because they depend on the robot’s actions and vice versa. One trick that robots use is to pretend the person is ignoring the robot. The robot assumes people are noisily optimal with respect to their objective, which is unknown to the robot and is modeled as no longer dependent on the robot’s actions:

JH (x, uH ). In particular, the

higher the value of an action for the objective (the lower the cost to go), the more likely the human is to take it. The robot can create a model for

P (uH ∣ x, JH ), for instance using the

softmax function from page 811:

(26.8)

P ( uH ∣ x, J H ) ∝ e with

−

Q(x, uH ;JH )

Q(x, uH ; JH ) the Q-value function corresponding to JH (the negative sign is there

because in robotics we like to minimize cost, not maximize reward). Note that the robot does not assume perfectly optimal actions, nor does it assume that the actions are chosen based on reasoning about the robot at all.

Armed with this model, the robot uses the human’s ongoing actions as evidence about

JH . If

we have an observation model for how human actions depend on the human’s objective, each human action can be incorporated to update the robot’s belief over what objective the person has:

(26.9)

b (JH ) ∝ b(JH )P (uH ∣ x, JH ). ′

An example is in Figure 26.27 : the robot is tracking a human’s location and as the human moves, the robot updates its belief over human goals. As the human heads toward the windows, the robot increases the probability that the goal is to look out the window, and decreases the probability that the goal is going to the kitchen, which is in the other direction.

Figure 26.27

Making predictions by assuming that people are noisily rational given their goal: the robot uses the past actions to update a belief over what goal the person is heading to, and then uses the belief to make predictions about future actions. (a) The map of a room. (b) Predictions after seeing a small part of the person’s trajectory (white path); (c) Predictions after seeing more human actions: the robot now knows that the person is not heading to the hallway on the left, because the path taken so far would be a poor path if that were the person’s goal. Images courtesy of Brian D. Ziebart. See Ziebart et al., (2009).

This is how the human’s past actions end up informing the robot about what the human will do in the future. Having a belief about the human’s goal helps the robot anticipate what next actions the human will take. The heatmap in the figure shows the robot’s future predictions: red is most probable; blue least probable.

The same can happen in driving. We might not know how much another driver values efficiency, but if we see them accelerate as someone is trying to merge in front of them, we now know a bit more about them. And once we know that, we can better anticipate what they will do in the future—the same driver is likely to come closer behind us, or weave through traffic to get ahead.

Once the robot can make predictions about human future actions, it has reduced its problem to solving an MDP. The human actions complicate the transition function, but as long as the robot can anticipate what action the person will take from any future state, the robot can

P (x ∣ x, uR ): it can compute P (uH ∣ x) from P (uH ∣ x, JH ) by marginalizing over JH , and combine it with P (x ∣ x, uR , uH ), the transition (dynamics) function for how the

calculate

′

′

world updates based on both the robot’s and the human’s actions. In Section 26.5  we focused on how to solve this in continuous state and action spaces for deterministic dynamics, and in Section 26.6  we discussed doing it with stochastic dynamics and uncertainty.

Splitting prediction from action makes it easier for the robot to handle interaction, but sacrifices performance much as splitting estimation from motion did, or splitting planning from control.

A robot with this split no longer understands that its actions can influence what people end up doing. In contrast, the robot in Figure 26.27  anticipates where people will go and then optimizes for reaching its own goal and avoiding collisions with them. In Figure 26.28 , we have an autonomous car merging on the highway. If it just planned in reaction to other cars, it might have to wait a long time while other cars occupy its target lane. In contrast, a car that reasons about prediction and action jointly knows that different actions it could take will result in different reactions from the human. If it starts to assert itself, the other cars are likely to slow down a bit and make room. Roboticists are working towards coordinated interactions like this so robots can work better with humans.

Figure 26.28

(a) Left: An autonomous car (middle lane) predicts that the human driver (left lane) wants to keep going forward, and plans a trajectory that slows down and merges behind. Right: The car accounts for the influence its actions can have on human actions, and realizes it can merge in front and rely on the human driver to slow down. (b) That same algorithm produces an unusual strategy at an intersection: the car realizes that it can make it more likely for the person (bottom) to proceed faster through the intersection by starting to inch backwards. Images courtesy of Anca Dragan. See Sadigh et al., (2016).

Human predictions about the robot Incomplete information is often two-sided: the robot does not know the human’s objective and the human, in turn, does not know the robot’s objective—people need to be making predictions about robots. As robot designers, we are not in charge of how the human makes predictions; we can only control what the robot does. However, the robot can act in a way to make it easier for the human to make correct predictions. The robot can assume that the human is using something roughly analogous to Equation (26.8)  to estimate the robot’s objective

JR , and thus the robot will act so that its true objective can be easily inferred.

A special case of the game is when the human and the robot are on the same team, working toward the same goal or objective:

JH = JR . Imagine getting a personal home robot that is

helping you make dinner or clean up—these are examples of collaboration.

We can now define a joint agent whose actions are tuples of human–robot actions, ( and who optimizes for

uH , uR )

JH (x, uH , uR ) = JR (x, uR , uH ), and we’re solving a regular planning

problem. We compute the optimal plan or policy for the joint agent, and voila, we now know what the robot and human should do.

Joint agent

This would work really well if people were perfectly optimal. The robot would do its part of the joint plan, the human theirs. Unfortunately, in practice, people don’t seem to follow the perfectly laid out joint-agent plan; they have a mind of their own! We’ve already learned one way to handle this though, back in Section 26.6 . We called it model predictive control (MPC): the idea was to come up with a plan, execute the first action, and then replan. That way, the robot always adapts its plan to what the human is actually doing.

Let’s work through an example. Suppose you and the robot are in your kitchen, and have decided to make waffles. You are slightly closer to the fridge, so the optimal joint plan would have you grab the eggs and milk from the fridge, while the robot fetches the flour from the cabinet. The robot knows this because it can measure quite precisely where everyone is. But suppose you start heading for the flour cabinet. You are going against the

optimal joint plan. Rather than sticking to it and stubbornly also going for the flour, the MPC robot recalculates the optimal plan, and now that you are close enough to the flour it is best for the robot to grab the waffle iron instead.

If we know that people might deviate from optimality, we can account for it ahead of time. In our example, the robot can try to anticipate that you are going for the flour the moment you take your first step (say, using the prediction technique above). Even if it is still technically optimal for you to turn around and head for the fridge, the robot should not assume that’s what is going to happen. Instead, the robot can compute a plan in which you keep doing what you seem to want.

Humans as black box agents We don’t have to treat people as objective-driven, intentional agents to get robots to coordinate with us. An alternative model is that the human is merely some agent whose policy

πH “messes” with the environment dynamics. The robot does not know πH

,

but can

model the problem as needing to act in an MDP with unknown dynamics. We have seen this before: for general agents in Chapter 22 , and for robots in particular in Section 26.7 .

The robot can fit a policy model

πH to human data, and use it to compute an optimal policy

for itself. Due to scarcity of data, this has been mostly used so far at the task level. For instance, robots have learned through interaction what actions people tend to take (in response to its own actions) for the task of placing and drilling screws in an industrial assembly task.

Then there is also the model-free reinforcement learning alternative: the robot can start with some initial policy or value function, and keep improving it over time via trial and error.

26.8.2 Learning to do what humans want Another way interaction with humans comes into robotics is in

JR itself—the robot’s cost or

reward function. The framework of rational agents and the associated algorithms reduce the problem of generating good behavior to specifying a good reward function. But for robots, as for many other AI agents, getting the cost right is still difficult.

Take autonomous cars: we want them to reach the destination, to be safe, to drive comfortably for their passengers, to obey traffic laws, etc. A designer of such a system needs to trade off these different components of the cost function. The designer’s task is hard because robots are built to help end users, and not every end user is the same. We all have different preferences for how aggressively we want our car to drive, etc.

Below, we explore two alternatives for trying to get robot behavior to match what we actually want the robot to do. The first is to learn a cost function from human input. The second is to bypass the cost function and imitate human demonstrations of the task.

Preference learning: Learning cost functions Imagine that an end user is showing a robot how to do a task. For instance, they are driving the car in the way they would like it to be driven by the robot. Can you think of a way for the robot to use these actions—we call them “demonstrations”—to figure out what cost function it should optimize? We have actually already seen the answer to this back in Section 26.8.1 . There, the setup was a little different: we had another person taking actions in the same space as the robot, and the robot needed to predict what the person would do. But one technique we went over for making these predictions was to assume that people act to noisily optimize some cost function

JH

,

and we can use their ongoing actions as evidence about what cost function that

is. We can do the same here, except not for the purpose of predicting human behavior in the future, but rather acquiring the cost function the robot itself should optimize. If the person drives defensively, the cost function that will explain their actions will put a lot of weight on safety and less so on efficiency. The robot can adopt this cost function as its own and optimize it when driving the car itself.

Roboticists have experimented with different algorithms for making this cost inference computationally tractable. In Figure 26.29 , we see an example of teaching a robot to prefer staying on the road to going over the grassy terrain. Traditionally in such methods, the cost function has been represented as a combination of hand-crafted features, but recent work has also studied how to represent it using a deep neural network, without feature engineering.

Figure 26.29

Left: A mobile robot is shown a demonstration that stays on the dirt road. Middle: The robot infers the desired cost function, and uses it in a new scene, knowing to put lower cost on the road there. Right: The robot plans a path for the new scene that also stays on the road, reproducing the preferences behind the demonstration. Images courtesy of Nathan Ratliff and James A. Bagnell. See Ratliff et al., (2006).

There are other ways for a person to provide input. A person could use language rather than demonstration to instruct the robot. A person could act as a critic, watching the robot perform a task one way (or two ways) and then saying how well the task was done (or which way was better), or giving advice on how to improve.

Learning policies directly via imitation An alternative is to bypass cost functions and learn the desired robot policy directly. In our car example, the human’s demonstrations make for a convenient data set of states labeled by the action the robot should take at each state: supervised learning to fit a policy

D = {(xi , ui )}. The robot can run

π : x ↦ u, and execute that policy. This is called imitation

learning or behavioral cloning.

Behavioral cloning

Generalization

A challenge with this approach is in generalization to new states. The robot does not know why the actions in its database have been marked as optimal. It has no causal rule; all it can do is run a supervised learning algorithm to try to learn a policy that will generalize to unknown states. However, there is no guarantee that the generalization will be correct.

Figure 26.30

A human teacher pushes the robot down to teach it to stay closer to the table. The robot appropriately updates its understanding of the desired cost function and starts optimizing it. Courtesy of Anca Dragan. See Bajcsy et al., (2017).

The ALVINN autonomous car project used this approach, and found that even when starting

D π will make small errors, which will take the car off the demonstrated trajectory. There, π will make a larger error, which will take the car even further off the from a state in

,

desired course.

We can address this at training time if we interleave collecting labels and learning: start with a demonstration, learn a policy, then roll out that policy and ask the human for what action to take at every state along the way, then repeat. The robot then learns how to correct its mistakes as it deviates from the human’s desired actions.

Alternatively, we can address it by leveraging reinforcement learning. The robot can fit a dynamics model based on the demonstrations, and then use optimal control (Section 26.5.4 ) to generate a policy that optimizes for staying close to the demonstration. A version of this has been used to perform very challenging maneuvers at an expert level in a small radio-controlled helicopter (see Figure 22.9(b) ).

The DAGGER (Data Aggregation) system starts with a human expert demonstration. From that it learns a policy,

π

1

and uses the policy to generate a data set

D

.

Then from

D it

π2 that best imitates the original human data. This repeats, and on the nth iteration it uses πn to generate more data, to be added to D, which is then used to create πn+1 . In other words, at each iteration the system gathers new data under the current generates a new policy

policy and trains the next policy using all the data gathered so far.

Related recent techniques use adversarial training: they alternate between training a classifier to distinguish between the robot’s learned policy and the human’s demonstrations, and training a new robot policy via reinforcement learning to fool the classifier. These advances enable the robot to handle states that are near demonstrations, but generalization to far-off states or to new dynamics is a work in progress.

Teaching interfaces and the correspondence problem. So far, we have imagined the case of an autonomous car or an autonomous helicopter, for which human demonstrations use the same actions that the robot can take itself: accelerating, braking, and steering. But what happens if we do this for tasks like cleaning up the kitchen table? We have two choices here: either the person demonstrates using their own body while the robot watches, or the person physically guides the robot’s effectors.

The first approach is appealing because it comes naturally to end users. Unfortunately, it suffers from the correspondence problem: how to map human actions onto robot actions. People have different kinematics and dynamics than robots. Not only does that make it difficult to translate or retarget human motion onto robot motion (e.g., retargeting a fivefinger human grasp to a two-finger robot grasp), but often the high-level strategy a person might use is not appropriate for the robot.

Correspondence problem

The second approach, where the human teacher moves the robot’s effectors into the right positions, is called kinesthetic teaching. It is not easy for humans to teach this way, especially to teach robots with multiple joints. The teacher needs to coordinate all the degrees of freedom as it is guiding the arm through the task. Researchers have thus investigated alternatives, like demonstrating keyframes as opposed to continuous

trajectories, as well as the use of visual programming to enable end users to program primitives for a task rather than demonstrate from scratch (Figure 26.31 ). Sometimes both approaches are combined.

Figure 26.31

A programming interface that involves placing specially designed blocks in the robot’s workspace to select objects and specify high-level actions. Images courtesy of Maya Cakmak. See Sefidgar et al., (2017).

Kinesthetic teaching

Keyframe

Visual programming

26.9 Alternative Robotic Frameworks Thus far, we have taken a view of robotics based on the notion of defining or learning a reward function, and having the robot optimize that reward function (be it via planning or learning), sometimes in coordination or collaboration with humans. This is a deliberative view of robotics, to be contrasted with a reactive view.

Deliberative

Reactive

26.9.1 Reactive controllers In some cases, it is easier to set up a good policy for a robot than to model the world and plan. Then, instead of a rational agent, we have a reflex agent.

For example, picture a legged robot that attempts to lift a leg over an obstacle. We could give this robot a rule that says lift the leg a small height h and move it forward, and if the leg encounters an obstacle, move it back and start again at a higher height. You could say that h is modeling an aspect of the world, but we can also think of h as an auxiliary variable of the robot controller, devoid of direct physical meaning. One such example is the six-legged (hexapod) robot, shown in Figure 26.32(a) , designed for walking through rough terrain. The robot’s sensors are inadequate to obtain accurate models of the terrain for path planning. Moreover, even if we added high-precision cameras and rangefinders, the 12 degrees of freedom (two for each leg) would render the resulting path planning problem computationally difficult.

Figure 26.32

(a) Genghis, a hexapod robot. (Image courtesy of Rodney A. Brooks.) (b) An augmented finite state machine (AFSM) that controls one leg. The AFSM reacts to sensor feedback: if a leg is stuck during the forward swinging phase, it will be lifted increasingly higher.

It is possible, nonetheless, to specify a controller directly without an explicit environmental model. (We have already seen this with the PD controller, which was able to keep a complex robot arm on target without an explicit model of the robot dynamics.)

For the hexapod robot we first choose a gait, or pattern of movement of the limbs. One statically stable gait is to first move the right front, right rear, and left center legs forward (keeping the other three fixed), and then move the other three. This gait works well on flat terrain. On rugged terrain, obstacles may prevent a leg from swinging forward. This problem can be overcome by a remarkably simple control rule: when a leg’s forward motion is blocked, simply retract it, lift it higher, and try again. The resulting controller is shown in Figure 26.32(b)  as a simple finite state machine; it constitutes a reflex agent with state, where the internal state is represented by the index of the current machine state (s1 through s4 ).

Gait

26.9.2 Subsumption architectures The subsumption architecture (Brooks, 1986) is a framework for assembling reactive controllers out of finite state machines. Nodes in these machines may contain tests for certain sensor variables, in which case the execution trace of a finite state machine is conditioned on the outcome of such a test. Arcs can be tagged with messages that will be

generated when traversing them, and that are sent to the robot’s motors or to other finite state machines. Additionally, finite state machines possess internal timers (clocks) that control the time it takes to traverse an arc. The resulting machines are called augmented finite state machines (AFSMs), where the augmentation refers to the use of clocks.

Subsumption architecture

Augmented finite state machine (AFSM)

An example of a simple AFSM is the four-state machine we just talked about, shown in Figure 26.32(b) . This AFSM implements a cyclic controller, whose execution mostly does not rely on environmental feedback. The forward swing phase, however, does rely on sensor feedback. If the leg is stuck, meaning that it has failed to execute the forward swing, the robot retracts the leg, lifts it up a little higher, and attempts to execute the forward swing once again. Thus, the controller is able to react to contingencies arising from the interplay of the robot and its environment.

The subsumption architecture offers additional primitives for synchronizing AFSMs, and for combining output values of multiple, possibly conflicting AFSMs. In this way, it enables the programmer to compose increasingly complex controllers in a bottom-up fashion. In our example, we might begin with AFSMs for individual legs, followed by an AFSM for coordinating multiple legs. On top of this, we might implement higher-level behaviors such as collision avoidance, which might involve backing up and turning.

The idea of composing robot controllers from AFSMs is quite intriguing. Imagine how difficult it would be to generate the same behavior with any of the configuration-space pathplanning algorithms described in the previous section. First, we would need an accurate model of the terrain. The configuration space of a robot with six legs, each of which is driven by two independent motors, totals 18 dimensions (12 dimensions for the configuration of the legs, and six for the location and orientation of the robot relative to its

environment). Even if our computers were fast enough to find paths in such highdimensional spaces, we would have to worry about nasty effects such as the robot sliding down a slope.

Because of such stochastic effects, a single path through configuration space would almost certainly be too brittle, and even a PID controller might not be able to cope with such contingencies. In other words, generating motion behavior deliberately is simply too complex a problem in some cases for present-day robot motion planning algorithms.

Unfortunately, the subsumption architecture has its own problems. First, the AFSMs are driven by raw sensor input, an arrangement that works if the sensor data is reliable and contains all necessary information for decision making, but fails if sensor data has to be integrated in nontrivial ways over time. Subsumption-style controllers have therefore mostly been applied to simple tasks, such as following a wall or moving toward visible light sources.

Second, the lack of deliberation makes it difficult to change the robot’s goals. A robot with a subsumption architecture usually does just one task, and it has no notion of how to modify its controls to accommodate different goals (just like the dung beetle on page 41).

Third, in many real-world problems, the policy we want is often too complex to encode explicitly. Think about the example from Figure 26.28 , of an autonomous car needing to negotiate a lane change with a human driver. We might start off with a simple policy that goes into the target lane. But when we test the car, we find out that not every driver in the target lane will slow down to let the car in. We might then add a bit more complexity: make the car nudge towards the target lane, wait for a response form the driver in that lane, and then either proceed or retreat back. But then we test the car, and realize that the nudging needs to happen at a different speed depending on the speed of the vehicle in the target lane, on whether there is another vehicle in front in the target lane, on whether there is a vehicle behind the car in the initial, and so on. The number of conditions that we need to consider to determine the right course of action can be very large, even for such a deceptively simple maneuver. This in turn presents scalability challenges for subsumptionstyle architectures.

All that said, robotics is a complex problem with many approaches: deliberative, reactive, or a mixture thereof; based on physics, cognitive models, data, or a mixture thereof. The right approach is still a subject for debate, scientific inquiry, and engineering prowess.

26.10 Application Domains Robotic technology is already permeating our world, and has the potential to improve our independence, health, and productivity. Here are some example applications.

HOME CARE: Robots have started to enter the home to care for older adults and people with motor impairments, assisting them with activities of daily living and enabling them to live more independently. These include wheelchairs and wheelchair-mounted arms like the Kinova arm from Figure 26.1(b) . Even though they start off as being operated by a human directly, these robots are gaining more and more autonomy. On the horizon are robots operated by brain–machine interfaces, which have been shown to enable people with quadriplegia to use a robot arm to grasp objects and even feed themselves (Figure 26.33(a) ). Related to these are prosthetic limbs that intelligently respond to our actions, and exoskeletons that give us superhuman strength or enable people who can’t control their muscles from the waist down to walk again.

Figure 26.33

(a) A patient with a brain–machine interface controlling a robot arm to grab a drink. Image courtesy of Brown University. (b) Roomba, the robot vacuum cleaner.

Photo by HANDOUT/KRT/Newscom.

Personal robots are meant to assist us with daily tasks like cleaning and organizing, freeing up our time. Although manipulation still has a way to go before it can operate seamlessly in messy, unstructured human environments, navigation has made some headway. In

particular, many homes already enjoy a mobile robot vacuum cleaner like the one in Figure 26.33(b) .

HEALTH CARE: Robots assist and augment surgeons, enabling more precise, minimally invasive, safer procedures with better patient outcomes. The Da Vinci surgical robot from Figure 26.34(a)  is now widely deployed at hospitals in the U.S.

Figure 26.34

(a) Surgical robot in the operating room. Photo by Patrick Landmann/Science Source. (b) Hospital delivery robot. Photo by Wired.

Telepresence robots

SERVICES: Mobile robots help out in office buildings, hotels, and hospitals. Savioke has put robots in hotels delivering products like towels or toothpaste to your room. The Helpmate and TUG robots carry food and medicine in hospitals (Figure 26.34(b) ), while Diligent Robotics’ Moxi robot helps out nurses with back-end logistical responsibilities. Co-Bot roams the halls of Carnegie Mellon University, ready to guide you to someone’s office. We can also use telepresence robots like the Beam to attend meetings and conferences remotely, or check in on our grandparents.

AUTONOMOUS CARS: Some of us are occasionally distracted while driving, by cell phone calls, texts, or other distractions. The sad result: more than a million people die every year in traffic accidents. Further, many of us spend a lot of time driving and would like to recapture some of that time. All this has led to a massive ongoing effort to deploy autonomous cars.

Prototypes have existed since the 1980s, but progress was stimulated by the 2005 DARPA Grand Challenge, an autonomous vehicle race over 200 challenging kilometers of unrehearsed desert terrain. Stanford’s Stanley vehicle completed the course in less than seven hours, winning a $2 million prize and a place in the National Museum of American History. Figure 26.35(a)  depicts BOSS, which in 2007 won the DARPA Urban Challenge, a complicated road race on city streets where robots faced other robots and had to obey traffic rules.

Figure 26.35

(a) Autonomous car BOSS which won the DARPA Urban Challenge. Photo by Tangi Quemener/AFP/Getty Images/Newscom. Courtesy of Sebastian Thrun. (b) Aerial view showing the perception and predictions of the Waymo autonomous car (white vehicle with green track). Other vehicles (blue boxes) and pedestrians (orange boxes) are shown with anticipated trajectories. Road/sidewalk boundaries are in yellow. Photo courtesy of Waymo.

In 2009, Google started an autonomous driving project (featuring many of the researchers who had worked on Stanley and BOSS), which has now spun off as Waymo. In 2018 Waymo started driverless testing (with nobody in the driver seat) in the suburbs of Pheonix, Arizona. In the meantime, other autonomous driving companies and ride-sharing companies are working on developing their own technology, while car manufacturers have been selling cars with more and more assistive intelligence, such as Tesla’s driver assist,

which is meant for highway driving. Other companies are targeting non-highway driving applications including college campuses and retirement communities. Still other companies are focused on non-passenger applications such as trucking, grocery delivery, and valet parking.

Driver assist

ENTERTAINMENT: Disney has been using robots (under the name animatronics) in their parks since 1963. Originally, these robots were restricted to hand-designed, open-loop, unvarying motion (and speech), but since 2009 a version called autonomatronics can generate autonomous actions. Robots also take the form of intelligent toys for children; for example, Anki’s Cozmo plays games with children and may pound the table with frustration when it loses. Finally, quadrotors like Skydio’s R1 from Figure 26.2(b)  act as personal photographers and videographers, following us around to take action shots as we ski or bike.

Animatronics

Autonomatronics

EXPLORATION AND HAZARDOUS ENVIRONMENTS: Robots have gone where no human has gone before, including the surface of Mars. Robotic arms assist astronauts in deploying and retrieving satellites and in building the International Space Station. Robots also help explore under the sea. They are routinely used to acquire maps of sunken ships. Figure 26.36  shows a robot mapping an abandoned coal mine, along with a 3D model of the mine acquired using range sensors. In 1996, a team of researches released a legged robot into the crater of an active volcano to acquire data for climate research. Robots are

becoming very effective tools for gathering information in domains that are difficult (or dangerous) for people to access.

Figure 26.36

(a) A robot mapping an abandoned coal mine. (b) A 3D map of the mine acquired by the robot. Courtesy of Sebastian Thrun.

Robots have assisted people in cleaning up nuclear waste, most notably in Three Mile Island, Chernobyl, and Fukushima. Robots were present after the collapse of the World Trade Center, where they entered structures deemed too dangerous for human search and rescue crews. Here too, these robots are initially deployed via teleoperation, and as technology advances they are becoming more and more autonomous, with a human operator in charge but not having to specify every single command.

INDUSTRY: The majority of robots today are deployed in factories, automating tasks that are difficult, dangerous, or dull for humans. (The majority of factory robots are in automobile factories.) Automating these tasks is a positive in terms of efficiently producing what society needs. At the same time, it also means displacing some human workers from their jobs. This has important policy and economics implications—the need for retraining and education, the need for a fair division of resources, etc. These topics are discussed further in Section 27.3.5 .

Summary Robotics is about physically embodied agents, which can change the state of the physical world. In this chapter, we have learned the following:

The most common types of robots are manipulators (robot arms) and mobile robots. They have sensors for perceiving the world and actuators that produce motion, which then affects the world via effectors. The general robotics problem involves stochasticity (which can be handled by MDPs), partial observability (which can be handled by POMDPs), and acting with and around other agents (which can be handled with game theory). The problem is made even harder by the fact that most robots work in continuous and high-dimensional state and action spaces. They also operate in the real world, which refuses to run faster than real time and in which failures lead to real things being damaged, with no “undo” capability. Ideally, the robot would solve the entire problem in one go: observations in the form of raw sensor feeds go in, and actions in the form of torques or currents to the motors come out. In practice though, this is too daunting, and roboticists typically decouple different aspects of the problem and treat them independently. We typically separate perception (estimation) from action (motion generation). Perception in robotics involves computer vision to recognize the surroundings through cameras, but also localization and mapping. Robotic perception concerns itself with estimating decision-relevant quantities from sensor data. To do so, we need an internal representation and a method for updating this internal representation over time. Probabilistic filtering algorithms such as particle filters and Kalman filters are useful for robot perception. These techniques maintain the belief state, a posterior distribution over state variables. For generating motion, we use configuration spaces, where a point specifies everything we need to know to locate every body point on the robot. For instance, for a robot arm with two joints, a configuration consists of the two joint angles. We typically decouple the motion generation problem into motion planning, concerned with producing a plan, and trajectory tracking control, concerned with producing a policy for control inputs (actuator commands) that results in executing the plan.

Motion planning can be solved via graph search using cell decomposition; using randomized motion planning algorithms, which sample milestones in the continuous configuration space; or using trajectory optimization, which can iteratively push a straight-line path out of collision by leveraging a signed distance field. A path found by a search algorithm can be executed using the path as the reference trajectory for a PID controller, which constantly corrects for errors between where the robot is and where it is supposed to be, or via computed torque control, which adds a feedforward term that makes use of inverse dynamics to compute roughly what torque to send to make progress along the trajectory. Optimal control unites motion planning and trajectory tracking by computing an optimal trajectory directly over control inputs. This is especially easy when we have quadratic costs and linear dynamics, resulting in a linear quadratic regulator (LQR). Popular methods make use of this by linearizing the dynamics and computing secondorder approximations of the cost (ILQR). Planning under uncertainty unites perception and action by online replanning (such as model predictive control) and information gathering actions that aid perception. Reinforcement learning is applied in robotics, with techniques striving to reduce the required number of interactions with the real world. Such techniques tend to exploit models, be it estimating models and using them to plan, or training policies that are robust with respect to different possible model parameters. Interaction with humans requires the ability to coordinate the robot’s actions with theirs, which can be formulated as a game. We usually decompose the solution into prediction, in which we use the person’s ongoing actions to estimate what they will do in the future, and action, in which we use the predictions to compute the optimal motion for the robot. Helping humans also requires the ability to learn or infer what they want. Robots can approach this by learning the desired cost function they should optimize from human input, such as demonstrations, corrections, or instruction in natural language. Alternatively, robots can imitate human behavior, and use reinforcement learning to help tackle the challenge of generalization to new states.

Bibliographical and Historical Notes The word robot was popularized by Czech playwright Karel Čapek in his 1920 play R.U.R. (Rossum’s Universal Robots). The robots, which were grown chemically rather than constructed mechanically, end up resenting their masters and decide to take over. It appears that it was Čapek’s brother, Josef, who first combined the Czech words “robota” (obligatory work) and “robotnik” (serf) to yield “robot” in his 1917 short story Opilec (Glanc, 1978). The term robotics was invented for a science fiction story (Asimov, 1950).

The idea of an autonomous machine predates the word “robot” by thousands of years. In 7th century BCE Greek mythology, a robot named Talos was built by Hephaistos, the Greek god of metallurgy, to protect the island of Crete. The legend is that the sorceress Medea defeated Talos by promising him immortality but then draining his life fluid. Thus, this is the first example of a robot making a mistake in the process of changing its objective function. In 322 BCE, Aristotle anticipated technological unemployment, speculating “If every tool, when ordered, or even of its own accord, could do the work that befits it... then there would be no need either of apprentices for the master workers or of slaves for the lords.”

In the 3rd century BCE an actual humanoid robot called the Servant of Philon could pour wine or water into a cup; a series of valves cut off the flow at the right time. Wonderful automata were built in the 18th century—Jacques Vaucanson’s mechanical duck from 1738 being one early example—but the complex behaviors they exhibited were entirely fixed in advance. Possibly the earliest example of a programmable robot-like device was the Jacquard loom (1805), described on page 15.

Grey Walter’s “turtle,” built in 1948, could be considered the first autonomous mobile robot, although its control system was not programmable. The “Hopkins Beast,” built in 1960 at Johns Hopkins University, was much more sophisticated; it had sonar and photocell sensors, pattern-recognition hardware, and could recognize the cover plate of a standard AC power outlet. It was capable of searching for outlets, plugging itself in, and then recharging its batteries! Still, the Beast had a limited repertoire of skills.

The first general-purpose mobile robot was “Shakey,” developed at what was then the Stanford Research Institute (now SRI) in the late 1960s (Fikes and Nilsson, 1971; Nilsson,

1984). Shakey was the first robot to integrate perception, planning, and execution, and much subsequent research in AI was influenced by this remarkable achievement. Shakey appears on the cover of this book with project leader Charlie Rosen (1917–2002). Other influential projects include the Stanford Cart and the CMU Rover (Moravec, 1983). Cox and Wilfong, (1990) describe classic work on autonomous vehicles.

The first commercial robot was an arm called UNIMATE, for universal automation, developed by Joseph Engelberger and George Devol in their compnay, Unimation. In 1961, the first UNIMATE robot was sold to General Motors for use in manufacturing TV picture tubes. 1961 was also the year when Devol obtained the first U.S. patent on a robot.

In 1973, Toyota and Nissan started using an updated version of UNIMATE for auto body spot welding. This initiated a major revolution in automobile manufacturing that took place mostly in Japan and the U.S., and that is still ongoing. Unimation followed up in 1978 with the development of the Puma robot (Programmable Universal Machine for Assembly), which was the de facto standard for robotic manipulation for the two decades that followed. About 500,000 robots are sold each year, with half of those going to the automotive industry.

In manipulation, the first major effort at creating a hand–eye machine was Heinrich Ernst’s MH-1, described in his MIT Ph.D. thesis (Ernst, 1961). The Machine Intelligence project at Edinburgh also demonstrated an impressive early system for vision-based assembly called FREDDY (Michie, 1972).

Research on mobile robotics has been stimulated by several important competitions. AAAI’s annual mobile robot competition began in 1992. The first competition winner was CARMEL (Congdon et al., 1992). Progress has been steady and impressive: in recent competitions robots entered the conference complex, found their way to the registration desk, registered for the conference, and even gave a short talk.

The RoboCup competition, launched in 1995 by Kitano and colleagues (1997), aims to “develop a team of fully autonomous humanoid robots that can win against the human world champion team in soccer” by 2050. Some competitions use wheeled robots, some humanoid robots, and some software simulations. Stone, (2016) describes recent innovations in RoboCup.

The DARPA Grand Challenge, organized by DARPA in 2004 and 2005, required autonomous vehicles to travel more than 200 kilometers through the desert in less than ten hours (Buehler et al., 2006). In the original event in 2004, no robot traveled more than eight miles, leading many to believe the prize would never be claimed. In 2005, Stanford’s robot Stanley won the competition in just under seven hours (Thrun, 2006). DARPA then organized the Urban Challenge, a competition in which robots had to navigate 60 miles in an urban environment with other traffic. Carnegie Mellon University’s robot BOSS took first place and claimed the $2 million prize (Urmson and Whittaker, 2008). Early pioneers in the development of robotic cars included Dickmanns and Zapp, (1987) and Pomerleau, (1993).

The field of robotic mapping has evolved from two distinct origins. The first thread began with work by Smith and Cheeseman (1986), who applied Kalman filters to the simultaneous localization and mapping (SLAM) problem. This algorithm was first implemented by Moutarlier and Chatila (1989), and later extended by Leonard and Durrant-Whyte, (1992); see Dissanayake et al., (2001) for an overview of early Kalman filter variations. The second thread began with the development of the occupancy grid representation for probabilistic mapping, which specifies the probability that each (x, y) location is occupied by an obstacle (Moravec and Elfes, 1985).

Occupancy grid

Kuipers and Levitt, (1988) were among the first to propose topological rather than metric mapping, motivated by models of human spatial cognition. A seminal paper by Lu and Milios, (1997) recognized the sparseness of the simultaneous localization and mapping problem, which gave rise to the development of nonlinear optimization techniques by Konolige, (2004) and Montemerlo and Thrun, (2004), as well as hierarchical methods by Bosse et al., (2004). Shatkay and Kaelbling, (1997) and Thrun et al., (1998) introduced the EM algorithm into the field of robotic mapping for data association. An overview of probabilistic mapping methods can be found in (Thrun et al., 2005).

Early mobile robot localization techniques are surveyed by Borenstein et al., (1996). Although Kalman filtering was well known as a localization method in control theory for

decades, the general probabilistic formulation of the localization problem did not appear in the AI literature until much later, through the work of Tom Dean and colleagues (Dean et al., 1990) and of Simmons and Koenig, (1995). The latter work introduced the term Markov localization. The first real-world application of this technique was by Burgard et al., (1999), through a series of robots that were deployed in museums. Monte Carlo localization based on particle filters was developed by Fox et al., (1999) and is now widely used. The RaoBlackwellized particle filter combines particle filtering for robot localization with exact filtering for map building (Murphy and Russell, 2001; Montemerlo et al., 2002).

Markov localization

Rao-Blackwellized particle filter

A great deal of early work on motion planning focused on geometric algorithms for deterministic and fully observable motion planning problems. The PSPACE-hardness of robot motion planning was shown in a seminal paper by Reif (1979). The configuration space representation is due to Lozano-Perez, (1983). A series of papers by Schwartz and Sharir on what they called piano movers problems (Schwartz et al., 1987) was highly influential.

Piano movers

Recursive cell decomposition for configuration space planning was originated in the work of Brooks and Lozano-Perez, (1985) and improved significantly by Zhu and Latombe, (1991). The earliest skeletonization algorithms were based on Voronoi diagrams (Rowat, 1979) and visibility graphs (Wesley and Lozano-Perez, 1979). Guibas et al., (1992) developed efficient

techniques for calculating Voronoi diagrams incrementally, and Choset, (1996) generalized Voronoi diagrams to broader motion planning problems.

Visibility graph

John Canny (1988) established the first singly exponential algorithm for motion planning. The seminal text by Latombe (1991) covers a variety of approaches to motion planning, as do the texts by Choset et al., (2005) and LaValle, (2006). Kavraki et al., (1996) developed the theory of probabilistic roadmaps. Kuffner and LaValle, (2000) developed rapidly exploring random trees (RRTs).

Involving optimization in geometric motion planning began with elastic bands (Quinlan and Khatib, 1993), which refine paths when the configuration-space obstacles change. Ratliff et al., (2009) formulated the idea as the solution to an optimal control problem, allowing the initial trajectory to start in collision, and deforming it by mapping workspace obstacle gradients via the Jacobian into the configuration space. Schulman et al., (2013) proposed a practical second-order alternative.

The control of robots as dynamical systems—whether for manipulation or navigation—has generated a vast literature. While this chapter explained the basics of trajectory tracking control and optimal control, it left out entire subfields, including adaptive control, robust control, and Lyapunov analysis. Rather than assuming everything about the system is known a priori, adaptive control aims to adapt the dynamics parameters and/or the control law online. Robust control, on the other hand, aims to design controllers that perform well in spite of uncertainty and external disturbances.

Lyapunov analysis was originally developed in the 1890s for the stability analysis of general nonlinear systems, but it was not until the early 1930s that control theorists realized its true potential. With the development of optimization methods, Lyapunov analysis was extended to control barrier functions, which lend themselves nicely to modern optimization tools. These methods are widely used in modern robotics for real-time controller design and safety analysis.

Crucial works in robotic control include a trilogy on impedance control by Hogan (1985), and a general study of robot dynamics by Featherstone, (1987). Dean and Wellman, (1991) were among the first to try to tie together control theory and AI planning systems. Three classic textbooks on the mathematics of robot manipulation are due to Paul (1981), Craig (1989), and Yoshikawa (1990). Control for manipulation is covered by Murray, (2017).

The area of grasping is also important in robotics—the problem of determining a stable grasp is quite difficult (Mason and Salisbury, 1985). Competent grasping requires touch sensing, or haptic feedback, to determine contact forces and detect slip (Fearing and Hollerbach, 1985). Understanding how to grasp the the wide variety of objects in the world is a daunting task. (Bousmalis et al., 2017) describe a system that combines real-world experimentation with simulations guided by sim-to-real transfer to produce robust grasping.

Haptic feedback

Potential-field control, which attempts to solve the motion planning and control problems simultaneously, was developed for robotics by Khatib (1986). In mobile robotics, this idea was viewed as a practical solution to the collision avoidance problem, and was later extended into an algorithm called vector field histograms by Borenstein (1991).

Vector field histogram

ILQR is currently widely used at the intersection of motion planning and control and is due to Li and Todorov, (2004). It is a variant of the much older differential dynamic programming technique (Jacobson and Mayne, 1970).

Fine-motion planning with limited sensing was investigated by Lozano-Perez et al., (1984) and Canny and Reif, (1987). Landmark-based navigation (Lazanas and Latombe, 1992) uses many of the same ideas in the mobile robot arena. Navigation functions, the robotics

version of a control policy for deterministic MDPs, were introduced by Koditschek, (1987). Key work applying POMDP methods (Section 17.4 ) to motion planning under uncertainty in robotics is due to Pineau et al., (2003) and Roy et al., (2005).

Reinforcement learning in robotics took off with the seminal work by Bagnell and Schneider, (2001) and Ng et al., (2003), who developed the paradigm in the context of autonomous helicopter control. Kober et al., (2013) offers an overview of how reinforcement learning changes when applied to the robotics problem. Many of the techniques implemented on physical systems build approximate dynamics models, dating back to locally weighted linear models due to Atkeson et al., (1997). But policy gradients played their role as well, enabling (simplified) humanoid robots to walk (Tedrake et al., 2004), or a robot arm to hit a baseball (Peters and Schaal, 2008).

Levine et al., (2016) demonstrated the first deep reinforcement learning application on a real robot. At the same time, model-free RL in simulation was being extended to continuous domains (Schulman et al., 2015a; Heess et al., 2016; Lillicrap et al., 2015). Other work scaled up physical data collection massively to showcase the learning of grasps and dynamics models (Pinto and Gupta, 2016; Agrawal et al., 2017; Levine et al., 2018). Transfer from simulation to reality or sim-to-real (Sadeghi and Levine, 2016; Andrychowicz et al., 2018a), metalearning (Finn et al., 2017), and sample-efficient model-free reinforcement learning (Andrychowicz et al., 2018b) are active areas of research.

Early methods for predicting human actions made use of filtering approaches (Madhavan and Schlenoff, 2003), but seminal work by Ziebart et al., (2009) proposed prediction by modeling people as approximately rational agents. Sadigh et al., (2016) captured how these predictions should actually depend on what the robot decides to do, building toward a game-theoretic setting. For collaborative settings, Sisbot et al., (2007) pioneered the idea of accounting for what people want in the robot’s cost function. Nikolaidis and Shah, (2013) decomposed collaboration into learning how the human will act, but also learning how the human wants the robot to act, both achievable from demonstrations. For learning from demonstration see Argall et al., (2009). Akgun et al., (2012) and Sefidgar et al., (2017) studied teaching by end users rather than by experts.

Tellex et al., (2011) showed how robots can infer what people want from natural language instructions. Finally, not only do robots need to infer what people want and plan on doing,

but people too need to make the same inferences about robots. Dragan et al., (2013) incorporated a model of the human’s inferences into robot motion planning.

The field of human–robot interaction is much broader than what we covered in this chapter, which focused primarily on the planning and learning aspects. Thomaz et al., (2016) provides a survey of interaction more broadly from a computational perspective. Ross et al., (2011) describe the DAGGER system.

The topic of software architectures for robots engenders much religious debate. The good old-fashioned AI candidate—the three-layer architecture—dates back to the design of Shakey and is reviewed by Gat (1998). The subsumption architecture is due to Brooks (1986), although similar ideas were developed independently by Braitenberg, whose book, Vehicles (1984), describes a series of simple robots based on the behavioral approach.

The success of Brooks’s six-legged walking robot was followed by many other projects. Connell, in his Ph.D. thesis (1989), developed an entirely reactive mobile robot that was capable of retrieving objects. Extensions of the paradigm to multirobot systems can be found in work by Parker (1996) and Mataric (1997). GRL (Horswill, 2000) and COLBERT (Konolige, 1997) abstract the ideas of concurrent behavior-based robotics into general robot control languages. Arkin (1998) surveys some of the most popular approaches in this field.

Two early textbooks, by Dudek and Jenkin (2000) and by Murphy (2000), cover robotics generally. More recent overviews are due to Bekey (2008) and Lynch and Park (2017). An excellent book on robot manipulation addresses advanced topics such as compliant motion (Mason 2001). Robot motion planning is covered in Choset et al., (2005) and LaValle, (2006). Thrun et al., (2005) introduces probabilistic robotics. The Handbook of Robotics (Siciliano and Khatib, 2016) is a massive, comprehensive overview of all of robotics.

The premiere conference for robotics is Robotics: Science and Systems Conference, followed by the IEEE International Conference on Robotics and Automation. Human–Robot Interaction is the premiere venue for interaction. Leading robotics journals include IEEE Robotics and Automation, the International Journal of Robotics Research, and Robotics and Autonomous Systems.

VII Conclusions

Chapter 27 Philosophy, Ethics, and Safety of AI In which we consider the big questions around the meaning of AI, how we can ethically develop and apply it, and how we can keep it safe.

Philosophers have been asking big questions for a long time: How do minds work? Is it possible for machines to act intelligently in the way that people do? Would such machines have real, conscious minds?

To these, we add new ones: What are the ethical implications of intelligent machines in dayto-day use? Should machines be allowed to decide to kill humans? Can algorithms be fair and unbiased? What will humans do if machines can do all kinds of work? And how do we control machines that may become more intelligent than us?

27.1 The Limits of AI In 1980, philosopher John Searle introduced a distinction between weak AI—the idea that machines could act as if they were intelligent—and strong AI—the assertion that machines that do so are actually consciously thinking (not just simulating thinking). Over time the definition of strong AI shifted to refer to what is also called “human-level AI” or “general AI”—programs that can solve an arbitrarily wide variety of tasks, including novel ones, and do so as well as a human.

Weak AI

Strong AI

Critics of weak AI who objected to the very possibility of intelligent behavior in machines now appear as shortsighted as Simon Newcomb, who in October 1903 wrote “aerial flight is one of the great class of problems with which man can never cope”—just two months before the Wright brothers’ flight at Kitty Hawk. The rapid progress of recent years does not, however, prove that there can be no limits to what AI can achieve. Alan Turing (1950), the first person to define AI, was also the first to raise possible objections to AI, foreseeing almost all the ones subsequently raised by others.

27.1.1 The argument from informality Turing’s “argument from informality of behavior” says that human behavior is far too complex to be captured by any formal set of rules—humans must be using some informal guidelines that (the argument claims) could never be captured in a formal set of rules and thus could never be codified in a computer program.

A key proponent of this view was Hubert Dreyfus, who produced a series of influential critiques of artificial intelligence: What Computers Can’t Do (1972), the sequel What Computers Still Can’t Do (1992), and, with his brother Stuart, Mind Over Machine (1986). Similarly, philosopher Kenneth Sayre (1993) said “Artificial intelligence pursued within the cult of computationalism stands not even a ghost of a chance of producing durable results.” The technology they criticize came to be called Good Old-Fashioned AI (GOFAI).

Good Old-Fashioned AI (GOFAI)

GOFAI corresponds to the simplest logical agent design described in Chapter 7 , and we saw there that it is indeed difficult to capture every contingency of appropriate behavior in a set of necessary and sufficient logical rules; we called that the qualification problem. But as we saw in Chapter 12 , probabilistic reasoning systems are more appropriate for openended domains, and as we saw in Chapter 21 , deep learning systems do well on a variety of “informal” tasks. Thus, the critique is not addressed against computers per se, but rather against one particular style of programming them with logical rules—a style that was popular in the 1980s but has been eclipsed by new approaches.

One of Dreyfus’s strongest arguments is for situated agents rather than disembodied logical inference engines. An agent whose understanding of “dog” comes only from a limited set of

Dog(x) ⇒ Mammal(x)” is at a disadvantage compared to an agent

logical sentences such as “

that has watched dogs run, has played fetch with them, and has been licked by one. As philosopher Andy Clark (1998) says, “Biological brains are first and foremost the control systems for biological bodies. Biological bodies move and act in rich real-world surroundings.” According to Clark, we are “good at frisbee, bad at logic.”

The embodied cognition approach claims that it makes no sense to consider the brain separately: cognition takes place within a body, which is embedded in an environment. We need to study the system as a whole; the brain’s functioning exploits regularities in its environment, including the rest of its body. Under the embodied cognition approach, robotics, vision, and other sensors become central, not peripheral.

Embodied cognition

Overall, Dreyfus saw areas where AI did not have complete answers and said that AI is therefore impossible; we now see many of these same areas undergoing continued research and development leading to increased capability, not impossibility.

27.1.2 The argument from disability The “argument from disability” makes the claim that “a machine can never do examples of

X, Turing lists the following:

X.” As

Be kind, resourceful, beautiful, friendly, have initiative, have a sense of humor, tell right from wrong, make mistakes, fall in love, enjoy strawberries and cream, make someone fall in love with it, learn from experience, use words properly, be the subject of its own thought, have as much diversity of behavior as man, do something really new.

In retrospect, some of these are rather easy—we’re all familiar with computers that “make mistakes.” Computers with metareasoning capabilities (Chapter 5 ) can examine heir own computations, thus being the subject of their own reasoning. A century-old technology has the proven ability to “make someone fall in love with it”—the teddy bear. Computer chess expert David Levy predicts that by 2050 people will routinely fall in love with humanoid robots. As for a robot falling in love, that is a common theme in fiction,1 but there has been only limited academic speculation on the subject (Kim et al., 2007). Computers have done things that are “really new,” making significant discoveries in astronomy, mathematics, chemistry, mineralogy, biology, computer science, and other fields, and creating new forms of art through style transfer (Gatys et al., 2016). Overall, programs exceed human performance in some tasks and lag behind on others. The one thing that it is clear they can’t do is be exactly human.

1 For example, the opera Coppélia (1870), the novel Do Androids Dream of Electric Sheep? (1968), the movies AI (2001), Wall-E (2008), and Her (2013).

27.1.3 The mathematical objection

Turing (1936) and Gödel (1931) proved that certain mathematical questions are in principle unanswerable by particular formal systems. Gödel’s incompleteness theorem (see Section 9.5 ) is the most famous example of this. Briefly, for any formal axiomatic framework F powerful enough to do arithmetic, it is possible to construct a so-called Gödel sentence G(F ) with the following properties:

G(F ) is a sentence of F , but cannot be proved within F . If F is consistent, then G(F ) is true.

Philosophers such as J. R. Lucas (1961) have claimed that this theorem shows that machines are mentally inferior to humans, because machines are formal systems that are limited by the incompleteness theorem—they cannot establish the truth of their own Gödel sentence— while humans have no such limitation. This has caused a lot of controversy, spawning a vast literature, including two books by the mathematician/physicist Sir Roger Penrose (1989, 1994). Penrose repeats Lucas’s claim with some fresh twists, such as the hypothesis that humans are different because their brains operate by quantum gravity—a theory that makes multiple false predictions about brain physiology.

We will examine three of the problems with Lucas’s claim. First, an agent should not be ashamed that it cannot establish the truth of some sentence while other agents can. Consider the following sentence:

Lucas cannot consistently assert that this sentence is true.

If Lucas asserted this sentence, then he would be contradicting himself, so therefore Lucas cannot consistently assert it, and hence it is true. We have thus demonstrated that there is a true sentence that Lucas cannot consistently assert while other people (and machines) can. But that does not make us think any less of Lucas.

Second, Gödel’s incompleteness theorem and related results apply to mathematics, not to computers. No entity—human or machine—can prove things that are impossible to prove. Lucas and Penrose falsely assume that humans can somehow get around these limits, as when Lucas (1976) says “we must assume our own consistency, if thought is to be possible at all.” But this is an unwarranted assumption: humans are notoriously inconsistent. This is certainly true for everyday reasoning, but it is also true for careful mathematical thought. A

famous example is the four-color map problem. Alfred Kempe (1879) published a proof that was widely accepted for 11 years until Percy Heawood (1890) pointed out a flaw.

Third, Gödel’s incompleteness theorem technically applies only to formal systems that are powerful enough to do arithmetic. This includes Turing machines, and Lucas’s claim is in part based on the assertion that computers are equivalent to Turing machines. This is not quite true. Turing machines are infinite, whereas computers (and brains) are finite, and any computer can therefore be described as a (very large) system in propositional logic, which is not subject to Gödel’s incompleteness theorem. Lucas assumes that humans can “change their minds” while computers cannot, but that is also false—a computer can retract a conclusion after new evidence or further deliberation; it can upgrade its hardware; and it can change its decision-making processes with machine learning or software rewriting.

27.1.4 Measuring AI Alan Turing, in his famous paper “Computing Machinery and Intelligence” (1950), suggested that instead of asking whether machines can think, we should ask whether machines can pass a behavioral test, which has come to be called the Turing test. The test requires a program to have a conversation (via typed messages) with an interrogator for five minutes. The interrogator then has to guess if the conversation is with a program or a person; the program passes the test if it fools the interrogator 30% of the time. To Turing, the key point was not the exact details of the test, but instead the idea of measuring intelligence by performance on some kind of open-ended behavioral task, rather than by philosophical speculation.

Nevertheless, Turing conjectured that by the year 2000 a computer with a storage of a billion units could pass the test, but here we are on the other side of 2000, and we still can’t agree whether any program has passed. Many people have been fooled when they didn’t know they might be chatting with a computer. The ELIZA program and Internet chatbots such as MGONZ (Humphrys, 2008) and NATACHATA (Jonathan et al., 2009) fool their correspondents repeatedly, and the chatbot CYBERLOVER has attracted the attention of law enforcement because of its penchant for tricking fellow chatters into divulging enough personal information that their identity can be stolen.

In 2014, a chatbot called Eugene Goostman fooled 33% of the untrained amateur judges in a Turing test. The program claimed to be a boy from Ukraine with limited command of

English; this helped explain its grammatical errors. Perhaps the Turing test is really a test of human gullibility. So far no well-trained judge has been fooled (Aaronson, 2014).

Turing test competitions have led to better chatbots, but have not been a focus of research within the AI community. Instead, AI researchers who crave competition are more likely to concentrate on playing chess or Go or StarCraft II, or taking an 8th grade science exam, or identifying objects in images. In many of these competitions, programs have reached or surpassed human-level performance, but that doesn’t mean the programs are human-like outside the specific task. The point is to improve basic science and technology and to provide useful tools, not to fool judges.

27.2 Can Machines Really Think? Some philosophers claim that a machine that acts intelligently would not be actually thinking, but would be only a simulation of thinking. But most AI researchers are not concerned with the distinction, and the computer scientist Edsger Dijkstra (1984) said that “The question of whether Machines Can Think ... is about as relevant as the question of whether Submarines Can Swim.” The American Heritage Dictionary’s first definition of swim is “To move through water by means of the limbs, fins, or tail,” and most people agree that submarines, being limbless, cannot swim. The dictionary also defines fly as “To move through the air by means of wings or winglike parts,” and most people agree that airplanes, having winglike parts, can fly. However, neither the questions nor the answers have any relevance to the design or capabilities of airplanes and submarines; rather they are about word usage in English. (The fact that ships do swim ( “privet”) in Russian amplifies this point.) English speakers have not yet settled on a precise definition for the word “think”— does it require “a brain” or just “brain-like parts?”

Again, the issue was addressed by Turing. He notes that we never have any direct evidence about the internal mental states of other humans—a kind of mental solipsism. Nevertheless, Turing says, “Instead of arguing continually over this point, it is usual to have the polite convention that everyone thinks.” Turing argues that we would also extend the polite convention to machines, if only we had experience with ones that act intelligently. However, now that we do have some experience, it seems that our willingness to ascribe sentience depends at least as much on humanoid appearance and voice as on pure intelligence.

Polite convention

27.2.1 The Chinese room The philosopher John Searle rejects the polite convention. His famous Chinese room argument (Searle, 1990) goes as follows: Imagine a human, who understands only English,

inside a room that contains a rule book, written in English, and various stacks of paper. Pieces of paper containing indecipherable symbols are slipped under the door to the room. The human follows the instructions in the rule book, finding symbols in the stacks, writing symbols on new pieces of paper, rearranging the stacks, and so on. Eventually, the instructions will cause one or more symbols to be transcribed onto a piece of paper that is passed back to the outside world. From the outside, we see a system that is taking input in the form of Chinese sentences and generating fluent, intelligent Chinese responses.

Chinese room

Searle then argues: it is given that the human does not understand Chinese. The rule book and the stacks of paper, being just pieces of paper, do not understand Chinese. Therefore, there is no understanding of Chinese. And Searle says that the Chinese room is doing the same thing that a computer would do, so therefore computers generate no understanding.

Searle (1980) is a proponent of biological naturalism, according to which mental states are high-level emergent features that are caused by low-level physical processes in the neurons, and it is the (unspecified) properties of the neurons that matter: according to Searle’s biases, neurons have “it” and transistors do not. There have been many refutations of Searle’s argument, but no consensus. His argument could equally well be used (perhaps by robots) to argue that a human cannot have true understanding; after all, a human is made out of cells, the cells do not understand, therefore there is no understanding. In fact, that is the plot of Terry Bisson’s (1990) science fiction story They’re Made Out of Meat, in which alien robots explore Earth and can’t believe that hunks of meat could possibly be sentient. How they can be remains a mystery.

Biological naturalism

27.2.2 Consciousness and qualia

Running through all the debates about strong AI is the issue of consciousness: awareness of the outside world, and of the self, and the subjective experience of living. The technical term for the intrinsic nature of experiences is qualia (from the Latin word meaning, roughly, “of what kind”). The big question is whether machines can have qualia. In the movie 2001, when astronaut David Bowman is disconnecting the “cognitive circuits” of the HAL 9000 computer, it says “I’m afraid, Dave. Dave, my mind is going. I can feel it.” Does HAL actually have feelings (and deserve sympathy)? Or is the reply just an algorithmic response, no different from “Error 404: not found”?

Consciousness

Qualia

There is a similar question for animals: pet owners are certain that their dog or cat has consciousness, but not all scientists agree. Crickets change their behavior based on temperature, but few people would say that crickets experience the feeling of being warm or cold.

One reason that the problem of consciousness is hard is that it remains ill-defined, even after centuries of debate. But help may be on the way. Recently philosophers have teamed with neuroscientists under the auspices of the Templeton Foundation to start a series of experiments that could resolve some of the issues. Advocates of two leading theories of consciousness (global workspace theory and integrated information theory) have agreed that the experiments could confirm one theory over the other—a rarity in philosophy.

Alan Turing (1950) concedes that the question of consciousness is a difficult one, but denies that it has much relevance to the practice of AI: “I do not wish to give the impression that I think there is no mystery about consciousness

… But I do not think these mysteries

necessarily need to be solved before we can answer the question with which we are concerned in this paper.” We agree with Turing—we are interested in creating programs that

behave intelligently. Individual aspects of consciousness—awareness, self-awareness, attention—can be programmed and can be part of an intelligent machine. The additional project of making a machine conscious in exactly the way humans are is not one that we are equipped to take on. We do agree that behaving intelligently will require some degree of awareness, which will differ from task to task, and that tasks involving interaction with humans will require a model of human subjective experience.

In the matter of modeling experience, humans have a clear advantage over machines, because they can use their own subjective apparatus to appreciate the subjective experience of others. For example, if you want to know what it’s like when someone hits their thumb with a hammer, you can hit your thumb with a hammer. Machines have no such capability— although unlike humans, they can run each other’s code.

27.3 The Ethics of AI Given that AI is a powerful technology, we have a moral obligation to use it well, to promote the positive aspects and avoid or mitigate the negative ones.

The positive aspects are many. For example, AI can save lives through improved medical diagnosis, new medical discoveries, better prediction of extreme weather events, and safer driving with driver assistance and (eventually) self-driving technologies. There are also many opportunities to improve lives. Microsoft’s AI for Humanitarian Action program applies AI to recovering from natural disasters, addressing the needs of children, protecting refugees, and promoting human rights. Google’s AI for Social Good program supports work on rainforest protection, human rights jurisprudence, pollution monitoring, measurement of fossil fuel emissions, crisis counseling, news fact checking, suicide prevention, recycling, and other issues. The University of Chicago’s Center for Data Science for Social Good applies machine learning to problems in criminal justice, economic development, education, public health, energy, and environment.

AI applications in crop management and food production help feed the world. Optimization of business processes using machine learning will make businesses more productive, increasing wealth and providing more employment. Automation can replace the tedious and dangerous tasks that many workers face, and free them to concentrate on more interesting aspects. People with disabilities will benefit from AI-based assistance in seeing, hearing, and mobility. Machine translation already allows people from different cultures to communicate. Software-based AI solutions have near zero marginal cost of production, and so have the potential to democratize access to advanced technology (even as other aspects of software have the potential to centralize power).

Despite these many positive aspects, we shouldn’t ignore the negatives. Many new technologies have had unintended negative side effects: nuclear fission brought Chernobyl and the threat of global destruction; the internal combustion engine brought air pollution, global warming, and the paving of paradise. Other technologies can have negative effects even when used as intended, such as sarin gas, AR-15 rifles, and telephone solicitation. Automation will create wealth, but under current economic conditions much of that wealth will flow to the owners of the automated systems, leading to increased income inequality.

This can be disruptive to a well-functioning society. In developing countries, the traditional path to growth through low-cost manufacturing for export may be cut off, as wealthy countries adopt fully automated manufacturing facilities on-shore. Our ethical and governance decisions will dictate the level of inequality that AI will engender.

Negative side effects

All scientists and engineers face ethical considerations of what projects they should or should not take on, and how they can make sure the execution of the project is safe and beneficial. In 2010, the UK’s Engineering and Physical Sciences Research Council held a meeting to develop a set of Principles of Robotics. In subsequent years other government agencies, nonprofit organizations, and companies created similar sets of principles. The gist is that every organization that creates AI technology, and everyone in the organization, has a responsibility to make sure the technology contributes to good, not harm. The most commonly-cited principles are:

Note that many of the principles, such as “ensure safety,” have applicability to all software or hardware systems, not just AI systems. Several principles are worded in a vague way, making them difficult to measure or enforce. That is in part because AI is a big field with many subfields, each of which has a different set of historical norms and different relationships between the AI developers and the stakeholders. Mittelstadt (2019) suggests that the subfields should each develop more specific actionable guidelines and case precedents.

27.3.1 Lethal autonomous weapons

The UN defines a lethal autonomous weapon as one that locates, selects, and engages (i.e., kills) human targets without human supervision. Various weapons fulfill some of these criteria. For example, land mines have been used since the 17th century: they can select and engage targets in a limited sense according to the degree of pressure exerted or the quantity of metal present, but they cannot go out and locate targets by themselves. (Land mines are banned under the Ottawa Treaty.) Guided missiles, in use since the 1940s, can chase targets, but they have to be pointed in the right general direction by a human. Auto-firing radarcontrolled guns have been used to defend naval ships since the 1970s; they are mainly intended to destroy incoming missiles, but they could also attack manned aircraft. Although the word “autonomous” is often used to describe unmanned air vehicles or drones, most such weapons are both remotely piloted and require human actuation of the lethal payload.

At the time of writing, several weapons systems seem to have crossed the line into full autonomy. For example Israel’s Harop missile is a “loitering munition” with a ten-foot wingspan and a fifty-pound warhead. It searches for up to six hours in a given geographical region for any target that meets a given criterion and then destroys it. The criterion could be “emits a radar signal resembling antiaircraft radar” or “looks like a tank.” The Turkish manufacturer STM advertises its Kargu quadcopter—which carries up to 1.5kg of explosives —as capable of “Autonomous hit ... targets selected on images ... tracking moving targets ... anti-personnel ... face recognition.”

Autonomous weapons have been called the “third revolution in warfare” after gunpowder and nuclear weapons. Their military potential is obvious. For example, few experts doubt that autonomous fighter aircraft would defeat any human pilot. Autonomous aircraft, tanks, and submarines can be cheaper, faster, more maneuverable, and have longer range than their manned counterparts.

Since 2014, the United Nations in Geneva has conducted regular discussions under the auspices of the Convention on Certain Conventional Weapons (CCW) on the question of whether to ban lethal autonomous weapons. At the time of writing, 30 nations, ranging in size from China to the Holy See, have declared their support for an international treaty, while other key countries—including Israel, Russia, South Korea, and the United States—are opposed to a ban.

The debate over autonomous weapons includes legal, ethical and practical aspects. The legal issues are governed primarily by the CCW, which requires the possibility of discriminating between combatants and non-combatants, the judgment of military necessity for an attack, and the assessment of proportionality between the military value of a target and the possibility of collateral damage. The feasibility of meeting these criteria is an engineering question—one whose answer will undoubtedly change over time. At present, discrimination seems feasible in some circumstances and will undoubtedly improve rapidly, but necessity and proportionality are not presently feasible: they require that machines make subjective and situational judgments that are considerably more difficult than the relatively simple tasks of searching for and engaging potential targets. For these reasons, it would be legal to use autonomous weapons only in circumstances where a human operator can reasonably predict that the execution of the mission will not result in civilians being targeted or the weapons conducting unnecessary or disproportionate attacks. This means that, for the time being, only very restricted missions could be undertaken by autonomous weapons.

On the ethical side, some find it simply morally unacceptable to delegate the decision to kill humans to a machine. For example, Germany’s ambassador in Geneva has stated that it “will not accept that the decision over life and death is taken solely by an autonomous system” while Japan “has no plan to develop robots with humans out of the loop, which may be capable of committing murder.” Gen. Paul Selva, at the time the second-ranking military officer in the United States, said in 2017, “I don’t think it’s reasonable for us to put robots in charge of whether or not we take a human life.” Finally, António Guterres, the head of the United Nations, stated in 2019 that “machines with the power and discretion to take lives without human involvement are politically unacceptable, morally repugnant and should be prohibited by international law.”

More than 140 NGOs in over 60 countries are part of the Campaign to Stop Killer Robots, and an open letter organized in 2015 by the Future of Life Institute organized an open letter was signed by over 4,000 AI researchers2 and 22,000 others.

2 Including the two authors of this book.

Against this, it can be argued that as technology improves it ought to be possible to develop weapons that are less likely than human soldiers or pilots to cause civilian casualties. (There is also the important benefit that autonomous weapons reduce the need for human soldiers

and pilots to risk death.) Autonomous systems will not succumb to fatigue, frustration, hysteria, fear, anger, or revenge, and need not “shoot first, ask questions later” (Arkin, 2015). Just as guided munitions have reduced collateral damage compared to unguided bombs, one may expect intelligent weapons to further improve the precision of attacks. (Against this, see Benjamin (2013) for an analysis of drone warfare casualties.) This, apparently, is the position of the United States in the latest round of negotiations in Geneva.

Perhaps counterintuitively, the United States is also one of the few nations whose own policies currently preclude the use of autonomous weapons. The 2011 U.S. Department of Defense (DOD) roadmap says: “For the foreseeable future, decisions over the use of force [by autonomous systems] and the choice of which individual targets to engage with lethal force will be retained under human control.” The primary reason for this policy is practical: autonomous systems are not reliable enough to be trusted with military decisions.

The issue of reliability came to the fore on September 26, 1983, when Soviet missile officer Stanislav Petrov’s computer display flashed an alert of an incoming missile attack. According to protocol, Petrov should have initiated a nuclear counterattack, but he suspected the alert was a bug and treated it as such. He was correct, and World War III was (narrowly) averted. We don’t know what would have happened if there had been no human in the loop.

Reliability is a very serious concern for military commanders, who know well the complexity of battlefield situations. Machine learning systems that operate flawlessly in training may perform poorly when deployed. Cyberattacks against autonomous weapons could result in friendly-fire casualties; disconnecting the weapon from all communication may prevent that (assuming it has not already been compromised), but then the weapon cannot be recalled if it is malfunctioning.

The overriding practical issue with autonomous weapons is that they they are scalable weapons of mass destruction, in the sense that the scale of an attack that can be launched is proportional to the amount of hardware one can afford to deploy. A quadcopter two inches in diameter can carry a lethal explosive charge, and one million can fit in a regular shipping container. Precisely because they are autonomous, these weapons would not need one million human supervisors to do their work.

As weapons of mass destruction, scalable autonomous weapons have advantages for the attacker compared to nuclear weapons and carpet bombing: they leave property intact and can be applied selectively to eliminate only those who might threaten an occupying force. They could certainly be used to wipe out an entire ethnic group or all the adherents of a particular religion. In many situations, they would also be untraceable. These characteristics make them particularly attractive to non-state actors.

These considerations—particularly those characteristics that advantage the attacker—suggest that autonomous weapons will reduce global and national security for all parties. The rational response for governments seems to be to engage in arms control discussions rather than an arms race.

The process of designing a treaty is not without its difficulties, however. AI is a dual use technology: AI technologies that have peaceful applications such as flight control, visual tracking, mapping, navigation, and multiagent planning, can easily be applied to military purposes. It is easy to turn an autonomous quadcopter into a weapon simply by attaching an explosive and commanding it to seek out a target. Dealing with this will require careful implementation of compliance regimes with industry cooperation, as has already been demonstrated with some success by the Chemical Weapons Convention.

Dual use

27.3.2 Surveillance, security, and privacy In 1976, Joseph Weizenbaum warned that automated speech recognition technology could lead to widespread wiretapping, and hence to a loss of civil liberties. Today, that threat has been realized, with most electronic communication going through central servers that can be monitored, and cities packed with microphones and cameras that can identify and track individuals based on their voice, face, and gait. Surveillance that used to require expensive and scarce human resources can now be done at a mass scale by machines.

As of 2018, there were as many as 350 million surveillance cameras in China and 70 million in the United States. China and other countries have begun exporting surveillance

technology to low-tech countries, some with reputations for mistreating their citizens and disproportionately targeting marginalized communities. AI engineers should be clear on what uses of surveillance are compatible with human rights, and decline to work on applications that are incompatible.

Surveillance camera

As more of our institutions operate online, we become more vulnerable to cybercrime (phishing, credit card fraud, botnets, ransomware) and cyberterrorism (including potentially deadly attacks such as shutting down hospitals and power plants or commandeering selfdriving cars). Machine learning can be a powerful tool for both sides in the cybersecurity battle. Attackers can use automation to probe for insecurities and they can apply reinforcement learning for phishing attempts and automated blackmail. Defenders can use unsupervised learning to detect anomalous incoming traffic patterns (Chandola et al., 2009; Malhotra et al.,2015) and various machine learning techniques to detect fraud (Fawcett and Provost, 1997; Bolton and Hand, 2002). As attacks get more sophisticated, there is a greater responsibility for all engineers, not just the security experts, to design secure systems from the start. One forecast (Kanal, 2017) puts the market for machine learning in cybersecurity at about $100 billion by 2021.

Cybersecurity

As we interact with computers for increasing amounts of our daily lives, more data on us is being collected by governments and corporations. Data collectors have a moral and legal responsibility to be good stewards of the data they hold. In the U.S., the Health Insurance Portability and Accountability Act (HIPAA) and the Family Educational Rights and Privacy Act (FERPA) protect the privacy of medical and student records. The European Union’s General Data Protection Regulation (GDPR) mandates that companies design their systems

with protection of data in mind and requires that they obtain user consent for any collection or processing of data.

Balanced against the individual’s right to privacy is the value that society gains from sharing data. We want to be able to stop terrorists without oppressing peaceful dissent, and we want to cure diseases without compromising any individual’s right to keep their health history private. One key practice is de-identification: eliminating personally identifying information (such as name and social security number) so that medical researchers can use the data to advance the common good. The problem is that the shared de-identified data may be subject to re-identification. For example, if the data strips out the name, social security number, and street address, but includes date of birth, gender, and zip code, then, as shown by Latanya Sweeney (2000), 87% of the U.S. population can be uniquely re-identified. Sweeney emphasized this point by re-identifying the health record for the governor of her state when he was admitted to the hospital. In the Netflix Prize competition, de-identified records of individual movie ratings were released, and competitors were asked to come up with a machine learning algorithm that could accurately predict which movies an individual would like. But researchers were able to re-identify individual users by matching the date of a rating in the Netflix database with the date of a similar ranking in the Internet Movie Database (IMDB), where users sometimes use their actual names (Narayanan and Shmatikov, 2006).

De-identification

Netflix Prize

This risk can be mitigated somewhat by generalizing fields: for example, replacing the exact birth date with just the year of birth, or a broader range like “20-30 years old.” Deleting a field altogether can be seen as a form of generalizing to “any.” But generalization alone does not guarantee that records are safe from re-identification; it may be that there is only one person in zip code 94720 who is 90–100 years old. A useful property is k-anonymity: a

database is k-anonymized if every record in the database is indistinguishable from at least

k − 1 other records. If there are records that are more unique than this, they would have to be further generalized.

K-anonymity

An alternative to sharing de-identified records is to keep all records private, but allow aggregate querying. An API for queries against the database is provided, and valid queries receive a response that summarizes the data with a count or average. But no response is given if it would violate certain guarantees of privacy. For example, we could allow an epidemiologist to ask, for each zip code, the percentage of people with cancer. For zip codes with at least n people a percentage would be given (with a small amount of random noise), but no response would be given for zip codes with fewer than n people..

Aggregate querying

Care must be taken to protect against de-identification using multiple queries. For example, if the query “average salary and number of employees of XYZ company age 30-40” gives the response [$81,234, 12] and the query “average salary and number of employees of XYZ company age 30-41” gives the response [$81,199, 13], and if we use LinkedIn to find the one 41-year-old at XYZ company, then we have successfully identified them, and can compute their exact salary, even though all the responses involved 12 or more people. The system must be carefully designed to protect against this, with a combination of limits on the queries that can be asked (perhaps only a predefined set of non-overlapping age ranges can be queried) and the precision of the results (perhaps both queries give the answer “about $81,000”).

A stronger guarantee is differential privacy, which assures that an attacker cannot use queries to re-identify any individual in the database, even if the attacker can make multiple

queries and has access to separate linking databases. The query response employs a randomized algorithm that adds a small amount of noise to the result. Given a database

r

D,

Q, and a possible response y to the query, we say that the database D has ε–differential privacy if the log probability of the response y varies by less than ε when we add the record r: any record in the database , any query

|log 

P (Q (D) = y) − log P (Q (D + r) = y) | ≤ ε

Differential privacy

In other words, whether any one person decides to participate in the data base or not makes no appreciable difference to the answers anyone can get, and therefore there is no privacy disincentive to participate. Many databases are designed to guarantee differential privacy.

So far we have considered the issue of sharing de-identified data from a central database. An approach called federated learning (Konečný et al., 2016) has no central database; instead, users maintain their own local databases that keep their data private. However, they can share parameters of a machine learning model that is enhanced with their data, without the risk of revealing any of the private data. Imagine a speech understanding application that users can run locally on their phone. The application contains a baseline neural network, which is then improved by local training on the words that are heard on the user’s phone. Periodically, the owners of the application poll a subset of the users and ask them for the parameter values of their improved local network, but not for any of their raw data. The parameter values are combined together to form a new improved model which is then made available to all users, so that they all get the benefit of the training that is done by other users.

Federated learning

For this scheme to preserve privacy, we have to be able to guarantee that the model parameters shared by each user cannot be reverse-engineered. If we sent the raw parameters, there is a chance that an adversary inspecting them could deduce whether, say, a certain word had been heard by the user’s phone. One way to eliminate this risk is with secure aggregation (Bonawitz et al., 2017). The idea is that the central server doesn’t need to know the exact parameter value from each distributed user; it only needs to know the average value for each parameter, over all polled users. So each user can disguise their parameter values by adding a unique mask to each value; as long as the sum of the masks is zero, the central server will be able to compute the correct average. Details of the protocol make sure that it is efficient in terms of communication (less than half the bits transmitted correspond to masking), is robust to individual users failing to respond, and is secure in the face of adversarial users, eavesdroppers, or even an adversarial central server.

Secure aggregation

27.3.3 Fairness and bias Machine learning is augmenting and sometimes replacing human decision-making in important situations: whose loan gets approved, to what neighborhoods police officers are deployed, who gets pretrial release or parole. But machine learning models can perpetuate societal bias. Consider the example of an algorithm to predict whether criminal defendants are likely to re-offend, and thus whether they should be released before trial. It could well be that such a system picks up the racial or gender prejudices of human judges from the examples in the training set. Designers of machine learning systems have a moral responsibility to ensure that their systems are in fact fair. In regulated domains such as credit, education, employment, and housing, they have a legal responsibility as well. But what is fairness? There are multiple criteria; here are six of the most commonly-used concepts:

Societal bias

INDIVIDUAL FAIRNESS: A requirement that individuals are treated similarly to other similar individuals, regardless of what class they are in. GROUP FAIRNESS: A requirement that two classes are treated similarly, as measured by some summary statistic. FAIRNESS THROUGH UNAWARENESS: If we delete the race and gender attributes from the data set, then it might seem that the system cannot discriminate on those attributes. Unfortunately, we know that machine learning models can predict latent variables (such as race and gender) given other correlated variables (such as zip code and occupation). Furthermore, deleting those attributes makes it impossible to verify equal opportunity or equal outcomes. Still, some countries (e.g., Germany) have chosen this approach for their demographic statistics (whether or not machine learning models are involved). EQUAL OUTCOME: The idea that each demographic class gets the same results; they have demographic parity. For example, suppose we have to decide whether we should approve loan applications; the goal is to approve those applicants who will pay back the loan and not those who will default on the loan. Demographic parity says that both males and females should have the same percentage of loans approved. Note that this is a group fairness criterion that does nothing to ensure individual fairness; a wellqualified applicant might be denied and a poorly-qualified applicant might be approved, as long as the overall percentages are equal. Also, this approach favors redress of past biases over accuracy of prediction. If a man and a woman are equal in every way, except the woman receives a lower salary for the same job, should she be approved because she would be equal if not for historical biases, or should she be denied because the lower salary does in fact make her more likely to default?

Demographic parity

EQUAL OPPORTUNITY: The idea that the people who truly have the ability to pay back the loan should have an equal chance of being correctly classified as such, regardless of their sex. This approach is also called “balance.” It can lead to unequal outcomes and ignores the effect of bias in the societal processes that produced the training data.

EQUAL IMPACT: People with similar likelihood to pay back the loan should have the same expected utility, regardless of the class they belong to. This goes beyond equal opportunity in that it considers both the benefits of a true prediction and the costs of a false prediction.

Let us examine how these issues play out in a particular context. COMPAS is a commercial system for recidivism (re-offense) scoring. It assigns to a defendant in a criminal case a risk score, which is then used by a judge to help make decisions: Is it safe to release the defendant before trial, or should they be held in jail? If convicted, how long should the sentence be? Should parole be granted? Given the significance of these decisions, the system has been the subject of intense scrutiny (Dressel and Farid, 2018).

COMPAS is designed to be well calibrated: all the individuals who are given the same score by the algorithm should have approximately the same probability of re-offending, regardless of race. For example, among all people that the model assigns a risk score of 7 out of 10, 60% of whites and 61% of blacks re-offend. The designers thus claim that it meets the desired fairness goal.

Well calibrated

On the other hand, COMPAS does not achieve equal opportunity: the proportion of those who did not re-offend but were falsely rated as high-risk was 45% for blacks and 23% for whites. In the case State v. Loomis, where a judge relied on COMPAS to determine the sentence of the defendant, Loomis argued that the secretive inner workings of the algorithm violated his due process rights. Though the Wisconsin Supreme Court found that the sentence given would be no different without COMPAS in this case, it did issue warnings about the algorithm’s accuracy and risks to minority defendants. Other researchers have questioned whether it is appropriate to use algorithms in applications such as sentencing.

We could hope for an algorithm that is both well calibrated and equal opportunity, but, as Kleinberg et al. (2016) show, that is impossible. If the base classes are different, then any algorithm that is well calibrated will necessarily not provide equal opportunity, and vice

versa. How can we weigh the two criteria? Equal impact is one possibility. In the case of COMPAS, this means weighing the negative utility of defendants being falsely classified as high risk and losing their freedom, versus the cost to society of an additional crime being committed, and finding the point that optimizes the tradeoff. This is complicated because there are multiple costs to consider. There are individual costs—a defendant who is wrongfully held in jail suffers a loss, as does the victim of a defendant who was wrongfully released and re-offends. But beyond that there are group costs—everyone has a certain fear that they will be wrongfully jailed, or will be the victim of a crime, and all taxpayers contribute to the costs of jails and courts. If we give value to those fears and costs in proportion to the size of a group, then utility for the majority may come at the expense of a minority.

Another problem with the whole idea of recidivism scoring, regardless of the model used, is that we don’t have unbiased ground truth data. The data does not tell us who has committed a crime—all we know is who has been convicted of a crime. If the arresting officers, judge, or jury is biased, then the data will be biased. If more officers patrol some locations, then the data will be biased against people in those locations. Only defendants who are released are candidates to recommit, so if the judges making the release decisions are biased, the data may be biased. If you assume that behind the biased data set there is an underlying, unknown, unbiased data set which has been corrupted by an agent with biases, then there are techniques to recover an approximation to the unbiased data. Jiang and Nachum (2019) describe various scenarios and the techniques involved.

One more risk is that machine learning can be used to justify bias. If decisions are made by a biased human after consulting with a machine learning system, the human can say “here is how my interpretation of the model supports my decision, so you shouldn’t question my decision.” But other interpretations could lead to an opposite decision.

Sometimes fairness means that we should reconsider the objective function, not the data or the algorithm. For example, in making job hiring decisions, if the objective is to hire candidates with the best qualifications in hand, we risk unfairly rewarding those who have had advantageous educational opportunities throughout their lives, thereby enforcing class boundaries. But if the objective is to hire candidates with the best ability to learn on the job, we have a better chance to cut across class boundaries and choose from a broader pool. Many companies have programs designed for such applicants, and find that after a year of

training, the employees hired this way do as well as the traditional candidates. Similarly, just 18% of computer science graduates in the U.S. are women, but some schools, such as Harvey Mudd University, have achieved 50% parity with an approach that is focused on encouraging and retaining those who start the computer science program, especially those who start with less programming experience.

A final complication is deciding which classes deserve protection. In the U.S., the Fair Housing Act recognized seven protected classes: race, color, religion, national origin, sex, disability, and familial status. Other local, state, and federal laws recognize other classes, including sexual orientation, and pregnancy, marital, and veteran status. Is it fair that these classes count for some laws and not others? International human rights law, which encompasses a broad set of protected classes, is a potential framework to harmonize protections across various groups.

Even in the absence of societal bias, sample size disparity can lead to biased results. In most data sets there will be fewer training examples of minority class individuals than of majority class individuals. Machine learning algorithms give better accuracy with more training data, so that means that members of minority classes will experience lower accuracy. For example, Buolamwini and Gebru (2018) examined a computer vision gender identification service, and found that it had near-perfect accuracy for light-skinned males, and a 33% error rate for dark-skinned females. A constrained model may not be able to simultaneously fit both the majority and minority class—a linear regression model might minimize average error by fitting just the majority class, and in an SVM model, the support vectors might all correspond to majority class members.

Sample size disparity

Bias can also come into play in the software development process (whether or not the software involves machine learning). Engineers who are debugging a system are more likely to notice and fix those problems that are applicable to themselves. For example, it is difficult to notice that a user interface design won’t work for colorblind people unless you are in fact colorblind, or that an Urdu language translation is faulty if you don’t speak Urdu.

How can we defend against these biases? First, understand the limits of the data you are using. It has been suggested that data sets (Gebru et al., 2018; Hind et al., 2018) and models (Mitchell et al., 2019) should come with annotations: declarations of provenance, security, conformity, and fitness for use. This is similar to the data sheets that accompany electronic components such as resistors; they allow designers to decide what components to use. In addition to the data sheets, it is important to train engineers to be aware of issues of fairness and bias, both in school and with on-the-job training. Having a diversity of engineers from different backgrounds makes it easier for them to notice problems in the data or models. A study by the AI Now Institute (West et al., 2019) found that only 18% of authors at leading AI conferences and 20% of AI professors are women. Black AI workers are at less than 4%. Rates at industry research labs are similar. Diversity could be increased by programs earlier in the pipeline—in college or high school—and by greater awareness at the professional level. Joy Buolamwini founded the Algorithmic Justice League to raise awareness of this issue and develop practices for accountability.

Data sheet

A second idea is to de-bias the data (Zemel et al., 2013). We could over-sample from minority classes to defend against sample size disparity. Techniques such as SMOTE, the synthetic minority over-sampling technique (Chawla et al., 2002) or ADASYN, the adaptive synthetic sampling approach for imbalanced learning (He et al., 2008), provide principled ways of oversampling. We could examine the provenance of data and, for example, eliminate examples from judges who have exhibited bias in their past court cases. Some analysts object to the idea of discarding data, and instead would recommend building a hierarchical model of the data that includes sources of bias, so they can be modeled and compensated for. Google and NeurIPS have attempted to raise awareness of this issue by sponsoring the Inclusive Images Competition, in which competitors train a network on a data set of labeled images collected in North America and Europe, and then test it on images taken from all around the world. The issue is that given this data set, it is easy to apply the label “bride” to a woman in a standard Western wedding dress, but harder to recognize traditional African and Indian matrimonial dress.

A third idea is to invent new machine learning models and algorithms that are more resistant to bias; and the final idea is to let a system make initial recommendations that may be biased, but then train a second system to de-bias the recommendations of the first one. Bellamy et al. (2018) introduced the IBM AI FAIRNESS 360 system, which provides a framework for all of these ideas. We expect there will be increased use of tools like this in the future.

How do you make sure that the systems you build will be fair? A set of best practices has been emerging (although they are not always followed):

Make sure that the software engineers talk with social scientists and domain experts to understand the issues and perspectives, and consider fairness from the start. Create an environment that fosters the development of a diverse pool of software engineers that are representative of society. Define what groups your system will support: different language speakers, different age groups, different abilities with sight and hearing, etc. Optimize for an objective function that incorporates fairness. Examine your data for prejudice and for correlations between protected attributes and other attributes. Understand how any human annotation of data is done, design goals for annotation accuracy, and verify that the goals are met. Don’t just track overall metrics for your system; make sure you track metrics for subgroups that might be victims of bias. Include system tests that reflect the experience of minority group users. Have a feedback loop so that when fairness problems come up, they are dealt with.

27.3.4 Trust and transparency It is one challenge to make an AI system accurate, fair, safe, and secure; a different challenge to convince everyone else that you have done so. People need to be able to trust the systems they use. A PwC survey in 2017 found that 76% of businesses were slowing the adoption of AI because of trustworthiness concerns. In Section 19.9.4  we covered some of the engineering approaches to trust; here we discuss the policy issues.

Trust

To earn trust, any engineered systems must go through a verification and validation (V&V) process. Verification means that the product satisfies the specifications. Validation means ensuring that the specifications actually meet the needs of the user and other affected parties. We have an elaborate V&V methodology for engineering in general, and for traditional software development done by human coders; much of that is applicable to AI systems. But machine learning systems are different and demand a different V&V process, which has not yet been fully developed. We need to verify the data that these systems learn from; we need to verify the accuracy and fairness of the results, even in the face of uncertainty that makes an exact result unknowable; and we need to verify that adversaries cannot unduly influence the model, nor steal information by querying the resulting model.

Verification and validation

One instrument of trust is certification; for example, Underwriters Laboratories (UL) was founded in 1894 at a time when consumers were apprehensive about the risks of electric power. UL certification of appliances gave consumers increased trust, and in fact UL is now considering entering the business of product testing and certification for AI.

Certification

Other industries have long had safety standards. For example, ISO 26262 is an international standard for the safety of automobiles, describing how to develop, produce, operate, and service vehicles in a safe way. The AI industry is not yet at this level of clarity, although there are some frameworks in progress, such as IEEE P7001, a standard defining ethical design for artificial intelligence and autonomous systems (Bryson and Winfield, 2017).

There is ongoing debate about what kind of certification is necessary, and to what extent it should be done by the government, by professional organizations like IEEE, by independent certifiers such as UL, or through self-regulation by the product companies.

Another aspect of trust is transparency: consumers want to know what is going on inside a system, and that the system is not working against them, whether due to intentional malice, an unintentional bug, or pervasive societal bias that is recapitulated by the system. In some cases this transparency is delivered directly to the consumer. In other cases their are intellectual property issues that keep some aspects of the system hidden to consumers, but open to regulators and certification agencies.

Transparency

When an AI system turns you down for a loan, you deserve an explanation. In Europe, the GDPR enforces this for you. An AI system that can explain itself is called explainable AI (XAI). A good explanation has several properties: it should be understandable and convincing to the user, it should accurately reflect the reasoning of the system, it should be complete, and it should be specific in that different users with different conditions or different outcomes should get different explanations.

Explainable AI (XAI)

It is quite easy to give a decision algorithm access to its own deliberative processes, simply by recording them and making them available as data structures. This means that machines may eventually be able to give better explanations of their decisions than humans can. Moreover, we can take steps to certify that the machine’s explanations are not deceptions (intentional or self-deception), something that is more difficult with a human.

An explanation is a helpful but not sufficient ingredient to trust. One issue is that explanations are not decisions: they are stories about decisions. As discussed in Section 19.9.4 , we say that a system is interpretable if we can inspect the source code of the model and see what it is doing, and we say it is explainable if we can make up a story about what it is doing—even if the system itself is an uninterpretable black box. To explain an uninterpretable black box, we need to build, debug, and test a separate explanation system, and make sure it is in sync with the original system. And because humans love a good story, we are all too willing to be swayed by an explanation that sounds good. Take any political controversy of the day, and you can always find two so-called experts with diametrically opposed explanations, both of which are internally consistent.

A final issue is that an explanation about one case does not give you a summary over other cases. If the bank explains, “Sorry, you didn’t get the loan because you have a history of previous financial problems,” you don’t know if that explanation is accurate or if the bank is secretly biased against you for some reason. In this case, you require not just an explanation, but also an audit of past decisions, with aggregated statistics across various demographic groups, to see if their approval rates are balanced.

Part of transparency is knowing whether you are interacting with an AI system or a human. Toby Walsh (2015) proposed that “an autonomous system should be designed so that it is unlikely to be mistaken for anything besides an autonomous system, and should identify itself at the start of any interaction.” He called this the “red flag” law, in honor of the UK’s 1865 Locomotive Act, which required any motorized vehicle to have a person with a red flag walk in front of it, to signal the oncoming danger.

In 2019, California enacted a law stating that “It shall be unlawful for any person to use a bot to communicate or interact with another person in California online, with the intent to mislead the other person about its artificial identity.”

27.3.5 The future of work From the first agricultural revolution (10,000 BCE) to the industrial revolution (late 18th century) to the green revolution in food production (1950s), new technologies have changed the way humanity works and lives. A primary concern arising from the advance of AI is that human labor will become obsolete. Aristotle, in Book I of his Politics, presents the main point quite clearly:

For if every instrument could accomplish its own work, obeying or anticipating the will of others ... if, in like manner, the shuttle would weave and the plectrum touch the lyre without a hand to guide them, chief workmen would not want servants, nor masters slaves.

Everyone agrees with Aristotle’s observation that there is an immediate reduction in employment when an employer finds a mechanical method to perform work previously done by a person. The issue is whether the so-called compensation effects that ensue—and that tend to increase employment—will eventually make up for this reduction. The primary compensation effect is the increase in overall wealth from greater productivity, which leads in turn to greater demand for goods and tends to increase employment. For example, PwC (Rao and Verweij, 2017) predicts that AI contribute $15 trillion annually to global GDP by 2030. The healthcare and automotive/transportation industries stand to gain the most in the short term. However, the advantages of automation have not yet taken over in our economy: the current rate of growth in labor productivity is actually below historical standards. Brynjolfsson et al. (2018) attempt to explain this paradox by suggesting that the lag between the development of basic technology and its implementation in the economy is longer than commonly supposed.

Technological innovations have historically put some people out of work. Weavers were replaced by automated looms in the 1810s, leading to the Luddite protests. The Luddites were not against technology per se; they just wanted the machines to be used by skilled workers paid a good wage to make high-quality goods, rather than by unskilled workers to make poor-quality goods at low wages. The global destruction of jobs in the 1930s led John Maynard Keynes to coin the term technological unemployment. In both cases, and several others, employment levels eventally recovered.

Technological unemployment

The mainstream economic view for most of the 20th century was that technological employment was at most a short-term phenomenon. Increased productivity would always lead to increased wealth and increased demand, and thus net job growth. A commonly cited example is that of bank tellers: although ATMs replaced humans in the job of counting out

cash for withdrawals, that made it cheaper to operate a bank branch, so the number of branches increased, leading to more bank employees overall. The nature of the work also changed, becoming less routine and requiring more advanced business skills. The net effect of automation seems to be in eliminating tasks rather than jobs.

The majority of commenters predict that the same will hold true with AI technology, at least in the short run. Gartner, McKinsey, Forbes, the World Economic Forum, and the Pew Research Center each released reports in 2018 predicting a net increase in jobs due to AIdriven automation. But some analysts think that this time around, things will be different. In 2019, IBM predicted that 120 million workers would need retraining due to automation by 2022, and Oxford Economics predicted that 20 million manufacturing jobs could be lost to automation by 2030.

Frey and Osborne (2017) survey 702 different occupations, and estimate that 47% of them are at risk of being automated, meaning that at least some of the tasks in the occupation can be performed by machine. For example, almost 3% of the workforce in the U.S. are vehicle drivers, and in some districts, as much as 15% of the male workforce are drivers. As we saw in Chapter 26 , the task of driving is likely to be eliminated by driverless cars/trucks/buses/taxis.

It is important to distinguish between occupations and the tasks within those occupations. McKinsey estimates that only 5% of occupations are fully automatable, but that 60% of occupations can have about 30% of their tasks automated. For example, future truck drivers will spend less time holding the steering wheel and more time making sure that the goods are picked up and delivered properly; serving as customer service representatives and salespeople at either end of the journey; and perhaps managing convoys of, say, three robotic trucks. Replacing three drivers with one convoy manager implies a net loss in employment, but if transportation costs decrease, there will be more demand, which wins some of the jobs back—but perhaps not all of them. As another example, despite many advances in applying machine learning to the problem of medical imaging, radiologists have so far been augmented, not replaced, by these tools. Ultimately, there is a choice of how to make use of automation: do we want to focus on cutting cost, and thus see job loss as a positive; or do we want to focus on improving quality, making life better for the worker and the customer?

It is difficult to predict exact timelines for automation, but currently, and for the next few years, the emphasis is on automation of structured analytical tasks, such as reading x-ray images, customer relationship management (e.g., bots that automatically sort customer complaints and respond with suggested remedies), and business process automation that combines text documents and structured data to make business decisions and improve workflow. Over time, we will see more automation with physical robots, first in controlled warehouse environments, then in more uncertain environments, building to a significant portion of the marketplace by around 2030.

Business process automation

As populations in developed countries grow older, the ratio between workers and retirees changes. In 2015 there were less than 30 retirees per 100 workers; by 2050 there may be over 60 per 100 workers. Care for the elderly will be an increasingly important role, one that can partially be filled by AI. Moreover, if we want to maintain the current standard of living, it will also be necessary to make the remaining workers more productive; automation seems like the best opportunity to do that.

Even if automation has a multi-trillion-dollar net positive impact, there may still be problems due to the pace of change. Consider how change came to the farming industry: in 1900, over 40% of the U.S. workforce was in agriculture, but by 2000 that had fallen to 2%.3 That is a huge disruption in the way we work, but it happened over a period of 100 years, and thus across generations, not in the lifetime of one worker.

3 In 2010, although only 2% of the U.S. workforce were actual farmers, over 25% of the population (80 million people) played the FARMVILLE game at least once.

Pace of change

Workers whose jobs are automated away this decade may have to retrain for a new profession within a few years—and then perhaps see their new profession automated and face yet another retraining period. Some may be happy to leave their old profession—we see that as the economy improves, trucking companies need to offer new incentives to hire enough drivers—but workers will be apprehensive about their new roles. To handle this, we as a society need to provide lifelong education, perhaps relying in part on online education driven by artificial intelligence (Martin, 2012). Bessen (2015) argues that workers will not see increases in income until they are trained to implement the new technologies, a process that takes time.

Technology tends to magnify income inequality. In an information economy marked by high-bandwidth global communication and zero-marginal-cost replication of intellectual property (what Frank and Cook (1996) call the “Winner-Take-All Society”), rewards tend to be concentrated. If farmer Ali is 10% better than farmer Bo, then Ali gets about 10% more income: Ali can charge slightly more for superior goods, but there is a limit on how much can be produced on the land, and how far it can be shipped. But if software app developer Cary is 10% better than Dana, it may be that Cary ends up with 99% of the global market. AI increases the pace of technological innovation and thus contributes to this overall trend, but AI also holds the promise of allowing us to take some time off and let our automated agents handle things for a while. Tim Ferriss (2007) recommends using automation and outsourcing to achieve a four-hour work week.

Income inequality

Before the industrial revolution, people worked as farmers or in other crafts, but didn’t report to a job at a place of work and put in hours for an employer. But today, most adults in developed countries do just that, and the job serves three purposes: it fuels the production of the goods that society needs to flourish, it provides the income that the worker needs to live, and it gives the worker a sense of purpose, accomplishment, and social integration. With increasing automation, it may be that these three purposes become disaggregated— society’s needs will be served in part by automation, and in the long run, individuals will get their sense of purpose from contributions other than work. Their income needs can be

served by social policies that include a combination of free or inexpensive access to social services and education, portable health care, retirement, and education accounts, progressive tax rates, earned income tax credits, negative income tax, or universal basic income.

27.3.6 Robot rights The question of robot consciousness, discussed in Section 27.2 , is critical to the question of what rights, if any, robots should have. If they have no consciousness, no qualia, then few would argue that they deserve rights.

But if robots can feel pain, if they can dread death, if they are considered “persons,” then the argument can be made (e.g., by Sparrow (2004)) that they have rights and deserve to have their rights recognized, just as slaves, women, and other historically oppressed groups have fought to have their rights recognized. The issue of robot personhood is often considered in fiction: from Pygmalion to Coppélia to Pinocchio to the movies AI and Centennial Man, we have the legend of a doll/robot coming to life and striving to be accepted as a human with human rights. In real life, Saudi Arabia made headlines by giving honorary citizenship to Sophia, a human-looking puppet capable of speaking preprogrammed lines.

If robots have rights, then they should not be enslaved, and there is a question of whether reprogramming them would be a kind of enslavement. Another ethical issue involves voting rights: a rich person could buy thousands of robots and program them to cast thousands of votes—should those votes count? If a robot clones itself, can they both vote? What is the boundary between ballot stuffing and exercising free will, and when does robotic voting violate the “one person, one vote” principle?

Ernie Davis argues for avoiding the dilemmas of robot consciousness by never building robots that could possibly be considered conscious. This argument was previously made by Joseph Weizenbaum in his book Computer Power and Human Reason (1976), and before that by Julien de La Mettrie in L’Homme Machine (1748). Robots are tools that we create, to do the tasks we direct them to do, and if we grant them personhood, we are just declining to take responsibility for the actions of our own property: “I’m not at fault for my self-driving car crash—the car did it itself.”

This issue takes a different turn if we develop human–robot hybrids. Of course we already have humans enhanced by technology such as contact lenses, pacemakers, and artificial hips. But adding computational protheses may blur the lines between human and machine.

27.3.7 AI Safety Almost any technology has the potential to cause harm in the wrong hands, but with AI and robotics, the hands might be operating on their own. Countless science fiction stories have warned about robots or cyborgs running amok. Early examples include Mary Shelley’s Frankenstein, or the Modern Prometheus (1818) and Karel Čapek’s play R.U.R. (1920), in which robots conquer the world. In movies, we have The Terminator (1984) and The Matrix (1999), which both feature robots trying to eliminate humans—the robopocalypse (Wilson, 2011). Perhaps robots are so often the villains because they represent the unknown, just like the witches and ghosts of tales from earlier eras. We can hope that a robot that is smart enough to figure out how to terminate the human race is also smart enough to figure out that that was not the intended utility function; but in building intelligent systems, we want to rely not just on hope, but on a design process with guarantees of safety.

Robopocalypse

It would be unethical to distribute an unsafe AI agent. We require our agents to avoid accidents, to be resistant to adversarial attacks and malicious abuse, and in general to cause benefits, not harms. That is especially true as AI agents are deployed in safety-critical applications, such as driving cars, controlling robots in dangerous factory or construction settings, and making life-or-death medical decisions.

There is a long history of safety engineering in traditional engineering fields. We know how to build bridges, airplanes, spacecraft, and power plants that are designed up front to behave safely even when components of the system fail. The first technique is failure modes and effect analysis (FMEA): analysts consider each component of the system, and imagine every possible way the component could go wrong (for example, what if this bolt were to snap?), drawing on past experience and on calculations based on the physical properties of the component. Then the analysts work forward to see what would result from the failure. If

the result is severe (a section of the bridge could fall down) then the analysts alter the design to mitigate the failure. (With this additional cross-member, the bridge can survive the failure of any 5 bolts; with this backup server, the online service can survive a tsunami taking out the primary server.) The technique of fault tree analysis (FTA) is used to make these determinations: analysts build an AND/OR tree of possible failures and assign probabilities to each root cause, allowing for calculations of overall failure probability. These techniques can and should be applied to all safety-critical engineered systems, including AI systems.

Safety engineering

Failure modes and effect analysis (FMEA)

Fault tree analysis (FTA)

The field of software engineering is aimed at producing reliable software, but the emphasis has historically been on correctness, not safety. Correctness means that the software faithfully implements the specification. But safety goes beyond that to insist that the specification has considered any feasible failure modes, and is designed to degrade gracefully even in the face of unforeseen failures. For example, the software for a self-driving car wouldn’t be considered safe unless it can handle unusual situations. For example, what if the power to the main computer dies? A safe system will have a backup computer with a separate power supply. What if a tire is punctured at high speed? A safe system will have tested for this, and will have software to correct for the resulting loss of control.

An agent designed as a utility maximizer, or as a goal achiever, can be unsafe if it has the wrong objective function. Suppose we give a robot the task of fetching a coffee from the kitchen. We might run into trouble with unintended side effects—the robot might rush to

accomplish the goal, knocking over lamps and tables along the way. In testing, we might notice this kind of behavior and modify the utility function to penalize such damage, but it is difficult for the designers and testers to anticipate all possible side effects ahead of time.

Unintended side effect

One way to deal with this is to design a robot to have low impact (Armstrong and Levinstein, 2017): instead of just maximizing utility, maximize the utility minus a weighted summary of all changes to the state of the world. In this way, all other things being equal, the robot prefers not to change those things whose effect on utility is unknown; so it avoids knocking over the lamp not because it knows specifically that knocking the lamp will cause it to fall over and break, but because it knows in general that disruption might be bad. This can be seen as a version of the physician’s creed “first, do no harm,” or as an analog to regularization in machine learning: we want a policy that achieves goals, but we prefer policies that take smooth, low-impact actions to get there. The trick is how to measure impact. It is not acceptable to knock over a fragile lamp, but perfectly fine if the air molecules in the room are disturbed a little, or if some bacteria in the room are inadvertently killed. It is certainly not acceptable to harm pets and humans in the room. We need to make sure that the robot knows the differences between these cases (and many subtle cases in between) through a combination of explicit programming, machine learning over time, and rigorous testing.

Low impact

Utility functions can go wrong due to externalities, the word used by economists for factors that are outside of what is measured and paid for. The world suffers when greenhouse gases are considered as externalities—companies and countries are not penalized for producing them, and as a result everyone suffers. Ecologist Garrett Hardin (1968) called the exploitation of shared resources the tragedy of the commons. We can mitigate the tragedy

by internalizing the externalities—making them part of the utility function, for example with a carbon tax—or by using the design principles that economist Elinor Ostrom identified as being used by local people throughout the world for centuries (work that won her the Nobel Prize in Economics in 2009):

Clearly define the shared resource and who has access. Adapt to local conditions. Allow all parties to participate in decisions. Monitor the resource with accountable monitors. Sanctions, proportional to the severity of the violation. Easy conflict resolution procedures. Hierarchical control for large shared resources.

Victoria Krakovna (2018) has cataloged examples of AI agents that have gamed the system, figuring out how to maximize utility without actually solving the problem that their designers intended them to solve. To the designers this looks like cheating, but to the agents, they are just doing their job. Some agents took advantage of bugs in the simulation (such as floating point overflow bugs) to propose solutions that would not work once the bug was fixed. Several agents in video games discovered ways to crash or pause the game when they were about to lose, thus avoiding a penalty. And in a specification where crashing the game was penalized, one agent learned to use up just enough of the game’s memory so that when it was the opponent’s turn, it would run out of memory and crash the game. Finally, a genetic algorithm operating in a simulated world was supposed to evolve fast-moving creatures but in fact produced creatures that were enormously tall and moved fast by falling over.

Designers of agents should be aware of these kinds of specification failures and take steps to avoid them. To help them do that, Krakovna was part of the team that released the AI Safety Gridworlds environments (Leike et al., 2017), which allows designers to test how well their agents perform.

The moral is that we need to be very careful in specifying what we want, because with utility maximizers we get what we actually asked for. The value alignment problem is the problem of making sure that what we ask for is what we really want; it is also known as the King Midas problem, as discussed on page 33. We run into trouble when a utility function

fails to capture background societal norms about acceptable behavior. For example, a human who is hired to clean floors, when faced with a messy person who repeatedly tracks in dirt, knows that it is acceptable to politely ask the person to be more careful, but it is not acceptable to kidnap or incapacitate said person.

Value alignment problem

A robotic cleaner needs to know these things too, either through explicit programming or by learning from observation. Trying to write down all the rules so that the robot always does the right thing is almost certainly hopeless. We have been trying to write loophole-free tax laws for several thousand years without success. Better to make the robot want to pay taxes, so to speak, than to try to make rules to force it to do so when it really wants to do something else. A sufficiently intelligent robot will find a way to do something else.

Robots can learn to conform better with human preferences by observing human behavior. This is clearly related to the notion of apprenticeship learning (Section 22.6 ). The robot may learn a policy that directly suggests what actions to take in what situations; this is often a straightforward supervised learning problem if the environment is observable. For example, a robot can watch a human playing chess: each state–action pair is an example for the learning process. Unfortunately, this form of imitation learning means that the robot will repeat human mistakes. Instead, the robot can apply inverse reinforcement learning to discover the utility function that the humans must be operating under. Watching even terrible chess players is probably enough for the robot to learn the objective of the game. Given just this information, the robot can then go on to exceed human performance—as, for example, ALPHAZERO did in chess—by computing optimal or near-optimal policies from the objective. This approach works not just in board games, but in real-world physical tasks such as helicopter aerobatics (Coates et al., 2009).

In more complex settings involving, for example, social interactions with humans, it is very unlikely that the robot will converge to exact and correct knowledge of each human’s individual preferences. (After all, many humans never quite learn what makes other humans tick, despite a lifetime of experience, and many of us are unsure of our own preferences

too.) It will be necessary, therefore, for machines to function appropriately when it is uncertain about human preferences. In Chapter 18 , we introduced assistance games, which capture exactly this situation. Solutions to assistance games include acting cautiously, so as not to disturb aspects of the world that the human might care about, and asking questions. For example, the robot could ask whether turning the oceans into sulphuric acid is an acceptable solution to global warming before it puts the plan into effect.

In dealing with humans, a robot solving an assistance game must accommodate human imperfections. If the robot asks permission, the human may give it, not foreseeing that the robot’s proposal is in fact catastrophic in the long term. Moreover, humans do not have complete introspective access to their true utility function, and they don’t always act in a way that is compatible with it. Humans sometimes lie or cheat, or do things they know are wrong. They sometimes take self-destructive actions like overeating or abusing drugs. AI systems need not learn to adopt these problematic tendencies, but they must understand that they exist when interpreting human behavior to get at the underlying human preferences.

Despite this toolbox of safeguards, there is a fear, expressed by prominent technologists such as Bill Gates and Elon Musk and scientists such as Stephen Hawking and Martin Rees, that AI could evolve out of control. They warn that we have no experience controlling powerful nonhuman entities with super-human capabilities. However, that’s not quite true; we have centuries of experience with nations and corporations; non-human entities that aggregate the power of thousands or millions of people. Our record of controlling these entities is not very encouraging: nations produce periodic convulsions called wars that kill tens of millions of human beings, and corporations are partly responsible for global warming and our inability to confront it.

AI systems may present much greater problems than nations and corporations because of their potential to self-improve at a rapid pace, as considered by I. J. Good (1965b)::

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an “intelligence explosion,” and the intelligence of man would be left far behind.

Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.

Ultraintelligent machine

Good’s “intelligence explosion” has also been called the technological singularity by mathematics professor and science fiction author Vernor Vinge, who wrote in 1993: “Within thirty years, we will have the technological means to create superhuman intelligence. Shortly after, the human era will be ended.” In 2017, inventor and futurist Ray Kurzweil predicted the singularity would appear by 2045, which means it got 2 years closer in 24 years. (At that rate, only 336 years to go!) Vinge and Kurzweil correctly note that technological progress on many measures is growing exponentially at present.

Technological singularity

It is, however, quite a leap to extrapolate all the way from the rapidly decreasing cost of computation to a singularity. So far, every technology has followed an S-shaped curve, where the exponential growth eventually tapers off. Sometimes new technologies step in when the old ones plateau, but sometimes it is not possible to keep the growth going, for technical, political, or sociological reasons. For example, the technology of flight advanced dramatically from the Wright brothers’ flight in 1903 to the moon landing in 1969, but has had no breakthroughs of comparable magnitude since then.

Another obstacle in the way of ultraintelligent machines taking over the world is the world. More specifically, some kinds of progress require not just thinking but acting in the physical world. (Kevin Kelly calls the overemphasis on pure intelligence thinkism.) An ultraintelligent machine tasked with creating a grand unified theory of physics might be capable of cleverly manipulating equations a billion times faster than Einstein, but to make any real progress, it would still need to raise millions of dollars to build a more powerful

supercollider and run physical experiments over the course of months or years. Only then could it start analyzing the data and theorizing. Depending on how the data turn out, the next step might require raising additional billions of dollars for an interstellar probe mission that would take centuries to complete. The “ultraintelligent thinking” part of this whole process might actually be the least important part. As another example, an ultraintelligent machine tasked with bringing peace to the Middle East might just end up getting 1000 times more frustrated than a human envoy. As yet, we don’t know how many of the big problems are like mathematics and how many are like the Middle East.

Thinkism

While some people fear the singularity, others relish it. The transhumanism social movement looks forward to a future in which humans are merged with—or replaced by— robotic and biotech inventions. Ray Kurzweil writes in The Singularity is Near (2005):

The Singularity will allow us to transcend these limitations of our biological bodies and brain. We will gain power over our fates. ... We will be able to live as long as we want ... We will fully understand human thinking and will vastly extend and expand its reach. By the end of this century, the nonbiological portion of our intelligence will be trillions of trillions of times more powerful than unaided human intelligence.

Transhumanism

Similarly, when asked whether robots will inherit the Earth, Marvin Minsky said “yes, but they will be our children.” These possibilities present a challenge for most moral theorists, who take the preservation of human life and the human species to be a good thing. Kurzweil also notes the potential dangers, writing “But the Singularity will also amplify the ability to act on our destructive inclinations, so its full story has not yet been written.” We humans would do well to make sure that any intelligent machine we design today that might evolve

into an ultraintelligent machine will do so in a way that ends up treating us well. As Eric Brynjolfsson puts it, “The future is not preordained by machines. It’s created by humans.”

Summary This chapter has addressed the following issues:

Philosophers use the term weak AI for the hypothesis that machines could possibly behave intelligently, and strong AI for the hypothesis that such machines would count as having actual minds (as opposed to simulated minds). Alan Turing rejected the question “Can machines think?” and replaced it with a behavioral test. He anticipated many objections to the possibility of thinking machines. Few AI researchers pay attention to the Turing test, preferring to concentrate on their systems’ performance on practical tasks, rather than the ability to imitate humans. Consciousness remains a mystery. AI is a powerful technology, and as such it poses potential dangers, through lethal autonomous weapons, security and privacy breaches, unintended side effects, unintentional errors, and malignant misuse. Those who work with AI technology have an ethical imperative to responsibly reduce those dangers. AI systems must be able to demonstrate they are fair, trustworthy, and transparent. There are multiple aspects of fairness, and it is impossible to maximize all of them at once. So a first step is to decide what counts as fair. Automation is already changing the way people work. As a society, we will have to deal with these changes.

Bibliographical and Historical Notes WEAK AI: When Alan Turing (1950) proposed the possibility of AI, he also posed many of the key philosophical questions, and provided possible replies. But various philosophers had raised similar issues long before AI was invented. Maurice Merleau-Ponty’s Phenomenology of Perception (1945) stressed the importance of the body and the subjective interpretation of reality afforded by our senses, and Martin Heidegger’s Being and Time (1927) asked what it means to actually be an agent. In the computer age, Alva Noe (2009) and Andy Clark (2015) propose that our brains form a rather minimal representation of the world, use the world itself on a just-in-time basis to maintain the illusion of a detailed internal model, and use props in the world (such as paper and pencil as well as computers) to increase the capabilities of the mind. Pfeifer et al. (2006) and Lakoff and Johnson (1999) present arguments for how the body helps shape cognition. Speaking of bodies, Levy (2008), Danaher and McArthur (2017), and Devlin (2018) address the issue of robot sex.

STRONG AI: René Descartes is known for his dualistic view of the human mind, but ironically his historical influence was toward mechanism and physicalism. He explicitly conceived of animals as automata, and he anticipated the Turing test, writing “it is not conceivable [that a machine] should produce different arrangements of words so as to give an appropriately meaningful answer to whatever is said in its presence, as even the dullest of men can do” (Descartes, 1637). Descartes’s spirited defense of the animals-as-automata viewpoint actually had the effect of making it easier to conceive of humans as automata as well, even though he himself did not take this step. The book L’Homme Machine (La Mettrie, 1748) did explicitly argue that humans are automata. As far back as Homer (circa 700 BCE), the Greek legends envisioned automata such as the bronze giant Talos and considered the issue of biotechne, or life through craft (Mayor, 2018).

The Turing test (Turing, 1950) has been debated (Shieber, 2004), anthologized (Epstein et al., 2008), and criticized (Shieber, 1994; Ford and Hayes, 1995). Bringsjord (2008) gives advice for a Turing test judge, and Christian (2011) for a human contestant. The annual Loebner Prize competition is the longest-running Turing test-like contest; Steve Worswick’s MITSUKU won four in a row from 2016 to 2019. The Chinese room has been debated endlessly (Searle, 1980; Chalmers, 1992; Preston and Bishop, 2002). Hernández-Orallo

(2016) gives an overview of approaches to measuring AI progress, and Chollet (2019) proposes a measure of intelligence based on skill-acquisition efficiency.

Consciousness remains a vexing problem for philosophers, neuroscientists, and anyone who has pondered their own existence. Block (2009), Churchland (2013) and Dehaene (2014) provide overviews of the major theories. Crick and Koch (2003) add their expertise in biology and neuroscience to the debate, and Gazzaniga (2018) shows what can be learned from studying brain disabilities in hospital cases. Koch (2019) gives a theory of consciousness—“intelligence is about doing while experience is about being”—that includes most animals, but not computers. Giulio Tononi and his colleagues propose integrated information theory (Oizumi et al., 2014). Damasio (1999) has a theory based on three levels: emotion, feeling, and feeling a feeling. Bryson (2012) shows the value of conscious attention for the process of learning action selection.

The philosophical literature on minds, brains, and related topics is large and jargon-filled. The Encyclopedia of Philosophy (Edwards, 1967) is an impressively authoritative and very useful navigation aid. The Cambridge Dictionary of Philosophy (Audi, 1999) is shorter and more accessible, and the online Stanford Encyclopedia of Philosophy offers many excellent articles and up-to-date references. The MIT Encyclopedia of Cognitive Science (Wilson and Keil, 1999) covers the philosophy, biology, and psychology of mind. There are multiple introductions to the philosophical “AI question” (Haugeland, 1985; Boden, 1990; Copeland, 1993; McCorduck, 2004; Minsky, 2007). The Behavioral and Brain Sciences, abbreviated BBS, is a major journal devoted to philosophical and scientific debates about AI and neuroscience.

Science fiction writer Isaac Asimov (1942, 1950) was one of the first to address the issue of robot ethics, with his laws of robotics:

0. A robot may not harm humanclistity, or through inaction, allow humanity to come to harm. 1. A robot may not injure a human being or, through inaction, allow a human being to come to harm. 2. A robot must obey orders given to it by human beings, except where such orders would conflict with the First Law.

3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

At first glance, these laws seem reasonable. But the trick is how to implement them. Should a robot allow a human to cross the street, or eat junk food, if the human might conceivably come to harm? In Asimov’s story Runaround (1942), humans need to debug a robot that is found wandering in a circle, acting “drunk.” They work out that the circle defines the locus of points that balance the second law (the robot was ordered to fetch some selenium at the center of the circle) with the third law (there is a danger there that threatens the robot’s existence).4 This suggests that the laws are not logical absolutes, but rather are weighed against each other, with a higher weight for the earlier laws. As this was 1942, before the emergence of digital computers, Asimov was probably thinking of an architecture based on control theory via analog computing.

4 Science fiction writers are in broad agreement that robots are very bad at resolving contradictions. In 2001, the HAL 9000 computer becomes homicidal due to a conflict in its orders, and in the Star Trek episode “I, Mudd,” Captain Kirk tells an enemy robot that “Everything Harry tells you is a lie,” and Harry says “I am lying.” At this, smoke comes out of the robot’s head and it shuts down.

Weld and Etzioni (1994) analyze Asimov’s laws and suggest some ways to modify the planning techniques of Chapter 11  to generate plans that do no harm. Asimov has considered many of the ethical issues around technology; in his 1958 story The Feeling of Power he tackles the issue of automation leading to a lapse of human skill—a technician rediscovers the lost art of multiplication—as well as the dilemma of what to do when the rediscovery is applied to warfare.

Norbert Wiener’s book God & Golem, Inc. (1964) correctly predicted that computers would achieve expert-level performance at games and other tasks, and that specifying what it is that we want would prove to be difficult. Wiener writes:

While it is always possible to ask for something other than we really want, this possibility is most serious when the process by which we are to obtain our wish is indirect, and the degree to which we have obtained our wish is not clear until the very end. Usually we realize our wishes, insofar as we do actually realize them, by a feedback process, in which we compare the degree of attainment of intermediate goals with our anticipation of them. In this process, the feedback goes through us, and we can turn back before it is too late. If the feedback is built into a machine that cannot be inspected until the final goal is attained, the possibilities for catastrophe are greatly increased. I should very

much hate to ride on the first trial of an automobile regulated by photoelectric feedback devices, unless there were somewhere a handle by which I could take over control if I found myself driving smack into a tree.

We summarized codes of ethics in the chapter, but the list of organizations that have issued sets of principles is growing rapidly, and now includes Apple, DeepMind, Facebook, Google, IBM, Microsoft, the Organisation for Economic Co-operation and Development (OECD), the United Nations Educational, Scientific and Cultural Organization (UNESCO), the U.S. Office of Science and Technology Policy the Beijing Academy of Artificial Intelligence (BAAI), the Institute of Electrical and Electronics Engineers (IEEE), the Association of Computing Machinery (ACM), the World Economic Forum, the Group of Twenty (G20), OpenAI, the Machine Intelligence Research Institute (MIRI), AI4People, the Centre for the Study of Existential Risk, the Center for Human-Compatible AI, the Center for Humane Technology, the Partnership on AI, the AI Now Institute, the Future of Life Institute, the Future of Humanity Institute, the European Union, and at least 42 national governments. We have the handbook on the Ethics of Computing (Berleur and Brunnstein, 2001) and introductions to the topic of AI ethics in book (Boddington, 2017) and survey (Etzioni and Etzioni, 2017a) form. The Journal of Artificial Intelligence and Law and AI and Society cover ethical issues. We’ll now look at some of the individual issues.

LETHAL AUTONOMOUS WEAPONS: P. W. Singer’s Wired for War (2009) raised ethical, legal, and technical issues around robots on the battlefield. Paul Scharre’s Army of None (2018), written by one of the authors of current US policy on autonomous weapons, offers a balanced and authoritative view. Etzioni and Etzioni (2017b) address the question of whether artificial intelligence should be regulated; they recommend a pause in the development of lethal autonomous weapons, and an international discussion on the subject of regulation.

PRIVACY: Latanya Sweeney (Sweeney, 2002b) presents the k-anonymity model and the idea of generalizing fields (Sweeney, 2002a). Achieving k-anonymity with minimal loss of data is an NP-hard problem, but Bayardo and Agrawal (2005) give an approximation algorithm. Cynthia Dwork (2008) describes differential privacy, and in subsequent work gives practical examples of clever ways to apply differential privacy to get better results than the naive approach (Dwork et al., 2014). Guo et al. (2019) describe a process for certified data removal: if you train a model on some data, and then there is a request to delete some

of the data, this extension of differential privacy lets you modify the model and prove that it does not make use of the deleted data. Ji et al. (2014) gives a review of the field of privacy. Etzioni (2004) argues for a balancing of privacy and security; individual rights and community. Fung et al. (2018), Bagdasaryan et al. (2018) discuss the various attacks on federated learning protocols. Narayanan et al. (2011) describe how they were able to deanonymize the obfuscated connection graph from the 2011 Social Network Challenge by crawling the site where the data was obtained (Flickr), and matching nodes with unusually high in-degree or out-degree between the provided data and the crawled data. This allowed them to gain additional information to win the challenge, and it also allowed them to uncover the true identity of nodes in the data. Tools for user privacy are becoming available; for example, TensorFlow provides modules for federated learning and privacy (McMahan and Andrew, 2018).

FAIRNESS: Cathy O’Neil’s book Weapons of Math Destruction (2017) describes how various black box machine learning models influence our lives, often in unfair ways. She calls on model builders to take responsibility for fairness, and for policy makers to impose appropriate regulation. Dwork et al. (2012) showed the flaws with the simplistic “fairness through unawareness” approach. Bellamy et al. (2018) present a toolkit for mitigating bias in machine learning systems. Tramèr et al. (2016) show how an adversary can “steal” a machine learning model by making queries against an API, Hardt et al. (2017) describe equal opportunity as a metric for fairness. Chouldechova and Roth (2018) give an overview of the frontiers of fairness, and Verma and Rubin (2018) give an exhaustive survey of fairness definitions.

Kleinberg et al. (2016) show that, in general, an algorithm cannot be both well-calibrated and equal opportunity. Berk et al. (2017) give some additional definitions of types of fairness, and again conclude that it is impossible to satisfy all aspects at once. Beutel et al. (2019) give advice for how to put fairness metrics into practice.

Dressel and Farid (2018) report on the COMPAS recidivism scoring model. Christin et al. (2015) and Eckhouse et al. (2019) discuss the use of predictive algorithms in the legal system. Corbett-Davies et al. (2017) show that that there is a tension between ensuring fairness and optimizing public safety, and Corbett-Davies and Goel (2018) discuss the differences between fairness frameworks. Chouldechova (2017) advocates for fair impact: all classes should have the same expected utility. Liu et al. (2018a) advocate for a long-term

measure of impact, pointing out that, for example, if we change the decision point for approving a loan in order to be more fair in the short run, this could have negative effect in the long run on people who end up defaulting on a loan and thus have their credit score reduced.

Since 2014 there has been an annual conference on Fairness, Accountability, and Transparency in Machine Learning. Mehrabi et al. (2019) give a comprehensive survey of bias and fairness in machine learning, cataloging 23 kinds of bias and 10 definitions of fairness.

TRUST: Explainable AI was an important topic going back to the early days of expert systems (Neches et al., 1985), and has been making a resurgence in recent years (Biran and Cotton, 2017; Miller et al., 2017; Kim, 2018). Barreno et al. (2010) give a taxonomy of the types of security attacks that can be made against a machine learning system, and Tygar (2011) surveys adversarial machine learning. Researchers at IBM have a proposal for gaining trust in AI systems through declarations of conformity (Hind et al., 2018). DARPA requires explainable decisions for its battlefield systems, and has issued a call for research in the area (Gunning, 2016).

AI SAFETY: The book Artificial Intelligence Safety and Security (Yampolskiy, 2018) collects essays on AI safety, both recent and classic, going back to Bill Joy’s Why the Future Doesn’t Need Us (Joy, 2000). The “King Midas problem” was anticipated by Marvin Minsky, who once suggested that an AI program designed to solve the Riemann Hypothesis might end up taking over all the resources of Earth to build more powerful supercomputers. Similarly, Omohundro (2008) foresees a chess program that hijacks resources, and Bostrom (2014) describes the runaway paper clip factory that takes over the world. Yudkowsky (2008) goes into more detail about how to design a Friendly AI. Amodei et al. (2016) present five practical safety problems for AI systems.

Omohundro (2008) describes the Basic AI Drives and concludes, “Social structures which cause individuals to bear the cost of their negative externalities would go a long way toward ensuring a stable and positive future.” Elinor Ostrom’s Governing the Commons (1990) describes practices for dealing with externalities by traditional cultures. Ostrom has also applied this approach to the idea of knowledge as a commons (Hess and Ostrom, 2007).

Ray Kurzweil (2005) proclaimed The Singularity is Near, and a decade later Murray Shanahan (2015) gave an update on the topic. Microsoft cofounder Paul Allen countered with The Singularity isn’t Near (2011). He didn’t dispute the possibility of ultraintelligent machines; he just thought it would take more than a century to get there. Rod Brooks is a frequent critic of singularitarianism; he points out that technologies often take longer than predicted to mature, that we are prone to magical thinking, and that exponentials don’t last forever (Brooks, 2017).

On the other hand, for every optimistic singularitarian there is a pessimist who fears new technology. The Web site pessimists.co shows that this has been true throughout history: for example, in the 1890s people were concerned that the elevator would inevitably cause nausea, that the telegraph would lead to loss of privacy and moral corruption, that the subway would release dangerous underground air and disturb the dead, and that the bicycle —especially the idea of a woman riding one—was the work of the devil.

Hans Moravec (2000) introduces some of the ideas of transhumanism, and Bostrom (2005) gives an updated history. Good’s ultraintelligent machine idea was foreseen a hundred years earlier in Samuel Butler’s Darwin Among the Machines (1863). Written four years after the publication of Charles Darwin’s On the Origins of Species and at a time when the most sophisticated machines were steam engines, Butler’s article envisioned “the ultimate development of mechanical consciousness” by natural selection. The theme was reiterated by George Dyson (1998) in a book of the same title, and was referenced by Alan Turing, who wrote in 1951 “At some stage therefore we should have to expect the machines to take control in the way that is mentioned in Samuel Butler’s Erewhon” (Turing, 1996).

ROBOT RIGHTS: A book edited by Yorick Wilks (2010) gives different perspectives on how we should deal with artificial companions, ranging from Joanna Bryson’s view that robots should serve us as tools, not as citizens, to Sherry Turkle’s observation that we already personify our computers and other tools, and are quite willing to blur the boundaries between machines and life. Wilks also contributed a recent update on his views (Wilks, 2019). The philosopher David Gunkel’s book Robot Rights (2018) considers four possibilities: can robots have rights or not, and should they or not? The American Society for the Prevention of Cruelty to Robots (ASPCR) proclaims that “The ASPCR is, and will continue to be, exactly as serious as robots are sentient.”

THE FUTURE OF WORK: In 1888, Edward Bellamy published the best-seller Looking Backward, which predicted that by the year 2000, technological advances would led to a utopia where equality is achieved and people work short hours and retire early. Soon after, E. M. Forster took the dystopian view in The Machine Stops (1909), in which a benevolent machine takes over the running of a society; things fall apart when the machine inevitably fails. Norbert Wiener’s prescient book The Human Use of Human Beings (1950) argues for the benefits of automation in freeing people from drudgery while offering more creative work, but also discusses several dangers that we recognize as problems today, particularly the problem of value alignment.

The book Disrupting Unemployment (Nordfors et al., 2018) discuss some of the ways that work is changing, opening opportunities for new careers. Erik Brynjolfsson and Andrew McAfee address these themes and more in their books Race Against the Machine and The Second Machine Age. Ford (2015) describes the challenges of increasing automation, and West (2018) provides recommendations to mitigate the problems, while MIT’s Thomas Malone (2004) shows that many of the same issues were apparent a decade earlier, but at that time were attributed to worldwide communication networks, not to automation.

Chapter 28 The Future of AI In which we try to see a short distance ahead. In Chapter 2 , we decided to view AI as the task of designing approximately rational agents. A variety of different agent designs were considered, ranging from reflex agents to knowledge-based decision-theoretic agents to deep learning agents using reinforcement learning. There is also variety in the component technologies from which these designs are assembled: logical, probabilistic, or neural reasoning; atomic, factored, or structured representations of states; various learning algorithms from various types of data; sensors and actuators to interact with the world. Finally, we have seen a variety of applications, in medicine, finance, transportation, communication, and other fields. There has been progress on all these fronts, both in our scientific understanding and in our technological capabilities.

Most experts are optimistic about continued progress; as we saw on page 28, the median estimate is for approximately human-level AI across a broad variety of tasks somewhere in the next 50 to 100 years. Within the next decade, AI is predicted to add trillions of dollars to the economy each year. But as we also saw, there are some critics who think general AI is centuries off, and there are numerous ethical concerns about the fairness, equity, and lethality of AI. In this chapter, we ask: where are we headed and what remains to be done? We do that by asking whether we have the right components, architectures, and goals to make AI a successful technology that delivers benefits to the world.

28.1 AI Components This section examines the components of AI systems and the extent to which each of them might accelerate or hinder future progress.

Sensors and actuators For much of the history of AI, direct access to the world has been glaringly absent. With a few notable exceptions, AI systems were built in such a way that humans had to supply the inputs and interpret the outputs. Meanwhile, robotic systems focused on low-level tasks in which high-level reasoning and planning were largely ignored and the need for perception was minimized. This was partly due to the great expense and engineering effort required to get real robots to work at all, and partly because of the lack of sufficient processing power and sufficiently effective algorithms to handle high-bandwidth visual input.

The situation has changed rapidly in recent years with the availability of ready-made programmable robots. These, in turn, have benefited from compact reliable motor drives and improved sensors. The cost of lidar for a self-driving car has fallen from $75,000 to $1,000, and a single-chip version may reach $10 per unit (Poulton and Watts, 2016). Radar sensors, once capable of only coarse-grained detection, are now sensitive enough to count the number of sheets in a stack of paper (Yeo et al., 2018).

The demand for better image processing in cellphone cameras has given us inexpensive high-resolution cameras for use in robotics. MEMS (micro-electromechanical systems) technology has supplied miniaturized accelerometers, gyroscopes, and actuators small enough to fit in artificial flying insects (Floreano et al., 2009; Fuller et al., 2014). It may be possible to combine millions of MEMS devices to produce powerful macroscopic actuators. 3-D printing (Muth et al., 2014) and bioprinting (Kolesky et al., 2014) have made it easier to experiment with prototypes.

Thus, we see that AI systems are at the cusp of moving from primarily software-only systems to useful embedded robotic systems. The state of robotics today is roughly comparable to the state of personal computers in the early 1980s: at that time personal computers were becoming available, but it would take another decade before they became

commonplace. It is likely that flexible, intelligent robots will first make strides in industry (where environments are more controlled, tasks are more repetitive, and the value of an investment is easier to measure) before the home market (where there is more variability in environment and tasks).

Representing the state of the world Keeping track of the world requires perception as well as updating of internal representations. Chapter 4  showed how to keep track of atomic state representations; Chapter 7  described how to do it for factored (propositional) state representations; Chapter 10  extended this to first-order logic; and Chapter 14  described probabilistic reasoning over time in uncertain environments. Chapter 21  introduced recurrent neural networks, which are also capable of maintaining a state representation over time.

Current filtering and perception algorithms can be combined to do a reasonable job of recognizing objects (“that’s a cat”) and reporting low-level predicates (“the cup is on the table”). Recognizing higher-level actions, such as “Dr. Russell is having a cup of tea with Dr. Norvig while discussing plans for next week,” is more difficult. Currently it can sometimes be done (see Figure 25.17  on page 908) given enough training examples, but future progress will require techniques that generalize to novel situations without requiring exhaustive examples (Poppe, 2010; Kang and Wildes, 2016). Another problem is that although the approximate filtering algorithms from Chapter 14  can handle quite large environments, they are still dealing with a factored representation— they have random variables, but do not represent objects and relations explicitly. Also, their notion of time is restricted to step-by-step change; given the recent trajectory of a ball, we can predict where it will be at time t

+ 1, but it is difficult to represent the abstract idea that

what goes up must come down. Section 15.1  explained how probability and first-order logic can be combined to solve these problems; Section 15.2  showed how we can handle uncertainty about the identity of objects; and Chapter 25  showed how recurrent neural networks enable computer vision to track the world; but we don’t yet have a good way of putting all these techniques together. Chapter 24  showed how word embeddings and similar representations can free us from the strict bounds of concepts defined by necessary and sufficient conditions. It remains a daunting task to define general, reusable representation schemes for complex domains.

Selecting actions The primary difficulty in action selection in the real world is coping with long-term plans— such as graduating from college in four years—that consist of billions of primitive steps. Search algorithms that consider sequences of primitive actions scale only to tens or perhaps hundreds of steps. It is only by imposing hierarchical structure on behavior that we humans cope at all. We saw in Section 11.4  how to use hierarchical representations to handle problems of this scale; furthermore, work in hierarchical reinforcement learning has succeeded in combining these ideas with the MDP formalism described in Chapter 17 .

As yet, these methods have not been extended to the partially observable case (POMDPs). Moreover, algorithms for solving POMDPs are typically using the same atomic state representation we used for the search algorithms of Chapter 3 . There is clearly a great deal of work to do here, but the technical foundations are largely in place for making progress. The main missing element is an effective method for constructing the hierarchical representations of state and behavior that are necessary for decision making over long time scales.

Deciding what we want Chapter 3  introduced search algorithms to find a goal state. But goal-based agents are brittle when the environment is uncertain, and when there are multiple factors to consider. In principle, utility-maximization agents address those issues in a completely general way. The fields of economics and game theory, as well as AI, make use of this insight: just declare what you want to optimize, and what each action does, and we can compute the optimal action.

In practice, however, we now realize that the task of picking the right utility function is a challenging problem in its own right. Imagine, for example, the complex web of interacting preferences that must be understood by an agent operating as an office assistant for a human being. The problem is exacerbated by the fact that each human is different, so an agent just “out of the box” will not have enough experience with any one individual to learn an accurate preference model; it will necessarily need to operate under preference uncertainty. Further complexity arises if we want to ensure that our agents are acting in a way that is fair and equitable for society, rather than just one individual.

We do not yet have much experience with building complex real-world preference models, let alone probability distributions over such models. Although there are factored formalisms, similar to Bayes nets, that are intended to decompose preferences over complex states, it has proven difficult to use these formalisms in practice. One reason may be that preferences over states are really compiled from preferences over state histories, which are described by reward functions (see Chapter 17 ). Even if the reward function is simple, the corresponding utility function may be very complex.

This suggests that we take seriously the task of knowledge engineering for reward functions as a way of conveying to our agents what we want them to do. The idea of inverse reinforcement learning (Section 22.6 ) is one approach to this problem when we have an expert who can perform a task, but not explain it. We could also use better languages for expressing what we want. For example, in robotics, linear temporal logic makes it easier to say what things we want to happen in the near future, what things we want to avoid, and what states we want to persist forever (Littman et al., 2017). We need better ways of saying what we want and better ways for robots to interpret the information we provide.

The computer industry as a whole has developed a powerful ecosystem for aggregating user preferences. When you click on something in an app, online game, social network, or shopping site, that serves as a recommendation that you (and your similar peers) would like to see similar things in the future. (Or it might be that the site is confusing and you clicked on the wrong thing—the data are always noisy.) The feedback inherent in this system makes it very effective in the short run for picking out ever more addictive games and videos.

But these systems often fail to provide an easy way of opting out—your device will auto-play a relevant video, but it is less likely to tell you “maybe it is time to put away your devices and take a relaxing walk in nature.” A shopping site will help you find clothes that match your style, but will not address world peace or ending hunger and poverty. To the extent that the menu of choices is driven by companies trying to profit from a customer’s attention, the menu will remain incomplete.

However, companies do respond to customers’ interests, and many customers have voiced the opinion that they are interested in a fair and sustainable world. Tim O’Reilly explains why profit is not the only motive with the following analogy: “Money is like gasoline during

a road trip. You don’t want to run out of gas on your trip, but you’re not doing a tour of gas stations. You have to pay attention to money, but it shouldn’t be about the money.”

Tristan Harris’s time well spent movement at the Center for Humane Technology is a step towards giving us more well-rounded choices (Harris, 2016). The movement addresses an issue that was recognized by Herbert Simon in 1971: “A wealth of information creates a poverty of attention.” Perhaps in the future we will have personal agents that stick up for our true long-term interests rather than the interests of the corporations whose apps currently fill our devices. It will be the agent’s job to mediate the offerings of various vendors, protect us from addictive attention-grabbers, and guide us towards the goals that really matter to us.

Time well spent

Personal agent

Learning Chapters 19  to 22  described how agents can learn. Current algorithms can cope with quite large problems, reaching or exceeding human capabilities in many tasks—as long as we have sufficient training examples and we are dealing with a predefined vocabulary of features and concepts. But learning can stall when data are sparse, or unsupervised, or when we are dealing with complex representations.

Much of the recent resurgence of AI in the popular press and in industry is due to the success of deep learning (Chapter 21 ). On the one hand, this can be seen as the incremental maturation of the subfield of neural networks. On the other hand, we can see it as a revolutionary leap in capabilities spurred by a confluence of factors: the availability of more training data thanks to the Internet, increased processing power from specialized hardware, and a few algorithmic tricks, such as generative adversarial networks (GANs), batch normalization, dropout, and the rectified linear (ReLU) activation function.

The future should see continued emphasis on improving deep learning for the tasks it excels at, and also extending it to cover other tasks. The brand name “deep learning” has proven to be so popular that we should expect its use to continue, even if the mix of techniques that fuel it changes considerably.

We have seen the emergence of the field of data science as the confluence of statistics, programming, and domain expertise. While we can expect to see continued development in the tools and techniques necessary to acquire, manage, and maintain big data, we will also need advances in transfer learning so that we can take advantage of data in one domain to improve performance on a related domain.

The vast majority of machine learning research today assumes a factored representation, learning a function h

:

Rn R for regression and h Rn →

:

→ {0,1}

for classification. Machine

learning has been less successful for problems that have only a small amount of data, or problems that require the construction of new structured, hierarchical representations. Deep learning, especially with convolutional networks applied to computer vision problems, has demonstrated some success in going from low-level pixels to intermediate-level concepts like Eye and Mouth, then to Face, and finally to Person or Cat.

A challenge for the future is to more smoothly combine learning and prior knowledge. If we give a computer a problem it has not encountered before—say, recognizing different models of cars—we don’t want the system to be powerless until it has been fed millions of labeled examples.

The ideal system should be able to draw on what it already knows: it should already have a model of how vision works, and how the design and branding of products in general work; now it should use transfer learning to apply that to the new problem of car models. It should be able to find on its own information about car models, drawing from text, images, and video available on the Internet. It should be capable of apprenticeship learning: having a conversation with a teacher, and not just asking “may I have a thousand images of a Corolla,” but rather being able to understand advice like “the Insight is similar to the Prius, but the Insight has a larger grille.” It should know that each model comes in a small range of possible colors, but that a car can be repainted, so there is a chance that it might see a car in a color that was not in the training set. (If it didn’t know that, it should be capable of learning it, or being told about it.)

All this requires a communication and representation language that humans and computers can share; we can’t expect a human analyst to directly modify a model with millions of weights. Probabilistic models (including probabilistic programming languages) give humans some ability to describe what we know, but these models are not yet well integrated with other learning mechanisms.

The work of Bengio and LeCun, (2007) is one step towards this integration. Recently Yann LeCun has suggested that the term “deep learning” should be replaced with the more general differentiable programming (Siskind and Pearlmutter, 2016; Li et al., 2018); this suggests that our general programming languages and our machine learning models could be merged together.

Differentiable programming

Right now, it is common to build a deep learning model that is differentiable, and thus can be trained to minimize loss, and retrained when circumstances change. But that deep learning model is only one part of a larger software system that takes in data, massages the data, feeds it to the model, and figures out what to do with the model’s output. All these other parts of the larger system were written by hand by a programmer, and thus are nondifferentiable, which means that when circumstances change, it is up to the programmer to recognize any problems and fix them by hand. With differentiable programming, the hope is that the entire system is subject to automated optimization.

The end goal is to be able to express what we know in whatever form is convenient to us: informal advice given in natural language, a strong mathematical law like F

= ma, a

statistical model accompanied by data, or a probabilistic program with unknown parameters that can be automatically optimized through gradient descent. Our computer models will learn from conversations with human experts as well as by using all the available data.

Yann LeCun, Geoffrey Hinton, and others have suggested that the current emphasis on supervised learning (and to a lesser extent reinforcement learning) is not sustainable—that computer models will have to rely on weakly supervised learning, in which some

supervision is given with a small number of labeled examples and/or a small number of rewards, but most of the learning is unsupervised, because unannotated data are so much more plentiful.

LeCun uses the term predictive learning for an unsupervised learning system that can model the world and learn to predict aspects of future states of the world—not just predict labels for inputs that are independent and identically distributed with respect to past data, and not just predict a value function over states. He suggests that GANs (generative adversarial networks) can be used to learn to minimize the difference between predictions and reality.

Predictive learning

Geoffrey Hinton stated in 2017 that “My view is throw it all away and start again,” meaning that the overall idea of learning by adjusting parameters in a network is enduring, but the specifics of the architecture of the networks and the technique of back-propagation need to be rethought. Smolensky (1988) had a prescription for how to think about connectionist models; his thoughts remain relevant today.

Resources Machine learning research and development has been accelerated by the increasing availability of data, storage, processing power, software, trained experts, and the investments needed to support them. Since the 1970s, there has been a 100,000-fold speedup in general-purpose processors and an additional 1,000-fold speedup due to specialized machine learning hardware. The Web has served as a rich source of images, videos, speech, text, and semi-structured data, currently adding over 1018 bytes every day.

Hundreds of high-quality data sets are available for a range of tasks in computer vision, speech recognition, and natural language processing. If the data you need is not already available, you can often assemble it from other sources, or engage humans to label data for you through a crowdsourcing platform. Validating the data obtained in this way becomes an important part of the overall workflow (Hirth et al., 2013).

An important recent development is the shift from shared data to shared models. The major cloud service providers (e.g., Amazon, Microsoft, Google, Alibaba, IBM, Salesforce) have begun competing to offer machine learning APIs with pre-built models for specific tasks such as visual object recognition, speech recognition, and machine translation. These models can be used as is, or can serve as a baseline to be customized with your particular data for your particular application.

Shared model

We expect that these models will improve over time, and that it will become unusual to start a machine learning project from scratch, just as it is now unusual to do a Web development project from scratch, with no libraries. It is possible that a big jump in model quality will occur when it becomes economical to process all the video on the Web; for example, the YouTube platform alone adds 300 hours of video every minute.

Moore’s law has made it more cost effective to process data; a megabyte of storage cost $1 million in 1969 and less then $0.02 in 2019, and supercomputer throughput has increased by a factor of more than 10

10

in that time. Specialized hardware components for machine

learning such as graphics processing units (GPUs), tensor cores, tensor processing units (TPUs), and field programmable gate arrays (FPGAs) are hundreds of times faster than conventional CPUs for machine learning training (Vasilache et al., 2014; Jouppi et al., 2017). In 2014 it took a full day to train an ImageNet model; in 2018 it takes just 2 minutes (Ying et al., 2018).

The OpenAI Institute reports that the amount of compute power used to train the largest machine learning models doubled every 3.5 months from 2012 to 2018, reaching over an exaflop/second-day for ALPHAZERO (although they also report that some very influential work used 100 million times less computing power (Amodei and Hernandez, 2018)). The same economic trends that have made cell-phone cameras cheaper and better also apply to processors—we will see continued progress in low-power, high-performance computing that benefits from economies of scale.

There is a possibility that quantum computers could accelerate AI. Currently there are some fast quantum algorithms for the linear algebra operations used in machine learning (Harrow et al., 2009; Dervovic et al., 2018), but no quantum computer capable of running them. We have some example applications of tasks such as image classification (Mott et al., 2017) where quantum algorithms are as good as classical algorithms on small problems.

Current quantum computers handle only a few tens of bits, whereas machine learning algorithms often handle inputs with millions of bits and create models with hundreds of millions of parameters. So we need breakthroughs in both quantum hardware and software to make quantum computing practical for large-scale machine learning. Alternatively, there may be a division of labor—perhaps a quantum algorithm to efficiently search the space of hyperparameters while the normal training process runs on conventional computers—but we don’t know how to do that yet. Research on quantum algorithms can sometimes inspire new and better algorithms on classical computers (Tang, 2018).

We have also seen exponential growth in the number of publications, people, and dollars in AI/machine learning/data science. Dean et al. (2018) show that the number of papers about “machine learning” on arXiv doubled every two years from 2009 to 2017. Investors are funding startup companies in these fields, large companies are hiring and spending as they determine their AI strategy, and governments are investing to make sure their country doesn’t fall too far behind.

28.2 AI Architectures It is natural to ask, “Which of the agent architectures in Chapter 2  should an agent use?” The answer is, “All of them!” Reflex responses are needed for situations in which time is of the essence, whereas knowledge-based deliberation allows the agent to plan ahead. Learning is convenient when we have lots of data, and necessary when the environment is changing, or when human designers have insufficient knowledge of the domain.

AI has long had a split between symbolic systems (based on logical and probabilistic inference) and connectionist systems (based on loss minimization over a large number of uninterpreted parameters). A continuing challenge for AI is to bring these two together, to capture the best of both. Symbolic systems allow us to string together long chains of reasoning and to take advantage of the expressive power of structured representations, while connectionist systems can recognize patterns even in the face of noisy data. One line of research aims to combine probabilistic programming with deep learning, although as yet the various proposals are limited in the extent to which the approaches are truly merged.

Agents also need ways to control their own deliberations. They must be able to use the available time well, and cease deliberating when action is demanded. For example, a taxidriving agent that sees an accident ahead must decide in a split second whether to brake or swerve. It should also spend that split second thinking about the most important questions, such as whether the lanes to the left and right are clear and whether there is a large truck close behind, rather than worrying about where to pick up the next passenger. These issues are usually studied under the heading of real-time AI. As AI systems move into more complex domains, all problems will become real-time, because the agent will never have long enough to solve the decision problem exactly.

Real-time AI

Clearly, there is a pressing need for general methods of controlling deliberation, rather than specific recipes for what to think about in each situation. The first useful idea is the anytime algorithms (Dean and Boddy, 1988; Horvitz, 1987): an algorithm whose output quality improves gradually over time, so that it has a reasonable decision ready whenever it is interrupted. Examples of anytime algorithms include iterative deepening in game-tree search and MCMC in Bayesian networks.

Anytime algorithm

The second technique for controlling deliberation is decision-theoretic metareasoning (Russell and Wefald, 1989; Horvitz and Breese, 1996; Hay et al., 2012). This method, which was mentioned briefly in Sections 3.6.5  and 5.7 , applies the theory of information value (Chapter 16 ) to the selection of individual computations (Section 3.6.5 ). The value of a computation depends on both its cost (in terms of delaying action) and its benefits (in terms of improved decision quality).

Decision-theoretic metareasoning

Metareasoning techniques can be used to design better search algorithms and to guarantee that the algorithms have the anytime property. Monte Carlo tree search is one example: the choice of leaf node at which to begin the next playout is made by an approximately rational metalevel decision derived from bandit theory.

Metareasoning is more expensive than reflex action, of course, but compilation methods can be applied so that the overhead is small compared to the costs of the computations being controlled. Metalevel reinforcement learning may provide another way to acquire effective policies for controlling deliberation: in essence, computations that lead to better decisions are reinforced, while those that turn out to have no effect are penalized. This approach avoids the myopia problems of the simple value-of-information calculation.

Metareasoning is one specific example of a reflective architecture—that is, an architecture that enables deliberation about the computational entities and actions occurring within the architecture itself. A theoretical foundation for reflective architectures can be built by defining a joint state space composed from the environment state and the computational state of the agent itself. Decision-making and learning algorithms can be designed that operate over this joint state space and thereby serve to implement and improve the agent’s computational activities. Eventually, we expect task-specific algorithms such as alpha–beta search, regression planning, and variable elimination to disappear from AI systems, to be replaced by general methods that direct the agent’s computations toward the efficient generation of high-quality decisions.

Reflective architecture

Metareasoning and reflection (and many other efficiency-related architectural and algorithmic devices explored in this book) are necessary because making decisions is hard. Ever since computers were invented, their blinding speed has led people to overestimate their ability to overcome complexity, or, equivalently, to underestimate what complexity really means. The truly gargantuan power of today’s machines tempts one to think that we could bypass all the clever devices and rely more on brute force. So let’s try to counteract this tendency. We begin with what physicists believe to be the speed of the ultimate 1kg computing device: about 1051 operations per second, or a billion trillion trillion times faster than the fastest supercomputer as of 2020 (Lloyd, 2000).1 Then we propose a simple task: enumerating strings of English words, much as Borges proposed in The Library of Babel. Borges stipulated books of 410 pages. Would that be feasible? Not quite. In fact, the computer running for a year could enumerate only the 11-word strings.

1 We gloss over the fact that this device consumes the entire energy output of a star and operates at a billion degrees centigrade.

Now consider the fact that a detailed plan for a human life consists of (very roughly) twenty trillion potential muscle actuations (Russell, 2019), and you begin to see the scale of the problem. A computer that is a billion trillion trillion times more powerful than the human brain is much further from being rational than a slug is from overtaking the starship Enterprise traveling at warp nine.

With these considerations in mind, it seems that the goal of building rational agents is perhaps a little too ambitious. Rather than aiming for something that cannot possibly exist, we should consider a different normative target—one that necessarily exists. Recall from Chapter 2  the following simple idea:

agent = architecture + program. Now fix the agent architecture (the underlying machine capabilities, perhaps with a fixed software layer on top) and allow the agent program to vary over all possible programs that the architecture can support. In any given task environment, one of these programs (or an equivalence class of them) delivers the best possible performance—perhaps not close to perfect rationality, but still better than any other agent program. We say that this program satisfies the criterion of bounded optimality. Clearly it exists, and clearly it constitutes a desirable goal. The trick is finding it, or something close to it.

Bounded optimality

For some elementary classes of agent programs in simple real-time environments, it is possible to identify bounded-optimal agent programs (Etzioni, 1989; Russell and Subramanian, 1995). The success of Monte Carlo tree search has revived interest in metalevel decision making, and there is reason to hope that bounded optimality within more complex families of agent programs can be achieved by techniques such as metalevel reinforcement learning. It should also be possible to develop a constructive theory of architecture, beginning with theorems on the bounded optimality of suitable methods of combining different bounded-optimal components such as reflex and action–value systems.

General AI Much of the progress in AI in the 21st century so far has been guided by competition on narrow tasks, such as the DARPA Grand Challenge for autonomous cars, the ImageNet object recognition competition, or playing Go, chess, poker, or Jeopardy! against a world champion. For each separate task, we build a separate AI system, usually with a separate machine learning model trained from scratch with data collected specifically for this task.

But a truly intelligent agent should be able to do more than one thing. Alan Turing, (1950) proposed his list (page 982) and science fiction author Robert Heinlein, (1973) countered with:

A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyse a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects.

So far, no AI system measures up to either of these lists, and some proponents of general or human-level AI (HLAI) insist that continued work on specific tasks (or on individual components) will not be enough to reach mastery on a wide variety of tasks; that we will need a fundamentally new approach. It seems to us that numerous new breakthroughs will indeed be necessary, but overall, AI as a field has made a reasonable exploration/exploitation tradeoff, assembling a portfolio of components, improving on particular tasks, while also exploring promising and sometimes far-out new ideas.

It would have been a mistake to tell the Wright brothers in 1903 to stop work on their single-task airplane and design an “artificial general flight” machine that can take off vertically, fly faster than sound, carry hundreds of passengers, and land on the moon. It also would have been a mistake to follow up their first flight with an annual competition to make spruce wood biplanes incrementally better.

We have seen that work on components can spur new ideas; for example, generative adversarial networks (GANs) and transformer language models each opened up new areas of research. We have also seen steps towards “diversity of behaviour.” For example, machine translation systems in the 1990s were built one at a time for each language pair (such as French to English), but today a single system can identifying the input text as being one of a hundred languages, and translate it into any of 100 target languages. Another natural language system can perform five distinct tasks with one joint model (Hashimoto et al., 2016).

AI engineering The field of computer programming started with a few extraordinary pioneers. But it didn’t reach the status of a major industry until a practice of software engineering was developed,

with a powerful collection of widely available tools, and a thriving ecosystem of teachers, students, practitioners, entrepreneurs, investors, and customers.

The AI industry has not yet reached that level of maturity. We do have a variety of powerful tools and frameworks, such as TensorFlow, Keras, PyTorch, CAFFE, Scikit-Learn and SCIPY. But many of the most promising approaches, such as GANs and deep reinforcement learning, have proven to be difficult to work with—they require experience and a degree of fiddling to get them to train properly in a new domain. We don’t have enough experts to do this across all the domains where we need it, and we don’t yet have the tools and ecosystem to let less-expert practitioners succeed.

Google’s Jeff Dean sees a future where we will want machine learning to handle millions of tasks; it won’t be feasible to develop each of them from scratch, so he suggests that rather than building each new system from scratch, we should start with a single huge system and, for each new task, extract from it the parts that are relevant to the task. We have seen some steps in this direction, such as the transformer language models (e.g., BERT, GPT-2) with billions of parameters, and an “outrageously large” ensemble neural network architecture that scales up to 68 billion parameters in one experiment (Shazeer et al., 2017). Much work remains to be done.

The future Which way will the future go? Science fiction authors seem to favor dystopian futures over utopian ones, probably because they make for more interesting plots. So far, AI seems to fit in with other powerful revolutionary technologies such as printing, plumbing, air travel, and telephony. All these technologies have made positive impacts, but also have some unintended side effects that disproportionately impact disadvantaged classes. We would do well to invest in minimizing the negative impacts.

AI is also different from previous revolutionary technologies. Improving printing, plumbing, air travel, and telephony to their logical limits would not produce anything to threaten human supremacy in the world. Improving AI to its logical limit certainly could.

In conclusion, AI has made great progress in its short history, but the final sentence of Alan Turing’s (1950) essay on Computing Machinery and Intelligence is still valid today:

We can see only a short distance ahead, but we can see that much remains to be done.

Appendix A Mathematical Background A.1 Complexity Analysis and O() Notation Computer scientists are often faced with the task of comparing algorithms to see how fast they run or how much memory they require. There are two approaches to this task. The first is benchmarking—running the algorithms on a computer and measuring speed in seconds and memory consumption in bytes. Ultimately, this is what really matters, but a benchmark can be unsatisfactory because it is so specific: it measures the performance of a particular program written in a particular language, running on a particular computer, with a particular compiler and particular input data. From the single result that the benchmark provides, it can be difficult to predict how well the algorithm would do on a different compiler, computer, or data set. The second approach relies on a mathematical analysis of algorithms, independent of the particular implementation and input, as discussed below.

Benchmarking

Analysis of algorithms

A.1.1 Asymptotic analysis We will consider algorithm analysis through the following example, a program to compute the sum of a sequence of numbers:

function SUMMATION(sequence) returns a number

sum ← 0 for i = 1 to LENGTH(sequence) sum ← sum + sequence [i]

do

return sum

The first step in the analysis is to abstract over the input, in order to find some parameter or parameters that characterize the size of the input. In this example, the input can be

n

characterized by the length of the sequence, which we will call . The second step is to abstract over the implementation, to find some measure that reflects the running time of the algorithm but is not tied to a particular compiler or computer. For the SUMMATION program, this could be just the number of lines of code executed, or it could be more detailed, measuring the number of additions, assignments, array references, and branches executed by the algorithm. Either way gives us a characterization of the total number of steps taken by the algorithm as a function of the size of the input. We will call this characterization . If we count lines of code, we have

T (n) = 2n + 2 for our example.

T ( n)

If all programs were as simple as SUMMATION, the analysis of algorithms would be a trivial field. But two problems make it more complicated. First, it is rare to find a parameter like

n

that completely characterizes the number of steps taken by an algorithm. Instead, the best we can usually do is compute the worst case

T

worst (

n) or the average case T (n). avg

Computing an average means that the analyst must assume some distribution of inputs.

The second problem is that algorithms tend to resist exact analysis. In that case, it is necessary to fall back on an approximation. We say that the SUMMATION algorithm is

n

O( n ) ,

meaning that its measure is at most a constant times , with the possible exception of a few

n

small values of . More formally,

T (n)  is O (f (n))  if T (n) ≤ kf (n)  for some k,  for all n > n  . 0

O() notation gives us what is called an asymptotic analysis. We can say without question that, as n asymptotically approaches infinity, an O(n) algorithm is better than an O(n ) algorithm. A single benchmark figure could not substantiate such a claim. The

2

Asymptotic analysis

O() notation abstracts over constant factors, which makes it easier to use, but less precise, than the T () notation. For example, an O(n ) algorithm will always be worse than an O(n) in the long run, but if the two algorithms are T (n + 1) and T (100n + 1000), then the O(n ) algorithm is actually better for n < 110. The

2

2

2

Despite this drawback, asymptotic analysis is the most widely used tool for analyzing algorithms. It is precisely because the analysis abstracts over both the exact number of

k

operations (by ignoring the constant factor ) and the exact content of the input (by

n

considering only its size ) that the analysis becomes mathematically feasible. The

O()

notation is a good compromise between precision and ease of analysis.

A.1.2 NP and inherently hard problems The analysis of algorithms and the

O() notation allow us to talk about the efficiency of a

particular algorithm. However, they have nothing to say about whether there could be a better algorithm for the problem at hand. The field of complexity analysis analyzes problems rather than algorithms. The first gross division is between problems that can be solved in polynomial time and problems that cannot be solved in polynomial time, no matter what algorithm is used. The class of polynomial problems—those which can be

O(nk ) for some k—is called P. These are sometimes called “easy” problems, because the class contains those problems with running times like O(log n) and O(n). But it also contains those with time O(n ), so the name “easy” should not be taken too literally.

solved in time

1000

Complexity analysis

P

Another important class of problems is NP, the class of nondeterministic polynomial problems. A problem is in this class if there is some algorithm that can guess a solution and then verify whether a guess is correct in polynomial time. The idea is that if you have an arbitrarily large number of processors so that you can try all the guesses at once, or if you are very lucky and always guess right the first time, then the NP problems become P problems. One of the biggest open questions in computer science is whether the class NP is equivalent to the class P when one does not have the luxury of an infinite number of processors or omniscient guessing. Most computer scientists are convinced that P

≠ NP;

that NP problems are inherently hard and have no polynomial-time algorithms. But this has never been proven.

NP

NP-complete

Those who are interested in deciding whether P

= NP

look at a subclass of NP called the

NP-complete problems. The word “complete” is used here in the sense of “most extreme” and thus refers to the hardest problems in the class NP. It has been proven that either all the NP-complete problems are in P or none of them is. This makes the class theoretically interesting, but the class is also of practical interest because many important problems are known to be NP-complete. An example is the satisfiability problem: given a sentence of propositional logic, is there an assignment of truth values to the proposition symbols of the sentence that makes it true? Unless a miracle occurs and P

= NP,

there can be no algorithm

that solves all satisfiability problems in polynomial time. However, AI is more interested in whether there are algorithms that perform efficiently on typical problems drawn from a predetermined distribution; as we saw in Chapter 7 , there are algorithms such as WALKSAT that do quite well on many problems.

The class of NP-hard problems consists of those problems that are reducible (in polynomial time) to all the problems in NP, so if you solved any NP-hard problem, you could solve all

the problems in NP. The NP-complete problems are all NP-hard, but there are some NPhard problems that are even harder than NP-complete.

NP-hard

The class co-NP is the complement of NP, in the sense that, for every decision problem in NP, there is a corresponding problem in co-NP with the “yes” and “no” answers reversed. We know that P is a subset of both NP and co-NP, and it is believed that there are problems in co-NP that are not in P. The co-NP-complete problems are the hardest problems in coNP.

Co-NP

Co-NP-complete

The class #P (pronounced “number P” according to Garey and Johnson (1979), but often pronounced “sharp P”) is the set of counting problems corresponding to the decision problems in NP. Decision problems have a yes-or-no answer: is there a solution to this 3SAT formula? Counting problems have an integer answer: how many solutions are there to this 3-SAT formula? In some cases, the counting problem is much harder than the decision problem. For example, deciding whether a bipartite graph has a perfect matching can be done in time

O(V E) (where the graph has V

vertices and

E edges), but the counting

problem “how many perfect matches does this bipartite graph have” is #P-complete, meaning that it is hard as any problem in #P and thus at least as hard as any NP problem.

Another class is the class of PSPACE problems—those that require a polynomial amount of space, even on a nondeterministic machine. It is believed that PSPACE-hard problems are

worse than NP-complete problems, although it could turn out that NP could turn out that P

= NP.

= PSPACE,

just as it

A.2 Vectors, Matrices, and Linear Algebra Mathematicians define a vector as a member of a vector space, but we will use a more concrete definition: a vector is an ordered sequence of values. For example, in twodimensional space, we have vectors such as

x = ⟨3, 4⟩ and y = ⟨0, 2⟩ . We follow the

convention of boldface characters for vector names, although some authors use arrows or bars over the names: x → or y¯. The elements of a vector can be accessed using subscripts:

z=⟨

z1 , z2 , … , zn ⟩ .

One confusing point: this book is synthesizing work from many

subfields, which variously call their sequences vectors, lists, or tuples, and variously use the notations ⟨1, 2⟩ ,   [1, 2] ,  or  (1, 2) .

Vector

The two fundamental operations on vectors are vector addition and scalar multiplication.

x + y is the elementwise sum: x + y = ⟨3 + 0, 4 + 2⟩ = ⟨3, 6⟩ . Scalar multiplication multiplies each element by a constant: 5x = ⟨5 × 3, 5 × 4⟩ = ⟨15, 20⟩ . The vector addition

x

The length of a vector is denoted | | and is computed by taking the square root of the sum

x √(3

of the squares of the elements: | | =

2

+ 42 ) = 5. The dot product

x ⋅ y (also called scalar

product) of two vectors is the sum of the products of corresponding elements, that is,

x⋅y=∑

i

xi y i ,

or in our particular case,

x ⋅ y = 3 × 0 + 4 × 2 = 8.

Vectors are often interpreted as directed line segments (arrows) in an n-dimensional Euclidean space. Vector addition is then equivalent to placing the tail of one vector at the head of the other, and the dot product

y

x ⋅ y is |x| |y| cos

θ,

where θ is the angle between

x

and .

A matrix is a rectangular array of values arranged into rows and columns. Here is a matrix

A of size 3 × 4:

⎛ ⎜ ⎜ ⎝

A A A

1,1 2,1 3,1

A A A

A A A

1,2 2,2 3,2

A A A

1,3 2,3 3,3

1,4

⎞

2,4 ⎟ ⎟ 3,4

⎠

Matrix

The first index of languages,

Ai j specifies the row and the second the column. In programming ,

Ai j is often written A[i,j] or A[i][j]. ,

The sum of two matrices is defined by adding their corresponding elements; for example

A + B)i j = Ai j + Bi j. (The sum is undefined if A and B have different sizes.) We can also define the multiplication of a matrix by a scalar: (cA)i j = cAi j . Matrix multiplication (the product of two matrices) is more complicated. The product AB is defined only if A is of size a × b and B is of size b × c (i.e., the second matrix has the same number of rows as the first (

,

,

,

,

,

has columns); the result is a matrix of size

a × c. If the matrices are of appropriate size, then

the result is

(

AB)i k = ,

∑A B j

i , j j, k .

Matrix multiplication is not commutative, even for square matrices:

AB ≠ BA in general. It

AB) C = A (BC) . Note that the dot product can be expressed in terms of a transpose and a matrix multiplication: x ⋅ y = x y. is, however, associative: (

⊤

I has elements Ii j equal to 1 when i = j and equal to 0 otherwise. It has the property that AI = A for all A. The transpose of A, written A is formed by turning rows into columns and vice versa, or, more formally, by A i j = Aj i . The inverse of a square matrix A is another square matrix A such that A A = I. For a singular matrix, The identity matrix

,

⊤

⊤

−1

,

,

−1

the inverse does not exist. For a nonsingular matrix, it can be computed in

Identity matrix

O(n ) time. 3

Transpose

Inverse

Singular

Matrices are used to solve systems of linear equations in

O(n ) time; the time is dominated 3

by inverting a matrix of coefficients. Consider the following set of equations, for which we

xy

z

want a solution in , , and :

x+y−z = 8 −3x − y + 2z = −11 −2x + y + 2z = −3. +2

We can represent this system as the matrix equation

⎛2 A = ⎜ −3 ⎝ −2

1

−1

−1

2

1

2

⎞ ⎟, ⎠

A x = b, where

⎛x⎞ x = ⎜y ⎟, ⎝z ⎠

⎛ 8⎞ b = ⎜ −11 ⎟ . ⎝ −3 ⎠

Ax = b we multiply both sides by A , yielding A Ax = A b, which simplifies to x = A b. After inverting A and multiplying by b, we get the answer −1

To solve

−1

−1

−1

⎛x⎞ ⎛ 2⎞ x = ⎜y ⎟ = ⎜ 3⎟. ⎝ z ⎠ ⎝ −1 ⎠ x

x

A few more miscellaneous points: we use log( ) for the natural logarithm, loge ( ). We use

fx

argmaxx ( ) for the value of

x for which f (x) is maximal.

A.3 Probability Distributions A probability is a measure over a set of events that satisfies three axioms:

1. The measure of each event is between 0 and 1. We write this as 0 ≤ where of

P ( X = xi ) ≤ 1 ,

X is a random variable representing an event and xi are the possible values

X. In general, random variables are denoted by uppercase letters and their values

by lowercase letters. 2. The measure of the whole set is 1; that is,

∑ni

=1

P ( X = xi ) = 1 .

3. The probability of a union of disjoint events is the sum of the probabilities of the individual events; that is, case where

x

1

and

x

2

P (X = x

1

∨

X = x ) = P (X = x ) + P (X = x ), in the 2

1

2

are disjoint.

A probabilistic model consists of a sample space of mutually exclusive possible outcomes, together with a probability measure for each outcome. For example, in a model of the weather tomorrow, the outcomes might be sun, cloud, rain, and snow. A subset of these outcomes constitutes an event. For example, the event of precipitation is the subset consisting of {

rain, snow}.

P(X) to denote the vector of values ⟨P (X = x ) , … , P (X = xn )⟩ . We also use P (xi ) as an abbreviation for P (X = xi ) and ∑x P (x) for ∑ni P (X = xi ). We use

1

=1

P (B | A) is defined as P (B ∩ A)/P (A). A and B are conditionally independent if P (B | A) = P (B) (or equivalently, P (A | B) = P (A)). The conditional probability

For continuous variables, there are an infinite number of values, and unless there are point spikes, the probability of any one exact value is 0. So it makes more sense to talk about the value being within a range. We do that with a probability density function, which has a slightly different meaning from the discrete probability function. Since

P (X = x)—the

X has the value x exactly—is zero, we instead measure measure how likely it is that X falls into an interval around x, compared to the the width of the interval, and probability that

take the limit as the interval width goes to zero:

P ( x) =

lim  

dx→0

P (x ≤ X ≤ x + dx) /dx.

Probability density function

The density function must be nonnegative for all x and must have ∞

∫

−∞

P (x) dx = 1 .

We can also define the cumulative distribution

FX (x), which is the probability of a random

variable being less than x:

F X ( x) = P ( X ≤ x ) = ∫

x −∞

P (u) du .

Cumulative distribution

Note that the probability density function has units, whereas the discrete probability function is unitless. For example, if values of X are measured in seconds, then the density is measured in Hz (i.e., 1/sec). If values of

X are points in three-dimensional space measured

in meters, then density is measured in 1/m3 .

Gaussian distribution

One of the most important probability distributions is the Gaussian distribution, also known as the normal distribution. We use the notation N (x;μ, σ 2 ) for the normal

distribution that is a function of x with mean μ and standard deviation σ (and therefore variance σ 2 ). It is defined as

N ( x; μ , σ 2 ) =

1

σ √2 π

e−(x−μ) /(2σ ) , 2

2

where x is a continuous variable ranging from −∞ to +∞. With mean μ = 0 and variance

σ 2 = 1, we get the special case of the standard normal distribution. For a distribution over a vector x in n dimensions, there is the multivariate Gaussian distribution: N (x;μ, Σ) =

√

1

π

n

(2 ) ∣Σ∣

e−

1 2

μ

((x− ) Σ ⊤

−1

μ

(x− ))

,

Standard normal distribution

Multivariate Gaussian

where

μ is the mean vector and Σ is the covariance matrix (see below). The cumulative

distribution for a univariate normal distribution is given by

x

F ( x) =

∫

−∞

N (z;μ, σ 2 )dz =

1 2

(1 + erf(

x−μ )) , σ √2

where erf(x) is the so-called error function, which has no closed-form representation. The central limit theorem states that the distribution formed by sampling n independent random variables and taking their mean tends to a normal distribution as n tends to infinity. This holds for almost any collection of random variables, even if they are not strictly independent, unless the variance of any finite subset of variables dominates the others.

Central limit theorem

The expectation of a random variable,

E(X), is the mean or average value, weighted by the

probability of each value. For a discrete variable it is:

E (X ) =

∑x P X i

i

(

=

xi ) .

Expectation

For a continuous variable, replace the summation with an integral and use the probability density function,

P ( x) : E (X ) =

∫ xP x dx ∞

( )

.

−∞

f

For any function , we also have

E(f (X)) =

∫ f x P x dx ∞

( ) ( )

.

−∞

Finally, when necessary, one may specify the distribution for the random variable as a subscript to the expectation operator:

EX

∼

Q(x) (g(X)) =

∫ g x Q x dx ∞

( )

( )

.

−∞

Besides the expectation, other important statistical properties of a distribution include the

μ

variance, which is the expected value of the square of the difference from the mean, , of the distribution:

V ar(X) = E((X − μ) ) 2

Variance

and the standard deviation, which is the square root of the variance.

Standard deviation

The root mean square (RMS) of a set of values (often samples of a random variable) is the square root of the mean of the squares of the values,

RMS(x , … , xn ) = √ 1

x

2 1

+…+

n

xn 2

.

The covariance of two random variables is the expectation of the product of their differences from their means: cov(

X, Y ) = E((X − μX )(Y − μY )) .

Covariance

The covariance matrix, often denoted Σ, is a matrix of covariances between elements of a vector of random variables. Given

X = ⟨X , … X n ⟩ 1

⊤

, the entries of the covariance matrix

are as follows: Σi,j = cov (

Xi , Xj ) = E ((Xi − μi ) (Xj − μj ))  .

Covariance matrix

We say we sample from a probability distribution, when we pick a value at random. We don’t know what any one pick will bring, but in the limit a large collection of samples will approach the same probability density function as the distribution it is sampled from. The uniform distribution is one where every element is equally (uniformly) probable. So when we say we “sample uniformly (at random) from the integers 0 to 99” it means that we are equally likely to pick any integer in that range.

Sampling

Uniform distribution

Bibliographical and Historical Notes The O() notation so widely used in computer science today was first introduced in the context of number theory by the mathematician P. G. H. Bachmann 1894. The concept of NP-completeness was invented by Cook 1971, and the modern method for establishing a reduction from one problem to another is due to Karp 1972. Cook and Karp have both won the Turing award for their work.

Textbooks on the analysis and design of algorithms include Sedgewick and Wayne (2011) and Cormen, Leiserson, Rivest and Stein (2009). These books place an emphasis on designing and analyzing algorithms to solve tractable problems. For the theory of NPcompleteness and other forms of intractability, see Garey and Johnson (1979) or Papadimitriou (1994). Good texts on probability include Chung (1979), Ross (2015), and Bertsekas and Tsitsiklis (2008).

Appendix B Notes on Languages and Algorithms B.1 Defining Languages with Backus–Naur Form (BNF) In this book we define several languages, including the languages of propositional logic (page 217), first-order logic (page 258), and a subset of English (page 842). A formal language is defined as a set of strings where each string is a sequence of symbols. The languages we are interested in consist of an infinite set of strings, so we need a concise way to characterize the set. We do that with a grammar. The particular type of grammar we use is called a context-free grammar, because each expression has the same form in any context. We write our grammars in a formalism called Backus–Naur form (BNF). There are four components to a BNF grammar:

A set of terminal symbols. These are the symbols or words that make up the strings of the language. They could be letters (A, B, C,  . . .) or words (a, aardvark, abacus,  . . .), or whatever symbols are appropriate for the domain.

Terminal symbol

A set of nonterminal symbols that categorize subphrases of the language. For example, the nonterminal symbol NounPhrase in English denotes an infinite set of strings including “you” and “the big slobbery dog.”

Nonterminal symbol

A start symbol, which is the nonterminal symbol that denotes the complete set of strings of the language. In English, this is Sentence; for arithmetic, it might be Expr, and for programming languages it is Program.

Start symbol

A set of rewrite rules, of the form

LHS RHS →

,

where

LHS is a nonterminal symbol and

RHS is a sequence of zero or more symbols. These can be either terminal or nonterminal symbols, or the symbol ϵ which is used to denote the empty string. ,

Rewrite rules

Context-free grammar

Backus–Naur form (BNF)

A rewrite rule of the form

Sentence NounPhrase V erbPhrase →

 

means that whenever we have two strings categorized as a NounPhrase and a VerbPhrase, we can append them together and categorize the result as a Sentence. As an abbreviation, the

two rules (S

→

A

)

and (S

→

B

)

can be written (S

→

AB |

).

To illustrate these concepts, here is

a BNF grammar for simple arithmetic expressions:

Expr Number Digit Operator

→ →

Expr Operator Expr Expr Number Digit Number Digit  

 

|

|( 

 )|

 

→

0|1|2|3|4|56|7|8|9

→

+| − | ÷ |

×

We cover languages and grammars in more detail in Chapter 23 . Be aware that other books use slightly different notations for BNF; for example, you might see ⟨Digit⟩ instead of Digit for a nonterminal, ‘word’ instead of word for a terminal, or :

:= instead of → in a rule.

B.2 Describing Algorithms with Pseudocode

Pseudocode

The algorithms in this book are described in pseudocode. Most of the pseudocode should be familiar to programmers who use languages like Java, C++, or especially Python. In some places we use mathematical formulas or ordinary English to describe parts that would otherwise be more cumbersome. A few idiosyncrasies should be noted:

PERSISTENT VARIABLES: We use the keyword persistent to say that a variable is given an initial value the first time a function is called and retains that value (or the value given to it by a subsequent assignment statement) on all subsequent calls to the function. Thus, persistent variables are like global variables in that they outlive a single call to their function; but they are accessible only within the function. The agent programs in the book use persistent variables for memory. Programs with persistent variables can be implemented as objects in object-oriented languages such as C++, Java, Python, and Smalltalk. In functional languages, they can be implemented by functional closures over an environment containing the required variables. FUNCTIONS AS VALUES: Functions have capitalized names, and variables have lowercase italic names. So most of the time, a function call looks like FN(x). However, we allow the value of a variable to be a function; for example, if the value of the variable

f is the square root function, then f (9) returns 3. INDENTATION IS SIGNIFICANT: Indentation is used to mark the scope of a loop or conditional, as in the languages Python and CoffeeScript, and unlike Java, C++, and Go (which use braces) or Lua and Ruby (which use end). DESTRUCTURING ASSIGNMENT: The notation “x, y ← pair” means that the righthand side must evaluate to a two-element collection, and the first element is assigned to

x and the second to y. The same idea is used in “for “x, y” in pairs do” and can be used to swap two variables: “x, y ← y, x” DEFAULT VALUES FOR PARAMETERS: The notation “function F (x, y = 0) returns a number” means that y is an optional argument with default value 0; that is, the calls F(3, 0) and F(3) are equivalent. yield: a function that contains the keyword yield is a generator that generates a sequence of values, one each time the yield expression is encountered. After yielding, the function continues execution with the next statement. The languages Python, Ruby, C#, and Javascript (ECMAScript) have this same feature.

Generator

LOOPS: There are four kinds of loops: – “for x in c do” executes the loop with the variable x bound to successive elements of the collection c. – “for i = 1 to n do” executes the loop with i bound to successive integers from 1 to n inclusive. – “while condition do” means the condition is evaluated before each iteration of the loop, and the loop exits if the condition is false. – “repeat ... until condition” means that the loop is executed unconditionally the first time, then the condition is evaluated, and the loop exits if the condition is true; otherwise the loop keeps executing (and testing at the end). LISTS: [x, y, z] denotes a list of three elements. The “+” operator concatenates lists: [1, 2] + [3, 4] = [1, 2, 3, 4]. A list can be used as a stack: POP removes and returns the last

element of a list, TOP returns the last element. SETS: {x, y, z} denotes a set of three elements. {x : p(x)} denotes the set of all elements

x for which p(x) is true. ARRAYS START AT 1: the first index of an array is 1 as in usual mathematical notation (and in R and Julia), not 0 (as in Python and Java and C).

B.3 Online Supplemental Material The book has a Web site with supplemental material, instructions for sending suggestions, and opportunities for joining discussion lists:

aima.cs.berkeley.edu The algorithms in the book, and multiple additional programming exercises, have been implemented in Python and Java (and some in other languages) at the online code repository, accessible from the Web site and currently hosted at:

github.com/aimacode

Bibliography The following abbreviations are used for frequently cited conferences and journals:

AAAI

Proceedings of the AAAI Conference on Artificial Intelligence

AAMAS

Proceedings of the International Conference on Autonomous Agents and Multiagent Systems

ACL

Proceedings of the Annual Meeting of the Association for Computational Linguistics

AIJ

Artificial Intelligence (Journal)

AIMag

AI Magazine

AIPS

Proceedings of the International Conference on AI Planning Systems

AISTATS

Proceedings of the International Conference on Artificial Intelligence and Statistics

BBS

Behavioral and Brain Sciences

CACM

Communications of the Association for Computing Machinery

COGSCI

Proceedings of the Annual Conference of the Cognitive Science Society

COLING

Proceedings of the International Conference on Computational Linguistics

COLT

Proceedings of the Annual ACM Workshop on Computational Learning Theory

CP

Proceedings of the International Conference on Principles and Practice of Constraint Programming

CVPR

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

EC

Proceedings of the ACM Conference on Electronic Commerce

ECAI

Proceedings of the European Conference on Artificial Intelligence

ECCV

Proceedings of the European Conference on Computer Vision

ECML

Proceedings of the The European Conference on Machine Learning

ECP

Proceedings of the European Conference on Planning

EMNLP

Proceedings of the Conference on Empirical Methods in Natural Language Processing

FGCS

Proceedings of the International Conference on Fifth Generation Computer Systems

FOCS

Proceedings of the Annual Symposium on Foundations of Computer Science

GECCO

Proceedings of the Genetics and Evolutionary Computing Conference

HRI

Proceedings of the International Conference on Human-Robot Interaction

ICAPS

Proceedings of the International Conference on Automated Planning and Scheduling

ICASSP

Proceedings of the International Conference on Acoustics, Speech, and Signal Processing

ICCV

Proceedings of the International Conference on Computer Vision

ICLP

Proceedings of the International Conference on Logic Programming

ICLR

Proceedings of the International Conference on Learning Representations

ICML

Proceedings of the International Conference on Machine Learning

ICPR

Proceedings of the International Conference on Pattern Recognition

ICRA

Proceedings of the IEEE International Conference on Robotics and Automation

ICSLP

Proceedings of the International Conference on Speech and Language Processing

IJAR

International Journal of Approximate Reasoning

IJCAI

Proceedings of the International Joint Conference on Artificial Intelligence

IJCNN

Proceedings of the International Joint Conference on Neural Networks

IJCV

International Journal of Computer Vision

ILP

Proceedings of the International Workshop on Inductive Logic Programming

IROS

Proceedings of the International Conference on Intelligent Robots and Systems

ISMIS

Proceedings of the International Symposium on Methodologies for Intelligent Systems

ISRR

Proceedings of the International Symposium on Robotics Research

JACM

Journal of the Association for Computing Machinery

JAIR

Journal of Artificial Intelligence Research

JAR

Journal of Automated Reasoning

JASA

Journal of the American Statistical Association

JMLR

Journal of Machine Learning Research

JSL

Journal of Symbolic Logic

KDD

Proceedings of the International Conference on Knowledge Discovery and Data Mining

KR

Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning

LICS

Proceedings of the IEEE Symposium on Logic in Computer Science

NeurIPS

Advances in Neural Information Processing Systems

PAMI

IEEE Transactions on Pattern Analysis and Machine Intelligence

PNAS

Proceedings of the National Academy of Sciences of the United States of America

PODS

Proceedings of the ACM International Symposium on Principles of Database Systems

RSS

Proceedings of the Conference on Robotics: Science and Systems

SIGIR

Proceedings of the Special Interest Group on Information Retrieval

SIGMOD

Proceedings of the ACM SIGMOD International Conference on Management of Data

SODA

Proceedings of the Annual ACM–SIAM Symposium on Discrete Algorithms

STOC

Proceedings of the Annual ACM Symposium on Theory of Computing

TARK

Proceedings of the Conference on Theoretical Aspects of Reasoning about Knowledge

UAI

Proceedings of the Conference on Uncertainty in Artificial Intelligence

Aaronson, S. (2014). My conversation with “Eugene Goostman,” the chatbot that’s all over the news for allegedly passing the Turing test. Shtetl-Optimized, www.scottaaronson.com/blog/?p=1858.

Aarts, E. and Lenstra, J. K. (2003). Local Search in Combinatorial Optimization. Princeton University Press.

Aarup, M., Arentoft, M. M., Parrod, Y., Stader, J., and Stokes, I. (1994). OPTIMUM-AIV: A knowledge-based planning and scheduling system for spacecraft AIV. In Fox, M. and Zweben, M.(Eds.), Knowledge Based Scheduling. Morgan Kaufmann.

Abbas, A. (2018). Foundations of Multiattribute Utility. Cambridge University Press.

Abbeel, P. and Ng, A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. In ICML-04.

Abney, S., McAllester, D. A., and Pereira, F. (1999). Relating probabilistic grammars and automata. In ACL-99.

Abramson, B. (1987). The expected-outcome model of two-player games. Ph.D. thesis, Columbia University.

Abramson, B. (1990). Expected-outcome: A general model of static evaluation. PAMI, 12, 182–193.

Abreu, D. and Rubinstein, A. (1988). The structure of Nash equilibrium in repeated games with finite automata. Econometrica, 56, 1259–1281.

Achlioptas, D. (2009). Random satisfiability. In Biere, A., Heule, M., van Maaren, H., and Walsh, T. (Eds.), Handbook of Satisfiability. IOS Press.

Ackerman, E. and Guizzo, E. (2016). The next generation of Boston Dynamics’ Atlas robot is quiet, robust, and tether free. IEEE Spectrum, 24, 2016.

Ackerman, N., Freer, C., and Roy, D. (2013). On the computability of conditional probability. arXiv 1005.3014.

Ackley, D. H. and Littman, M. L. (1991). Interactions between learning and evolution. In Langton, C., Taylor, C., Farmer, J. D., and Rasmussen, S. (Eds.), Artificial Life II. Addison-Wesley.

Adida, B. and Birbeck, M. (2008). RDFa primer. Tech. rep., W3C.

Adolph, K. E., Kretch, K. S., and LoBue, V. (2014). Fear of heights in infants? Current Directions in Psychological Science, 23, 60–66.

Agerbeck, C. and Hansen, M. O. (2008). A multi-agent approach to solving NP-complete problems. Master’s thesis, Technical Univ. of Denmark.

Aggarwal, G., Goel, A., and Motwani, R. (2006). Truthful auctions for pricing search keywords. In EC-06.

Agha, G. (1986). ACTORS: A Model of Concurrent Computation in Distributed Systems. MIT Press.

Agichtein, E. and Gravano, L. (2003). Querying text databases for efficient information extraction. In Proc. IEEE Conference on Data Engineering.

Agmon, S. (1954). The relaxation method for linear inequalities. Canadian Journal of Mathematics, 6, 382–392.

Agostinelli, F., McAleer, S., Shmakov, A., and Baldi, P. (2019). Solving the Rubik’s Cube with deep reinforcement learning and search. Nature Machine Intelligence, 1, 356–363.

Agrawal, P., Nair, A. V., Abbeel, P., Malik, J., and Levine, S. (2017). Learning to poke by poking: Experiential learning of intuitive physics. In NeurIPS 29.

Agre, P. E. and Chapman, D. (1987). Pengi: an implementation of a theory of activity. In IJCAI-87.

Aizerman, M., Braverman, E., and Rozonoer, L. (1964). Theoretical foundations of the potential function method in pattern recognition learning. Automation and Remote Control, 25, 821–837.

Akametalu, A. K., Fisac, J. F., Gillula, J. H., Kaynama, S., Zeilinger, M. N., and Tomlin, C. J. (2014). Reachability-based safe learning with Gaussian processes. In 53rd IEEE Conference on Decision and Control.

Akgun, B., Cakmak, M., Jiang, K., and Thomaz, A. (2012). Keyframe-based learning from demonstration. International Journal of Social Robotics, 4, 343–355.

Aldous, D. and Vazirani, U. (1994). “Go with the winners” algorithms. In FOCS-94.

Alemi, A. A., Chollet, F., Een, N., Irving, G., Szegedy, C., and Urban, J. (2017). DeepMath Deep sequence models for premise selection. In NeurIPS 29.

Allais, M. (1953). Le comportment de l’homme rationnel devant la risque: critique des postulats et axiomes de l’école Américaine. Econometrica, 21, 503–546.

Allan, J., Harman, D., Kanoulas, E., Li, D., Van Gysel, C., and Vorhees, E. (2017). Trec 2017 common core track overview. In Proc. TREC.

Allen, J. F. (1983). Maintaining knowledge about temporal intervals. CACM, 26, 832–843.

Allen, J. F. (1984). Towards a general theory of action and time. AIJ, 23, 123–154.

Allen, J. F. (1991). Time and time again: The many ways to represent time. Int. J. Intelligent Systems, 6, 341–355.

Allen, J. F., Hendler, J., and Tate, A. (Eds.). (1990). Readings in Planning. Morgan Kaufmann.

Allen, P. and Greaves, M. (2011). The singularity isn’t near. Technology review, 12, 7–8.

Allen-Zhu, Z., Li, Y., and Song, Z. (2018). A convergence theory for deep learning via overparameterization. arXiv:1811.03962.

Alterman, R. (1988). Adaptive planning. Cognitive Science, 12, 393–422.

Amarel, S. (1967). An approach to heuristic problem-solving and theorem proving in the propositional calculus. In Hart, J. and Takasu, S. (Eds.), Systems and Computer Science. University of Toronto Press.

Amarel, S. (1968). On representations of problems of reasoning about actions. In Michie, D. (Ed.), Machine Intelligence 3, Vol. 3. Elsevier.

Amir, E. and Russell, S. J. (2003). Logical filtering. In IJCAI-03.

Amit, Y. and Geman, D. (1997). Shape quantization and recognition with randomized trees. Neural Computation, 9, 1545–1588.

Amodei, D. and Hernandez, D. (2018). AI and compute. OpenAI blog, blog.openai.com/aiand-compute/.

Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016). Concrete problems in AI safety. arXiv:1606.06565.

Andersen, S. K., Olesen, K. G., Jensen, F. V., and Jensen, F. (1989). HUGIN—A shell for building Bayesian belief universes for expert systems. In IJCAI-89.

Anderson, J. R. (1980). Cognitive Psychology and Its Implications. W. H. Freeman.

Anderson, J. R. (1983). The Architecture of Cognition. Harvard University Press.

Anderson, K., Sturtevant, N. R., Holte, R. C., and Schaeffer, J. (2008). Coarse-to-fine search techniques. Tech. rep., University of Alberta.

Andoni, A. and Indyk, P. (2006). Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. In FOCS-06.

Andor, D., Alberti, C., Weiss, D., Severyn, A., Presta, A., Ganchev, K., Petrov, S., and Collins, M. (2016). Globally normalized transition-based neural networks. arXiv:1603.06042.

Andre, D., Friedman, N., and Parr, R. (1998). Generalized prioritized sweeping. In NeurIPS 10.

Andre, D. and Russell, S. J. (2002). State abstraction for programmable reinforcement learning agents. In AAAI-02.

Andreae, P. (1985). Justified Generalisation: Learning Procedures from Examples. Ph.D. thesis, MIT.

Andrieu, C., Doucet, A., and Holenstein, R. (2010). Particle Markov chain Monte Carlo methods. J. Royal Statistical Society, 72, 269–342.

Andrychowicz, M., Baker, B., Chociej, M., Jozefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., et al. (2018a). Learning dexterous in-hand manipulation. arXiv:1808.00177.

Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. (2018b). Hindsight experience replay. In NeurIPS 30.

Aneja, J., Deshpande, A., and Schwing, A. (2018). Convolutional image captioning. In CVPR-18.

Aoki, M. (1965). Optimal control of partially observable Markov systems. J. Franklin Institute, 280, 367–386.

Appel, K. and Haken, W. (1977). Every planar map is four colorable: Part I: Discharging. Illinois J. Math., 21, 429–490.

Appelt, D. (1999). Introduction to information extraction. AI Communications, 12, 161–172.

Apt, K. R. (1999). The essence of constraint propagation. Theoretical Computer Science, 221, 179–210.

Apt, K. R. (2003). Principles of Constraint Programming. Cambridge University Press.

Apté, C., Damerau, F., and Weiss, S. (1994). Automated learning of decision rules for text categorization. ACM Transactions on Information Systems, 12, 233–251.

Arbuthnot, J. (1692). Of the Laws of Chance. Motte, London. Translation into English, with additions, of Huygens (1657).

Archibald, C., Altman, A., and Shoham, Y. (2009). Analysis of a winning computational billiards player. In IJCAI-09.

Arfaee, S. J., Zilles, S., and Holte, R. C. (2010). Bootstrap learning of heuristic functions. In Third Annual Symposium on Combinatorial Search.

Argall, B. D., Chernova, S., Veloso, M., and Browning, B. (2009). A survey of robot learning from demonstration. Robotics and autonomous systems, 57, 469–483.

Ariely, D. (2009). Predictably Irrational (Revised edition). Harper.

Arkin, R. (1998). Behavior-Based Robotics. MIT Press.

Arkin, R. (2015). The case for banning killer robots: Counterpoint. CACM, 58.

Armando, A., Carbone, R., Compagna, L., Cuellar, J., and Tobarra, L. (2008). Formal analysis of SAML 2.0 web browser single sign-on: Breaking the SAML-based single sign-on for Google apps. In Proc. 6th ACM Workshop on Formal Methods in Security Engineering.

Armstrong, S. and Levinstein, B. (2017). Low impact artificial intelligences. arXiv:1705.10720.

Arnauld, A. (1662). La logique, ou l’art de penser. Chez Charles Savreux, Paris.

Arora, N. S., Russell, S. J., and Sudderth, E. (2013). NET-VISA: Network processing vertically integrated seismic analysis. Bull. Seism. Soc. Amer., 103, 709–729.

Arora, S. (1998). Polynomial time approximation schemes for Euclidean traveling salesman and other geometric problems. JACM, 45, 753–782.

Arpit, D., Jastrzebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., and Lacoste-Julien, S. (2017). A closer look at memorization in deep networks. arXiv:1706.05394.

Arrow, K. J. (1951). Social Choice and Individual Values. Wiley.

Arulampalam, M. S., Maskell, S., Gordon, N., and Clapp, T. (2002). A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking. IEEE Transactions on Signal Processing, 50, 174–188.

Arulkumaran, K., Deisenroth, M. P., Brundage, M., and Bharath, A. A. (2017). Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34, 26–38.

Arunachalam, R. and Sadeh, N. M. (2005). The supply chain trading agent competition. Electronic Commerce Research and Applications, Spring, 66–84.

Ashby, W. R. (1940). Adaptiveness and equilibrium. J. Mental Science, 86, 478–483.

Ashby, W. R. (1948). Design for a brain. Electronic Engineering, December, 379–383.

Ashby, W. R. (1952). Design for a Brain. Wiley.

Asimov, I. (1942). Runaround. Astounding Science Fiction, March.

Asimov, I. (1950). I, Robot. Doubleday.

Asimov, I. (1958). The feeling of power. If: Worlds of Science Fiction, February.

Astrom, K. J. (1965). Optimal control of Markov decision processes with incomplete state estimation. J. Math. Anal. Applic., 10, 174–205.

Atkeson, C. G., Moore, A. W., and Schaal, S. (1997). Locally weighted learning for control. In Lazy learning. Springer.

Audi, R. (Ed.). (1999). The Cambridge Dictionary of Philosophy. Cambridge University Press.

Auer, P., Cesa-Bianchi, N., and Fischer, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47, 235–256.

Aumann, R. and Brandenburger, A. (1995). Epistemic conditions for nash equilibrium. Econometrica, 67, 1161–1180.

Axelrod, R. (1985). The Evolution of Cooperation. Basic Books.

Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv:1607.06450.

Baader, F., Calvanese, D., McGuinness, D., Nardi, D., and Patel-Schneider, P. (2007). The Description Logic Handbook (2nd edition). Cambridge University Press.

Baader, F. and Snyder, W. (2001). Unification theory. In Robinson, J. and Voronkov, A. (Eds.), Handbook of Automated Reasoning. Elsevier.

Bacchus, F. (1990). Representing and Reasoning with Probabilistic Knowledge. MIT Press.

Bacchus, F. and Grove, A. (1995). Graphical models for preference and utility. In UAI-95.

Bacchus, F. and Grove, A. (1996). Utility independence in a qualitative decision theory. In KR-96.

Bacchus, F., Grove, A., Halpern, J. Y., and Koller, D. (1992). From statistics to beliefs. In AAAI-92.

Bacchus, F. and van Beek, P. (1998). On the conversion between non-binary and binary constraint satisfaction problems. In AAAI-98.

Bacchus, F. and van Run, P. (1995). Dynamic variable ordering in CSPs. In CP-95.

Bacchus, F., Dalmao, S., and Pitassi, T. (2003). Value elimination: Bayesian inference via backtracking search. In UAI-03.

Bachmann, P. G. H. (1894). Die analytische Zahlentheorie. B. G. Teubner, Leipzig.

Backus, J. W. (1959). The syntax and semantics of the proposed international algebraic language of the Zurich ACM-GAMM conference. Proc. Int’l Conf. on Information Processing.

Bacon, F. (1609). Wisdom of the Ancients. Cassell and Company.

Baeza-Yates, R. and Ribeiro-Neto, B. (2011). Modern Information Retrieval (2nd edition). Addison-Wesley.

Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., and Shmatikov, V. (2018). How to backdoor federated learning. arXiv:1807.00459.

Bagnell, J. A. and Schneider, J. (2001). Autonomous helicopter control using reinforcement learning policy search methods. In ICRA-01.

Bahdanau, D., Cho, K., and Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. In ICLR-15.

Bahubalendruni, M. R. and Biswal, B. B. (2016). A review on assembly sequence generation and its automation. Proc. Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science, 230, 824–838.

Bai, A. and Russell, S. J. (2017). Efficient reinforcement learning with hierarchies of machines by leveraging internal transitions. In IJCAI-17.

Bai, H., Cai, S., Ye, N., Hsu, D., and Lee, W. S. (2015). Intention-aware online POMDP planning for autonomous driving in a crowd. In ICRA-15.

Bajcsy, A., Losey, D. P., O’Malley, M. K., and Dragan, A. D. (2017). Learning robot objectives from physical human interaction. Proceedings of Machine Learning Research, 78, 217–226.

Baker, C. L., Saxe, R., and Tenenbaum, J. B. (2009). Action understanding as inverse planning. Cognition, 113, 329–349.

Baker, J. (1975). The Dragon system—An overview. IEEE Transactions on Acoustics, Speech, and Signal Processing, 23, 24–29.

Baker, J. (1979). Trainable grammars for speech recognition. In Speech Communication Papers for the 97th Meeting of the Acoustical Society of America.

Baldi, P., Chauvin, Y., Hunkapiller, T., and McClure, M. (1994). Hidden Markov models of biological primary sequence information. PNAS, 91, 1059–1063.

Baldwin, J. M. (1896). A new factor in evolution. American Naturalist, 30, 441–451. Continued on pages 536–553.

Ballard, B. W. (1983). The *-minimax search procedure for trees containing chance nodes. AIJ, 21, 327–350.

Baluja, S. (1997). Genetic algorithms and explicit search statistics. In NeurIPS 9.

Bancilhon, F., Maier, D., Sagiv, Y., and Ullman, J. D. (1986). Magic sets and other strange ways to implement logic programs. In PODS-86.

Banko, M. and Brill, E. (2001). Scaling to very very large corpora for natural language disambiguation. In ACL-01.

Banko, M., Brill, E., Dumais, S. T., and Lin, J. (2002). AskMSR: Question answering using the worldwide web. In Proc. AAAI Spring Symposium on Mining Answers from Texts and Knowledge Bases.

Banko, M., Cafarella, M. J., Soderland, S., Broadhead, M., and Etzioni, O. (2007). Open information extraction from the web. In IJCAI-07.

Banko, M. and Etzioni, O. (2008). The tradeoffs between open and traditional relation extraction. In ACL-08.

Bansal, K., Loos, S., Rabe, M. N., Szegedy, C., and Wilcox, S. (2019). HOList: An environment for machine learning of higher-order theorem proving (extended version). arXiv:1904.03241.

Bar-Hillel, Y. (1954). Indexical expressions. Mind, 63, 359–379.

Bar-Shalom, Y. (Ed.). (1992). Multitarget-Multisensor Tracking: Advanced Applications. Artech House.

Bar-Shalom, Y. and Fortmann, T. E. (1988). Tracking and Data Association. Academic Press.

Bar-Shalom, Y., Li, X.-R., and Kirubarajan, T. (2001). Estimation, Tracking and Navigation: Theory, Algorithms and Software. Wiley.

Barber, D. (2012). Bayesian Reasoning and Machine Learning. Cambridge University Press.

Barr, A. and Feigenbaum, E. A. (Eds.). (1981). The Handbook of Artificial Intelligence, Vol. 1. HeurisTech Press and William Kaufmann.

Barreiro, J., Boyce, M., Do, M., Frank, J., Iatauro, M., Kichkaylo, T., Morris, P., Ong, J., Remolina, E., Smith, T., et al. (2012). EUROPA: A platform for AI planning, scheduling, constraint programming, and optimization. 4th International Competition on Knowledge Engineering for Planning and Scheduling (ICKEPS).

Barreno, M., Nelson, B., Joseph, A. D., and Tygar, J. D. (2010). The security of machine learning. Machine Learning, 81, 121–148.

Barrett, S. and Stone, P. (2015). Cooperating with unknown teammates in complex domains: A robot soccer case study of ad hoc teamwork. In AAAI-15.

Barták, R., Salido, M. A., and Rossi, F. (2010). New trends in constraint satisfaction, planning, and scheduling: A survey. The Knowledge Engineering Review, 25, 249–279.

Bartholdi, J. J., Tovey, C. A., and Trick, M. A. (1989). The computational difficulty of manipulating an election. Social Choice and Welfare, 6, 227–241.

Barto, A. G., Bradtke, S. J., and Singh, S. (1995). Learning to act using real-time dynamic programming. AIJ, 73, 81–138.

Barto, A. G., Sutton, R. S., and Brouwer, P. S. (1981). Associative search network: A reinforcement learning associative memory. Biological Cybernetics, 40, 201–211.

Barwise, J. and Etchemendy, J. (2002). Language, Proof and Logic. CSLI Press.

Baum, E., Boneh, D., and Garrett, C. (1995). On genetic algorithms. In COLT-95.

Baum, E. and Smith, W. D. (1997). A Bayesian approach to relevance in game playing. AIJ, 97, 195–242.

Baum, L. E. and Petrie, T. (1966). Statistical inference for probabilistic functions of finite state Markov chains. Annals of Mathematical Statistics, 41, 1554–1563.

Baxter, J. and Bartlett, P. (2000). Reinforcement learning in POMDPs via direct gradient ascent. In ICML-00.

Bayardo, R. J. and Agrawal, R. (2005). Data privacy through optimal k-anonymization. In Proc. 21st Int’l Conf. on Data Engineering.

Bayardo, R. J. and Miranker, D. P. (1994). An optimal backtrack algorithm for treestructured constraint satisfaction problems. AIJ, 71, 159–181.

Bayardo, R. J. and Schrag, R. C. (1997). Using CSP look-back techniques to solve real-world SAT instances. In AAAI-97.

Bayes, T. (1763). An essay towards solving a problem in the doctrine of chances. Phil. Trans. Roy. Soc., 53, 370–418.

Beal, J. and Winston, P. H. (2009). The new frontier of human-level artificial intelligence. IEEE Intelligent Systems, 24, 21–23.

Beardon, A. F., Candeal, J. C., Herden, G., Induráin, E., and Mehta, G. B. (2002). The nonexistence of a utility function and the structure of non-representable preference relations. Journal of Mathematical Economics, 37, 17 – 38.

Beattie, C., Leibo, J. Z., Teplyashin, D., Ward, T., Wainwright, M., Küttler, H., Lefrancq, A., Green, S., Valdés, V., Sadik, A., Schrittwieser, J., Anderson, K., York, S., Cant, M., Cain, A., Bolton, A., Gaffney, S., King, H., Hassabis, D., Legg, S., and Petersen, S. (2016). DeepMind lab. arXiv:1612.03801.

Bechhofer, R. (1954). A single-sample multiple decision procedure for ranking means of normal populations with known variances. Annals of Mathematical Statistics, 25, 16–39.

Beck, J. C., Feng, T. K., and Watson, J.-P. (2011). Combining constraint programming and local search for job-shop scheduling. INFORMS Journal on Computing, 23, 1–14.

Beckert, B. and Posegga, J. (1995). Leantap: Lean, tableau-based deduction. JAR, 15, 339– 358.

Beeri, C., Fagin, R., Maier, D., and Yannakakis, M. (1983). On the desirability of acyclic database schemes. JACM, 30, 479–513.

Bekey, G. (2008). Robotics: State Of The Art And Future Challenges. Imperial College Press.

Belkin, M., Hsu, D., Ma, S., and Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off. PNAS, 116, 15849–15854.

Bell, C. and Tate, A. (1985). Using temporal constraints to restrict search in a planner. In Proc. Third Alvey IKBS SIG Workshop.

Bell, J. L. and Machover, M. (1977). A Course in Mathematical Logic. Elsevier.

Bellamy, E. (2003). Looking Backward: 2000-1887. Broadview Press.

Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., Nagar, S., Ramamurthy, K. N., Richards, J. T., Saha, D., Sattigeri, P., Singh, M., Varshney, K. R., and Zhang, Y. (2018). AI fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv:1810.01943.

Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. (2013). The arcade learning environment: An evaluation platform for general agents. JAIR, 47, 253–279.

Bellman, R. E. (1952). On the theory of dynamic programming. PNAS, 38, 716–719.

Bellman, R. E. (1958). On a routing problem. Quarterly of Applied Mathematics, 16.

Bellman, R. E. (1961). Adaptive Control Processes: A Guided Tour. Princeton University Press.

Bellman, R. E. (1965). On the application of dynamic programming to the determination of optimal play in chess and checkers. PNAS, 53, 244–246.

Bellman, R. E. (1984). Eye of the Hurricane. World Scientific.

Bellman, R. E. and Dreyfus, S. E. (1962). Applied Dynamic Programming. Princeton University Press.

Bellman, R. E. (1957). Dynamic Programming. Princeton University Press.

Ben-Tal, A. and Nemirovski, A. (2001). Lectures on Modern Convex Optimization: Analysis, Algorithms, and Engineering Applications. SIAM (Society for Industrial and Applied Mathematics).

Bengio, Y., Simard, P., and Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5, 157–166.

Bengio, Y. and Bengio, S. (2001). Modeling high-dimensional discrete data with multi-layer neural networks. In NeurIPS 13.

Bengio, Y., Ducharme, R., Vincent, P., and Jauvin, C. (2003). A neural probabilistic language model. JMLR, 3, 1137–1155.

Bengio, Y. and LeCun, Y. (2007). Scaling learning algorithms towards AI. In Bottou, L., Chapelle, O., DeCoste, D., and Weston, J. (Eds.), Large-Scale Kernel Machines. MIT Press.

Benjamin, M. (2013). Drone Warfare: Killing by Remote Control. Verso Books.

Bentham, J. (1823). Principles of Morals and Legislation. Oxford University Press, Oxford. Original work published in 1789.

Benzmüller, C. and Paleo, B. W. (2013). Formalization, mechanization and automation of Gödel’s proof of God’s existence. arXiv:1308.4526.

Beresniak, A., Medina-Lara, A., Auray, J. P., De Wever, A., Praet, J.-C., Tarricone, R., Torbica, A., Dupont, D., Lamure, M., and Duru, G. (2015). Validation of the underlying assumptions of the quality-adjusted life-years outcome: Results from the ECHOUTCOME European project. PharmacoEconomics, 33, 61–69.

Berger, J. O. (1985). Statistical Decision Theory and Bayesian Analysis. Springer Verlag.

Bergstra, J. and Bengio, Y. (2012). Random search for hyper-parameter optimization. JMLR, 13, 281–305.

Berk, R., Heidari, H., Jabbari, S., Kearns, M., and Roth, A. (2017). Fairness in criminal justice risk assessments: The state of the art. arXiv:1703.09207.

Berkson, J. (1944). Application of the logistic function to bio-assay. JASA, 39, 357–365.

Berleur, J. and Brunnstein, K. (2001). Ethics of Computing: Codes, Spaces for Discussion and Law. Chapman and Hall.

Berlin, K., Koren, S., Chin, C.-S., Drake, J. P., Landolin, J. M., and Phillippy, A. M. (2015). Assembling large genomes with single-molecule sequencing and locality-sensitive hashing. Nature Biotechnology, 33, 623.

Berliner, H. J. (1979). The B* tree search algorithm: A best-first proof procedure. AIJ, 12, 23–40.

Berliner, H. J. (1980a). Backgammon computer program beats world champion. AIJ, 14, 205–220.

Berliner, H. J. (1980b). Computer backgammon. Scientific American, 249, 64–72.

Bermúdez-Chacón, R., Gonnet, G. H., and Smith, K. (2015). Automatic problem-specific hyperparameter optimization and model selection for supervised machine learning. Tech. rep., ETH Zurich.

Bernardo, J. M. and Smith, A. (1994). Bayesian Theory. Wiley.

Berners-Lee, T., Hendler, J., and Lassila, O. (2001). The semantic web. Scientific American, 284, 34–43.

Bernoulli, D. (1738). Specimen theoriae novae de mensura sortis. Proc. St. Petersburg Imperial Academy of Sciences, 5, 175–192.

Bernstein, P. L. (1996). Against the Gods: The Remarkable Story of Risk. Wiley.

Berrada, L., Zisserman, A., and Kumar, M. P. (2019). Training neural networks for and by interpolation. arXiv:1906.05661.

Berrou, C., Glavieux, A., and Thitimajshima, P. (1993). Near Shannon limit error controlcorrecting coding and decoding: Turbo-codes. 1. In Proc. IEEE International Conference on Communications.

Berry, D. A. and Fristedt, B. (1985). Bandit Problems: Sequential Allocation of Experiments. Chapman and Hall.

Bertele, U. and Brioschi, F. (1972). Nonserial Dynamic Programming. Academic Press.

Bertoli, P., Cimatti, A., and Roveri, M. (2001a). Heuristic search + symbolic model checking = efficient conformant planning. In IJCAI-01.

Bertoli, P., Cimatti, A., Roveri, M., and Traverso, P. (2001b). Planning in nondeterministic domains under partial observability via symbolic model checking. In IJCAI-01.

Bertot, Y., Casteran, P., Huet, G., and Paulin-Mohring, C. (2004). Interactive Theorem Proving and Program Development. Springer.

Bertsekas, D. (1987). Dynamic Programming: Deterministic and Stochastic Models. PrenticeHall.

Bertsekas, D. and Tsitsiklis, J. N. (1996). Neuro-Dynamic Programming. Athena Scientific.

Bertsekas, D. and Tsitsiklis, J. N. (2008). Introduction to Probability (2nd edition). Athena Scientific.

Bertsekas, D. and Shreve, S. E. (2007). Stochastic Optimal Control: The Discrete-Time Case. Athena Scientific.

Bertsimas, D., Delarue, A., and Martin, S. (2019). Optimizing schools’ start time and bus routes. PNAS, 116 13, 5943–5948.

Bertsimas, D. and Dunn, J. (2017). Optimal classification trees. Machine Learning, 106, 1039–1082.

Bessen, J. (2015). Learning by Doing: The Real Connection between Innovation, Wages, and Wealth. Yale University Press.

Bessière, C. (2006). Constraint propagation. In Rossi, F., van Beek, P., and Walsh, T. (Eds.), Handbook of Constraint Programming. Elsevier.

Beutel, A., Chen, J., Doshi, T., Qian, H., Woodruff, A., Luu, C., Kreitmann, P., Bischof, J., and Chi, E. H. (2019). Putting fairness principles into practice: Challenges, metrics, and improvements. arXiv:1901.04562.

Bhar, R. and Hamori, S. (2004). Hidden Markov Models: Applications to Financial Economics. Springer.

Bibel, W. (1993). Deduction: Automated Logic. Academic Press.

Bien, J., Tibshirani, R., et al. (2011). Prototype selection for interpretable classification. Annals of Applied Statistics, 5, 2403–2424.

Biere, A., Heule, M., van Maaren, H., and Walsh, T. (Eds.). (2009). Handbook of Satisfiability. IOS Press.

Bies, A., Mott, J., and Warner, C. (2015). English news text treebank: Penn treebank revised. Linguistic Data Consortium.

Billings, D., Burch, N., Davidson, A., Holte, R. C., Schaeffer, J., Schauenberg, T., and Szafron, D. (2003). Approximating game-theoretic optimal strategies for full-scale poker. In IJCAI-03.

Billingsley, P. (2012). Probability and Measure (4th edition). Wiley.

Binder, J., Koller, D., Russell, S. J., and Kanazawa, K. (1997a). Adaptive probabilistic networks with hidden variables. Machine Learning, 29, 213–244.

Binder, J., Murphy, K., and Russell, S. J. (1997b). Space-efficient inference in dynamic probabilistic networks. In IJCAI-97.

Bingham, E., Chen, J., Jankowiak, M., Obermeyer, F., Pradhan, N., Karaletsos, T., Singh, R., Szerlip, P., Horsfall, P., and Goodman, N. D. (2019). Pyro: Deep universal probabilistic programming. JMLR, 20, 1–26.

Binmore, K. (1982). Essays on Foundations of Game Theory. Pitman.

Biran, O. and Cotton, C. (2017). Explanation and justification in machine learning: A survey. In Proc. IJCAI-17 Workshop on Explainable AI.

Bishop, C. M. (1995). Neural Networks for Pattern Recognition. Oxford University Press.

Bishop, C. M. (2007). Pattern Recognition and Machine Learning. Springer-Verlag.

Bisson, T. (1990). They’re made out of meat. Omni Magazine.

Bistarelli, S., Montanari, U., and Rossi, F. (1997). Semiring-based constraint satisfaction and optimization. JACM, 44, 201–236.

Bitner, J. R. and Reingold, E. M. (1975). Backtrack programming techniques. CACM, 18, 651–656.

Bizer, C., Auer, S., Kobilarov, G., Lehmann, J., and Cyganiak, R. (2007). DBPedia – querying Wikipedia like a database. In 16th International Conference on World Wide Web.

Blazewicz, J., Ecker, K., Pesch, E., Schmidt, G., and Weglarz, J. (2007). Handbook on Scheduling: Models and Methods for Advanced Planning. Springer-Verlag.

Blei, D. M., Ng, A. Y., and Jordan, M. I. (2002). Latent Dirichlet allocation. In NeurIPS 14.

Bliss, C. I. (1934). The method of probits. Science, 79, 38–39.

Block, H. D., Knight, B., and Rosenblatt, F. (1962). Analysis of a four-layer series-coupled perceptron. Rev. Modern Physics, 34, 275–282.

Block, N. (2009). Comparing the major theories of consciousness. In Gazzaniga, M. S. (Ed.), The Cognitive Neurosciences. MIT Press.

Blum, A. L. and Furst, M. (1997). Fast planning through planning graph analysis. AIJ, 90, 281–300.

Blum, A. L. (1996). On-line algorithms in machine learning. In Proc. Workshop on On-Line Algorithms, Dagstuhl.

Blum, A. L., Hopcroft, J., and Kannan, R. (2020). Foundations of Data Science. Cambridge University Press.

Blum, A. L. and Mitchell, T. M. (1998). Combining labeled and unlabeled data with cotraining. In COLT-98.

Blumer, A., Ehrenfeucht, A., Haussler, D., and Warmuth, M. (1989). Learnability and the Vapnik-Chervonenkis dimension. JACM, 36, 929–965.

Bobrow, D. G. (1967). Natural language input for a computer problem solving system. In Minsky, M. L. (Ed.), Semantic Information Processing. MIT Press.

Bod, R. (2008). The data-oriented parsing approach: Theory and application. In Computational Intelligence: A Compendium. Springer-Verlag.

Bod, R., Scha, R., and Sima’an, K. (2003). Data-Oriented Parsing. CSLI Press.

Boddington, P. (2017). Towards a Code of Ethics for Artificial Intelligence. Springer-Verlag.

Boden, M. A. (Ed.). (1990). The Philosophy of Artificial Intelligence. Oxford University Press.

Bolognesi, A. and Ciancarini, P. (2003). Computer programming of kriegspiel endings: The case of KR vs. K. In Advances in Computer Games 10.

Bolton, R. J. and Hand, D. J. (2002). Statistical fraud detection: A review. Statistical science, 17, 235–249.

Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Patel, S., Ramage, D., Segal, A., and Seth, K. (2017). Practical secure aggregation for privacy-preserving machine learning. In Proc. ACM SIGSAC Conference on Computer and Communications Security.

Bond, A. H. and Gasser, L. (Eds.). (1988). Readings in Distributed Artificial Intelligence. Morgan Kaufmann.

Bonet, B. (2002). An epsilon-optimal grid-based algorithm for partially observable Markov decision processes. In ICML-02.

Bonet, B. and Geffner, H. (1999). Planning as heuristic search: New results. In ECP-99.

Bonet, B. and Geffner, H. (2000). Planning with incomplete information as heuristic search in belief space. In ICAPS-00.

Bonet, B. and Geffner, H. (2005). An algorithm better than AO*? In AAAI-05.

Boole, G. (1847). The Mathematical Analysis of Logic: Being an Essay towards a Calculus of Deductive Reasoning. Macmillan, Barclay, and Macmillan.

Booth, T. L. (1969). Probabilistic representation of formal languages. In IEEE Conference Record of the 1969 Tenth Annual Symposium on Switching and Automata Theory.

Borel, E. (1921). La théorie du jeu et les équations intégrales à noyau symétrique. Comptes Rendus Hebdomadaires des Séances de l’Académie des Sciences, 173, 1304–1308.

Borenstein, J., Everett, B., and Feng, L. (1996). Navigating Mobile Robots: Systems and Techniques. A. K. Peters, Ltd.

Borenstein, J. and Koren., Y. (1991). The vector field histogram—Fast obstacle avoidance for mobile robots. IEEE Transactions on Robotics and Automation, 7, 278–288.

Borgida, A., Brachman, R. J., McGuinness, D., and Alperin Resnick, L. (1989). CLASSIC: A structural data model for objects. SIGMOD Record, 18, 58–67.

Boroditsky, L. (2003). Linguistic relativity. In Nadel, L. (Ed.), Encyclopedia of Cognitive Science. Macmillan.

Boser, B., Guyon, I., and Vapnik, V. N. (1992). A training algorithm for optimal margin classifiers. In COLT-92.

Bosse, M., Newman, P., Leonard, J., Soika, M., Feiten, W., and Teller, S. (2004). Simultaneous localization and map building in large-scale cyclic environments using the Atlas framework. Int. J. Robotics Research, 23, 1113–1139.

Bostrom, N. (2005). A history of transhumanist thought. Journal of Evolution and Technology, 14, 1–25.

Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press.

Bottou, L. and Bousquet, O. (2008). The tradeoffs of large scale learning. In NeurIPS 20.

Bottou, L., Curtis, F. E., and Nocedal, J. (2018). Optimization methods for large-scale machine learning. SIAM Review, 60, 223–311.

Boué, L. (2019). Real numbers, data science and chaos: How to fit any dataset with a single parameter. arXiv:1904.12320.

Bousmalis, K., Irpan, A., Wohlhart, P., Bai, Y., Kelcey, M., Kalakrishnan, M., Downs, L., Ibarz, J., Pastor, P., Konolige, K., Levine, S., and Vanhoucke, V. (2017). Using simulation and domain adaptation to improve efficiency of deep robotic grasping. arXiv:1709.07857.

Boutilier, C. (2002). A POMDP formulation of preference elicitation problems. In AAAI-02.

Boutilier, C. and Brafman, R. I. (2001). Partial-order planning with concurrent interacting actions. JAIR, 14, 105–136.

Boutilier, C., Dearden, R., and Goldszmidt, M. (2000). Stochastic dynamic programming with factored representations. AIJ, 121, 49–107.

Boutilier, C., Reiter, R., and Price, B. (2001). Symbolic dynamic programming for first-order MDPs. In IJCAI-01.

Boutilier, C., Brafman, R. I., Domshlak, C., Hoos, H. H., and Poole, D. (2004). CP-nets: A tool for representing and reasoning with conditional ceteris paribus preference statements. JAIR, 21, 135–191.

Boutilier, C., Friedman, N., Goldszmidt, M., and Koller, D. (1996). Context-specific independence in Bayesian networks. In UAI-96.

Bouzy, B. and Cazenave, T. (2001). Computer Go: An AI oriented survey. AIJ, 132, 39–103.

Bowling, M., Burch, N., Johanson, M., and Tammelin, O. (2015). Heads-up limit hold’em poker is solved. Science, 347, 145–149.

Bowling, M., Johanson, M., Burch, N., and Szafron, D. (2008). Strategy evaluation in extensive games with importance sampling. In ICML-08.

Bowman, S., Angeli, G., Potts, C., and Manning, C. (2015). A large annotated corpus for learning natural language inference. In EMNLP-15.

Box, G. E. P. (1957). Evolutionary operation: A method of increasing industrial productivity. Applied Statistics, 6, 81–101.

Box, G. E. P., Jenkins, G., Reinsel, G., and Ljung, G. M. (2016). Time Series Analysis: Forecasting and Control (5th edition). Wiley.

Box, G. E. P. and Tiao, G. C. (1973). Bayesian Inference in Statistical Analysis. AddisonWesley.

Boyan, J. A. and Moore, A. W. (1998). Learning evaluation functions for global optimization and Boolean satisfiability. In AAAI-98.

Boyd, S. and Vandenberghe, L. (2004). Convex Optimization. Cambridge University Press.

Boyen, X., Friedman, N., and Koller, D. (1999). Discovering the hidden structure of complex dynamic systems. In UAI-99.

Boyer, R. S. and Moore, J. S. (1979). A Computational Logic. Academic Press.

Boyer, R. S. and Moore, J. S. (1984). Proof checking the RSA public key encryption algorithm. American Mathematical Monthly, 91, 181–189.

Brachman, R. J. (1979). On the epistemological status of semantic networks. In Findler, N. V. (Ed.), Associative Networks: Representation and Use of Knowledge by Computers. Academic Press.

Brachman, R. J. and Levesque, H. J. (Eds.). (1985). Readings in Knowledge Representation. Morgan Kaufmann.

Bradt, R. N., Johnson, S. M., and Karlin, S. (1956). On sequential designs for maximizing the sum of n observations. Ann. Math. Statist., 27, 1060–1074.

Brafman, O. and Brafman, R. (2009). Sway: The Irresistible Pull of Irrational Behavior. Broadway Business.

Brafman, R. I. and Domshlak, C. (2008). From one to many: Planning for loosely coupled multi-agent systems. In ICAPS-08.

Brafman, R. I. and Tennenholtz, M. (2000). A near optimal polynomial time algorithm for learning in certain classes of stochastic games. AIJ, 121, 31–47.

Braitenberg, V. (1984). Vehicles: Experiments in Synthetic Psychology. MIT Press.

Brandt, F., Conitzer, V., Endriss, U., Lang, J., and Procaccia, A. D. (Eds.). (2016). Handbook of Computational Social Choice. Cambridge University Press.

Brants, T. (2000). TnT: A statistical part-of-speech tagger. In Proc. Sixth Conference on Applied Natural Language Processing.

Brants, T., Popat, A. C., Xu, P., Och, F. J., and Dean, J. (2007). Large language models in machine translation. In EMNLP-CoNLL-07.

Bratko, I. (2009). Prolog Programming for Artificial Intelligence (4th edition). Addison-Wesley.

Bratman, M. E. (1987). Intention, Plans, and Practical Reason. Harvard University Press.

Breck, E., Cai, S., Nielsen, E., Salib, M., and Sculley, D. (2016). What’s your ML test score? A rubric for ML production systems. In Proc. NIPS 2016 Workshop on Reliable Machine Learning in the Wild.

Breese, J. S. (1992). Construction of belief and decision networks. Computational Intelligence, 8, 624–647.

Breese, J. S. and Heckerman, D. (1996). Decision-theoretic troubleshooting: A framework for repair and experiment. In UAI-96.

Breiman, L., Friedman, J., Olshen, R. A., and Stone, C. J. (1984). Classification and Regression Trees. Wadsworth International Group.

Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32.

Brelaz, D. (1979). New methods to color the vertices of a graph. CACM, 22, 251–256.

Brent, R. P. (1973). Algorithms for Minimization without Derivatives. Prentice-Hall.

Bresnan, J. (1982). The Mental Representation of Grammatical Relations. MIT Press.

Brewka, G., Dix, J., and Konolige, K. (1997). Nononotonic Reasoning: An Overview. Center for the Study of Language and Information (CSLI).

Brickley, D. and Guha, R. V. (2004). RDF vocabulary description language 1.0: RDF schema. Tech. rep., W3C.

Briggs, R. (1985). Knowledge representation in Sanskrit and artificial intelligence. AIMag, 6, 32–39.

Brill, E. (1992). A simple rule-based part of speech tagger. In Proc. Third Conference on Applied Natural Language Processing.

Brin, D. (1998). The Transparent Society. Perseus.

Brin, S. and Page, L. (1998). The anatomy of a large-scale hypertextual web search engine. In Proc. Seventh World Wide Web Conference.

Bringsjord, S. (2008). If I were judge. In Epstein, R., Roberts, G., and Beber, G. (Eds.), Parsing the Turing Test. Springer.

Broadbent, D. E. (1958). Perception and Communication. Pergamon.

Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). OpenAI gym. arXiv:1606.01540.

Brooks, R. A. (1986). A robust layered control system for a mobile robot. IEEE J. of Robotics and Automation, 2, 14–23.

Brooks, R. A. (1989). Engineering approach to building complete, intelligent beings. Proc. SPIE—the International Society for Optical Engineering, 1002, 618–625.

Brooks, R. A. (1991). Intelligence without representation. AIJ, 47, 139–159.

Brooks, R. A. and Lozano-Perez, T. (1985). A subdivision algorithm in configuration space for findpath with rotation. IEEE Transactions on Systems, Man and Cybernetics, 15, 224– 233.

Brooks, R. A. (2017). The seven deadly sins of AI predictions. MIT Technology Review, Oct 6.

Brooks, S., Gelman, A., Jones, G., and Meng, X.-L. (2011). Handbook of Markov Chain Monte Carlo. Chapman & Hall/CRC.

Brown, C., Finkelstein, L., and Purdom, P. (1988). Backtrack searching in the presence of symmetry. In Mora, T. (Ed.), Applied Algebra, Algebraic Algorithms and Error-Correcting Codes. Springer-Verlag.

Brown, K. C. (1974). A note on the apparent bias of net revenue estimates. J. Finance, 29, 1215–1216.

Brown, N. and Sandholm, T. (2017). Libratus: The superhuman AI for no-limit poker. In IJCAI-17.

Brown, N. and Sandholm, T. (2019). Superhuman AI for multiplayer poker. Science, 365, 885–890.

Brown, P. F., Cocke, J., Della Pietra, S. A., Della Pietra, V. J., Jelinek, F., Mercer, R. L., and Roossin, P. (1988). A statistical approach to language translation. In COLING-88.

Brown, P. F., Desouza, P. V., Mercer, R. L., Pietra, V. J. D., and Lai, J. C. (1992). Class-based n-gram models of natural language. Computational linguistics, 18(4).

Browne, C., Powley, E. J., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., Tavener, S., Liebana, D. P., Samothrakis, S., and Colton, S. (2012). A survey of Monte

Carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in Games, 4, 1–43.

Brownston, L., Farrell, R., Kant, E., and Martin, N. (1985). Programming Expert Systems in OPS5: An Introduction to Rule-Based Programming. Addison-Wesley.

Bruce, V., Green, P., and Georgeson, M. (2003). Visual Perception: Physiology, Psychology and Ecology. Routledge and Kegan Paul.

Brügmann, B. (1993). Monte Carlo Go. Tech. rep., Department of Physics, Syracuse University.

Bryce, D. and Kambhampati, S. (2007). A tutorial on planning graph-based reachability heuristics. AIMag, Spring, 47–83.

Bryce, D., Kambhampati, S., and Smith, D. E. (2006). Planning graph heuristics for belief space search. JAIR, 26, 35–99.

Brynjolfsson, E. and McAfee, A. (2011). Race Against the Machine. Digital Frontier Press.

Brynjolfsson, E. and McAfee, A. (2014). The Second Machine Age. W. W. Norton.

Brynjolfsson, E., Rock, D., and Syverson, C. (2018). Artificial intelligence and the modern productivity paradox: A clash of expectations and statistics. In Agrawal, A., Gans, J., and Goldfarb, A. (Eds.), The Economics of Artificial Intelligence: An Agenda. University of Chicago Press.

Bryson, A. E. and Ho, Y.-C. (1969). Applied Optimal Control. Blaisdell.

Bryson, A. E. (1962). A gradient method for optimizing multi-stage allocation processes. In Proc. of a Harvard Symposium on Digital Computers and Their Applications.

Bryson, J. J. (2012). A role for consciousness in action selection. International Journal of Machine Consciousness, 4, 471–482.

Bryson, J. J. and Winfield, A. (2017). Standardizing ethical design for artificial intelligence and autonomous systems. Computer, 50, 116–119.

Buchanan, B. G., Mitchell, T. M., Smith, R. G., and Johnson, C. R. (1978). Models of learning systems. In Encyclopedia of Computer Science and Technology, Vol. 11. Dekker.

Buchanan, B. G. and Shortliffe, E. H. (Eds.). (1984). Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project. Addison-Wesley.

Buchanan, B. G., Sutherland, G. L., and Feigenbaum, E. A. (1969). Heuristic DENDRAL: A program for generating explanatory hypotheses in organic chemistry. In Meltzer, B., Michie, D., and Swann, M. (Eds.), Machine Intelligence 4. Edinburgh University Press.

Buck, C., Heafield, K., and Van Ooyen, B. (2014). N-gram counts and language models from the common crawl. In Proc. International Conference on Language Resources and Evaluation.

Buehler, M., Iagnemma, K., and Singh, S. (Eds.). (2006). The 2005 DARPA Grand Challenge: The Great Robot Race. Springer-Verlag.

Buffon, G. (1777). Essai d’arithmetique morale. Supplement to Histoire naturelle, vol. IV.

Bunt, H. C. (1985). The formal representation of (quasi-) continuous concepts. In Hobbs, J. R. and Moore, R. C. (Eds.), Formal Theories of the Commonsense World. Ablex.

Buolamwini, J. and Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency.

Burgard, W., Cremers, A. B., Fox, D., Hahnel, D., Lakemeyer, G., Schulz, D., Steiner, W., and Thrun, S. (1999). Experiences with an interactive museum tour-guide robot. AIJ, 114, 3–55.

Burkov, A. (2019). The Hundred-Page Machine Learning Book. Burkov.

Burns, E., Hatem, M., Leighton, M. J., and Ruml, W. (2012). Implementing fast heuristic search code. In Symposium on Combinatorial Search.

Buro, M. (1995). ProbCut: An effective selective extension of the alpha-beta algorithm. J. International Computer Chess Association, 18, 71–76.

Buro, M. (2002). Improving heuristic mini-max search by supervised learning. AIJ, 134, 85– 99.

Burstein, J., Leacock, C., and Swartz, R. (2001). Automated evaluation of essays and short answers. In Fifth International Computer Assisted Assessment Conference.

Burton, R. (2009). On Being Certain: Believing You Are Right Even When You’re Not. St. Martin’s Griffin.

Buss, D. M. (2005). Handbook of Evolutionary Psychology. Wiley.

Butler, S. (1863). Darwin among the machines. The Press (Christchurch, New Zealand), June 13.

Bylander, T. (1994). The computational complexity of propositional STRIPS planning. AIJ, 69, 165–204.

Byrd, R. H., Lu, P., Nocedal, J., and Zhu, C. (1995). A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific and Statistical Computing, 16, 1190–1208.

Cabeza, R. and Nyberg, L. (2001). Imaging cognition II: An empirical review of 275 PET and fMRI studies. J. Cognitive Neuroscience, 12, 1–47.

Cafarella, M. J., Halevy, A., Zhang, Y., Wang, D. Z., and Wu, E. (2008). Webtables: Exploring the power of tables on the web. In VLDB-08.

Calvanese, D., Lenzerini, M., and Nardi, D. (1999). Unifying class-based representation formalisms. JAIR, 11, 199–240.

Camacho, R. and Michie, D. (1995). Behavioral cloning: A correction. AIMag, 16, 92.

Campbell, D. E. and Kelly, J. (2002). Impossibility theorems in the Arrovian framework. In Arrow, K. J., Sen, A. K., and Suzumura, K. (Eds.), Handbook of Social Choice and Welfare Volume 1. Elsevier Science.

Campbell, M. S., Hoane, A. J., and Hsu, F.-H. (2002). Deep Blue. AIJ, 134, 57–83.

Cannings, C., Thompson, E., and Skolnick, M. H. (1978). Probability functions on complex pedigrees. Advances in Applied Probability, 10, 26–61.

Canny, J. and Reif, J. (1987). New lower bound techniques for robot motion planning problems. In FOCS-87.

Canny, J. (1986). A computational approach to edge detection. PAMI, 8, 679–698.

Canny, J. (1988). The Complexity of Robot Motion Planning. MIT Press.

Capen, E., Clapp, R., and Campbell, W. (1971). Competitive bidding in high-risk situations. J. Petroleum Technology, 23, 641–653.

Carbonell, J. G. (1983). Derivational analogy and its role in problem solving. In AAAI-83.

Carbonell, J. G., Knoblock, C. A., and Minton, S. (1989). PRODIGY: An integrated architecture for planning and learning. Technical report, Computer Science Department, Carnegie-Mellon University.

Carbonnel, C. and Cooper, M. C. (2016). Tractability in constraint satisfaction problems: A survey. Constraints, 21(2), 115–144.

Cardano, G. (1663). Liber de ludo aleae. Lyons.

Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, D., Goodfellow, I., Madry, A., and Kurakin, A. (2019). On evaluating adversarial robustness. arXiv:1902.06705.

Carnap, R. (1928). Der logische Aufbau der Welt. Weltkreis-verlag. Translated into English as The Logical Structure of the World (Carnap, 1967).

Carnap, R. (1948). On the application of inductive logic. Philosophy and Phenomenological Research, 8, 133–148.

Carnap, R. (1950). Logical Foundations of Probability. University of Chicago Press.

Carpenter, B., Gelman, A., Hoffman, M., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., and Riddell, A. (2017). Stan: A probabilistic programming language. Journal of Statistical Software, 76, 1–32.

Carroll, S. (2007). The Making of the Fittest: DNA and the Ultimate Forensic Record of Evolution. Norton.

Casati, R. and Varzi, A. (1999). Parts and Places: The Structures of Spatial Representation. MIT Press.

Cassandra, A. R., Kaelbling, L. P., and Littman, M. L. (1994). Acting optimally in partially observable stochastic domains. In AAAI-94.

Cassandras, C. G. and Lygeros, J. (2006). Stochastic Hybrid Systems. CRC Press.

Castro, R., Coates, M., Liang, G., Nowak, R., and Yu, B. (2004). Network tomography: Recent developments. Statistical Science, 19, 499–517.

Cauchy, A. (1847). Méthode générale pour la résolution des systèmes d’équations simultanées. Comp. Rend. Sci. Paris, 25, 536–538.

Cesa-Bianchi, N. and Lugosi, G. (2006). Prediction, Learning, and Games. Cambridge University Press.

Chajewska, U., Koller, D., and Parr, R. (2000). Making rational decisions using adaptive utility elicitation. In AAAI-00.

Chakrabarti, P. P., Ghose, S., Acharya, A., and de Sarkar, S. C. (1989). Heuristic search in restricted memory. AIJ, 41, 197–222.

Chalkiadakis, G., Elkind, E., and Wooldridge, M. (2011). Computational Aspects of Cooperative Game Theory. Morgan Kaufmann.

Chalmers, D. J. (1992). Subsymbolic computation and the Chinese room. In Dinsmore, J. (Ed.), The symbolic and connectionist paradigms: Closing the gap. Lawrence Erlbaum.

Chandola, V., Banerjee, A., and Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys, 41.

Chandra, A. K. and Harel, D. (1980). Computable queries for relational data bases. J. Computer and System Sciences, 21, 156–178.

Chang, C.-L. and Lee, R. C.-T. (1973). Symbolic Logic and Mechanical Theorem Proving. Academic Press.

Chang, H. S., Fu, M. C., Hu, J., and Marcus, S. I. (2005). An adaptive sampling algorithm for solving Markov decision processes. Operations Research, 53, 126–139.

Chao, W.-L., Hu, H., and Sha, F. (2018). Being negative but constructively: Lessons learnt from creating better visual question answering datasets. In ACL-18.

Chapman, D. (1987). Planning for conjunctive goals. AIJ, 32, 333–377.

Charniak, E. (1993). Statistical Language Learning. MIT Press.

Charniak, E. (1996). Tree-bank grammars. In AAAI-96.

Charniak, E. (1997). Statistical parsing with a context-free grammar and word statistics. In AAAI-97.

Charniak, E. and Goldman, R. (1992). A Bayesian model of plan recognition. AIJ, 64, 53–79.

Charniak, E., Riesbeck, C., McDermott, D., and Meehan, J. (1987). Artificial Intelligence Programming (2nd edition). Lawrence Erlbaum.

Charniak, E. (1991). Bayesian networks without tears. AIMag, 12, 50–63.

Charniak, E. (2018). Introduction to Deep Learning. MIT Press.

Chaslot, G., Bakkes, S., Szita, I., and Spronck, P. (2008). Monte-Carlo tree search: A new framework for game AI. In Proc. Fourth Artificial Intelligence and Interactive Digital Entertainment Conference.

Chater, N. and Oaksford, M. (Eds.). (2008). The Probabilistic Mind: Prospects for Bayesian Cognitive Science. Oxford University Press.

Chatfield, C. (1989). The Analysis of Time Series: An Introduction (4th edition). Chapman and Hall.

Chavira, M. and Darwiche, A. (2008). On probabilistic inference by weighted model counting. AIJ, 172, 772–799.

Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P. (2002). SMOTE: Synthetic minority over-sampling technique. JAIR, 16, 321–357.

Cheeseman, P. (1985). In defense of probability. In IJCAI-85.

Cheeseman, P. (1988). An inquiry into computer understanding. Computational Intelligence, 4, 58–66.

Cheeseman, P., Kanefsky, B., and Taylor, W. (1991). Where the really hard problems are. In IJCAI-91.

Cheeseman, P., Self, M., Kelly, J., and Stutz, J. (1988). Bayesian classification. In AAAI-88.

Cheeseman, P. and Stutz, J. (1996). Bayesian classification (AutoClass): Theory and results. In Fayyad, U., Piatesky-Shapiro, G., Smyth, P., and Uthurusamy, R. (Eds.), Advances in Knowledge Discovery and Data Mining. AAAI Press/MIT Press.

Chen, D. and Manning, C. (2014). A fast and accurate dependency parser using neural networks. In EMNLP-14.

Chen, J., Holte, R. C., Zilles, S., and Sturtevant, N. R. (2017). Front-to-end bidirectional heuristic search with near-optimal node expansions. IJCAI-17.

Chen, M. X., Firat, O., Bapna, A., Johnson, M., Macherey, W., Foster, G., Jones, L., Parmar, N., Schuster, M., Chen, Z., Wu, Y., and Hughes, M. (2018). The best of both worlds: Combining recent advances in neural machine translation. In ACL-18.

Chen, S. F. and Goodman, J. (1996). An empirical study of smoothing techniques for language modeling. In ACL-96.

Chen, T. and Guestrin, C. (2016). XGBoost: A scalable tree boosting system. In KDD-16.

Cheng, J. and Druzdzel, M. J. (2000). AIS-BN: An adaptive importance sampling algorithm for evidential reasoning in large Bayesian networks. JAIR, 13, 155–188.

Cheng, J., Greiner, R., Kelly, J., Bell, D. A., and Liu, W. (2002). Learning Bayesian networks from data: An information-theory based approach. AIJ, 137, 43–90.

Chiu, C., Sainath, T., Wu, Y., Prabhavalkar, R., Nguyen, P., Chen, Z., Kannan, A., Weiss, R., Rao, K., Gonina, K., Jaitly, N., Li, B., Chorowski, J., and Bacchiani, M. (2017). State-ofthe-art speech recognition with sequence-to-sequence models. arXiv:1712.01769.

Chklovski, T. and Gil, Y. (2005). Improving the design of intelligent acquisition interfaces for collecting world knowledge from web contributors. In Proc. Third International Conference on Knowledge Capture.

Chollet, F. (2019). On the measure of intelligence. arXiv:1911.01547.

Chollet, F. (2017). Deep Learning with Python. Manning.

Chomsky, N. (1956). Three models for the description of language. IRE Transactions on Information Theory, 2, 113–124.

Chomsky, N. (1957). Syntactic Structures. Mouton.

Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B., and LeCun, Y. (2014). The loss surface of multilayer networks. arXiv:1412.0233.

Choset, H. (1996). Sensor Based Motion Planning: The Hierarchical Generalized Voronoi Graph. Ph.D. thesis, California Institute of Technology.

Choset, H., Hutchinson, S., Lynch, K., Kantor, G., Burgard, W., Kavraki, L., and Thrun, S. (2005). Principles of Robot Motion: Theory, Algorithms, and Implementation. MIT Press.

Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5, 153–163.

Chouldechova, A. and Roth, A. (2018). The frontiers of fairness in machine learning. arXiv:1810.08810.

Christian, B. (2011). The Most Human Human. Doubleday.

Christin, A., Rosenblat, A., and Boyd, D. (2015). Courts and predictive algorithms. Data & Civil Rights.

Chung, K. L. (1979). Elementary Probability Theory with Stochastic Processes (3rd edition). Springer-Verlag.

Church, A. (1936). A note on the Entscheidungsproblem. JSL, 1, 40–41 and 101–102.

Church, A. (1956). Introduction to Mathematical Logic. Princeton University Press.

Church, K. (1988). A stochastic parts program and noun phrase parser for unrestricted texts. In Proc. Second Conference on Applied Natural Language Processing.

Church, K. and Patil, R. (1982). Coping with syntactic ambiguity or how to put the block in the box on the table. Computational Linguistics, 8, 139–149.

Church, K. (2004). Speech and language processing: Can we use the past to predict the future. In Proc. Conference on Text, Speech, and Dialogue.

Church, K. and Gale, W. A. (1991). A comparison of the enhanced Good–Turing and deleted estimation methods for estimating probabilities of English bigrams. Computer Speech and Language, 5, 19–54.

Church, K. and Hestness, J. (2019). A survey of 25 years of evaluation. Natural Language Engineering, 25, 753–767.

Churchland, P. M. (2013). Matter and Consciousness (3rd edition). MIT Press.

Ciancarini, P. and Favini, G. P. (2010). Monte Carlo tree search in Kriegspiel. AIJ, 174, 670– 684.

Ciancarini, P. and Wooldridge, M. (2001). Agent-Oriented Software Engineering. SpringerVerlag.

Cimatti, A., Roveri, M., and Traverso, P. (1998). Automatic OBDD-based generation of universal plans in non-deterministic domains. In AAAI-98.

Claret, G., Rajamani, S. K., Nori, A. V., Gordon, A. D., and Borgström, J. (2013). Bayesian inference using data flow analysis. In Proc. 9th Joint Meeting on Foundations of Software Engineering.

Clark, A. (1998). Being There: Putting Brain, Body, and World Together Again. MIT Press.

Clark, A. (2015). Surfing Uncertainty: Prediction, Action, and the Embodied Mind. Oxford University Press.

Clark, K. L. (1978). Negation as failure. In Gallaire, H. and Minker, J. (Eds.), Logic and Data Bases. Plenum.

Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. (2018). Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv:1803.05457.

Clark, P., Etzioni, O., Khot, T., Mishra, B. D., Richardson, K., et al. (2019). From ‘F’ to ‘A’ on the NY Regents science exams: An overview of the Aristo project. arXiv:1909.01958.

Clark, S. and Curran, J. R. (2004). Parsing the WSJ using CCG and log-linear models. In ACL-04.

Clarke, A. C. (1968). 2001: A Space Odyssey. Signet.

Clarke, E. and Grumberg, O. (1987). Research on automatic verification of finite-state concurrent systems. Annual Review of Computer Science, 2, 269–290.

Clearwater, S. H. (Ed.). (1996). Market-Based Control. World Scientific.

Clocksin, W. F. and Mellish, C. S. (2003). Programming in Prolog (5th edition). SpringerVerlag.

Clocksin, W. F. (2003). Clause and Effect: Prolog Programming for the Working Programmer. Springer.

Coase, R. H. (1960). The problem of social cost. Journal of Law and Economics, pp. 1–44.

Coates, A., Abbeel, P., and Ng, A. Y. (2009). Apprenticeship learning for helicopter control. Association for Computing Machinery, 52(7).

Cobham, A. (1964). The intrinsic computational difficulty of functions. In Proc. International Congress for Logic, Methodology, and Philosophy of Science.

Cohen, P. R. (1995). Empirical Methods for Artificial Intelligence. MIT Press.

Cohen, P. R. and Levesque, H. J. (1990). Intention is choice with commitment. AIJ, 42, 213– 261.

Cohen, P. R., Morgan, J., and Pollack, M. E. (1990). Intentions in Communication. MIT Press.

Cohen, P. R. and Perrault, C. R. (1979). Elements of a plan-based theory of speech acts. Cognitive Science, 3, 177–212.

Cohn, A. G., Bennett, B., Gooday, J. M., and Gotts, N. (1997). RCC: A calculus for region based qualitative spatial reasoning. GeoInformatica, 1, 275–316.

Collin, Z., Dechter, R., and Katz, S. (1999). Self-stabilizing distributed constraint satisfaction. Chicago J. of Theoretical Computer Science, 1999.

Collins, M. (1999). Head-driven Statistical Models for Natural Language Processing. Ph.D. thesis, University of Pennsylvania.

Collins, M. and Duffy, K. (2002). New ranking algorithms for parsing and tagging: Kernels over discrete structures, and the voted perceptron. In ACL-02.

Colmerauer, A. and Roussel, P. (1993). The birth of Prolog. SIGPLAN Notices, 28, 37–52.

Colmerauer, A., Kanoui, H., Pasero, R., and Roussel, P. (1973). Un système de communication homme–machine en Français. Rapport, Groupe d’Intelligence Artificielle, Université d’Aix-Marseille II.

Condon, J. H. and Thompson, K. (1982). Belle chess hardware. In Clarke, M. R. B. (Ed.), Advances in Computer Chess 3. Pergamon.

Congdon, C. B., Huber, M., Kortenkamp, D., Bidlack, C., Cohen, C., Huffman, S., Koss, F., Raschke, U., and Weymouth, T. (1992). CARMEL versus Flakey: A comparison of two robots. Tech. rep., American Association for Artificial Intelligence.

Conlisk, J. (1989). Three variants on the Allais example. American Economic Review, 79, 392– 407.

Connell, J. (1989). A Colony Architecture for an Artificial Creature. Ph.D. thesis, Artificial Intelligence Laboratory, MIT.

Conway, D. and White, J. (2012). Machine Learning for Hackers. O’Reilly.

Cook, S. A. (1971). The complexity of theorem-proving procedures. In STOC-71.

Cook, S. A. and Mitchell, D. (1997). Finding hard instances of the satisfiability problem: A survey. In Du, D., Gu, J., and Pardalos, P. (Eds.), Satisfiability problems: Theory and applications. American Mathematical Society.

Cooper, G. (1990). The computational complexity of probabilistic inference using Bayesian belief networks. AIJ, 42, 393–405.

Cooper, G. and Herskovits, E. (1992). A Bayesian method for the induction of probabilistic networks from data. Machine Learning, 9, 309–347.

Copeland, J. (1993). Artificial Intelligence: A Philosophical Introduction. Blackwell.

Corbett-Davies, S. and Goel, S. (2018). The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv:1808.00023.

Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., and Huq, A. (2017). Algorithmic decision making and the cost of fairness. arXiv:1701.08230.

Cormen, T. H., Leiserson, C. E., Rivest, R., and Stein, C. (2009). Introduction to Algorithms (3rd edition). MIT Press.

Cortes, C. and Vapnik, V. N. (1995). Support vector networks. Machine Learning, 20, 273– 297.

Cournot, A. (Ed.). (1838). Recherches sur les principes mathématiques de la théorie des richesses. L. Hachette, Paris.

Cover, T. and Thomas, J. (2006). Elements of Information Theory (2nd edition). Wiley.

Cowan, J. D. and Sharp, D. H. (1988a). Neural nets. Quarterly Reviews of Biophysics, 21, 365– 427.

Cowan, J. D. and Sharp, D. H. (1988b). Neural nets and artificial intelligence. Daedalus, 117, 85–121.

Cowell, R., Dawid, A. P., Lauritzen, S., and Spiegelhalter, D. J. (2002). Probabilistic Networks and Expert Systems. Springer.

Cox, I. (1993). A review of statistical data association techniques for motion correspondence. IJCV, 10, 53–66.

Cox, I. and Hingorani, S. L. (1994). An efficient implementation and evaluation of Reid’s multiple hypothesis tracking algorithm for visual tracking. In ICPR-94.

Cox, I. and Wilfong, G. T. (Eds.). (1990). Autonomous Robot Vehicles. Springer Verlag.

Cox, R. T. (1946). Probability, frequency, and reasonable expectation. American Journal of Physics, 14, 1–13.

Craig, J. (1989). Introduction to Robotics: Mechanics and Control (2nd edition). Addison-Wesley.

Craik, K. (1943). The Nature of Explanation. Cambridge University Press.

Cramton, P., Shoham, Y., and Steinberg, R. (Eds.). (2006). Combinatorial Auctions. MIT Press.

Craven, M., DiPasquo, D., Freitag, D., McCallum, A., Mitchell, T. M., Nigam, K., and Slattery, S. (2000). Learning to construct knowledge bases from the World Wide Web. AIJ, 118, 69–113.

Crawford, J. M. and Auton, L. D. (1993). Experimental results on the crossover point in satisfiability problems. In AAAI-93.

Crick, F. (1999). The impact of molecular biology on neuroscience. Phil. Trans. Roy. Soc., B, 354, 2021–2025.

Crick, F. and Koch, C. (2003). A framework for consciousness. Nature Neuroscience, 6, 119.

Crisan, D. and Doucet, A. (2002). A survey of convergence results on particle filtering methods for practitioners. IEEE Trans. Signal Processing, 50, 736–746.

Cristianini, N. and Hahn, M. (2007). Introduction to Computational Genomics: A Case Studies Approach. Cambridge University Press.

Cristianini, N. and Schölkopf, B. (2002). Support vector machines and kernel methods: The new generation of learning machines. AIMag, 23, 31–41.

Cristianini, N. and Shawe-Taylor, J. (2000). An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge University Press.

Crockett, L. (1994). The Turing Test and the Frame Problem: AI’s Mistaken Understanding of Intelligence. Ablex.

Croft, W. B., Metzler, D., and Strohman, T. (2010). Search Engines: Information Retrieval in Practice. Addison-Wesley.

Cross, S. E. and Walker, E. (1994). DART: Applying knowledge based planning and scheduling to crisis action planning. In Zweben, M. and Fox, M. S. (Eds.), Intelligent Scheduling. Morgan Kaufmann.

Cruse, A. (2011). Meaning in Language: An Introduction to Semantics and Pragmatics. Oxford University Press.

Culberson, J. and Schaeffer, J. (1996). Searching with pattern databases. In Advances in Artificial Intelligence (Lecture Notes in Artificial Intelligence 1081). Springer-Verlag.

Culberson, J. and Schaeffer, J. (1998). Pattern databases. Computational Intelligence, 14, 318– 334.

Cummins, D. and Allen, C. (1998). The Evolution of Mind. Oxford University Press.

Cushing, W., Kambhampati, S., Mausam, and Weld, D. S. (2007). When is temporal planning really temporal? In IJCAI-07.

Cusumano-Towner, M. F., Saad, F., Lew, A. K., and Mansinghka, V. K. (2019). Gen: A general-purpose probabilistic programming system with programmable inference. In PLDI-19.

Cybenko, G. (1988). Continuous valued neural networks with two hidden layers are sufficient. Technical report, Department of Computer Science, Tufts University.

Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of Controls, Signals, and Systems, 2, 303–314.

Cyert, R. and de Groot, M. (1979). Adaptive utility. In Allais, M. and Hagen, O. (Eds.), Expected Utility Hypothesis and the Allais Paradox. D. Reidel.

Dagan, I., Glickman, O., and Magnini, B. (2005). The PASCAL recognising textual entailment challenge. In Machine Learning Challenges Workshop.

Daganzo, C. (1979). Multinomial Probit: The Theory and Its Application to Demand Forecasting. Academic Press.

Dagum, P. and Luby, M. (1993). Approximating probabilistic inference in Bayesian belief networks is NP-hard. AIJ, 60, 141–153.

Dagum, P. and Luby, M. (1997). An optimal approximation algorithm for Bayesian inference. AIJ, 93, 1–27.

Dai, A. M. and Le, Q. V. (2016). Semi-supervised sequence learning. In NeurIPS 28.

Dalal, N. and Triggs, B. (2005). Histograms of oriented gradients for human detection. In CVPR-05.

Dalvi, N. N., Ré, C., and Suciu, D. (2009). Probabilistic databases. CACM, 52, 86–94.

Daly, R., Shen, Q., and Aitken, S. (2011). Learning Bayesian networks: Approaches and issues. Knowledge Engineering Review, 26, 99–157.

Damasio, A. R. (1999). The Feeling of What Happens: Body and Emotion in the Making of Consciousness. Houghton Mifflin.

Danaher, J. and McArthur, N. (2017). Robot Sex: Social and Ethical Implications. MIT Press.

Dantzig, G. B. (1949). Programming of interdependent activities: II. Mathematical model. Econometrica, 17, 200–211.

Darwiche, A. (2001). Recursive conditioning. AIJ, 126, 5–41.

Darwiche, A. and Ginsberg, M. L. (1992). A symbolic generalization of probability theory. In AAAI-92.

Darwiche, A. (2009). Modeling and reasoning with Bayesian networks. Cambridge University Press.

Darwin, C. (1859). On The Origin of Species by Means of Natural Selection. J. Murray.

Dasgupta, P., Chakrabarti, P. P., and de Sarkar, S. C. (1994). Agent searching in a tree and the optimality of iterative deepening. AIJ, 71, 195–208.

Dasgupta, P. and Maskin, E. (2008). On the robustness of majority rule. Journal of the European Economic Association, 6, 949–973.

Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., and Bengio, Y. (2015). Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In NeurIPS 27.

Davidson, D. (1980). Essays on Actions and Events. Oxford University Press.

Davidson, D. (1986). A nice derangement of epitaphs. Philosophical Grounds of Rationality, 4, 157–174.

Davis, E. (1986). Representing and Acquiring Geographic Knowledge. Pitman and Morgan Kaufmann.

Davis, E. (1990). Representations of Commonsense Knowledge. Morgan Kaufmann.

Davis, E. (2005). Knowledge and communication: A first-order theory. AIJ, 166, 81–140.

Davis, E. (2006). The expressivity of quantifying over regions. J. Logic and Computation, 16, 891–916.

Davis, E. (2007). Physical reasoning. In van Harmelan, F., Lifschitz, V., and Porter, B. (Eds.), The Handbook of Knowledge Representation. Elsevier.

Davis, E. (2008). Pouring liquids: A study in commonsense physical reasoning. AIJ, 172.

Davis, E. (2017). Logical formalizations of commonsense reasoning: A survey. JAIR, 59, 651–723.

Davis, E. and Morgenstern, L. (2004). Introduction: Progress in formal commonsense reasoning. AIJ, 153, 1–12.

Davis, E. and Morgenstern, L. (2005). A first-order theory of communication and multiagent plans. J. Logic and Computation, 15, 701–749.

Davis, M. (1957). A computer program for Presburger’s algorithm. In Proving Theorems (as Done by Man, Logician, or Machine). Proc. Summer Institute for Symbolic Logic. Second edition; publication date is 1960.

Davis, M., Logemann, G., and Loveland, D. (1962). A machine program for theoremproving. CACM, 5, 394–397.

Davis, M. and Putnam, H. (1960). A computing procedure for quantification theory. JACM, 7, 201–215.

Dayan, P. (1992). The convergence of TD(λ) for general λ. Machine Learning, 8, 341–362.

Dayan, P. and Abbott, L. F. (2001). Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems. MIT Press.

Dayan, P. and Hinton, G. E. (1993). Feudal reinforcement learning. In NeurIPS 5.

Dayan, P. and Niv, Y. (2008). Reinforcement learning and the brain: The good, the bad and the ugly. Current Opinion in Neurobiology, 18, 185–196.

de Condorcet, M. (1785). Essay on the Application of Analysis to the Probability of Majority Decisions. Imprimerie Royale.

de Dombal, F. T., Leaper, D. J., Horrocks, J. C., and Staniland, J. R. (1974). Human and computer-aided diagnosis of abdominal pain: Further report with emphasis on performance of clinicians. British Medical Journal, 1, 376–380.

de Dombal, F. T., Staniland, J. R., and Clamp, S. E. (1981). Geographical variation in disease presentation. Medical Decision Making, 1, 59–69.

de Farias, D. P. and Roy, B. V. (2003). The linear programming approach to approximate dynamic programming. Operations Research, 51, 839–1016.

de Finetti, B. (1937). Le prévision: ses lois logiques, ses sources subjectives. Ann. Inst. Poincaré, 7, 1–68.

de Finetti, B. (1993). On the subjective meaning of probability. In Monari, P. and Cocchi, D. (Eds.), Probabilita e Induzione. Clueb.

de Freitas, J. F. G., Niranjan, M., and Gee, A. H. (2000). Sequential Monte Carlo methods to train neural network models. Neural Computation, 12, 933–953.

de Ghellinck, G. (1960). Les problèmes de décisions séquentielles. Cahiers du Centre d’Études de Recherche Opérationnelle, 2, 161–179.

de Kleer, J. (1975). Qualitative and quantitative knowledge in classical mechanics. Tech. rep., MIT Artificial Intelligence Laboratory.

de Kleer, J. (1989). A comparison of ATMS and CSP techniques. In IJCAI-89.

de Kleer, J. and Brown, J. S. (1985). A qualitative physics based on confluences. In Hobbs, J. R. and Moore, R. C. (Eds.), Formal Theories of the Commonsense World. Ablex.

de Marcken, C. (1996). Unsupervised Language Acquisition. Ph.D. thesis, MIT.

De Morgan, A. (1864). On the syllogism, No. IV, and on the logic of relations. Transaction of the Cambridge Philosophical Society, X, 331–358.

de Salvo Braz, R., Amir, E., and Roth, D. (2007). Lifted first-order probabilistic inference. In Getoor, L. and Taskar, B. (Eds.), Introduction to Statistical Relational Learning. MIT Press.

Deacon, T. W. (1997). The Symbolic Species: The Co-evolution of Language and the Brain. W. W. Norton.

Deale, M., Yvanovich, M., Schnitzius, D., Kautz, D., Carpenter, M., Zweben, M., Davis, G., and Daun, B. (1994). The space shuttle ground processing scheduling system. In Zweben, M. and Fox, M. (Eds.), Intelligent Scheduling. Morgan Kaufmann.

Dean, J., Patterson, D. A., and Young, C. (2018). A new golden age in computer architecture: Empowering the machine-learning revolution. IEEE Micro, 38, 21–29.

Dean, T., Basye, K., Chekaluk, R., and Hyun, S. (1990). Coping with uncertainty in a control system for navigation and exploration. In AAAI-90.

Dean, T. and Boddy, M. (1988). An analysis of time-dependent planning. In AAAI-88.

Dean, T., Firby, R. J., and Miller, D. (1990). Hierarchical planning involving deadlines, travel time, and resources. Computational Intelligence, 6, 381–398.

Dean, T., Kaelbling, L. P., Kirman, J., and Nicholson, A. (1993). Planning with deadlines in stochastic domains. In AAAI-93.

Dean, T. and Kanazawa, K. (1989a). A model for projection and action. In IJCAI-89.

Dean, T. and Kanazawa, K. (1989b). A model for reasoning about persistence and causation. Computational Intelligence, 5, 142–150.

Dean, T. and Wellman, M. P. (1991). Planning and Control. Morgan Kaufmann.

Dearden, R., Friedman, N., and Andre, D. (1999). Model-based Bayesian exploration. In UAI-99.

Dearden, R., Friedman, N., and Russell, S. J. (1998). Bayesian Q-learning. In AAAI-98.

Debevec, P., Taylor, C., and Malik, J. (1996). Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. In Proc. 23rd Annual Conference on Computer Graphics (SIGGRAPH).

Debreu, G. (1960). Topological methods in cardinal utility theory. In Arrow, K. J., Karlin, S., and Suppes, P. (Eds.), Mathematical Methods in the Social Sciences, 1959. Stanford University Press.

Dechter, A. and Dechter, R. (1987). Removing redundancies in constraint networks. In AAAI-87.

Dechter, R. (1990a). Enhancement schemes for constraint processing: Backjumping, learning and cutset decomposition. AIJ, 41, 273–312.

Dechter, R. (1990b). On the expressiveness of networks with hidden variables. In AAAI-90.

Dechter, R. (1999). Bucket elimination: A unifying framework for reasoning. AIJ, 113, 41–85.

Dechter, R. and Pearl, J. (1985). Generalized best-first search strategies and the optimality of A*. JACM, 32, 505–536.

Dechter, R. and Pearl, J. (1987). Network-based heuristics for constraint-satisfaction problems. AIJ, 34, 1–38.

Dechter, R. and Pearl, J. (1989). Tree clustering for constraint networks. AIJ, 38, 353–366.

Dechter, R. and Rish, I. (2003). Mini-buckets: A general scheme for bounded inference. JACM, 50, 107–153.

Dechter, R. (2003). Constraint Processing. Morgan Kaufmann.

Dechter, R. (2019). Reasoning with Probabilistic and Deterministic Graphical Models: Exact Algorithms (2nd edition). Morgan & Claypool.

Dechter, R. and Frost, D. (2002). Backjump-based backtracking for constraint satisfaction problems. AIJ, 136, 147–188.

Dechter, R. and Mateescu, R. (2007). AND/OR search spaces for graphical models. AIJ, 171, 73–106.

DeCoste, D. and Schölkopf, B. (2002). Training invariant support vector machines. Machine Learning, 46, 161–190.

Dedekind, R. (1888). Was sind und was sollen die Zahlen. Braunschweig, Germany.

Deerwester, S. C., Dumais, S. T., Landauer, T. K., Furnas, G. W., and Harshman, R. A. (1990). Indexing by latent semantic analysis. J. American Society for Information Science, 41, 391–407.

DeGroot, M. H. (1970). Optimal Statistical Decisions. McGraw-Hill.

DeGroot, M. H. and Schervish, M. J. (2001). Probability and Statistics (3rd edition). Addison Wesley.

Dehaene, S. (2014). Consciousness and the Brain: Deciphering How the Brain Codes Our Thoughts. Penguin Books.

Del Moral, P., Doucet, A., and Jasra, A. (2006). Sequential Monte Carlo samplers. J. Royal Statistical Society, 68, 411–436.

Del Moral, P. (2004). Feynman–Kac Formulae, Genealogical and Interacting Particle Systems with Applications. Springer-Verlag.

Delgrande, J. and Schaub, T. (2003). On the relation between Reiter’s default logic and its (major) variants. In Seventh European Conference on Symbolic and Quantitative Approaches to Reasoning with Uncertainty.

Delling, D., Sanders, P., Schultes, D., and Wagner, D. (2009). Engineering route planning algorithms. In Lerner, J., Wagner, D., and Zweig, K. (Eds.), Algorithmics, LNCS. Springer-Verlag.

Dempster, A. P. (1968). A generalization of Bayesian inference. J. Royal Statistical Society, 30 (Series B), 205–247.

Dempster, A. P., Laird, N., and Rubin, D. (1977). Maximum likelihood from incomplete data via the EM algorithm. J. Royal Statistical Society, 39 (Series B), 1–38.

Denardo, E. V. (1967). Contraction mappings in the theory underlying dynamic programming. SIAM Review, 9, 165–177.

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In CVPR-09.

Deng, L. (2016). Deep learning: From speech recognition to language and multimodal processing. APSIPA Transactions on Signal and Information Processing, 5.

Deng, L., Yu, D., et al. (2014). Deep learning: Methods and applications. Foundations and Trends in Signal Processing, 7, 197–387.

Deng, X. and Papadimitriou, C. H. (1990). Exploring an unknown graph. In FOCS-90.

Deng, X. and Papadimitriou, C. H. (1994). On the complexity of cooperative solution concepts. Mathematics of Operations Research, 19, 257–266.

Denney, E., Fischer, B., and Schumann, J. (2006). An empirical evaluation of automated theorem provers in software certification. Int. J. AI Tools, 15, 81–107.

D’Épenoux, F. (1963). A probabilistic production and inventory problem. A probabilistic production and inventory problem, 10, 98–108.

Dervovic, D., Herbster, M., Mountney, P., Severini, S., Usher, N., and Wossnig, L. (2018). Quantum linear systems algorithms: A primer. arXiv:1802.08227.

Descartes, R. (1637). Discourse on method. In Cottingham, J., Stoothoff, R., and Murdoch, D. (Eds.), The Philosophical Writings of Descartes, Vol. I. Cambridge University Press, Cambridge.

Descotte, Y. and Latombe, J.-C. (1985). Making compromises among antagonist constraints in a planner. AIJ, 27, 183–217.

Deshpande, I., Hu, Y.-T., Sun, R., Pyrros, A., Siddiqui, N., Koyejo, S., Zhao, Z., Forsyth, D., and Schwing, A. (2019). Max-sliced Wasserstein distance and its use for GANs. In CVPR-19.

Deutscher, G. (2010). Through the Language Glass: Why the World Looks Different in Other Languages. Metropolitan Books.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805.

Devlin, K. (2018). Turned On: Science, Sex and Robots. Bloomsbury.

Devroye, L. (1987). A course in density estimation. Birkhauser.

Dias, M. B., Zlot, R., Kalra, N., and Stentz, A. (2006). Market-based multirobot coordination: A survey and analysis. Proc. IEEE, 94, 1257–1270.

Dickmanns, E. D. and Zapp, A. (1987). Autonomous high speed road vehicle guidance by computer vision. In Automatic Control—World Congress, 1987: Selected Papers from the 10th Triennial World Congress of the International Federation of Automatic Control.

Dietterich, T. (2000). Hierarchical reinforcement learning with the MAXQ value function decomposition. JAIR, 13, 227–303.

Dijkstra, E. W. (1959). A note on two problems in connexion with graphs. Numerische Mathematik, 1, 269–271.

Dijkstra, E. W. (1984). The threats to computing science. In ACM South Central Regional Conference.

Ding, Y., Sohn, J. H., Kawczynski, M. G., Trivedi, H., Harnish, R., Jenkins, N. W., Lituiev, D., Copeland, T. P., Aboian, M. S., Mari Aparici, C., et al. (2018). A deep learning model to predict a diagnosis of alzheimer disease by using 18F-FDG PET of the brain. Radiology, p. 180958.

Dinh, H., Russell, A., and Su, Y. (2007). On the value of good advice: The complexity of A* with accurate heuristics. In AAAI-07.

Dissanayake, G., Newman, P., Clark, S., Durrant-Whyte, H., and Csorba, M. (2001). A solution to the simultaneous localisation and map building (SLAM) problem. IEEE Transactions on Robotics and Automation, 17, 229–241.

Dittmer, S. and Jensen, F. (1997). Myopic value of information in influence diagrams. In UAI-97.

Do, M. and Kambhampati, S. (2003). Planning as constraint satisfaction: solving the planning graph by compiling it into CSP. AIJ, 132, 151–182.

Do, M. B. and Kambhampati, S. (2001). Sapa: A domain-independent heuristic metric temporal planner. In ECP-01.

Doctorow, C. (2001). Metacrap: Putting the torch to seven straw-men of the meta-utopia. www.well.com/doctorow/metacrap.htm.

Doctorow, C. and Stross, C. (2012). The Rapture of the Nerds: A Tale of the Singularity, Posthumanity, and Awkward Social Situations. Tor Books.

Dodd, L. (1988). The inside/outside algorithm: Grammatical inference applied to stochastic context-free grammars. Tech. rep., Royal Signals and Radar Establishment, Malvern.

Domingos, P. and Pazzani, M. (1997). On the optimality of the simple Bayesian classifier under zero–one loss. Machine Learning, 29, 103–30.

Domingos, P. (2012). A few useful things to know about machine learning. Commun. ACM, 55(10), 78–87.

Domingos, P. (2015). The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World. Basic Books.

Dong, X., Gabrilovich, E., Heitz, G., Horn, W., Lao, N., Murphy, K., Strohmann, T., Sun, S., and Zhang, W. (2014). Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In KDD-14.

Doorenbos, R. (1994). Combining left and right unlinking for matching a large number of learned rules. In AAAI-94.

Doran, J. and Michie, D. (1966). Experiments with the graph traverser program. Proc. Roy. Soc., 294, Series A, 235–259.

Dorf, R. C. and Bishop, R. H. (2004). Modern Control Systems (10th edition). Prentice-Hall.

Dorigo, M., Birattari, M., Blum, C., Clerc, M., Stützle, T., and Winfield, A. (2008). Ant Colony Optimization and Swarm Intelligence: 6th International Conference, ANTS 2008, Brussels, Belgium, September 22-24, 2008, Proceedings, Vol. 5217. Springer-Verlag.

Doshi-Velez, F. and Kim, B. (2017). Towards a rigorous science of interpretable machine learning. arXiv:1702.08608.

Doucet, A. (1997). Monte Carlo methods for Bayesian estimation of hidden Markov models: Application to radiation signals. Ph.D. thesis, Université de Paris-Sud.

Doucet, A., de Freitas, J. F. G., and Gordon, N. (2001). Sequential Monte Carlo Methods in Practice. Springer-Verlag.

Doucet, A., de Freitas, J. F. G., Murphy, K., and Russell, S. J. (2000). Rao-Blackwellised particle filtering for dynamic Bayesian networks. In UAI-00.

Doucet, A. and Johansen, A. M. (2011). A tutorial on particle filtering and smoothing: Fifteen years later. In Crisan, D. and Rozovskii, B. (Eds.), Oxford Handbook of Nonlinear

Filtering. Oxford.

Dowty, D., Wall, R., and Peters, S. (1991). Introduction to Montague Semantics. D. Reidel.

Doyle, J. (1979). A truth maintenance system. AIJ, 12, 231–272.

Doyle, J. (1983). What is rational psychology? Toward a modern mental philosophy. AIMag, 4, 50–53.

Drabble, B. (1990). Mission scheduling for spacecraft: Diaries of T-SCHED. In Expert Planning Systems. Institute of Electrical Engineers.

Dragan, A. D., Lee, K. C., and Srinivasa, S. (2013). Legibility and predictability of robot motion. In HRI-13.

Dredze, M., Crammer, K., and Pereira, F. (2008). Confidence-weighted linear classification. In ICML-08.

Dressel, J. and Farid, H. (2018). The accuracy, fairness, and limits of predicting recidivism. Science Advances, 4, eaao5580.

Dreyfus, H. L. (1972). What Computers Can’t Do: A Critique of Artificial Reason. Harper and Row.

Dreyfus, H. L. (1992). What Computers Still Can’t Do: A Critique of Artificial Reason. MIT Press.

Dreyfus, H. L. and Dreyfus, S. E. (1986). Mind over Machine: The Power of Human Intuition and Expertise in the Era of the Computer. Blackwell.

Dreyfus, S. E. (1962). The numerical solution of variational problems. J. Math. Anal. and Appl., 5, 30–45.

Dreyfus, S. E. (1969). An appraisal of some shortest-paths algorithms. Operations Research, 17, 395–412.

Dreyfus, S. E. (1990). Artificial neural networks, back propagation, and the Kelley–Bryson gradient procedure. J. Guidance, Control, and Dynamics, 13, 926–928.

Du, S. S., Lee, J. D., Li, H., Wang, L., and Zhai, X. (2018). Gradient descent finds global minima of deep neural networks. arXiv:1811.03804.

Dubois, D. and Prade, H. (1994). A survey of belief revision and updating rules in various uncertainty models. Int. J. Intelligent Systems, 9, 61–100.

Duda, R. O. and Hart, P. E. (1973). Pattern classification and scene analysis. Wiley.

Duda, R. O., Hart, P. E., and Stork, D. G. (2001). Pattern Classification (2nd edition). Wiley.

Dudek, G. and Jenkin, M. (2000). Computational Principles of Mobile Robotics. Cambridge University Press.

Duffy, D. (1991). Principles of Automated Theorem Proving. John Wiley & Sons.

Dunn, H. L. (1946). Record linkage”. Am. J. Public Health, 36, 1412–1416.

Durfee, E. H. and Lesser, V. R. (1989). Negotiating task decomposition and allocation using partial global planning. In Huhns, M. and Gasser, L. (Eds.), Distributed AI, Vol. 2. Morgan Kaufmann.

Durme, B. V. and Pasca, M. (2008). Finding cars, goddesses and enzymes: Parametrizable acquisition of labeled instances for open-domain information extraction. In AAAI-08.

Dwork, C. (2008). Differential privacy: A survey of results. In International Conference on Theory and Applications of Models of Computation.

Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. (2012). Fairness through awareness. In Proc. 3rd innovations in theoretical computer science conference.

Dwork, C., Roth, A., et al. (2014). The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9, 211–407.

Dyson, F. (2004). A meeting with Enrico Fermi. Nature, 427, 297.

Dyson, G. (1998). Darwin among the machines : the evolution of global intelligence. Perseus Books.

Earley, J. (1970). An efficient context-free parsing algorithm. CACM, 13, 94–102.

Ebendt, R. and Drechsler, R. (2009). Weighted A* search–unifying view and application. AIJ, 173, 1310–1342.

Eckerle, J., Chen, J., Sturtevant, N. R., Zilles, S., and Holte, R. C. (2017). Sufficient conditions for node expansion in bidirectional heuristic search. In ICAPS-17.

Eckhouse, L., Lum, K., Conti-Cook, C., and Ciccolini, J. (2019). Layers of bias: A unified approach for understanding problems with risk assessment. Criminal Justice and Behavior, 46, 185–209.

Edelkamp, S. (2009). Scaling search with symbolic pattern databases. In Model Checking and Artificial Intelligence (MOCHART).

Edelkamp, S. and Schrödl, S. (2012). Heuristic Search. Morgan Kaufmann.

Edmonds, J. (1965). Paths, trees, and flowers. Canadian J. of Mathematics, 17, 449–467.

Edwards, P. (Ed.). (1967). The Encyclopedia of Philosophy. Macmillan.

Eiter, T., Leone, N., Mateis, C., Pfeifer, G., and Scarcello, F. (1998). The KR system dlv: Progress report, comparisons and benchmarks. In KR-98.

Elio, R. (Ed.). (2002). Common Sense, Reasoning, and Rationality. Oxford University Press.

Elkan, C. (1997). Boosting and naive Bayesian learning. Tech. rep., Department of Computer Science and Engineering, University of California, San Diego.

Ellsberg, D. (1962). Risk, Ambiguity, and Decision. Ph.D. thesis, Harvard University.

Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14, 179–211.

Elman, J. L., Bates, E., Johnson, M., Karmiloff-Smith, A., Parisi, D., and Plunkett, K. (1997). Rethinking Innateness. MIT Press.

Elo, A. E. (1978). The rating of chess players: Past and present. Arco Publishing.

Elsken, T., Metzen, J. H., and Hutter, F. (2018). Neural architecture search: A survey. arXiv:1808.05377.

Empson, W. (1953). Seven Types of Ambiguity. New Directions.

Enderton, H. B. (1972). A Mathematical Introduction to Logic. Academic Press.

Engel, J., Resnick, C., Roberts, A., Dieleman, S., Norouzi, M., Eck, D., and Simonyan, K. (2017). Neural audio synthesis of musical notes with wavenet autoencoders. In Proc. 34th International Conference on Machine Learning-Volume 70.

Epstein, R., Roberts, G., and Beber, G. (Eds.). (2008). Parsing the Turing test. Springer.

Erdmann, M. A. and Mason, M. (1988). An exploration of sensorless manipulation. IEEE Journal of Robotics and Automation, 4, 369–379.

Ernst, H. A. (1961). MH-1, a Computer-Operated Mechanical Hand. Ph.D. thesis, MIT.

Ernst, M., Millstein, T., and Weld, D. S. (1997). Automatic SAT-compilation of planning problems. In IJCAI-97.

Erol, K., Hendler, J., and Nau, D. S. (1994). HTN planning: Complexity and expressivity. In AAAI-94.

Erol, K., Hendler, J., and Nau, D. S. (1996). Complexity results for HTN planning. AIJ, 18, 69–93.

Erol, Y., Li, L., Ramsundar, B., and Russell, S. J. (2013). The extended parameter filter. In ICML-13.

Erol, Y., Wu, Y., Li, L., and Russell, S. J. (2017). A nearly-black-box online algorithm for joint parameter and state estimation in temporal models. In AAAI-17.

Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., and Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542, 115.

Etzioni, A. (2004). From Empire to Community: A New Approach to International Relation. Palgrave Macmillan.

Etzioni, A. and Etzioni, O. (2017a). Incorporating ethics into artificial intelligence. The Journal of Ethics, 21, 403–418.

Etzioni, A. and Etzioni, O. (2017b). Should artificial intelligence be regulated? Issues in Science and Technology, Summer.

Etzioni, O. (1989). Tractable decision-analytic control. In Proc. First International Conference on Knowledge Representation and Reasoning.

Etzioni, O., Banko, M., Soderland, S., and Weld, D. S. (2008). Open information extraction from the web. CACM, 51.

Etzioni, O., Hanks, S., Weld, D. S., Draper, D., Lesh, N., and Williamson, M. (1992). An approach to planning with incomplete information. In KR-92.

Etzioni, O., Banko, M., and Cafarella, M. J. (2006). Machine reading. In AAAI-06.

Etzioni, O., Cafarella, M. J., Downey, D., Popescu, A.-M., Shaked, T., Soderland, S., Weld, D. S., and Yates, A. (2005). Unsupervised named-entity extraction from the web: An experimental study. AIJ, 165(1), 91–134.

Evans, T. G. (1968). A program for the solution of a class of geometric-analogy intelligencetest questions. In Minsky, M. L. (Ed.), Semantic Information Processing. MIT Press.

Fagin, R., Halpern, J. Y., Moses, Y., and Vardi, M. Y. (1995). Reasoning about Knowledge. MIT Press.

Fahlman, S. E. (1974). A planning system for robot construction tasks. AIJ, 5, 1–49.

Faugeras, O. (1992). What can be seen in three dimensions with an uncalibrated stereo rig? In ECCV, Vol. 588 of Lecture Notes in Computer Science.

Faugeras, O., Luong, Q.-T., and Papadopoulo, T. (2001). The Geometry of Multiple Images. MIT Press.

Fawcett, T. and Provost, F. (1997). Adaptive fraud detection. Data mining and knowledge discovery, 1, 291–316.

Fearing, R. S. and Hollerbach, J. M. (1985). Basic solid mechanics for tactile sensing. Int. J. Robotics Research, 4, 40–54.

Featherstone, R. (1987). Robot Dynamics Algorithms. Kluwer Academic Publishers.

Feigenbaum, E. A. (1961). The simulation of verbal learning behavior. Proc. Western Joint Computer Conference, 19, 121–131.

Feigenbaum, E. A., Buchanan, B. G., and Lederberg, J. (1971). On generality and problem solving: A case study using the DENDRAL program. In Meltzer, B. and Michie, D. (Eds.), Machine Intelligence 6. Edinburgh University Press.

Feldman, J. and Sproull, R. F. (1977). Decision theory and artificial intelligence II: The hungry monkey. Technical report, Computer Science Department, University of Rochester.

Feldman, J. and Yakimovsky, Y. (1974). Decision theory and artificial intelligence I: Semantics-based region analyzer. AIJ, 5, 349–371.

Feldman, M. (2017). Oak Ridge readies Summit supercomputer for 2018 debut. Top500.org, bit.ly/2ERRFr9.

Fellbaum, C. (2001). Wordnet: An Electronic Lexical Database. MIT Press.

Fellegi, I. and Sunter, A. (1969). A theory for record linkage”. JASA, 64, 1183–1210.

Felner, A., Korf, R. E., and Hanan, S. (2004). Additive pattern database heuristics. JAIR, 22, 279–318.

Felner, A. (2018). Position paper: Using early goal test in A*. In Eleventh Annual Symposium on Combinatorial Search.

Felner, A., Korf, R. E., Meshulam, R., and Holte, R. C. (2007). Compressed pattern databases. JAIR, 30.

Felner, A., Zahavi, U., Holte, R. C., Schaeffer, J., Sturtevant, N. R., and Zhang, Z. (2011). Inconsistent heuristics in theory and practice. AIJ, 175, 1570–1603.

Felzenszwalb, P. and McAllester, D. A. (2007). The generalized A* architecture. JAIR.

Fenton, N. and Neil, M. (2018). Risk Assessment and Decision Analysis with Bayesian Networks (2nd edition). Chapman and Hall.

Ferguson, T. (1992). Mate with knight and bishop in kriegspiel. Theoretical Computer Science, 96, 389–403.

Ferguson, T. (1995). Mate with the two bishops in kriegspiel. www.math.ucla.edu/˜tom/ paper.

Ferguson, T. (2001). Optimal Stopping and Applications. www.math.ucla.edu/tom/Stopping/ Contents.html.

Ferguson, T. (1973). Bayesian analysis of some nonparametric problems. Annals of Statistics, 1, 209–230.

Fern, A., Natarajan, S., Judah, K., and Tadepalli, P. (2014). A decision-theoretic model of assistance. JAIR, 50, 71–104.

Fernandez, J. M. F. and Mahlmann, T. (2018). The Dota 2 bot competition. IEEE Transactions on Games.

Ferraris, P. and Giunchiglia, E. (2000). Planning as satisability in nondeterministic domains. In AAAI-00.

Ferriss, T. (2007). The 4-Hour Workweek. Crown.

Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A. A., Lally, A., Murdock, J. W., Nyberg, E., Prager, J., Schlaefer, N., and Welty, C. (2010). Building Watson: An overview of the DeepQA project. AI Magazine, Fall.

Fikes, R. E., Hart, P. E., and Nilsson, N. J. (1972). Learning and executing generalized robot plans. AIJ, 3, 251–288.

Fikes, R. E. and Nilsson, N. J. (1971). STRIPS: A new approach to the application of theorem proving to problem solving. AIJ, 2, 189–208.

Fikes, R. E. and Nilsson, N. J. (1993). STRIPS, a retrospective. AIJ, 59, 227–232.

Fine, S., Singer, Y., and Tishby, N. (1998). The hierarchical hidden Markov model: Analysis and applications. Machine Learning, 32.

Finn, C., Abbeel, P., and Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. In Proc. 34th International Conference on Machine Learning-Volume 70.

Finney, D. J. (1947). Probit analysis: A statistical treatment of the sigmoid response curve. Cambridge University Press.

Firoiu, V., Whitney, W. F., and Tenenbaum, J. B. (2017). Beating the world’s best at Super Smash Bros. with deep reinforcement learning. arXiv:1702.06230.

Firth, J. (1957). Papers in Linguistics. Oxford University Press.

Fisher, R. A. (1922). On the mathematical foundations of theoretical statistics. Phil. Trans. Roy. Soc., A, 222, 309–368.

Fix, E. and Hodges, J. L. (1951). Discriminatory analysis—Nonparametric discrimination: Consistency properties. Tech. rep., USAF School of Aviation Medicine.

Floreano, D., Zufferey, J. C., Srinivasan, M. V., and Ellington, C. (2009). Flying Insects and Robots. Springer.

Floyd, R. W. (1962). Algorithm 97: Shortest path. CACM, 5, 345.

Fogel, D. B. (2000). Evolutionary Computation: Toward a New Philosophy of Machine Intelligence. IEEE Press.

Fogel, L. J., Owens, A. J., and Walsh, M. J. (1966). Artificial Intelligence through Simulated Evolution. Wiley.

Forbes, J., Huang, T., Kanazawa, K., and Russell, S. J. (1995). The BATmobile: Towards a Bayesian automated taxi. In IJCAI-95.

Forbus, K. D. (1985). Qualitative process theory. In Bobrow, D. (Ed.), Qualitative Reasoning About Physical Systems. MIT Press.

Forbus, K. D. and de Kleer, J. (1993). Building Problem Solvers. MIT Press.

Forbus, K. D., Hinrichs, T. R., De Kleer, J., and Usher, J. M. (2010). FIRE: Infrastructure for experience-based systems with common sense. In AAAI Fall Symposium: Commonsense Knowledge.

Ford, K. M. and Hayes, P. J. (1995). Turing Test considered harmful. In IJCAI-95.

Ford, L. R. (1956). Network flow theory. Tech. rep., RAND Corporation.

Ford, M. (2015). Rise of the Robots: Technology and the Threat of a Jobless Future. Basic Books.

Ford, M. (2018). Architects of Intelligence. Packt.

Forestier, J.-P. and Varaiya, P. (1978). Multilayer control of large Markov chains. IEEE Transactions on Automatic Control, 23, 298–304.

Forgy, C. (1981). OPS5 user’s manual. Technical report, Computer Science Department, Carnegie-Mellon University.

Forgy, C. (1982). A fast algorithm for the many patterns/many objects match problem. AIJ, 19, 17–37.

Forster, E. M. (1909). The Machine Stops. Sheba Blake.

Forsyth, D. and Ponce, J. (2002). Computer Vision: A Modern Approach. Prentice Hall.

Fouhey, D., Kuo, W.-C., Efros, A., and Malik, J. (2018). From lifestyle vlogs to everyday interactions. In CVPR-18.

Fourier, J. (1827). Analyse des travaux de l’Académie Royale des Sciences, pendant l’année 1824; partie mathématique. Histoire de l’Académie Royale des Sciences de France, 7, xlvii– lv.

Fox, C. and Tversky, A. (1995). Ambiguity aversion and comparative ignorance. Quarterly Journal of Economics, 110, 585–603.

Fox, D., Burgard, W., Dellaert, F., and Thrun, S. (1999). Monte Carlo localization: Efficient position estimation for mobile robots. In AAAI-99.

Fox, M. S. (1990). Constraint-guided scheduling: A short history of research at CMU. Computers in Industry, 14, 79–88.

Fox, M. S., Allen, B., and Strohm, G. (1982). Job shop scheduling: An investigation in constraint-directed reasoning. In AAAI-82.

Franco, J. and Paull, M. (1983). Probabilistic analysis of the Davis Putnam procedure for solving the satisfiability problem. Discrete Applied Mathematics, 5, 77–87.

Francois-Lavet, V., Henderson, P., Islam, R., Bellemare, M. G., and Pineau, J. (2018). An introduction to deep reinforcement learning. Foundations and Trends in Machine Learning, 11, 219–354.

Frank, I., Basin, D. A., and Matsubara, H. (1998). Finding optimal strategies for imperfect information games. In AAAI-98.

Frank, R. H. and Cook, P. J. (1996). The Winner-Take-All Society. Penguin.

Frans, K., Ho, J., Chen, X., Abbeel, P., and Schulman, J. (2018). Meta learning shared hierarchies. In ICLR-18.

Franz, A. and Brants, T. (2006). All our n-gram are belong to you. Google blog, ai.googleblog.com/2006/08/all-our-n-gram-are-belong-to-you.html.

Frege, G. (1879). Begriffsschrift, eine der arithmetischen nachgebildete Formelsprache des reinen Denkens. Halle, Berlin. English translation appears in van Heijenoort (1967).

Freitag, D. and McCallum, A. (2000). Information extraction with hmm structures learned by stochastic optimization. In AAAI-00.

Freuder, E. C. (1978). Synthesizing constraint expressions. CACM, 21, 958–966.

Freuder, E. C. (1982). A sufficient condition for backtrack-free search. JACM, 29, 24–32.

Freuder, E. C. (1985). A sufficient condition for backtrack-bounded search. JACM, 32, 755– 761.

Freund, Y. and Schapire, R. E. (1996). Experiments with a new boosting algorithm. In ICML96.

Freund, Y. and Schapire, R. E. (1999). Large margin classification using the perceptron algorithm. Machine Learning, 37, 277–296.

Frey, B. J. (1998). Graphical models for machine learning and digital communication. MIT Press.

Frey, C. B. and Osborne, M. A. (2017). The future of employment: How susceptible are jobs to computerisation? Technological forecasting and social change, 114, 254–280.

Friedberg, R. M. (1958). A learning machine: Part I. IBM Journal of Research and Development, 2, 2–13.

Friedberg, R. M., Dunham, B., and North, T. (1959). A learning machine: Part II. IBM Journal of Research and Development, 3, 282–287.

Friedman, G. J. (1959). Digital simulation of an evolutionary process. General Systems Yearbook, 4, 171–184.

Friedman, J., Hastie, T., and Tibshirani, R. (2000). Additive logistic regression: A statistical view of boosting. Annals of Statistics, 28, 337–374.

Friedman, J. (2001). Greedy function approximation: A gradient boosting machine. Annals of statistics, 29, 1189–1232.

Friedman, N. (1998). The Bayesian structural EM algorithm. In UAI-98.

Friedman, N. and Goldszmidt, M. (1996). Learning Bayesian networks with local structure. In UAI-96.

Friedman, N. and Koller, D. (2003). Being Bayesian about Bayesian network structure: A Bayesian approach to structure discovery in Bayesian networks. Machine Learning, 50, 95–125.

Friedman, N., Murphy, K., and Russell, S. J. (1998). Learning the structure of dynamic probabilistic networks. In UAI-98.

Friedman, N. (2004). Inferring cellular networks using probabilistic graphical models. Science, 303.

Fruhwirth, T. and Abdennadher, S. (2003). Essentials of constraint programming. Cambridge University Press.

Fuchs, J. J., Gasquet, A., Olalainty, B., and Currie, K. W. (1990). PlanERS-1: An expert planning system for generating spacecraft mission plans. In First International Conference on Expert Planning Systems. Institute of Electrical Engineers.

Fudenberg, D. and Tirole, J. (1991). Game theory. MIT Press.

Fukunaga, A. S., Rabideau, G., Chien, S., and Yan, D. (1997). ASPEN: A framework for automated planning and scheduling of spacecraft control and operations. In Proc. International Symposium on AI, Robotics and Automation in Space.

Fukushima, K. (1980). Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36, 193–202.

Fukushima, K. and Miyake, S. (1982). Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition. In Competition and cooperation in neural nets. Springer.

Fuller, S. B., Straw, A. D., Peek, M. Y., Murray, R. M., and Dickinson, M. H. (2014). Flying Drosophila stabilize their vision-based velocity controller by sensing wind with their antennae. Proc. National Academy of Sciences of the United States of America, 111 13, E1182–91.

Fung, C., Yoon, C. J. M., and Beschastnikh, I. (2018). Mitigating sybils in federated learning poisoning. arXiv:1808.04866.

Fung, R. and Chang, K. C. (1989). Weighting and integrating evidence for stochastic simulation in Bayesian networks. In UAI 5.

Gaddum, J. H. (1933). Reports on biological standard III: Methods of biological assay depending on a quantal response. Special report series of the medical research council, Medical Research Council.

Gaifman, H. (1964a). Concerning measures in first order calculi. Israel J. Mathematics, 2, 1– 18.

Gaifman, H. (1964b). Concerning measures on Boolean algebras. Pacific J. Mathematics, 14, 61–73.

Gallaire, H. and Minker, J. (Eds.). (1978). Logic and Databases. Plenum.

Gallier, J. H. (1986). Logic for Computer Science: Foundations of Automatic Theorem Proving. Harper and Row.

Galton, F. (1886). Regression towards mediocrity in hereditary stature. J. Anthropological Institute of Great Britain and Ireland, 15, 246–263.

Gamba, A., Gamberini, L., Palmieri, G., and Sanna, R. (1961). Further experiments with PAPA. Nuovo Cimento Supplemento, 20, 221–231.

Gandomi, A. and Haider, M. (2015). Beyond the hype: Big data concepts, methods, and analytics. International journal of information management, 35, 137–144.

Gao, J. (2014). Machine learning applications for data center optimization. Google Research.

Garcìa, J. and Fernández, F. (2015). A comprehensive survey on safe reinforcement learning. JMLR, 16, 1437–1480.

Gardner, M. (1968). Logic Machines, Diagrams and Boolean Algebra. Dover.

Garey, M. R. and Johnson, D. S. (1979). Computers and Intractability. W. H. Freeman.

Gaschnig, J. (1977). A general backtrack algorithm that eliminates most redundant tests. In IJCAI-77.

Gaschnig, J. (1979). Performance measurement and analysis of certain search algorithms. Technical report, Computer Science Department, Carnegie-Mellon University.

Gasser, R. (1995). Efficiently harnessing computational resources for exhaustive search. Ph.D. thesis, ETH Zürich.

Gat, E. (1998). Three-layered architectures. In Kortenkamp, D., Bonasso, R. P., and Murphy, R. (Eds.), AI-based Mobile Robots: Case Studies of Successful Robot Systems. MIT Press.

Gatys, L. A., Ecker, A. S., and Bethge, M. (2016). Image style transfer using convolutional neural networks. In CVPR-16.

Gauci, J., Conti, E., Liang, Y., Virochsiri, K., He, Y., Kaden, Z., Narayanan, V., and Ye, X. (2018). Horizon: Facebook’s open source applied reinforcement learning platform.

arXiv:1811.00260.

Gauss, C. F. (1809). Theoria Motus Corporum Coelestium in Sectionibus Conicis Solem Ambientium. Sumtibus F. Perthes et I. H. Besser, Hamburg.

Gauss, C. F. (1829). Beiträge zur theorie der algebraischen gleichungen. Werke, 3, 71–102.

Gazzaniga, M. (2018). The Consciousness Instinct. Farrar, Straus and Girou.

Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H. M., III, H. D., and Crawford, K. (2018). Datasheets for datasets. arXiv:1803.09010.

Geiger, D., Verma, T., and Pearl, J. (1990a). d-separation: From theorems to algorithms. In Henrion, M., Shachter, R. D., Kanal, L. N., and Lemmer, J. F. (Eds.), UAI-90. Elsevier.

Geiger, D., Verma, T., and Pearl, J. (1990b). Identifying independence in Bayesian networks. Networks, 20, 507–534.

Gelb, A. (1974). Applied Optimal Estimation. MIT Press.

Gelernter, H. (1959). Realization of a geometry-theorem proving machine. In Proc. an International Conference on Information Processing. UNESCO House.

Gelfond, M. and Lifschitz, V. (1988). Compiling circumscriptive theories into logic programs. In Non-Monotonic Reasoning: 2nd International Workshop Proceedings.

Gelfond, M. (2008). Answer sets. In van Harmelan, F., Lifschitz, V., and Porter, B. (Eds.), Handbook of Knowledge Representation. Elsevier.

Gelman, A. (2004). Exploratory data analysis for complex models. Journal of Computational and Graphical Statistics, 13, 755–779.

Gelman, A., Carlin, J. B., Stern, H. S., and Rubin, D. (1995). Bayesian Data Analysis. Chapman & Hall.

Geman, S. and Geman, D. (1984). Stochastic relaxation, Gibbs distributions, and Bayesian restoration of images. PAMI, 6, 721–741.

Gene Ontology Consortium, The. (2008). The gene ontology project in 2008. Nucleic Acids Research, 36(D440–D444).

Genesereth, M. R. (1984). The use of design descriptions in automated diagnosis. AIJ, 24, 411–436.

Genesereth, M. R. and Nilsson, N. J. (1987). Logical Foundations of Artificial Intelligence. Morgan Kaufmann.

Genesereth, M. R. and Nourbakhsh, I. (1993). Time-saving tips for problem solving with incomplete information. In AAAI-93.

Genesereth, M. R. and Smith, D. E. (1981). Meta-level architecture. Memo, Computer Science Department, Stanford University.

Gent, I., Petrie, K., and Puget, J.-F. (2006). Symmetry in constraint programming. In Rossi, F., van Beek, P., and Walsh, T. (Eds.), Handbook of Constraint Programming. Elsevier.

Géron, A. (2019). Hands-On Machine Learning with Scikit-Learn, Kerasm and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. O’Reilly.

Gers, F. A., Schmidhuber, J., and Cummins, F. (2000). Learning to forget: Continual prediction with LSTM. Neural Computation, 12, 2451–2471.

Getoor, L. and Taskar, B. (Eds.). (2007). Introduction to Statistical Relational Learning. MIT Press.

Ghaheri, A., Shoar, S., Naderan, M., and Hoseini, S. S. (2015). The applications of genetic algorithms in medicine. Oman medical journal, 30, 406–416.

Ghahramani, Z. (1998). Learning dynamic Bayesian networks. In Adaptive Processing of Sequences and Data Structures.

Ghahramani, Z. (2005). Tutorial on nonparametric Bayesian methods. Given at the UAI-05 Conference.

Ghallab, M., Howe, A., Knoblock, C. A., and McDermott, D. (1998). PDDL—The planning domain definition language. Tech. rep., Yale Center for Computational Vision and Control.

Ghallab, M. and Laruelle, H. (1994). Representation and control in IxTeT, a temporal planner. In AIPS-94.

Ghallab, M., Nau, D. S., and Traverso, P. (2004). Automated Planning: Theory and practice. Morgan Kaufmann.

Ghallab, M., Nau, D. S., and Traverso, P. (2016). Automated Planning and aAting. Cambridge University Press.

Gibbs, R. W. (2006). Metaphor interpretation as embodied simulation. Mind, 21, 434–458.

Gibson, J. J. (1950). The Perception of the Visual World. Houghton Mifflin.

Gibson, J. J. (1979). The Ecological Approach to Visual Perception. Houghton Mifflin.

Gibson, J. J., Olum, P., and Rosenblatt, F. (1955). Parallax and perspective during aircraft landings. American Journal of Psychology, 68, 372–385.

Gilks, W. R., Richardson, S., and Spiegelhalter, D. J. (Eds.). (1996). Markov chain Monte Carlo in practice. Chapman and Hall.

Gilks, W. R., Thomas, A., and Spiegelhalter, D. J. (1994). A language and program for complex Bayesian modelling. The Statistician, 43, 169–178.

Gilks, W. R. and Berzuini, C. (2001). Following a moving target—Monte Carlo inference for dynamic Bayesian models. J. Royal Statistical Society, 63, 127–146.

Gilks, W. R. and Wild, P. P. (1992). Adaptive rejection sampling for Gibbs sampling. Applied Statistics, 41, 337–348.

Gillies, D. B. (1959). Solutions to general non-zero-sum games. In Tucker, A. W. and Luce, L. D. (Eds.), Contributions to the Theory of Games, volume IV. Princeton University Press.

Gilmore, P. C. (1960). A proof method for quantification theory: Its justification and realization. IBM Journal of Research and Development, 4, 28–35.

Gilpin, A., Sandholm, T., and Sorensen, T. (2008). A heads-up no-limit Texas Hold’em poker player: Discretized betting models and automatically generated equilibriumfinding programs. In AAMAS-08.

Ginsberg, M. L. (1993). Essentials of Artificial Intelligence. Morgan Kaufmann.

Ginsberg, M. L. (2001). GIB: Imperfect infoormation in a computationally challenging game. JAIR, 14, 303–358.

Gionis, A., Indyk, P., and Motwani, R. (1999). Similarity search in high dimensions vis hashing. In Proc. 25th Very Large Database (VLDB) Conference.

Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2016). Region-based convolutional networks for accurate object detection and segmentation. PAMI, 38, 142–58.

Gittins, J. C. (1989). Multi-Armed Bandit Allocation Indices. Wiley.

Gittins, J. C. and Jones, D. M. (1974). A dynamic allocation index for the sequential design of experiments. In Gani, J. (Ed.), Progress in Statistics. North-Holland.

Glanc, A. (1978). On the etymology of the word “robot”. SIGART Newsletter, 67, 12.

Glickman, M. E. (1999). Parameter estimation in large dynamic paired comparison experiments. Applied Statistics, 48, 377–394.

Glorot, X., Bordes, A., and Bengio, Y. (2011). Deep sparse rectifier neural networks. In AISTATS’2011.

Glover, F. and Laguna, M. (Eds.). (1997). Tabu search. Kluwer.

Gluss, B. (1959). An optimum policy for detecting a fault in a complex system. Operations Research, 7, 468–477.

Godefroid, P. (1990). Using partial orders to improve automatic verification methods. In Proc. 2nd Int’l Workshop on Computer Aided Verification.

Gödel, K. (1930). Über die Vollständigkeit des Logikkalküls. Ph.D. thesis, University of Vienna.

Gödel, K. (1931). Über formal unentscheidbare Sätze der Principia mathematica und verwandter Systeme I. Monatshefte für Mathematik und Physik, 38, 173–198.

Goebel, J., Volk, K., Walker, H., and Gerbault, F. (1989). Automatic classification of spectra from the infrared astronomical satellite (IRAS). Astronomy and Astrophysics, 222, L5– L8.

Goertzel, B. and Pennachin, C. (2007). Artificial General Intelligence. Springer.

Gogate, V. and Domingos, P. (2011). Approximation by quantization. In UAI-11.

Gold, E. M. (1967). Language identification in the limit. Information and Control, 10, 447– 474.

Goldberg, A. V., Kaplan, H., and Werneck, R. F. (2006). Reach for A*: Efficient point-topoint shortest path algorithms. In Workshop on algorithm engineering and experiments.

Goldberg, Y. (2017). Neural network methods for natural language processing. Synthesis Lectures on Human Language Technologies, 10.

Goldberg, Y., Zhao, K., and Huang, L. (2013). Efficient implementation of beam-search incremental parsers. In ACL-13.

Goldman, R. and Boddy, M. (1996). Expressive planning and explicit knowledge. In AIPS96.

Goldszmidt, M. and Pearl, J. (1996). Qualitative probabilities for default reasoning, belief revision, and causal modeling. AIJ, 84, 57–112.

Golomb, S. and Baumert, L. (1965). Backtrack proramming. JACM, 14, 516–524.

Golub, G., Heath, M., and Wahba, G. (1979). Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics, 21.

Gomes, C., Selman, B., Crato, N., and Kautz, H. (2000). Heavy-tailed phenomena in satisfiability and constrain processing. JAR, 24, 67–100.

Gomes, C., Kautz, H., Sabharwal, A., and Selman, B. (2008). Satisfiability solvers. In van Harmelen, F., Lifschitz, V., and Porter, B. (Eds.), Handbook of Knowledge Representation. Elsevier.

Gomes, C. and Selman, B. (2001). Algorithm portfolios. AIJ, 126, 43–62.

Gomes, C., Selman, B., and Kautz, H. (1998). Boosting combinatorial search through randomization. In AAAI-98.

Gonthier, G. (2008). Formal proof–The four-color theorem. Notices of the AMS, 55, 1382– 1393.

Good, I. J. (1961). A causal calculus. British Journal of the Philosophy of Science, 11, 305–318.

Good, I. J. (1965a). The mystery of Go. New Scientist, 427, 172–174.

Good, I. J. (1965b). Speculations concerning the first ultraintelligent machine. In Alt, F. L. and Rubinoff, M. (Eds.), Advances in Computers, Vol. 6. Academic Press.

Good, I. J. (1983). Good Thinking: The Foundations of Probability and Its Applications. University of Minnesota Press.

Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press.

Goodfellow, I., Bulatov, Y., Ibarz, J., Arnoud, S., and Shet, V. (2014). Multi-digit number recognition from Street View imagery using deep convolutional neural networks. In International Conference on Learning Representations.

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2015a). Generative adversarial nets. In NeurIPS 27.

Goodfellow, I., Vinyals, O., and Saxe, A. M. (2015b). Qualitatively characterizing neural network optimization problems. In International Conference on Learning Representations.

Goodman, J. (2001). A bit of progress in language modeling. Tech. rep., Microsoft Research.

Goodman, N. D., Mansinghka, V. K., Roy, D., Bonawitz, K., and Tenenbaum, J. B. (2008). Church: A language for generative models. In UAI-08.

Goodman, N. (1977). The Structure of Appearance (3rd edition). D. Reidel.

Gopnik, A. and Glymour, C. (2002). Causal maps and Bayes nets: A cognitive and computational account of theory-formation. In Caruthers, P., Stich, S., and Siegal, M. (Eds.), The Cognitive Basis of Science. Cambridge University Press.

Gordon, A. D., Graepel, T., Rolland, N., Russo, C., Borgström, J., and Guiver, J. (2014). Tabular: A schema-driven probabilistic programming language. In POPL-14.

Gordon, A. S. and Hobbs, J. R. (2017). A Formal Theory of Commonsense Psychology: How People Think People Think. Cambridge University Press.

Gordon, M. J., Milner, A. J., and Wadsworth, C. P. (1979). Edinburgh LCF. Springer-Verlag.

Gordon, N. (1994). Bayesian methods for tracking. Ph.D. thesis, Imperial College.

Gordon, N., Salmond, D. J., and Smith, A. F. M. (1993). Novel approach to nonlinear/nonGaussian Bayesian state estimation. IEE Proceedings F (Radar and Signal Processing), 140, 107–113.

Gordon, S. A. (1994). A faster Scrabble move generation algorithm. Software Practice and Experience, 24, 219–232.

Gorry, G. A. (1968). Strategies for computer-aided diagnosis. Math. Biosciences, 2, 293–318.

Gorry, G. A., Kassirer, J. P., Essig, A., and Schwartz, W. B. (1973). Decision analysis as the basis for computer-aided management of acute renal failure. American Journal of Medicine, 55, 473–484.

Gottlob, G., Leone, N., and Scarcello, F. (1999a). A comparison of structural CSP decomposition methods. In IJCAI-99.

Gottlob, G., Leone, N., and Scarcello, F. (1999b). Hypertree decompositions and tractable queries. In PODS-99.

Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. (2017). Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR-17.

Grace, K., Salvatier, J., Dafoe, A., Zhang, B., and Evans, O. (2017). When will AI exceed human performance? Evidence from AI experts. arXiv:1705.08807.

Graham, S. L., Harrison, M. A., and Ruzzo, W. L. (1980). An improved context-free recognizer. ACM Transactions on Programming Languages and Systems, 2, 415–462.

Grassmann, H. (1861). Lehrbuch der Arithmetik. Th. Chr. Fr. Enslin, Berlin.

Grayson, C. J. (1960). Decisions under uncertainty: Drilling decisions by oil and gas operators. Tech. rep., Harvard Business School.

Green, B., Wolf, A., Chomsky, C., and Laugherty, K. (1961). BASEBALL: An automatic question answerer. In Proc. Western Joint Computer Conference.

Green, C. (1969a). Application of theorem proving to problem solving. In IJCAI-69.

Green, C. (1969b). Theorem-proving by resolution as a basis for question-answering systems. In Meltzer, B., Michie, D., and Swann, M. (Eds.), Machine Intelligence 4. Edinburgh University Press.

Green, C. and Raphael, B. (1968). The use of theorem-proving techniques in questionanswering systems. In Proc. 23rd ACM National Conference.

Gribkoff, E., Van den Broeck, G., and Suciu, D. (2014). Understanding the complexity of lifted inference and asymmetric weighted model counting. In UAI-14.

Griffiths, T. L., Kemp, C., and Tenenbaum, J. B. (2008). Bayesian models of cognition. In Sun, R. (Ed.), The Cambridge handbook of computational cognitive modeling. Cambridge University Press.

Grinstead, C. and Snell, J. (1997). Introduction to Probability. American Mathematical Society.

Grosz, B. J. and Stone, P. (2018). A century long commitment to assessing artificial intelligence and its impact on society. Communications of the ACM, 61.

Grove, W. and Meehl, P. (1996). Comparative efficiency of informal (subjective, impressionistic) and formal (mechanical, algorithmic) prediction procedures: The clinical statistical controversy. Psychology, Public Policy, and Law, 2, 293–323.

Gruber, T. (2004). Interview of Tom Gruber. AIS SIGSEMIS Bulletin, 1.

Gu, J. (1989). Parallel Algorithms and Architectures for Very Fast AI Search. Ph.D. thesis, Univ. of Utah.

Guard, J., Oglesby, F., Bennett, J., and Settle, L. (1969). Semi-automated mathematics. JACM, 16, 49–62.

Guestrin, C., Koller, D., Gearhart, C., and Kanodia, N. (2003a). Generalizing plans to new environments in relational MDPs. In IJCAI-03.

Guestrin, C., Koller, D., Parr, R., and Venkataraman, S. (2003b). Efficient solution algorithms for factored MDPs. JAIR, 19, 399–468.

Guestrin, C., Lagoudakis, M. G., and Parr, R. (2002). Coordinated reinforcement learning. In ICML-02.

Guibas, L. J., Knuth, D. E., and Sharir, M. (1992). Randomized incremental construction of Delaunay and Voronoi diagrams. Algorithmica, 7, 381–413.

Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016). Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama, 316, 2402–2410.

Gunkel, D. J. (2018). Robot Rights. MIT Press.

Gunning, D. (2016). Explainable artificial intelligence (xai). Tech. rep., DARPA.

Guo, C., Goldstein, T., Hannun, A., and van der Maaten, L. (2019). Certified data removal from machine learning models. arXiv:1911.03030.

Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., and Smith, N. A. (2018). Annotation artifacts in natural language inference data. arXiv:1803.02324.

Guyon, I., Bennett, K., Cawley, G. C., Escalante, H. J., Escalera, S., Ho, T. K., Macià, N., Ray, B., Saeed, M., Statnikov, A. R., and Viegas, E. (2015). Design of the 2015 ChaLearn AutoML challenge. In IJCNN-15.

Guyon, I. and Elisseeff, A. (2003). An introduction to variable and feature selection. JMLR, 3, 1157–1182.

Hacking, I. (1975). The Emergence of Probability. Cambridge University Press.

Hadfield-Menell, D., Dragan, A. D., Abbeel, P., and Russell, S. J. (2017a). Cooperative inverse reinforcement learning. In NeurIPS 29.

Hadfield-Menell, D., Dragan, A. D., Abbeel, P., and Russell, S. J. (2017b). The off-switch game. In IJCAI-17.

Hadfield-Menell, D. and Russell, S. J. (2015). Multitasking: Efficient optimal planning for bandit superprocesses. In UAI-15.

Hailperin, T. (1984). Probability logic. Notre Dame J. Formal Logic, 25, 198–212.

Hald, A. (1990). A History of Probability and Statistics and Their Applications before 1750. Wiley.

Hales, T. (2005). A proof of the Kepler conjecture. Annals of mathematics, 162, 1065–1185.

Hales, T., Adams, M., Bauer, G., Dang, T. D., Harrison, J., Le Truong, H., Kaliszyk, C., Magron, V., McLaughlin, S., Nguyen, T. T., et al. (2017). A formal proof of the Kepler conjecture. In Forum of Mathematics, Pi.

Halevy, A. (2007). Dataspaces: A new paradigm for data integration. In Brazilian Symposium on Databases.

Halevy, A., Norvig, P., and Pereira, F. (2009). The unreasonable effectiveness of data. IEEE Intelligent Systems, March/April, 8–12.

Halpern, J. Y. (1990). An analysis of first-order logics of probability. AIJ, 46, 311–350.

Halpern, J. Y. (1999). Technical addendum, Cox’s theorem revisited. JAIR, 11, 429–435.

Halpern, J. Y. and Weissman, V. (2008). Using first-order logic to reason about policies. ACM Transactions on Information and System Security, 11, 1–41.

Hammersley, J. M. and Handscomb, D. C. (1964). Monte Carlo Methods. Methuen.

Han, J., Pei, J., and Kamber, M. (2011). Data Mining: Concepts and Techniques. Elsevier.

Han, X. and Boyden, E. (2007). Multiple-color optical activation, silencing, and desynchronization of neural activity, with single-spike temporal resolution. PLoS One, e299.

Handschin, J. E. and Mayne, D. Q. (1969). Monte Carlo techniques to estimate the conditional expectation in multi-stage nonlinear filtering. Int. J. Control, 9, 547–559.

Hans, A., Schneegaß, D., Schäfer, A. M., and Udluft, S. (2008). Safe exploration for reinforcement learning. In ESANN.

Hansen, E. (1998). Solving POMDPs by searching in policy space. In UAI-98.

Hansen, E. and Zilberstein, S. (2001). LAO*: a heuristic search algorithm that finds solutions with loops. AIJ, 129, 35–62.

Hansen, P. and Jaumard, B. (1990). Algorithms for the maximum satisfiability problem. Computing, 44, 279–303.

Hanski, I. and Cambefort, Y. (Eds.). (1991). Dung Beetle Ecology. Princeton University Press.

Hansson, O. and Mayer, A. (1989). Heuristic search as evidential reasoning. In UAI 5.

Haralick, R. M. and Elliott, G. L. (1980). Increasing tree search efficiency for constraint satisfaction problems. AIJ, 14, 263–313.

Hardin, G. (1968). The tragedy of the commons. Science, 162, 1243–1248.

Hardt, M., Price, E., Srebro, N., et al. (2017). Equality of opportunity in supervised learning. In NeurIPS 29.

Harris, T. (2016). How technology is hijacking your mind—From a magician and Google design ethicist. medium.com/thrive-global/how-technology-hijacks-peoples-mindsfrom-a-magician-and-google-s-design-ethicist-56d62ef5edf3.

Harris, Z. (1954). Distributional structure. Word, 10.

Harrison, J. and March, J. G. (1984). Decision making and postdecision surprises. Administrative Science Quarterly, 29, 26–42.

Harrow, A. W., Hassidim, A., and Lloyd, S. (2009). Quantum algorithm for linear systems of equations. Physical Review Letters, 103 15, 150502.

Harsanyi, J. (1967). Games with incomplete information played by Bayesian players. Management Science, 14, 159–182.

Hart, P. E., Nilsson, N. J., and Raphael, B. (1968). A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics, SSC-4(2), 100–107.

Hart, T. P. and Edwards, D. J. (1961). The tree prune (TP) algorithm. Artificial intelligence project memo, MIT.

Hartley, H. (1958). Maximum likelihood estimation from incomplete data. Biometrics, 14, 174–194.

Hartley, R. and Zisserman, A. (2000). Multiple view geometry in computer vision. Cambridge University Press.

Hashimoto, K., Xiong, C., Tsuruoka, Y., and Socher, R. (2016). A joint many-task model: Growing a neural network for multiple NLP tasks. arXiv:1611.01587.

Haslum, P., Botea, A., Helmert, M., Bonet, B., and Koenig, S. (2007). Domain-independent construction of pattern database heuristics for cost-optimal planning. In AAAI-07.

Haslum, P. and Geffner, H. (2001). Heuristic planning with time and resources. In Proc. IJCAI-01 Workshop on Planning with Resources.

Haslum, P. (2006). Improving heuristics through relaxed search – An analysis of TP4 and HSP*a in the 2004 planning competition. JAIR, 25, 233–267.

Hastie, T. and Tibshirani, R. (1996). Discriminant adaptive nearest neighbor classification and regression. In NeurIPS 8.

Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference and Prediction (2nd edition). Springer-Verlag.

Hastings, W. K. (1970). Monte Carlo sampling methods using Markov chains and their applications. Biometrika, 57, 97–109.

Hatem, M. and Ruml, W. (2014). Simpler bounded suboptimal search. In AAAI-14.

Haugeland, J. (1985). Artificial Intelligence: The Very Idea. MIT Press.

Havelund, K., Lowry, M., Park, S., Pecheur, C., Penix, J., Visser, W., and White, J. L. (2000). Formal analysis of the remote agent before and after flight. In Proc. 5th NASA Langley Formal Methods Workshop.

Havenstein, H. (2005). Spring comes to AI winter. Computer World, Fe. 14.

Hawkins, J. (1961). Self-organizing systems: A review and commentary. Proc. IRE, 49, 31– 48.

Hay, N., Russell, S. J., Shimony, S. E., and Tolpin, D. (2012). Selecting computations: Theory and applications. In UAI-12.

Hayes, P. J. (1978). The naive physics manifesto. In Michie, D. (Ed.), Expert Systems in the Microelectronic Age. Edinburgh University Press.

Hayes, P. J. (1979). The logic of frames. In Metzing, D. (Ed.), Frame Conceptions and Text Understanding. de Gruyter.

Hayes, P. J. (1985a). Naive physics I: Ontology for liquids. In Hobbs, J. R. and Moore, R. C. (Eds.), Formal Theories of the Commonsense World, chap. 3. Ablex.

Hayes, P. J. (1985b). The second naive physics manifesto. In Hobbs, J. R. and Moore, R. C. (Eds.), Formal Theories of the Commonsense World, chap. 1. Ablex.

Hays, J. and Efros, A. (2007). Scene completion Using millions of photographs. ACM Transactions on Graphics (SIGGRAPH), 26.

He, H., Bai, Y., Garcia, E. A., and Li, S. (2008). ADASYN: Adaptive synthetic sampling approach for imbalanced learning. In 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In CVPR-16.

Heawood, P. J. (1890). Map colouring theorem. Quarterly Journal of Mathematics, 24, 332– 338.

Hebb, D. O. (1949). The Organization of Behavior. Wiley.

Heckerman, D. (1986). Probabilistic interpretation for MYCIN’s certainty factors. In Kanal, L. N. and Lemmer, J. F. (Eds.), UAI 2. Elsevier.

Heckerman, D. (1991). Probabilistic Similarity Networks. MIT Press.

Heckerman, D. (1998). A tutorial on learning with Bayesian networks. In Jordan, M. I. (Ed.), Learning in graphical models. Kluwer.

Heckerman, D., Geiger, D., and Chickering, D. M. (1994). Learning Bayesian networks: The combination of knowledge and statistical data. Technical report, Microsoft Research.

Heess, N., Wayne, G., Silver, D., Lillicrap, T., Erez, T., and Tassa, Y. (2016). Learning continuous control policies by stochastic value gradients. In NeurIPS 28.

Heidegger, M. (1927). Being and Time. SCM Press.

Heinlein, R. A. (1973). Time Enough for Love. Putnam.

Held, M. and Karp, R. M. (1970). The traveling salesman problem and minimum spanning trees. Operations Research, 18, 1138–1162.

Helmert, M. (2001). On the complexity of planning in transportation domains. In ECP-01.

Helmert, M. (2006). The fast downward planning system. JAIR, 26, 191–246.

Helmert, M. and Röger, G. (2008). How good is almost perfect? In AAAI-08.

Helmert, M., Röger, G., and Karpas, E. (2011). Fast downward stone soup: A baseline for building planner portfolios. In ICAPS.

Hendeby, G., Karlsson, R., and Gustafsson, F. (2010). Particle filtering: The need for speed. EURASIP J. Adv. Sig. Proc., June.

Henrion, M. (1988). Propagation of uncertainty in Bayesian networks by probabilistic logic sampling. In Lemmer, J. F. and Kanal, L. N. (Eds.), UAI 2. Elsevier.

Henzinger, T. A. and Sastry, S. (Eds.). (1998). Hybrid Systems: Computation and Control. Springer-Verlag.

Herbrand, J. (1930). Recherches sur la Théorie de la Démonstration. Ph.D. thesis, University of Paris.

Herbrich, R., Minka, T., and Graepel, T. (2007). TrueSkill: A Bayesian skill rating system. In NeurIPS 19.

Hernández-Orallo, J. (2016). Evaluation in artificial intelligence: From task-oriented to ability-oriented measurement. Artificial Intelligence Review, 48, 397–447.

Hess, C. and Ostrom, E. (2007). Understanding Knowledge as a Commons. MIT Press.

Hewitt, C. (1977). Viewing control structures as patterns of passing messages. AIJ, 8, 323– 364.

Hewitt, C. (1969). PLANNER: a language for proving theorems in robots. In IJCAI-69.

Hezaveh, Y. D., Levasseur, L. P., and Marshall, P. J. (2017). Fast automated analysis of strong gravitational lenses with convolutional neural networks. Nature, 548, 555–557.

Hierholzer, C. (1873). Über die Möglichkeit, einen Linienzug ohne Wiederholung und ohne Unterbrechung zu umfahren. Mathematische Annalen, 6, 30–32.

Hilbert, M. and Lopez, P. (2011). The world’s technological capacity to store, communicate, and compute information. Science, 332, 60–65.

Hilgard, E. R. and Bower, G. H. (1975). Theories of Learning (4th edition). Prentice-Hall.

Hind, M., Mehta, S., Mojsilovic, A., Nair, R., Ramamurthy, K. N., Olteanu, A., and Varshney, K. R. (2018). Increasing trust in AI services through supplier’s declarations of conformity. arXiv:1808.07261.

Hintikka, J. (1962). Knowledge and Belief. Cornell University Press.

Hinton, G. E. and Anderson, J. A. (1981). Parallel Models of Associative Memory. Lawrence Erlbaum.

Hinton, G. E. and Nowlan, S. J. (1987). How learning can guide evolution. Complex Systems, 1, 495–502.

Hinton, G. E. and Sejnowski, T. (1983). Optimal perceptual inference. In CVPR-83.

Hinton, G. E. and Sejnowski, T. (1986). Learning and relearning in Boltzmann machines. In Rumelhart, D. E. and McClelland, J. L. (Eds.), Parallel Distributed Processing. MIT Press.

Hinton, G. E. (1987). Learning translation invariant recognition in a massively parallel network. In Goos, G. and Hartmanis, J. (Eds.), PARLE: Parallel Architectures and Languages Europe. Springer-Verlag.

Hinton, G. E., Deng, L., Yu, D., Dahl, G., Mohamed, A. R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., and Kingsbury, B. (2012). Deep neural networks for acoustic modeling in speech recognition. Signal Processing Magazine, 29, 82 – 97.

Hinton, G. E., Osindero, S., and Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527–1554.

Hirth, M., Hoßfeld, T., and Tran-Gia, P. (2013). Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms. Mathematical and Computer Modelling, 57, 2918–2932.

Ho, M. K., Littman, M. L., MacGlashan, J., Cushman, F., and Austerweil, J. L. (2017). Showing versus doing: Teaching by demonstration. In NeurIPS 29.

Ho, T. K. (1995). Random decision forests. In Proc. 3rd Int’l Conf. on Document Analysis and Recognition.

Hobbs, J. R. (1990). Literature and Cognition. CSLI Press.

Hobbs, J. R. and Moore, R. C. (Eds.). (1985). Formal Theories of the Commonsense World. Ablex.

Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Technische Universität München.

Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9, 1735–1780.

Hoffman, M., Bach, F. R., and Blei, D. M. (2011). Online learning for latent Dirichlet allocation. In NeurIPS 23.

Hoffmann, J. (2001). FF: The fast-forward planning system. AIMag, 22, 57–62.

Hoffmann, J. and Brafman, R. I. (2006). Conformant planning via heuristic forward search: A new approach. AIJ, 170, 507–541.

Hoffmann, J. and Brafman, R. I. (2005). Contingent planning via heuristic forward search with implicit belief states. In ICAPS-05.

Hoffmann, J. (2005). Where “ignoring delete lists” works: Local search topology in planning benchmarks. JAIR, 24, 685–758.

Hoffmann, J. and Nebel, B. (2001). The FF planning system: Fast plan generation through heuristic search. JAIR, 14, 253–302.

Hoffmann, J., Sabharwal, A., and Domshlak, C. (2006). Friends or foes? An AI planning perspective on abstraction and search. In ICAPS-06.

Hofleitner, A., Herring, R., Abbeel, P., and Bayen, A. M. (2012). Learning the dynamics of arterial traffic from probe data using a dynamic Bayesian network. IEEE Transactions on Intelligent Transportation Systems, 13, 1679–1693.

Hogan, N. (1985). Impedance control: An approach to manipulation. Parts I, II, and III. J. Dynamic Systems, Measurement, and Control, 107, 1–24.

Hoiem, D., Efros, A., and Hebert, M. (2007). Recovering surface layout from an image. IJCV, 75, 151–172.

Holland, J. H. (1975). Adaption in Natural and Artificial Systems. University of Michigan Press.

Holland, J. H. (1995). Hidden Order: How Adaptation Builds Complexity. Addison-Wesley.

Holte, R. C., Felner, A., Sharon, G., and Sturtevant, N. R. (2016). Bidirectional search that is guaranteed to meet in the middle. In AAAI-16.

Holzmann, G. J. (1997). The Spin model checker. IEEE Transactions on Software Engineering, 23, 279–295.

Hood, A. (1824). Case 4th—28 July 1824 (Mr. Hood’s cases of injuries of the brain). Phrenological Journal and Miscellany, 2, 82–94.

Hooker, J. (1995). Testing heuristics: We have it all wrong. J. Heuristics, 1, 33–42.

Hoos, H. H. and Stützle, T. (2004). Stochastic Local Search: Foundations and Applications. Morgan Kaufmann.

Hoos, H. H. and Tsang, E. (2006). Local search methods. In Rossi, F., van Beek, P., and Walsh, T. (Eds.), Handbook of Constraint Processing. Elsevier.

Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. PNAS, 79, 2554–2558.

Horn, A. (1951). On sentences which are true of direct unions of algebras. JSL, 16, 14–21.

Horn, B. K. P. (1970). Shape from shading: A method for obtaining the shape of a smooth opaque object from one view. Technical report, MIT Artificial Intelligence Laboratory.

Horn, B. K. P. and Brooks, M. J. (1989). Shape from Shading. MIT Press.

Horn, K. V. (2003). Constructing a logic of plausible inference: A guide to Cox’s theorem. IJAR, 34, 3–24.

Horning, J. J. (1969). A Study of Grammatical Inference. Ph.D. thesis, Stanford University.

Horswill, I. (2000). Functional programming of behavior-based systems. Autonomous Robots, 9, 83–93.

Horvitz, E. J. (1987). Problem-solving design: Reasoning about computational value, tradeoffs, and resources. In Proc. Second Annual NASA Research Forum.

Horvitz, E. J. and Barry, M. (1995). Display of information for time-critical decision making. In UAI-95.

Horvitz, E. J., Breese, J. S., Heckerman, D., and Hovel, D. (1998). The Lumiere project: Bayesian user modeling for inferring the goals and needs of software users. In UAI-98.

Horvitz, E. J., Breese, J. S., and Henrion, M. (1988). Decision theory in expert systems and artificial intelligence. IJAR, 2, 247–302.

Horvitz, E. J. and Breese, J. S. (1996). Ideal partition of resources for metareasoning. In AAAI-96.

Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components. J. Ed. Psych., 24, 417–441.

Howard, J. and Gugger, S. (2020). Deep Learning for Coders with fastai and PyTorch. O’Reilly.

Howard, J. and Ruder, S. (2018). Fine-tuned language models for text classification. arXiv:1801.06146.

Howard, R. A. (1960). Dynamic Programming and Markov Processes. MIT Press.

Howard, R. A. (1966). Information value theory. IEEE Transactions on Systems Science and Cybernetics, SSC-2, 22–26.

Howard, R. A. (1989). Microrisks for medical decision analysis. Int. J. Technology Assessment in Health Care, 5, 357–370.

Howard, R. A. and Matheson, J. E. (1984). Influence diagrams. In Howard, R. A. and Matheson, J. E. (Eds.), Readings on the Principles and Applications of Decision Analysis. Strategic Decisions Group.

Howe, D. (1987). The computational behaviour of Girard’s paradox. In LICS-87.

Howson, C. (2003). Probability and logic. J. Applied Logic, 1, 151–165.

Hsiao, K., Kaelbling, L. P., and Lozano-Perez, T. (2007). Grasping POMDPs. In ICRA-07.

Hsu, F.-H. (2004). Behind Deep Blue: Building the Computer that Defeated the World Chess Champion. Princeton University Press.

Hsu, F.-H., Anantharaman, T. S., Campbell, M. S., and Nowatzyk, A. (1990). A grandmaster chess machine. Scientific American, 263, 44–50.

Hu, J. and Wellman, M. P. (1998). Multiagent reinforcement learning: Theoretical framework and an algorithm. In ICML-98.

Hu, J. and Wellman, M. P. (2003). Nash Q-learning for general-sum stochastic games. JMLR, 4, 1039–1069.

Huang, T., Koller, D., Malik, J., Ogasawara, G., Rao, B., Russell, S. J., and Weber, J. (1994). Automatic symbolic traffic scene analysis using belief networks. In AAAI-94.

Huang, T. and Russell, S. J. (1998). Object identification: A Bayesian analysis with application to traffic surveillance. AIJ, 103, 1–17.

Hubel, D. H. and Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J. Physiology, 160, 106–154.

Hubel, D. H. and Wiesel, T. N. (1968). Receptive fields and functional architecture of monkey striate cortex. J. Physiology, 195, 215–243.

Hubel, D. H. (1988). Eye, Brain, and Vision. W. H. Freeman.

Hubel, D. H. and Wiesel, T. N. (1959). Receptive fields of single neurons in the cat’s striate cortex. Journal of Physiology, 148, 574–591.

Huddleston, R. D. and Pullum, G. K. (2002). The Cambridge Grammar of the English Language. Cambridge University Press.

Huffman, D. A. (1971). Impossible objects as nonsense sentences. In Meltzer, B. and Michie, D. (Eds.), Machine Intelligence 6. Edinburgh University Press.

Hughes, B. D. (1995). Random Walks and Random Environments, Vol. 1: Random Walks. Oxford University Press.

Hughes, G. E. and Cresswell, M. J. (1996). A New Introduction to Modal Logic. Routledge.

Huhns, M. N. and Singh, M. (Eds.). (1998). Readings in Agents. Morgan Kaufmann.

Hume, D. (1739). A Treatise of Human Nature (2nd edition). Republished by Oxford University Press, 1978, Oxford.

Humphrys, M. (2008). How my program passed the Turing test. In Epstein, R., Roberts, G., and Beber, G. (Eds.), Parsing the Turing Test. Springer.

Hunsberger, L. and Grosz, B. J. (2000). A combinatorial auction for collaborative planning. In Int. Conference on Multi-Agent Systems.

Hunt, W. and Brock, B. (1992). A formal HDL and its use in the FM9001 verification. Phil. Trans. Roy. Soc., 339.

Hunter, L. and States, D. J. (1992). Bayesian classification of protein structure. IEEE Expert, 7, 67–75.

Hur, C.-K., Nori, A. V., Rajamani, S. K., and Samuel, S. (2014). Slicing probabilistic programs. In PLDI-14.

Hurst, M. (2000). The Interpretation of Text in Tables. Ph.D. thesis, Edinburgh.

Hurwicz, L. (1973). The design of mechanisms for resource allocation. American Economic Review Papers and Proceedings, 63, 1–30.

Huth, M. and Ryan, M. (2004). Logic in Computer Science: Modelling and Reasoning About Systems (2nd edition). Cambridge University Press.

Huttenlocher, D. and Ullman, S. (1990). Recognizing solid objects by alignment with an image. IJCV, 5, 195–212.

Hutter, F., Kotthoff, L., and Vanschoren, J. (2019). Automated Machine Learning. Springer.

Huygens, C. (1657). De ratiociniis in ludo aleae. In van Schooten, F. (Ed.), Exercitionum Mathematicorum. Elsevirii, Amsterdam. Translated into English by John Arbuthnot

(1692).

Huyn, N., Dechter, R., and Pearl, J. (1980). Probabilistic analysis of the complexity of A*. AIJ, 15, 241–254.

Huynh, V. A. and Roy, N. (2009). icLQG: Combining local and global optimization for control in information space. In ICRA-09.

Hwa, R. (1998). An empirical evaluation of probabilistic lexicalized tree insertion grammars. In ACL-98.

Hwang, C. H. and Schubert, L. K. (1993). EL: A formal, yet natural, comprehensive knowledge representation. In AAAI-93.

Hyafil, L. and Rivest, R. (1976). Constructing optimal binary decision trees is NP-complete. Information Processing Letters, 5, 15–17.

Ieong, S. and Shoham, Y. (2005). Marginal contribution nets: A compact representation scheme for coalitional games. In Proc. Sixth ACM Conference on Electronic Commerce (EC’05).

Ingerman, P. Z. (1967). Panini–Backus form suggested. CACM, 10, 137.

Intille, S. and Bobick, A. (1999). A framework for recognizing multi-agent action from visual evidence. In AAAI-99.

Ioffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167.

Irpan, A. (2018). Deep reinforcement learning doesn’t work yet. www.alexirpan.com/2018/ 02/14/rl-hard.html.

Isard, M. and Blake, A. (1996). Contour tracking by stochastic propagation of conditional density. In ECCV-96.

Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. (2017). Image-to-image translation with conditional adversarial networks. In CVPR-17.

Jaakkola, T. and Jordan, M. I. (1996). Computing upper and lower bounds on likelihoods in intractable networks. In UAI-96.

Jacobson, D. H. and Mayne, D. Q. (1970). Differential Dynamic Programming. North-Holland.

Jaderberg, M., Czarnecki, W. M., Dunning, I., Marris, L., Lever, G., Castaneda, A. G., Beattie, C., Rabinowitz, N. C., Morcos, A. S., Ruderman, A., et al. (2019). Human-level performance in 3D multiplayer games with population-based reinforcement learning. Science, 364, 859–865.

Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W. M., Donahue, J., Razavi, A., Vinyals, O., Green, T., Dunning, I., Simonyan, K., Fernando, C., and Kavukcuoglu, K. (2017). Population based training of neural networks. arXiv:1711.09846.

Jaffar, J. and Lassez, J.-L. (1987). Constraint logic programming. In Proc. Fourteenth ACM POPL Conference. Association for Computing Machinery.

Jaffar, J., Michaylov, S., Stuckey, P. J., and Yap, R. H. C. (1992). The CLP(R) language and system. ACM Transactions on Programming Languages and Systems, 14, 339–395.

Jain, D., Barthels, A., and Beetz, M. (2010). Adaptive Markov logic networks: Learning statistical relational models with dynamic parameters. In ECAI-10.

Jain, D., Kirchlechner, B., and Beetz, M. (2007). Extending Markov logic to model probability distributions in relational domains. In 30th Annual German Conference on AI (KI).

James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013). An Introduction to Statistical Learning with Applications in R. Springer-Verlag.

Jarrett, K., Kavukcuoglu, K., Ranzato, M., and LeCun, Y. (2009). What is the best multi-stage architecture for object recognition? In ICCV-09.

Jaynes, E. T. (2003). Probability Theory: The Logic of Science. Cambridge Univ. Press.

Jeffrey, R. C. (1983). The Logic of Decision (2nd edition). University of Chicago Press.

Jeffreys, H. (1948). Theory of Probability. Oxford.

Jelinek, F. (1976). Continuous speech recognition by statistical methods. Proc. IEEE, 64, 532–556.

Jelinek, F. and Mercer, R. L. (1980). Interpolated estimation of Markov source parameters from sparse data. In Proc. Workshop on Pattern Recognition in Practice.

Jennings, H. S. (1906). Behavior of the Lower Organisms. Columbia University Press.

Jenniskens, P., Betlem, H., Betlem, J., and Barifaijo, E. (1994). The Mbale meteorite shower. Meteoritics, 29, 246–254.

Jensen, F. V. (2007). Bayesian Networks and Decision Graphs. Springer-Verlag.

Ji, Z., Lipton, Z. C., and Elkan, C. (2014). Differential privacy and machine learning: A survey and review. arXiv:1412.7584.

Jiang, H. and Nachum, O. (2019). Identifying and correcting label bias in machine learning. arXiv:1901.04966.

Jimenez, P. and Torras, C. (2000). An efficient algorithm for searching implicit AND/OR graphs with cycles. AIJ, 124, 1–30.

Joachims, T. (2001). A statistical learning model of text classification with support vector machines. In SIGIR-01.

Johnson, M. (1998). PCFG models of linguistic tree representations. Comput. Linguist., 24, 613–632.

Johnson, W. W. and Story, W. E. (1879). Notes on the “15” puzzle. American Journal of Mathematics, 2, 397–404.

Johnston, M. D. and Adorf, H.-M. (1992). Scheduling with neural networks: The case of the Hubble space telescope. Computers and Operations Research, 19, 209–240.

Jonathan, P. J. Y., Fung, C. C., and Wong, K. W. (2009). Devious chatbots-interactive malware with a plot. In FIRA RoboWorld Congress.

Jones, M. and Love, B. C. (2011). Bayesian fundamentalism or enlightenment? On the explanatory status and theoretical contributions of Bayesian models of cognition. BBS, 34, 169–231.

Jones, R. M., Laird, J., and Nielsen, P. E. (1998). Automated intelligent pilots for combat flight simulation. In AAAI-98.

Jones, R., McCallum, A., Nigam, K., and Riloff, E. (1999). Bootstrapping for text learning tasks. In Proc. IJCAI-99 Workshop on Text Mining: Foundations, Techniques, and Applications.

Jones, T. (2007). Artificial Intelligence: A Systems Approach. Infinity Science Press.

Jonsson, A., Morris, P., Muscettola, N., Rajan, K., and Smith, B. (2000). Planning in interplanetary space: Theory and practice. In AIPS-00.

Jordan, M. I. (2005). Dirichlet processes, Chinese restaurant processes and all that. Tutorial presentation at the NeurIPS Conference.

Jordan, M. I. (1986). Serial order: A parallel distributed processing approach. Tech. rep., UCSD Institute for Cognitive Science.

Jordan, M. I., Ghahramani, Z., Jaakkola, T., and Saul, L. K. (1999). An introduction to variational methods for graphical models. Machine Learning, 37, 183–233.

Jouannaud, J.-P. and Kirchner, C. (1991). Solving equations in abstract algebras: A rulebased survey of unification. In Lassez, J.-L. and Plotkin, G. (Eds.), Computational Logic. MIT Press.

Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. (2016). Bag of tricks for efficient text classification. arXiv:1607.01759.

Jouppi, N. P., Young, C., Patil, N., Patterson, D. A., et al. (2017). In-datacenter performance analysis of a tensor processing unit. In ACM/IEEE 44th International Symposium on Computer Architecture.

Joy, B. (2000). Why the future doesn’t need us. Wired, 8.

Jozefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., and Wu, Y. (2016). Exploring the limits of language modeling. arXiv:1602.02410.

Jozefowicz, R., Zaremba, W., and Sutskever, I. (2015). An empirical exploration of recurrent network architectures. In ICML-15.

Juels, A. and Wattenberg, M. (1996). Stochastic hillclimbing as a baseline method for evaluating genetic algorithms. In NeurIPS 8.

Julesz, B. (1971). Foundations of Cyclopean Perception. University of Chicago Press.

Julian, K. D., Kochenderfer, M. J., and Owen, M. P. (2018). Deep neural network compression for aircraft collision avoidance systems. arXiv:1810.04240.

Juliani, A., Berges, V., Vckay, E., Gao, Y., Henry, H., Mattar, M., and Lange, D. (2018). Unity: A general platform for intelligent agents. arXiv:1809.02627.

Junker, U. (2003). The logic of ilog (j)configurator: Combining constraint programming with a description logic. In Proc. IJCAI-03 Configuration Workshop.

Jurafsky, D. and Martin, J. H. (2020). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition (3rd edition). Prentice-Hall.

Kadane, J. B. and Simon, H. A. (1977). Optimal strategies for a class of constrained sequential problems. Annals of Statistics, 5, 237–255.

Kadane, J. B. and Larkey, P. D. (1982). Subjective probability and the theory of games. Management Science, 28, 113–120.

Kaelbling, L. P., Littman, M. L., and Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. AIJ, 101, 99–134.

Kaelbling, L. P. and Rosenschein, S. J. (1990). Action and planning in embedded agents. Robotics and Autonomous Systems, 6, 35–48.

Kager, R. (1999). Optimality Theory. Cambridge University Press.

Kahn, H. and Marshall, A. W. (1953). Methods of reducing sample size in Monte Carlo computations. Operations Research, 1, 263–278.

Kahn, H. (1950a). Random sampling (Monte Carlo) techniques in neutron attenuation problems–I. Nucleonics, 6, 27–passim.

Kahn, H. (1950b). Random sampling (Monte Carlo) techniques in neutron attenuation problems–II. Nucleonics, 6, 60–65.

Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux.

Kahneman, D., Slovic, P., and Tversky, A. (Eds.). (1982). Judgment under Uncertainty: Heuristics and Biases. Cambridge University Press.

Kahneman, D. and Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47, 263–291.

Kaindl, H. and Khorsand, A. (1994). Memory-bounded bidirectional search. In AAAI-94.

Kalman, R. (1960). A new approach to linear filtering and prediction problems. J. Basic Engineering, 82, 35–46.

Kambhampati, S. (1994). Exploiting causal structure to control retrieval and refitting during plan reuse. Computational Intelligence, 10, 213–244.

Kanade, T., Thorpe, C., and Whittaker, W. (1986). Autonomous land vehicle project at CMU. In ACM Fourteenth Annual Conference on Computer Science.

Kanal, E. (2017). Machine learning in cybersecurity. CMU SEI Blog, insights.sei.cmu.edu/sei_blog/2017/06/machine-learning-in-cybersecurity.html.

Kanazawa, A., Black, M., Jacobs, D., and Malik, J. (2018a). End-to-end recovery of human shape and pose. In CVPR-18.

Kanazawa, A., Tulsiani, M., Efros, A., and Malik, J. (2018b). Learning category-specific mesh reconstruction from image collections. In ECCV-18.

Kanazawa, K., Koller, D., and Russell, S. J. (1995). Stochastic simulation algorithms for dynamic probabilistic networks. In UAI-95.

Kang, S. M. and Wildes, R. P. (2016). Review of action recognition and detection methods. arXiv:1610.06906.

Kanter, J. M. and Veeramachaneni, K. (2015). Deep feature synthesis: Towards automating data science endeavors. In Proc. IEEE Int’l Conf. on Data Science and Advanced Analytics.

Kantorovich, L. V. (1939). Mathematical methods of organizing and planning production. Published in translation in Management Science, 6(4), 366–422, 1960.

Kaplan, D. and Montague, R. (1960). A paradox regained. Notre Dame Formal Logic, 1, 79– 90.

Karaboga, D. and Basturk, B. (2007). A powerful and efficient algorithm for numerical function optimization: Artificial bee colony (ABC) algorithm. Journal of global optimization, 39, 459–471.

Karamchandani, A., Bjerager, P., and Cornell, C. A. (1989). Adaptive importance sampling. In Proc. Fifth International Conference on Structural Safety and Reliability.

Karmarkar, N. (1984). A new polynomial-time algorithm for linear programming. Combinatorica, 4, 373–395.

Karp, R. M. (1972). Reducibility among combinatorial problems. In Miller, R. E. and Thatcher, J. W. (Eds.), Complexity of Computer Computations. Plenum.

Karpathy, A. (2015). The unreasonable effectiveness of recurrent neural networks. Andrej Karpathy blog, karpathy.github.io/2015/05/21/rnn-effectiveness/.

Karpathy, A. and Fei-Fei, L. (2015). Deep visual-semantic alignments for generating image descriptions. In CVPR-15.

Karras, T., Aila, T., Laine, S., and Lehtinen, J. (2017). Progressive growing of GANs for improved quality, stability, and variation. arXiv:1710.10196.

Karsch, K., Hedau, V., Forsyth, D., and Hoiem, D. (2011). Rendering synthetic objects into legacy photographs. In SIGGRAPH Asia.

Kartam, N. A. and Levitt, R. E. (1990). A constraint-based approach to construction planning of multi-story buildings. In Expert Planning Systems. Institute of Electrical Engineers.

Kasami, T. (1965). An efficient recognition and syntax analysis algorithm for context-free languages. Tech. rep., Air Force Cambridge Research Laboratory.

Katehakis, M. N. and Veinott, A. F. (1987). The multi-armed bandit problem: Decomposition and computation. Mathematics of Operations Research, 12, 185–376.

Katz, B. (1997). Annotating the world wide web using natural language. In RIAO ’97.

Kaufmann, M., Manolios, P., and Moore, J. S. (2000). Computer-Aided Reasoning: An Approach. Kluwer.

Kautz, H. (2006). Deconstructing planning as satisfiability. In AAAI-06.

Kautz, H., McAllester, D. A., and Selman, B. (1996). Encoding plans in propositional logic. In KR-96.

Kautz, H. and Selman, B. (1992). Planning as satisfiability. In ECAI-92.

Kautz, H. and Selman, B. (1998). BLACKBOX: A new approach to the application of theorem proving to problem solving. Working Notes of the AIPS-98 Workshop on Planning as Combinatorial Search.

Kavraki, L., Svestka, P., Latombe, J.-C., and Overmars, M. (1996). Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE Transactions on Robotics and Automation, 12, 566–580.

Kazemi, S. M., Kimmig, A., Van den Broeck, G., and Poole, D. (2017). New liftable classes for first-order probabilistic inference. In NeurIPS 29.

Kearns, M. (1990). The Computational Complexity of Machine Learning. MIT Press.

Kearns, M., Mansour, Y., and Ng, A. Y. (2000). Approximate planning in large POMDPs via reusable trajectories. In NeurIPS 12.

Kearns, M. and Singh, S. (1998). Near-optimal reinforcement learning in polynomial time. In ICML-98.

Kearns, M. and Vazirani, U. (1994). An Introduction to Computational Learning Theory. MIT Press.

Kearns, M. (1988). Thoughts on hypothesis boosting.

Kearns, M., Mansour, Y., and Ng, A. Y. (2002). A sparse sampling algorithm for nearoptimal planning in large Markov decision processes. Machine Learning, 49, 193–208.

Kebeasy, R. M., Hussein, A. I., and Dahy, S. A. (1998). Discrimination between natural earthquakes and nuclear explosions using the Aswan Seismic Network. Annali di Geofisica, 41, 127–140.

Keeney, R. L. (1974). Multiplicative utility functions. Operations Research, 22, 22–34.

Keeney, R. L. and Raiffa, H. (1976). Decisions with Multiple Objectives: Preferences and Value Tradeoffs. Wiley.

Kelley, H. J. (1960). Gradient theory of optimal flight paths. ARS Journal, 30, 947–954.

Kemp, M. (Ed.). (1989). Leonardo on Painting: An Anthology of Writings. Yale University Press.

Kempe, A. B. (1879). On the geographical problem of the four-colors. American Journal of Mathematics, 2, 193–200.

Kephart, J. O. and Chess, D. M. (2003). The vision of autonomic computing. IEEE Computer, 36, 41–50.

Kersting, K., Raedt, L. D., and Kramer, S. (2000). Interpreting Bayesian logic programs. In Proc. AAAI-00 Workshop on Learning Statistical Models from Relational Data.

Keskar, N. S., McCann, B., Varshney, L., Xiong, C., and Socher, R. (2019). CTRL: A conditional transformer language model for controllable generation. arXiv:1909.

Keynes, J. M. (1921). A Treatise on Probability. Macmillan.

Khare, R. (2006). Microformats: The next (small) thing on the semantic web. IEEE Internet Computing, 10, 68–75.

Khatib, O. (1986). Real-time obstacle avoidance for robot manipulator and mobile robots. Int. J. Robotics Research, 5, 90–98.

Kim, B., Khanna, R., and Koyejo, O. O. (2017). Examples are not enough, learn to criticize! Criticism for interpretability. In NeurIPS 29.

Kim, J. H. (1983). CONVINCE: A Conversational Inference Consolidation Engine. Ph.D. thesis, Department of Computer Science, UCLA.

Kim, J. H. and Pearl, J. (1983). A computational model for combined causal and diagnostic reasoning in inference systems. In IJCAI-83.

Kim, J.-H., Lee, C.-H., Lee, K.-H., and Kuppuswamy, N. (2007). Evolving personality of a genetic robot in ubiquitous environment. In Proc. 16th IEEE International Symposium on Robot and Human Interactive Communication.

Kim, T. W. (2018). Explainable artificial intelligence (XAI), the goodness criteria and the grasp-ability test. arXiv:1810.09598.

Kingma, D. P. and Welling, M. (2013). Auto-encoding variational Bayes. arXiv:1312.6114.

Kirk, D. E. (2004). Optimal Control Theory: An Introduction. Dover.

Kirkpatrick, S., Gelatt, C. D., and Vecchi, M. P. (1983). Optimization by simulated annealing. Science, 220, 671–680.

Kisynski, J. and Poole, D. (2009). Lifted aggregation in directed first-order probabilistic models. In IJCAI-09.

Kitaev, N., Kaiser, L., and Levskaya, A. (2020). Reformer: The efficient transformer. arXiv:2001.04451.

Kitaev, N. and Klein, D. (2018). Constituency parsing with a self-attentive encoder. arXiv:1805.01052.

Kitani, K. M., abd James Andrew Bagnell, B. D. Z., and Hebert, M. (2012). Activity forecasting. In ECCV-12.

Kitano, H., Asada, M., Kuniyoshi, Y., Noda, I., and Osawa, E. (1997). RoboCup: The robot world cup initiative. In Proc. First International Conference on Autonomous Agents.

Kjaerulff, U. (1992). A computational scheme for reasoning in dynamic probabilistic networks. In UAI-92.

Klarman, H. E., Francis, J., and Rosenthal, G. D. (1968). Cost effectiveness analysis applied to the treatment of chronic renal disease. Medical Care, 6, 48–54.

Klein, D. and Manning, C. (2001). Parsing with treebank grammars: Empirical bounds, theoretical models, and the structure of the Penn treebank. In ACL-01.

Klein, D. and Manning, C. (2003). A* parsing: Fast exact Viterbi parse selection. In HLTNAACL-03.

Kleinberg, J. M., Mullainathan, S., and Raghavan, M. (2016). Inherent trade-offs in the fair determination of risk scores. arXiv:1609.05807.

Klemperer, P. (2002). What really matters in auction design. J. Economic Perspectives, 16.

Kneser, R. and Ney, H. (1995). Improved backing-off for M-gram language modeling. In ICASSP-95.

Knoblock, C. A. (1991). Search reduction in hierarchical problem solving. In AAAI-91.

Knuth, D. E. (1964). Representing numbers using only one 4. Mathematics Magazine, 37, 308–310.

Knuth, D. E. (1975). An analysis of alpha–beta pruning. AIJ, 6, 293–326.

Knuth, D. E. (2015). The Art of Computer Programming, Vol. 4, Fascicle 6: Satisfiability. Addison-Wesley.

Knuth, D. E. and Bendix, P. B. (1970). Simple word problems in universal algebras. In Leech, J. (Ed.), Computational Problems in Abstract Algebra. Pergamon.

Kober, J., Bagnell, J. A., and Peters, J. (2013). Reinforcement learning in robotics: A survey. International Journal of Robotics Research, 32, 1238–1274.

Koch, C. (2019). The Feeling of Life Itself. MIT Press.

Kochenderfer, M. J. (2015). Decision Making Under Uncertainty: Theory and Application. MIT Press.

Kocsis, L. and Szepesvari, C. (2006). Bandit-based Monte-Carlo planning. In ECML-06.

Koditschek, D. (1987). Exact robot navigation by means of potential functions: Some topological considerations. In ICRA-87.

Koehn, P. (2009). Statistical Machine Translation. Cambridge University Press.

Koelsch, S. and Siebel, W. A. (2005). Towards a neural basis of music perception. Trends in Cognitive Sciences, 9, 578–584.

Koenderink, J. J. (1990). Solid Shape. MIT Press.

Koenderink, J. J. and van Doorn, A. J. (1991). Affine structure from motion. J. Optical Society of America A, 8, 377–385.

Koenig, S. (1991). Optimal probabilistic and decision-theoretic planning using Markovian decision theory. Master’s report, Computer Science Division, University of California, Berkeley.

Koenig, S. (2000). Exploring unknown environments with real-time search or reinforcement learning. In NeurIPS 12.

Koenig, S. (2001). Agent-centered search. AIMag, 22, 109–131.

Koenig, S. and Likhachev, M. (2002). D* Lite. AAAI-15, 15.

Koenig, S., Likhachev, M., and Furcy, D. (2004). Lifelong planning A*. AIJ, 155, 93–146.

Kolesky, D. B., Truby, R. L., Gladman, A. S., Busbee, T. A., Homan, K. A., and Lewis, J. A. (2014). 3D bioprinting of vascularized, heterogeneous cell-laden tissue constructs. Advanced Materials, 26, 3124–3130.

Koller, D., Meggido, N., and von Stengel, B. (1996). Efficient computation of equilibria for extensive two-person games. Games and Economic Behaviour, 14, 247–259.

Koller, D. and Pfeffer, A. (1997). Representations and solutions for game-theoretic problems. AIJ, 94, 167–215.

Koller, D. and Pfeffer, A. (1998). Probabilistic frame-based systems. In AAAI-98.

Koller, D. and Friedman, N. (2009). Probabilistic Graphical Models: Principles and Techniques. MIT Press.

Koller, D., McAllester, D. A., and Pfeffer, A. (1997). Effective Bayesian inference for stochastic programs. In AAAI-97.

Koller, D. and Parr, R. (2000). Policy iteration for factored MDPs. In UAI-00.

Koller, D. and Sahami, M. (1997). Hierarchically classifying documents using very few words. In ICML-97.

Kolmogorov, A. N. (1941). Interpolation und extrapolation von stationaren zufalligen folgen. Bulletin of the Academy of Sciences of the USSR, Ser. Math. 5, 3–14.

Kolmogorov, A. N. (1950). Foundations of the Theory of Probability. Chelsea.

Kolmogorov, A. N. (1963). On tables of random numbers. Sankhya, the Indian Journal of Statistics: Series A, 25(4), 369–376.

Kolmogorov, A. N. (1965). Three approaches to the quantitative definition of information. Problems in Information Transmission, 1, 1–7.

Kolter, J. Z., Abbeel, P., and Ng, A. Y. (2008). Hierarchical apprenticeship learning, with application to quadruped locomotion. In NeurIPS 20.

Kondrak, G. and van Beek, P. (1997). A theoretical evaluation of selected backtracking algorithms. AIJ, 89, 365–387.

Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., and Bacon, D. (2016). Federated learning: Strategies for improving communication efficiency. arXiv:1610.05492.

Konolige, K. (1997). COLBERT: A language for reactive control in Saphira. In Künstliche Intelligenz: Advances in Artificial Intelligence, LNAI.

Konolige, K. (2004). Large-scale map-making. In AAAI-04.

Konolige, K. (1982). A first order formalization of knowledge and action for a multi-agent planning system. In Hayes, J. E., Michie, D., and Pao, Y.-H. (Eds.), Machine Intelligence 10. Ellis Horwood.

Konolige, K. (1994). Easy to be hard: Difficult problems for greedy algorithms. In KR-94.

Koopmans, T. C. (1972). Representation of preference orderings over time. In McGuire, C. B. and Radner, R. (Eds.), Decision and Organization. Elsevier.

Korb, K. B. and Nicholson, A. (2010). Bayesian Artificial Intelligence. CRC Press.

Korf, R. E. (1985a). Depth-first iterative-deepening: an optimal admissible tree search. AIJ, 27, 97–109.

Korf, R. E. (1985b). Iterative-deepening A*: An optimal admissible tree search. In IJCAI-85.

Korf, R. E. (1987). Planning as search: A quantitative approach. AIJ, 33, 65–88.

Korf, R. E. (1990). Real-time heuristic search. AIJ, 42, 189–212.

Korf, R. E. (1993). Linear-space best-first search. AIJ, 62, 41–78.

Korf, R. E. and Chickering, D. M. (1996). Best-first minimax search. AIJ, 84, 299–337.

Korf, R. E. and Felner, A. (2002). Disjoint pattern database heuristics. AIJ, 134, 9–22.

Korf, R. E. and Zhang, W. (2000). Divide-and-conquer frontier search applied to optimal sequence alignment. In AAAI-00.

Korf, R. E. (1997). Finding optimal solutions to Rubik’s Cube using pattern databases. In AAAI-97.

Korf, R. E. and Reid, M. (1998). Complexity analysis of admissible heuristic search. In AAAI98.

Koutsoupias, E. and Papadimitriou, C. H. (1992). On the greedy algorithm for satisfiability. Information Processing Letters, 43, 53–55.

Kovacs, D. L. (2011). BNF definition of PDDL3.1. Unpublished manuscript from the IPC2011 website.

Kowalski, R. (1974). Predicate logic as a programming language. In Proc. IFIP Congress.

Kowalski, R. (1979). Logic for Problem Solving. Elsevier.

Kowalski, R. (1988). The early years of logic programming. CACM, 31, 38–43.

Kowalski, R. and Sergot, M. (1986). A logic-based calculus of events. New Generation Computing, 4, 67–95.

Koza, J. R. (1992). Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press.

Koza, J. R. (1994). Genetic Programming II: Automatic Discovery of Reusable Programs. MIT Press.

Koza, J. R., Bennett, F. H., Andre, D., and Keane, M. A. (1999). Genetic Programming III: Darwinian Invention and Problem Solving. Morgan Kaufmann.

Krakovna, V. (2018). Specification gaming examples in AI.

Kraska, T., Beutel, A., Chi, E. H., Dean, J., and Polyzotis, N. (2017). The case for learned index structures. arXiv:1712.01208.

Kraus, S. (2001). Strategic Negotiation in Multiagent Environments. MIT Press.

Kraus, S., Ephrati, E., and Lehmann, D. (1991). Negotiation in a non-cooperative environment. AIJ, 3, 255–281.

Krause, A. and Guestrin, C. (2005). Optimal nonmyopic value of information in graphical models: Efficient algorithms and theoretical limits. In IJCAI-05.

Krause, A. and Guestrin, C. (2009). Optimal value of information in graphical models. JAIR, 35, 557–591.

Krause, A., McMahan, B., Guestrin, C., and Gupta, A. (2008). Robust submodular observation selection. JMLR, 9, 2761–2801.

Kripke, S. A. (1963). Semantical considerations on modal logic. Acta Philosophica Fennica, 16, 83–94.

Krishna, V. (2002). Auction Theory. Academic Press.

Krishnamurthy, V. (2016). Partially Observed Markov Decision Processes: From Filtering to Controlled Sensing. Cambridge University Press.

Krishnanand, K. and Ghose, D. (2009). Glowworm swarm optimisation: A new method for optimising multi-modal functions. International Journal of Computational Intelligence Studies, 1, 93–119.

Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2013). ImageNet classification with deep convolutional neural networks. In NeurIPS 25.

Krogh, A., Brown, M., Mian, I. S., Sjolander, K., and Haussler, D. (1994). Hidden Markov models in computational biology: Applications to protein modeling. J. Molecular

Biology, 235, 1501–1531.

Krogh, A. and Hertz, J. A. (1992). A simple weight decay can improve generalization. In NeurIPS 4.

Kruppa, E. (1913). Zur Ermittlung eines Objecktes aus zwei Perspektiven mit innerer Orientierung. Sitz.-Ber. Akad. Wiss., Wien, Math. Naturw., Kl. Abt. IIa, 122, 1939–1948.

Kübler, S., McDonald, R., and Nivre, J. (2009). Dependency Parsing. Morgan & Claypool.

Kuffner, J. J. and LaValle, S. (2000). RRT-connect: An efficient approach to single-query path planning. In ICRA-00.

Kuhn, H. W. (1953). Extensive games and the problem of information. In Kuhn, H. W. and Tucker, A. W. (Eds.), Contributions to the Theory of Games II. Princeton University Press.

Kuhn, H. W. (1955). The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2, 83–97.

Kuipers, B. J. (1985). Qualitative simulation. In Bobrow, D. (Ed.), Qualitative Reasoning About Physical Systems. MIT Press.

Kuipers, B. J. and Levitt, T. S. (1988). Navigation and mapping in large-scale space. AIMag, 9, 25–43.

Kuipers, B. J. (2001). Qualitative simulation. In Meyers, R. A. (Ed.), Encyclopedia of Physical Science and Technology. Academic Press.

Kulkarni, T., Kohli, P., Tenenbaum, J. B., and Mansinghka, V. K. (2015). Picture: A probabilistic programming language for scene perception. In CVPR-15.

Kumar, P. R. and Varaiya, P. (1986). Stochastic Systems: Estimation, Identification, and Adaptive Control. Prentice-Hall.

Kumar, S. (2017). A survey of deep learning methods for relation extraction. arXiv:1705.03645.

Kumar, V. and Kanal, L. N. (1988). The CDP: A unifying formulation for heuristic search, dynamic programming, and branch-and-bound. In Kanal, L. N. and Kumar, V. (Eds.), Search in Artificial Intelligence. Springer-Verlag.

Kurien, J., Nayak, P., and Smith, D. E. (2002). Fragment-based conformant planning. In AIPS-02.

Kurth, T., Treichler, S., Romero, J., Mudigonda, M., Luehr, N., Phillips, E. H., Mahesh, A., Matheson, M., Deslippe, J., Fatica, M., Prabhat, and Houston, M. (2018). Exascale deep learning for climate analytics. arXiv:1810.01993.

Kurzweil, R. (2005). The Singularity is Near. Viking.

Kwok, C., Etzioni, O., and Weld, D. S. (2001). Scaling question answering to the web. In Proc. 10th International Conference on the World Wide Web.

La Mettrie, J. O. (1748). L’homme machine. E. Luzac, Leyde, France.

La Mura, P. and Shoham, Y. (1999). Expected utility networks. In UAI-99.

Laborie, P. (2003). Algorithms for propagating resource constraints in AI planning and scheduling. AIJ, 143, 151–188.

Ladkin, P. (1986a). Primitives and units for time specification. In AAAI-86.

Ladkin, P. (1986b). Time representation: a taxonomy of interval relations. In AAAI-86.

Lafferty, J., McCallum, A., and Pereira, F. (2001). Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML-01.

Lai, T. L. and Robbins, H. (1985). Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6, 4–22.

Laird, J., Newell, A., and Rosenbloom, P. S. (1987). SOAR: An architecture for general intelligence. AIJ, 33, 1–64.

Laird, J., Rosenbloom, P. S., and Newell, A. (1986). Chunking in Soar: The anatomy of a general learning mechanism. Machine Learning, 1, 11–46.

Laird, J. (2008). Extending the Soar cognitive architecture. In Artificial General Intelligence Conference.

Lake, B., Salakhutdinov, R., and Tenenbaum, J. B. (2015). Human-level concept learning through probabilistic program induction. Science, 350, 1332–1338.

Lakoff, G. (1987). Women, Fire, and Dangerous Things: What Categories Reveal About the Mind. University of Chicago Press.

Lakoff, G. and Johnson, M. (1980). Metaphors We Live By. University of Chicago Press.

Lakoff, G. and Johnson, M. (1999). Philosophy in the Flesh : The Embodied Mind and Its Challenge to Western Thought. Basic Books.

Lam, J. and Greenspan, M. (2008). Eye-in-hand visual servoing for accurate shooting in pool robotics. In 5th Canadian Conference on Computer and Robot Vision.

Lamarck, J. B. (1809). Philosophie zoologique. Chez Dentu et L’Auteur, Paris.

Lample, G. and Conneau, A. (2019). Cross-lingual language model pretraining. arXiv:1901.07291.

Landhuis, E. (2004). Lifelong debunker takes on arbiter of neutral choices: Magicianturned-mathematician uncovers bias in a flip of a coin. Stanford Report, June 7.

Langdon, W. and Poli, R. (2002). Foundations of Genetic Programming. Springer.

Langton, C. (Ed.). (1995). Artificial Life. MIT Press.

LaPaugh, A. S. (2010). Algorithms and theory of computation handbook. In Atallah, M. J. and Blanton, M. (Eds.), VLSI Layout Algorithms. Chapman & Hall/CRC.

Laplace, P. (1816). Essai philosophique sur les probabilités (3rd edition). Courcier Imprimeur, Paris.

Larochelle, H. and Murray, I. (2011). The neural autoregressive distribution estimator. In AISTATS-11.

Larson, S. C. (1931). The shrinkage of the coefficient of multiple correlation. J. Educational Psychology, 22, 45–55.

Laskey, K. B. (1995). Sensitivity analysis for probability assessments in Bayesian networks. IEEE Transactions on Systems, Man and Cybernetics, 25, 901–909.

Laskey, K. B. (2008). MEBN: A language for first-order Bayesian knowledge bases. AIJ, 172, 140–178.

Latombe, J.-C. (1991). Robot Motion Planning. Kluwer.

Lauritzen, S. (1995). The EM algorithm for graphical association models with missing data. Computational Statistics and Data Analysis, 19, 191–201.

Lauritzen, S., Dawid, A. P., Larsen, B., and Leimer, H. (1990). Independence properties of directed Markov fields. Networks, 20, 491–505.

Lauritzen, S. and Spiegelhalter, D. J. (1988). Local computations with probabilities on graphical structures and their application to expert systems. J. Royal Statistical Society, B 50, 157–224.

Lauritzen, S. and Wermuth, N. (1989). Graphical models for associations between variables, some of which are qualitative and some quantitative. Annals of Statistics, 17, 31–57.

LaValle, S. (2006). Planning Algorithms. Cambridge University Press.

Lawler, E. L., Lenstra, J. K., Kan, A., and Shmoys, D. B. (1992). The Travelling Salesman Problem. Wiley Interscience.

Lawler, E. L., Lenstra, J. K., Kan, A., and Shmoys, D. B. (1993). Sequencing and scheduling: Algorithms and complexity. In Graves, S. C., Zipkin, P. H., and Kan, A. H. G. R. (Eds.), Logistics of Production and Inventory: Handbooks in Operations Research and Management Science, Volume 4. North-Holland.

Lawler, E. L. and Wood, D. E. (1966). Branch-and-bound methods: A survey. Operations Research, 14, 699–719.

Lazanas, A. and Latombe, J.-C. (1992). Landmark-based robot navigation. In AAAI-92.

Le, T. A., Baydin, A. G., and Wood, F. (2017). Inference compilation and universal probabilistic programming. In AISTATS-17.

Lebedev, M. A. and Nicolelis, M. A. (2006). Brain-machine interfaces: Past, present and future. Trends in Neurosciences, 29, 536–546.

Lecoutre, C. (2009). Constraint Networks: Techniques and Algorithms. Wiley-IEEE Press.

LeCun, Y., Denker, J., and Solla, S. (1990). Optimal brain damage. In NeurIPS 2.

LeCun, Y., Jackel, L., Boser, B., and Denker, J. (1989). Handwritten digit recognition: Applications of neural network chips and automatic learning. IEEE Communications Magazine, 27, 41–46.

LeCun, Y., Jackel, L., Bottou, L., Brunot, A., Cortes, C., Denker, J., Drucker, H., Guyon, I., Muller, U., Sackinger, E., Simard, P., and Vapnik, V. N. (1995). Comparison of learning algorithms for handwritten digit recognition. In Int. Conference on Artificial Neural Networks.

LeCun, Y., Bengio, Y., and Hinton, G. E. (2015). Deep learning. Nature, 521, 436–444.

Lee, D., Seo, H., and Jung, M. W. (2012). Neural basis of reinforcement learning and decision making. Annual Review of Neuroscience, 35, 287–308.

Lee, K.-F. (2018). AI Superpowers: China, Silicon Valley, and the New World Order. Houghton Mifflin.

Leech, G., Rayson, P., and Wilson, A. (2001). Word Frequencies in Written and Spoken English: Based on the British National Corpus. Longman.

Legendre, A. M. (1805). Nouvelles méthodes pour la détermination des orbites des comètes. Chez Firmin Didot, Paris.

Lehmann, J., Isele, R., Jakob, M., Jentzsch, A., Kontokostas, D., Mendes, P. N., Hellmann, S., Morsey, M., van Kleef, P., Auer, S., and Bizer, C. (2015). DBpedia – A large-scale, multilingual knowledge base extracted from Wikipedia. Semantic Web, 6, 167–195.

Lehrer, J. (2009). How We Decide. Houghton Mifflin.

Leike, J., Martic, M., Krakovna, V., Ortega, P. A., Everitt, T., Lefrancq, A., Orseau, L., and Legg, S. (2017). AI safety gridworlds. arXiv:1711.09883.

Lelis, L., Arfaee, S. J., Zilles, S., and Holte, R. C. (2012). Learning heuristic functions faster by using predicted solution costs. In Proc. Fifth Annual Symposium on Combinatorial Search.

Lenat, D. B. (1975). BEINGS: Knowledge as interacting experts. In IJCAI-75.

Lenat, D. B. and Guha, R. V. (1990). Building Large Knowledge-Based Systems: Representation and Inference in the CYC Project. Addison-Wesley.

Leonard, H. S. and Goodman, N. (1940). The calculus of individuals and its uses. JSL, 5, 45– 55.

Leonard, J. and Durrant-Whyte, H. (1992). Directed Sonar Sensing for Mobile Robot Navigation. Kluwer.

Lepage, G. P. (1978). A new algorithm for adaptive multidimensional integration. Journal of Computational Physics, 27, 192–203.

Lerner, U. (2002). Hybrid Bayesian Networks for Reasoning About Complex Systems. Ph.D. thesis, Stanford University.

Leśniewski, S. (1916). Podstawy ogólnej teorii mnogości. Popławski.

Lesser, V. R. and Corkill, D. D. (1988). The distributed vehicle monitoring testbed: A tool for investigating distributed problem solving networks. In Engelmore, R. and Morgan, T. (Eds.), Blackboard Systems. Addison-Wesley.

Letz, R., Schumann, J., Bayerl, S., and Bibel, W. (1992). SETHEO: A high-performance theorem prover. JAR, 8, 183–212.

Levesque, H. J. and Brachman, R. J. (1987). Expressiveness and tractability in knowledge representation and reasoning. Computational Intelligence, 3, 78–93.

Levin, D. A., Peres, Y., and Wilmer, E. L. (2008). Markov Chains and Mixing Times. American Mathematical Society.

Levine, S., Finn, C., Darrell, T., and Abbeel, P. (2016). End-to-end training of deep visuomotor policies. JMLR, 17, 1334–1373.

Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., and Quillen, D. (2018). Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. International Journal of Robotics Research, 37, 421–436.

Levy, D. (1989). The million pound bridge program. In Levy, D. and Beal, D. (Eds.), Heuristic Programming in Artificial Intelligence. Ellis Horwood.

Levy, D. (2008). Love and Sex with Robots: The Evolution of Human—Robot Relationships. Harper.

Levy, O. and Goldberg, Y. (2014). Linguistic regularities in sparse and explicit word representations. In Proc. Eighteenth Conference on Computational Natural Language Learning.

Leyton-Brown, K. and Shoham, Y. (2008). Essentials of Game Theory: A Concise, Multidisciplinary Introduction. Morgan & Claypool.

Li, C. M. and Anbulagan (1997). Heuristics based on unit propagation for satisfiability problems. In IJCAI-97.

Li, K. and Malik, J. (2018a). Implicit maximum likelihood estimation. arXiv:1809.09087.

Li, K. and Malik, J. (2018b). On the implicit assumptions of GANs. arXiv:1811.12402.

Li, M., Vitányi, P., et al. (2008). An Introduction to Kolmogorov Complexity and Its Applications (3rd edition). Springer-Verlag.

Li, T.-M., Gharbi, M., Adams, A., Durand, F., and Ragan-Kelley, J. (2018). Differentiable programming for image processing and deep learning in Halide. ACM Transactions on Graphics, 37, 139.

Li, W. and Todorov, E. (2004). Iterative linear quadratic regulator design for nonlinear biological movement systems. In Proc. 1st International Conference on Informatics in Control, Automation and Robotics.

Li, X. and Yao, X. (2012). Cooperatively coevolving particle swarms for large scale optimization. IEEE Trans. Evolutionary Computation, 16, 210–224.

Li, Z., Li, P., Krishnan, A., and Liu, J. (2011). Large-scale dynamic gene regulatory network inference combining differential equation models with local dynamic Bayesian network analysis. Bioinformatics, 27 19, 2686–91.

Liang, P., Jordan, M. I., and Klein, D. (2011). Learning dependency-based compositional semantics. arXiv:1109.6841.

Liang, P. and Potts, C. (2015). Bringing machine learning and compositional semantics together. Annual Review of Linguistics, 1, 355–376.

Liberatore, P. (1997). The complexity of the language A. Electronic Transactions on Artificial Intelligence, 1, 13–38.

Lifschitz, V. (2001). Answer set programming and plan generation. AIJ, 138, 39–54.

Lighthill, J. (1973). Artificial intelligence: A general survey. In Lighthill, J., Sutherland, N. S., Needham, R. M., Longuet-Higgins, H. C., and Michie, D. (Eds.), Artificial Intelligence: A Paper Symposium. Science Research Council of Great Britain.

Lillicrap, T., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv:1509.02971.

Lin, S. (1965). Computer solutions of the travelling salesman problem. Bell Systems Technical Journal, 44(10), 2245–2269.

Lin, S. and Kernighan, B. W. (1973). An effective heuristic algorithm for the travellingsalesman problem. Operations Research, 21, 498–516.

Lindley, D. V. (1956). On a measure of the information provided by an experiment. Annals of Mathematical Statistics, 27, 986–1005.

Lindsay, R. K., Buchanan, B. G., Feigenbaum, E. A., and Lederberg, J. (1980). Applications of Artificial Intelligence for Organic Chemistry: The DENDRAL Project. McGraw-Hill.

Lindsten, F., Jordan, M. I., and Schön, T. B. (2014). Particle Gibbs with ancestor sampling. JMLR, 15, 2145–2184.

Littman, M. L. (1994). Markov games as a framework for multi-agent reinforcement learning. In ICML-94.

Littman, M. L., Cassandra, A. R., and Kaelbling, L. P. (1995). Learning policies for partially observable environments: Scaling up. In ICML-95.

Littman, M. L. (2015). Reinforcement learning improves behaviour from evaluative feedback. Nature, 521, 445–451.

Littman, M. L., Topcu, U., Fu, J., Isbell, C., Wen, M., and MacGlashan, J. (2017). Environment-independent task specifications via GLTL. arXiv:1704.04341.

Liu, B., Gemp, I., Ghavamzadeh, M., Liu, J., Mahadevan, S., and Petrik, M. (2018). Proximal gradient temporal difference learning: Stable reinforcement learning with polynomial sample complexity. JAIR, 63, 461–494.

Liu, H., Simonyan, K., Vinyals, O., Fernando, C., and Kavukcuoglu, K. (2017). Hierarchical representations for efficient architecture search. arXiv:1711.00436.

Liu, H., Simonyan, K., and Yang, Y. (2019). DARTS: Differentiable architecture search. In ICLR-19.

Liu, J. and Chen, R. (1998). Sequential Monte Carlo methods for dynamic systems. JASA, 93, 1022–1031.

Liu, J. and West, M. (2001). Combined parameter and state estimation in simulation-based filtering. In Doucet, A., de Freitas, J. F. G., and Gordon, N. (Eds.), Sequential Monte Carlo Methods in Practice. Springer.

Liu, L. T., Dean, S., Rolf, E., Simchowitz, M., and Hardt, M. (2018a). Delayed impact of fair machine learning. arXiv:1803.04383.

Liu, M.-Y., Breuel, T., and Kautz, J. (2018b). Unsupervised image-to-image translation networks. In NeurIPS 30.

Liu, X., Faes, L., Kale, A. U., Wagner, S. K., Fu, D. J., Bruynseels, A., Mahendiran, T., Moraes, G., Shamdas, M., Kern, C., Ledsam, J. R., Schmid, M., Balaskas, K., Topol, E., Bachmann, L. M., Keane, P. A., and Denniston, A. K. (2019a). A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: A systematic review and meta-analysis. The Lancet Digital Health.

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019b). RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692.

Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., Gupta, V., Singh, N., Natarajan, V., Hofmann-Wellenhof, R., Corrado, G., Peng, L., Webster, D. R., Ai, D., Huang, S., Liu, Y., Dunn, R. C., and Coz, D. (2019c). A deep learning system for differential diagnosis of skin diseases. arXiv:1909.

Liu, Y., Gadepalli, K. K., Norouzi, M., Dahl, G., Kohlberger, T., Venugopalan, S., Boyko, A. S., Timofeev, A., Nelson, P. Q., Corrado, G., Hipp, J. D., Peng, L., and Stumpe, M. C. (2017). Detecting cancer metastases on gigapixel pathology images. arXiv:1703.02442.

Liu, Y., Kohlberger, T., Norouzi, M., Dahl, G., Smith, J. L., Mohtashamian, A., Olson, N., Peng, L., Hipp, J. D., and Stumpe, M. C. (2018). Artificial intelligence-based breast cancer nodal metastasis detection: Insights into the black box for pathologists. Archives of Pathology & Laboratory Medicine, 143, 859–868.

Livescu, K., Glass, J., and Bilmes, J. (2003). Hidden feature modeling for speech recognition using dynamic Bayesian networks. In EUROSPEECH-2003.

Lloyd, S. (2000). Ultimate physical limits to computation. Nature, 406, 1047–1054.

Lloyd, W. F. (1833). Two Lectures on the Checks to Population. Oxford University.

Llull, R. (1305). Ars Magna. Published as Salzinger, I. et al. (Eds.), Raymundi Lulli Opera omnia, Mainz, 1721–1742.

Loftus, E. and Palmer, J. (1974). Reconstruction of automobile destruction: An example of the interaction between language and memory. J. Verbal Learning and Verbal Behavior, 13, 585–589.

Lohn, J. D., Kraus, W. F., and Colombano, S. P. (2001). Evolutionary optimization of yagiuda antennas. In Proc. Fourth International Conference on Evolvable Systems.

Longuet-Higgins, H. C. (1981). A computer algorithm for reconstructing a scene from two projections. Nature, 293, 133–135.

Loos, S., Irving, G., Szegedy, C., and Kaliszyk, C. (2017). Deep network guided proof search. In Proc. 21st Int’l Conf. on Logic for Programming, Artificial Intelligence and Reasoning.

Lopez de Segura, R. (1561). Libro de la invencion liberal y arte del juego del axedrez. Andres de Angulo.

Lorentz, R. (2015). Early playout termination in MCTS. In Plaat, A., van den Herik, J., and Kosters, W. (Eds.), Advances in Computer Games. Springer-Verlag.

Love, N., Hinrichs, T., and Genesereth, M. R. (2006). General game playing: Game description language specification. Tech. rep., Stanford University Computer Science Dept.

Lovejoy, W. S. (1991). A survey of algorithmic methods for partially observed Markov decision processes. Annals of Operations Research, 28, 47–66.

Lovelace, A. (1843). Sketch of the analytical engine invented by Charles Babbage. Notes appended to Lovelace’s translation of an article of the above title written by L. F. Menabrea based on lectures by Charles babbage in 1840. The translation appeared in R. Taylor (Ed.), Scientific Memoirs, vol. III. R. and J. E. Taylor, London.

Loveland, D. (1970). A linear format for resolution. In Proc. IRIA Symposium on Automatic Demonstration.

Lowe, D. (1987). Three-dimensional object recognition from single two-dimensional images. AIJ, 31, 355–395.

Lowe, D. (2004). Distinctive image features from scale-invariant keypoints. IJCV, 60, 91–110.

Löwenheim, L. (1915). Über möglichkeiten im Relativkalkül. Mathematische Annalen, 76, 447–470.

Lowerre, B. T. (1976). The HARPY Speech Recognition System. Ph.D. thesis, Computer Science Department, Carnegie-Mellon University.

Lowry, M. (2008). Intelligent software engineering tools for NASA’s crew exploration vehicle. In ISMIS-08.

Loyd, S. (1959). Mathematical Puzzles of Sam Loyd: Selected and Edited by Martin Gardner. Dover.

Lozano-Perez, T. (1983). Spatial planning: A configuration space approach. IEEE Transactions on Computers, C-32, 108–120.

Lozano-Perez, T., Mason, M., and Taylor, R. (1984). Automatic synthesis of fine-motion strategies for robots. Int. J. Robotics Research, 3, 3–24.

Lu, F. and Milios, E. (1997). Globally consistent range scan alignment for environment mapping. Autonomous Robots, 4, 333–349.

Lubberts, A. and Miikkulainen, R. (2001). Co-evolving a Go-playing neural network. In GECCO-01.

Luby, M., Sinclair, A., and Zuckerman, D. (1993). Optimal speedup of Las Vegas algorithms. Information Processing Letters, 47, 173–180.

Lucas, J. R. (1961). Minds, machines, and Gödel. Philosophy, 36.

Lucas, J. R. (1976). This Gödel is killing me: A rejoinder. Philosophia, 6, 145–148.

Lucas, P., van der Gaag, L., and Abu-Hanna, A. (2004). Bayesian networks in biomedicine and health-care. Artificial Intelligence in Medicine.

Luce, D. R. and Raiffa, H. (1957). Games and Decisions. Wiley.

Lukasiewicz, T. (1998). Probabilistic logic programming. In ECAI-98.

Lundberg, S. M. and Lee, S.-I. (2018). A unified approach to interpreting model predictions. In NeurIPS 30.

Lunn, D., Jackson, C., Best, N., Thomas, A., and Spiegelhalter, D. J. (2013). The BUGS Book: A Practical Introduction to Bayesian Analysis. Chapman and Hall.

Lunn, D., Thomas, A., Best, N., and Spiegelhalter, D. J. (2000). WinBUGS—a Bayesian modelling framework: Concepts, structure, and extensibility. Statistics and Computing, 10, 325–337.

Luo, S., Bimbo, J., Dahiya, R., and Liu, H. (2017). Robotic tactile perception of object properties: A review. Mechatronics, 48, 54–67.

Lyman, P. and Varian, H. R. (2003). How much information? www.sims.berkeley.edu/howmuch-info-2003.

Lynch, K. and Park, F. C. (2017). Modern Robotics. Cambridge University Press.

Machina, M. (2005). Choice under uncertainty. In Encyclopedia of Cognitive Science. Wiley.

MacKay, D. J. C. (2002). Information Theory, Inference and Learning Algorithms. Cambridge University Press.

MacKenzie, D. (2004). Mechanizing Proof. MIT Press.

Mackworth, A. K. (1977). Consistency in networks of relations. AIJ, 8, 99–118.

Mackworth, A. K. and Freuder, E. C. (1985). The complexity of some polynomial network consistency algorithms for constraint satisfaction problems. AIJ, 25, 65–74.

Madhavan, R. and Schlenoff, C. I. (2003). Moving object prediction for off-road autonomous navigation. In Unmanned Ground Vehicle Technology V.

Mailath, G. and Samuelson, L. (2006). Repeated Games and Reputations: Long-Run Relationships. Oxford University Press.

Majercik, S. M. and Littman, M. L. (2003). Contingent planning under uncertainty via stochastic satisfiability. AIJ, 147, 119–162.

Malhotra, P., Vig, L., Shroff, G., and Agarwal, P. (2015). Long short term memory networks for anomaly detection in time series. In ISANN-15.

Malik, D., Palaniappan, M., Fisac, J. F., Hadfield-Menell, D., Russell, S. J., and Dragan, A. D. (2018). An efficient, generalized bellman update for cooperative inverse reinforcement learning. In ICML-18.

Malone, T. W. (2004). The Future of Work. Harvard Business Review Press.

Maneva, E., Mossel, E., and Wainwright, M. (2007). A new look at survey propagation and its generalizations. arXiv:cs/0409012.

Manna, Z. and Waldinger, R. (1971). Toward automatic program synthesis. CACM, 14, 151– 165.

Manna, Z. and Waldinger, R. (1985). The Logical Basis for Computer Programming: Volume 1: Deductive Reasoning. Addison-Wesley.

Manne, A. S. (1960). Linear programming and sequential decisions. Management Science, 6, 259–267.

Manning, C. and Schütze, H. (1999). Foundations of Statistical Natural Language Processing. MIT Press.

Manning, C., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.

Mannion, M. (2002). Using first-order logic for product line model validation. In Software Product Lines: Second International Conference.

Mansinghka, V. K., Selsam, D., and Perov, Y. (2013). Venture: A higher-order probabilistic programming platform with programmable inference. arXiv:1404.0099.

Marbach, P. and Tsitsiklis, J. N. (1998). Simulation-based optimization of Markov reward processes. Technical report, Laboratory for Information and Decision Systems, MIT.

Marcus, G. (2009). Kluge: The Haphazard Evolution of the Human Mind. Mariner Books.

Marcus, M. P., Santorini, B., and Marcinkiewicz, M. A. (1993). Building a large annotated corpus of English: The Penn treebank. Computational Linguistics, 19, 313–330.

Marinescu, R. and Dechter, R. (2009). AND/OR branch-and-bound search for combinatorial optimization in graphical models. AIJ, 173, 1457–1491.

Markov, A. (1913). An example of statistical investigation in the text of “Eugene Onegin” illustrating coupling of “tests” in chains. Proc. Academy of Sciences of St. Petersburg, 7, 153–162.

Marler, R. T. and Arora, J. S. (2004). Survey of multi-objective optimization methods for engineering. Structural and Multidisciplinary Optimization, 26, 369–395.

Maron, M. E. (1961). Automatic indexing: An experimental inquiry. JACM, 8, 404–417.

Màrquez, L. and Rodríguez, H. (1998). Part-of-speech tagging using decision trees. In ECML-98.

Marr, D. and Poggio, T. (1976). Cooperative computation of stereo disparity. Science, 194, 283–287.

Marr, D. (1982). Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. W. H. Freeman.

Marriott, K. and Stuckey, P. J. (1998). Programming with Constraints: An Introduction. MIT Press.

Marsland, S. (2014). Machine Learning: An Algorithmic Perspective (2nd edition). CRC Press.

Martelli, A. and Montanari, U. (1973). Additive AND/OR graphs. In IJCAI-73.

Martelli, A. (1977). On the complexity of admissible search algorithms. AIJ, 8, 1–13.

Marthi, B., Pasula, H., Russell, S. J., and Peres, Y. (2002). Decayed MCMC filtering. In UAI02.

Marthi, B., Russell, S. J., Latham, D., and Guestrin, C. (2005). Concurrent hierarchical reinforcement learning. In IJCAI-05.

Marthi, B., Russell, S. J., and Wolfe, J. (2007). Angelic semantics for high-level actions. In ICAPS-07.

Marthi, B., Russell, S. J., and Wolfe, J. (2008). Angelic hierarchical planning: Optimal and online algorithms. In ICAPS-08.

Martin, D., Fowlkes, C., and Malik, J. (2004). Learning to detect natural image boundaries using local brightness, color, and texture cues. PAMI, 26, 530–549.

Martin, F. G. (2012). Will massive open online courses change how we teach? CACM, 55, 26–28.

Martin, J. H. (1990). A Computational Model of Metaphor Interpretation. Academic Press.

Mason, M. (1993). Kicking the sensing habit. AIMag, 14, 58–59.

Mason, M. (2001). Mechanics of Robotic Manipulation. MIT Press.

Mason, M. and Salisbury, J. (1985). Robot Hands and the Mechanics of Manipulation. MIT Press.

Mataric, M. J. (1997). Reinforcement learning in the multi-robot domain. Autonomous Robots, 4, 73–83.

Mates, B. (1953). Stoic Logic. University of California Press.

Matuszek, C., Cabral, J., Witbrock, M., and DeOliveira, J. (2006). An introduction to the syntax and semantics of Cyc. In Proc. AAAI Spring Symposium on Formalizing and Compiling Background Knowledge and Its Applications to Knowledge Representation and Question Answering.

Mausam and Kolobov, A. (2012). Planning with Markov Decision Processes: An AI Perspective. Morgan & Claypool.

Maxwell, J. (1868). On governors. Proc. Roy. Soc., 16, 270–283.

Mayer, J., Khairy, K., and Howard, J. (2010). Drawing an elephant with four complex parameters. American Journal of Physics, 78, 648–649.

Mayor, A. (2018). Gods and Robots: Myths, Machines, and Ancient Dreams of Technology. Princeton University Press.

McAllester, D. A. (1980). An outlook on truth maintenance. AI memo, MIT AI Laboratory.

McAllester, D. A. (1988). Conspiracy numbers for min-max search. AIJ, 35, 287–310.

McAllester, D. A. (1998). What is the most pressing issue facing AI and the AAAI today? Candidate statement, election for Councilor of the American Association for Artificial Intelligence.

McAllester, D. A. and Rosenblitt, D. (1991). Systematic nonlinear planning. In AAAI-91.

McAllester, D. A. (1990). Truth maintenance. In AAAI-90.

McAllester, D. A., Milch, B., and Goodman, N. D. (2008). Random-world semantics and syntactic independence for expressive languages. Technical report, MIT.

McCallum, A. (2003). Efficiently inducing features of conditional random fields. In UAI-03.

McCallum, A., Schultz, K., and Singh, S. (2009). FACTORIE: Probabilistic programming via imperatively defined factor graphs. In NeurIPS 22.

McCarthy, J. (1958). Programs with common sense. In Proc. Symposium on Mechanisation of Thought Processes.

McCarthy, J. (1963). Situations, actions, and causal laws. Memo, Stanford University Artificial Intelligence Project.

McCarthy, J. (1968). Programs with common sense. In Minsky, M. L. (Ed.), Semantic Information Processing. MIT Press.

McCarthy, J. (1980). Circumscription: A form of non-monotonic reasoning. AIJ, 13, 27–39.

McCarthy, J. (2007). From here to human-level AI. AIJ, 171.

McCarthy, J. and Hayes, P. J. (1969). Some philosophical problems from the standpoint of artificial intelligence. In Meltzer, B., Michie, D., and Swann, M. (Eds.), Machine Intelligence 4. Edinburgh University Press.

McCawley, J. D. (1988). The Syntactic Phenomena of English. University of Chicago Press.

McCorduck, P. (2004). Machines Who Think: A Personal Inquiry Into the History and Prospects of Artificial Intelligence (Revised edition). A K Peters.

McCulloch, W. S. and Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5, 115–137.

McCune, W. (1997). Solution of the Robbins problem. JAR, 19, 263–276.

McCune, W. (1990). Otter 2.0. In International Conference on Automated Deduction.

McDermott, D. (1976). Artificial intelligence meets natural stupidity. SIGART Newsletter, 57, 4–9.

McDermott, D. (1978a). Planning and acting. Cognitive Science, 2, 71–109.

McDermott, D. (1978b). Tarskian semantics, or no notation without denotation! Cognitive Science, 2, 277 – 282.

McDermott, D. (1985). Reasoning about plans. In Hobbs, J. and Moore, R. (Eds.), Formal theories of the commonsense world. Ablex.

McDermott, D. (1987). A critique of pure reason. Computational Intelligence, 3, 151–237.

McDermott, D. (1996). A heuristic estimator for means-ends analysis in planning. In ICAPS96.

McDermott, D. and Doyle, J. (1980). Non-monotonic logic: i. AIJ, 13, 41–72.

McDermott, J. (1982). R1: A rule-based configurer of computer systems. AIJ, 19, 39–88.

McEliece, R. J., MacKay, D. J. C., and Cheng, J.-F. (1998). Turbo decoding as an instance of Pearl’s “belief propagation” algorithm. IEEE Journal on Selected Areas in Communications, 16, 140–152.

McGregor, J. J. (1979). Relational consistency algorithms and their application in finding subgraph and graph isomorphisms. Information Sciences, 19, 229–250.

McIlraith, S. and Zeng, H. (2001). Semantic web services. IEEE Intelligent Systems, 16, 46– 53.

McKinney, W. (2012). Python for Data Analysis: Data Wrangling with Pandas. O’Reilly.

McLachlan, G. J. and Krishnan, T. (1997). The EM Algorithm and Extensions. Wiley.

McMahan, H. B. and Andrew, G. (2018). A general approach to adding differential privacy to iterative training procedures. arXiv:1812.06210.

McMillan, K. L. (1993). Symbolic Model Checking. Kluwer.

McWhorter, J. H. (2014). The Language Hoax: Why the World Looks the Same in Any Language. Oxford University Press.

Meehl, P. (1955). Clinical vs. Statistical Prediction. University of Minnesota Press.

Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., and Galstyan, A. (2019). A survey on bias and fairness in machine learning. arXiv:1908.09635.

Mendel, G. (1866). Versuche über pflanzen-hybriden. Verhandlungen des Naturforschenden Vereins, Abhandlungen, Brünn, 4, 3–47. Translated into English by C. T. Druery, published by Bateson (1902).

Mercer, J. (1909). Functions of positive and negative type and their connection with the theory of integral equations. Phil. Trans. Roy. Soc., A, 209, 415–446.

Merleau-Ponty, M. (1945). Phenomenology of Perception. Routledge.

Metropolis, N., Rosenbluth, A., Rosenbluth, M., Teller, A., and Teller, E. (1953). Equations of state calculations by fast computing machines. J. Chemical Physics, 21, 1087–1091.

Metropolis, N. and Ulam, S. (1949). The beginning of the Monte Carlo method. Journal of the American Statistical Association, 44, 335–341.

Mézard, M., Parisi, G., and Virasoro, M. (1987). Spin Glass Theory and Beyond: An Introduction to the Replica Method and Its Applications. World Scientific.

Michie, D. (1966). Game-playing and game-learning automata. In Fox, L. (Ed.), Advances in Programming and Non-Numerical Computation. Pergamon.

Michie, D. (1972). Machine intelligence at Edinburgh. Management Informatics, 2, 7–12.

Michie, D. and Chambers, R. A. (1968). BOXES: An experiment in adaptive control. In Dale, E. and Michie, D. (Eds.), Machine Intelligence 2. Elsevier.

Michie, D. (1963). Experiments on the mechanization of game-learning Part I. Characterization of the model and its parameters. The Computer Journal, 6, 232–236.

Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., et al. (2019). Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing. Elsevier.

Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv:1301.3781.

Mikolov, T., Karafiát, M., Burget, L., Černocký, J., and Khudanpur, S. (2010). Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association.

Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. (2014). Distributed representations of words and phrases and their compositionality. In NeurIPS 26.

Milch, B. (2006). Probabilistic Models with Unknown Objects. Ph.D. thesis, UC Berkeley.

Milch, B., Marthi, B., Sontag, D., Russell, S. J., Ong, D., and Kolobov, A. (2005). BLOG: Probabilistic models with unknown objects. In IJCAI-05.

Milch, B., Zettlemoyer, L., Kersting, K., Haimes, M., and Kaelbling, L. P. (2008). Lifted probabilistic inference with counting formulas. In AAAI-08.

Milgrom, P. (1997). Putting auction theory to work: The simultaneous ascending auction. Tech. rep., Stanford University Department of Economics.

Mill, J. S. (1863). Utilitarianism. Parker, Son and Bourn, London.

Miller, A. C., Merkhofer, M. M., Howard, R. A., Matheson, J. E., and Rice, T. R. (1976). Development of automated aids for decision analysis. Technical report, SRI International.

Miller, T., Howe, P., and Sonenberg, L. (2017). Explainable AI: Beware of inmates running the asylum. In Proc. IJCAI-17 Workshop on Explainable AI.

Minka, T. (2010). Bayesian linear regression. Unpublished manuscript.

Minka, T., Cleven, R., and Zaykov, Y. (2018). TrueSkill 2: An improved Bayesian skill rating system. Tech. rep., Microsoft Research.

Minker, J. (2001). Logic-Based Artificial Intelligence. Kluwer.

Minsky, M. L. (1975). A framework for representing knowledge. In Winston, P. H. (Ed.), The Psychology of Computer Vision. McGraw-Hill.

Minsky, M. L. (1986). The Society of Mind. Simon and Schuster.

Minsky, M. L. (2007). The Emotion Machine: Commonsense Thinking, Artificial Intelligence, and the Future of the Human Mind. Simon and Schuster.

Minsky, M. L. and Papert, S. (1969). Perceptrons: An Introduction to Computational Geometry. MIT Press.

Minsky, M. L. and Papert, S. (1988). Perceptrons: An Introduction to Computational Geometry (Expanded edition). MIT Press.

Minsky, M. L., Singh, P., and Sloman, A. (2004). The St. Thomas common sense symposium: Designing architectures for human-level intelligence. AIMag, 25, 113– 124.

Minton, S., Johnston, M. D., Philips, A. B., and Laird, P. (1992). Minimizing conflicts: A heuristic repair method for constraint satisfaction and scheduling problems. AIJ, 58, 161–205.

Mirjalili, S. M. and Lewis, A. (2014). Grey wolf optimizer. Advances in Engineering Software, 69, 46–61.

Misak, C. (2004). The Cambridge Companion to Peirce. Cambridge University Press.

Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., and Gebru, T. (2019). Model cards for model reporting. Proc. of the Conference on Fairness, Accountability, and Transparency.

Mitchell, M. (1996). An Introduction to Genetic Algorithms. MIT Press.

Mitchell, M. (2019). Artificial Intelligence: A Guide for Thinking Humans. Farrar, Straus and Giroux.

Mitchell, M., Holland, J. H., and Forrest, S. (1996). When will a genetic algorithm outperform hill climbing? In NeurIPS 6.

Mitchell, T. M. (1997). Machine Learning. McGraw-Hill.

Mitchell, T. M. (2005). Reading the web: A breakthrough goal for AI. AIMag, 26.

Mitchell, T. M. (2007). Learning, information extraction and the web. In ECML-07.

Mitchell, T. M., Cohen, W., Hruschka, E., Talukdar, P., Yang, B., Betteridge, J., Carlson, A., Dalvi, B., Gardner, M., Kisiel, B., et al. (2018). Never-ending learning. CACM, 61, 103– 115.

Mitchell, T. M., Shinkareva, S. V., Carlson, A., Chang, K.-M., Malave, V. L., Mason, R. A., and Just, M. A. (2008). Predicting human brain activity associated with the meanings of nouns. Science, 320, 1191–1195.

Mittelstadt, B. (2019). Principles alone cannot guarantee ethical AI. Nature Machine Intelligence, 1, 501–507.

Mitten, L. G. (1960). An analytic solutlon to the least cost testing sequence problem. Journal of Industrial Engineering, 11, 17.

Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. (2018). Spectral normalization for generative adversarial networks. arXiv:1802.05957.

Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. A. (2013). Playing Atari with deep reinforcement learning. arXiv:1312.5602.

Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M. A., Fidjeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature, 518, 529–533.

Mohr, R. and Henderson, T. C. (1986). Arc and path consistency revisited. AIJ, 28, 225–233.

Montague, R. (1970). English as a formal language. In Visentini, B. (Ed.), Linguaggi nella Società e nella Tecnica. Edizioni di Comunità.

Montague, R. (1973). The proper treatment of quantification in ordinary English. In Hintikka, K. J. J., Moravcsik, J. M. E., and Suppes, P. (Eds.), Approaches to Natural Language. D. Reidel.

Montanari, U. (1974). Networks of constraints: Fundamental properties and applications to picture processing. Information Sciences, 7, 95–132.

Montemerlo, M. and Thrun, S. (2004). Large-scale robotic 3-D mapping of urban structures. In Proc. International Symposium on Experimental Robotics.

Montemerlo, M., Thrun, S., Koller, D., and Wegbreit, B. (2002). FastSLAM: A factored solution to the simultaneous localization and mapping problem. In AAAI-02.

Mooney, R. (1999). Learning for semantic interpretation: Scaling up without dumbing down. In Proc. 1st Workshop on Learning Language in Logic.

Moore, A. M. and Wong, W.-K. (2003). Optimal reinsertion: A new search operator for accelerated and more accurate Bayesian network structure learning. In ICML-03.

Moore, A. W. and Atkeson, C. G. (1993). Prioritized sweeping—Reinforcement learning with less data and less time. Machine Learning, 13, 103–130.

Moore, A. W. and Lee, M. S. (1997). Cached sufficient statistics for efficient machine learning with large datasets. JAIR, 8, 67–91.

Moore, E. F. (1959). The shortest path through a maze. In Proc. International Symposium on the Theory of Switching, Part II. Harvard University Press.

Moore, R. C. (1980). Reasoning about knowledge and action. Artificial intelligence center technical note, SRI International.

Moore, R. C. (1985). A formal theory of knowledge and action. In Hobbs, J. R. and Moore, R. C. (Eds.), Formal Theories of the Commonsense World. Ablex.

Moore, R. C. and DeNero, J. (2011). L1 and L2 regularization for multiclass hinge loss models. In Symposium on Machine Learning in Speech and Natural Language Processing.

Moravčík, M., Schmid, M., Burch, N., Lisý, V., Morrill, D., Bard, N., Davis, T., Waugh, K., Johanson, M., and Bowling, M. (2017). Deepstack: Expert-level artificial intelligence in no-limit poker. arXiv:1701.01724.

Moravec, H. P. (1983). The Stanford cart and the CMU rover. Proc. IEEE, 71, 872–884.

Moravec, H. P. and Elfes, A. (1985). High resolution maps from wide angle sonar. In ICRA85.

Moravec, H. P. (2000). Robot: Mere Machine to Transcendent Mind. Oxford University Press.

Morgan, C. L. (1896). Habit and Instinct. Edward Arnold.

Morgan, T. J. H. and Griffiths, T. L. (2015). What the Baldwin Effect affects. In COGSCI-15.

Morjaria, M. A., Rink, F. J., Smith, W. D., Klempner, G., Burns, C., and Stein, J. (1995). Elicitation of probabilities for belief networks: Combining qualitative and quantitative information. In UAI-95.

Morrison, P. and Morrison, E. (Eds.). (1961). Charles Babbage and His Calculating Engines: Selected Writings by Charles Babbage and Others. Dover.

Moskewicz, M. W., Madigan, C. F., Zhao, Y., Zhang, L., and Malik, S. (2001). Chaff: Engineering an efficient SAT solver. In Proc. 38th Design Automation Conference.

Mott, A., Job, J., Vlimant, J.-R., Lidar, D., and Spiropulu, M. (2017). Solving a Higgs optimization problem with quantum annealing for machine learning. Nature, 550, 375.

Motzkin, T. S. and Schoenberg, I. J. (1954). The relaxation method for linear inequalities. Canadian Journal of Mathematics, 6, 393–404.

Moutarlier, P. and Chatila, R. (1989). Stochastic multisensory data fusion for mobile robot location and environment modeling. In ISRR-89.

Mueller, E. T. (2006). Commonsense Reasoning. Morgan Kaufmann.

Muggleton, S. H. and De Raedt, L. (1994). Inductive logic programming: Theory and methods. J. Logic Programming, 19/20, 629–679.

Müller, M. (2002). Computer Go. AIJ, 134, 145–179.

Mumford, D. and Shah, J. (1989). Optimal approximations by piece-wise smooth functions and associated variational problems. Commun. Pure Appl. Math., 42, 577–685.

Mundy, J. and Zisserman, A. (Eds.). (1992). Geometric Invariance in Computer Vision. MIT Press.

Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. G. (2017). Safe and efficient off-policy reinforcement learning. In NeurIPS 29.

Murphy, K. (2002). Dynamic Bayesian Networks: Representation, Inference and Learning. Ph.D. thesis, UC Berkeley.

Murphy, K. (2012). Machine Learning: A Probabilistic Perspective. MIT Press.

Murphy, K. and Mian, I. S. (1999). Modelling gene expression data using Bayesian networks. Tech. rep., Computer Science Division, UC Berkeley.

Murphy, K. and Russell, S. J. (2001). Rao-Blackwellised particle filtering for dynamic Bayesian networks. In Doucet, A., de Freitas, J. F. G., and Gordon, N. J. (Eds.), Sequential Monte Carlo Methods in Practice. Springer-Verlag.

Murphy, K. and Weiss, Y. (2001). The factored frontier algorithm for approximate inference in DBNs. In UAI-01.

Murphy, R. (2000). Introduction to AI Robotics. MIT Press.

Murray, L. M. (2013). Bayesian state-space modelling on high-performance hardware using LibBi. arXiv:1306.3277.

Murray, R. M. (2017). A Mathematical Introduction to Robotic Manipulation. CRC Press.

Murray-Rust, P., Rzepa, H. S., Williamson, J., and Willighagen, E. L. (2003). Chemical markup, XML and the world–wide web. 4. CML schema. J. Chem. Inf. Comput. Sci., 43, 752–772.

Murthy, C. and Russell, J. R. (1990). A constructive proof of Higman’s lemma. In LICS-90.

Muscettola, N. (2002). Computing the envelope for stepwise-constant resource allocations. In CP-02.

Muscettola, N., Nayak, P., Pell, B., and Williams, B. (1998). Remote agent: To boldly go where no AI system has gone before. AIJ, 103, 5–48.

Muslea, I. (1999). Extraction patterns for information extraction tasks: A survey. In Proc. AAAI-99 Workshop on Machine Learning for Information Extraction.

Muth, J. T., Vogt, D. M., Truby, R. L., Mengüç, Y., Kolesky, D. B., Wood, R. J., and Lewis, J. A. (2014). Embedded 3D printing of strain sensors within highly stretchable elastomers. Advanced Materials, 26, 6307–6312.

Myerson, R. (1981). Optimal auction design. Mathematics of Operations Research, 6, 58–73.

Myerson, R. (1986). Multistage games with communication. Econometrica, 54, 323–358.

Myerson, R. (1991). Game Theory: Analysis of Conflict. Harvard University Press.

Nair, V. and Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. In ICML-10.

Nalwa, V. S. (1993). A Guided Tour of Computer Vision. Addison-Wesley.

Narayanan, A., Shi, E., and Rubinstein, B. I. (2011). Link prediction by de-anonymization: How we won the Kaggle social network challenge. In IJCNN-11.

Narayanan, A. and Shmatikov, V. (2006). How to break anonymity of the Netflix prize dataset. arXiv:cs/0610105.

Nash, J. (1950). Equilibrium points in N-person games. PNAS, 36, 48–49.

Nash, P. (1973). Optimal Allocation of Resources Between Research Projects. Ph.D. thesis, University of Cambridge.

Nayak, P. and Williams, B. (1997). Fast context switching in real-time propositional reasoning. In AAAI-97.

Neches, R., Swartout, W. R., and Moore, J. D. (1985). Enhanced maintenance and explanation of expert systems through explicit models of their development. IEEE Transactions on Software Engineering, SE-11, 1337–1351.

Nemhauser, G. L., Wolsey, L. A., and Fisher, M. L. (1978). An analysis of approximations for maximizing submodular set functions I. Mathematical Programming, 14, 265–294.

Nesterov, Y. and Nemirovski, A. (1994). Interior-Point Polynomial Methods in Convex Programming. SIAM (Society for Industrial and Applied Mathematics).

Newell, A. (1982). The knowledge level. AIJ, 18, 82–127.

Newell, A. (1990). Unified Theories of Cognition. Harvard University Press.

Newell, A. and Ernst, G. (1965). The search for generality. In Proc. IFIP Congress.

Newell, A., Shaw, J. C., and Simon, H. A. (1957). Empirical explorations with the logic theory machine. Proc. Western Joint Computer Conference, 15, 218–239. Reprinted in Feigenbaum and Feldman (1963).

Newell, A. and Simon, H. A. (1961). GPS, a program that simulates human thought. In Billing, H. (Ed.), Lernende Automaten. R. Oldenbourg.

Newell, A. and Simon, H. A. (1972). Human Problem Solving. Prentice-Hall.

Newell, A. and Simon, H. A. (1976). Computer science as empirical inquiry: Symbols and search. CACM, 19, 113–126.

Newton, I. (1664–1671). Methodus fluxionum et serierum infinitarum. Unpublished notes.

Ng, A. Y. (2004). Feature selection, L1 vs. L2 regularization, and rotational invariance. In ICML-04.

Ng, A. Y. (2019). Machine Learning Yearning. www.mlyearning.org.

Ng, A. Y., Harada, D., and Russell, S. J. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In ICML-99.

Ng, A. Y. and Jordan, M. I. (2000). PEGASUS: A policy search method for large MDPs and POMDPs. In UAI-00.

Ng, A. Y. and Jordan, M. I. (2002). On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes. In NeurIPS 14.

Ng, A. Y., Kim, H. J., Jordan, M. I., and Sastry, S. (2003). Autonomous helicopter flight via reinforcement learning. In NeurIPS 16.

Ng, A. Y. and Russell, S. J. (2000). Algorithms for inverse reinforcement learning. In ICML00.

Nicholson, A. and Brady, J. M. (1992). The data association problem when monitoring robot vehicles using dynamic belief networks. In ECAI-92.

Nielsen, M. A. (2015). Neural Networks and Deep Learning. Determination Press.

Nielsen, T. and Jensen, F. (2003). Sensitivity analysis in influence diagrams. IEEE Transactions on Systems, Man and Cybernetics, 33, 223–234.

Niemelä, I., Simons, P., and Syrjänen, T. (2000). Smodels: A system for answer set programming. In Proc. 8th International Workshop on Non-Monotonic Reasoning.

Nikolaidis, S. and Shah, J. (2013). Human-robot cross-training: computational formulation, modeling and evaluation of a human team training strategy. In HRI-13.

Niles, I. and Pease, A. (2001). Towards a standard upper ontology. In Proc. International Conference on Formal Ontology in Information Systems.

Nilsson, D. and Lauritzen, S. (2000). Evaluating influence diagrams using LIMIDs. In UAI00.

Nilsson, N. J. (1965). Learning Machines: Foundations of Trainable Pattern-Classifying Systems. McGraw-Hill.

Nilsson, N. J. (1971). Problem-Solving Methods in Artificial Intelligence. McGraw-Hill.

Nilsson, N. J. (1984). Shakey the robot. Technical note, SRI International.

Nilsson, N. J. (1986). Probabilistic logic. AIJ, 28, 71–87.

Nilsson, N. J. (1995). Eye on the prize. AIMag, 16, 9–17.

Nilsson, N. J. (2009). The Quest for Artificial Intelligence: A History of Ideas and Achievements. Cambridge University Press.

Nisan, N. (2007). Introduction to mechanism design (for computer scientists). In Nisan, N., Roughgarden, T., Tardos, E., and Vazirani, V. V. (Eds.), Algorithmic Game Theory. Cambridge University Press.

Nisan, N., Roughgarden, T., Tardos, E., and Vazirani, V. (Eds.). (2007). Algorithmic Game Theory. Cambridge University Press.

Niv, Y. (2009). Reinforcement learning in the brain. Journal of Mathematical Psychology, 53, 139–154.

Nivre, J., De Marneffe, M.-C., Ginter, F., Goldberg, Y., Hajic, J., Manning, C., McDonald, R., Petrov, S., et al. (2016). Universal dependencies v1: A multilingual treebank collection. In Proc. International Conference on Language Resources and Evaluation.

Nodelman, U., Shelton, C., and Koller, D. (2002). Continuous time Bayesian networks. In UAI-02.

Noe, A. (2009). Out of Our Heads: Why You Are Not Your Brain, and Other Lessons from the Biology of Consciousness. Hill and Wang.

Nordfors, D., Cerf, V., and Senges, M. (2018). Disrupting Unemployment. Amazon Digital Services.

Norvig, P. (1988). Multiple simultaneous interpretations of ambiguous sentences. In COGSCI-88.

Norvig, P. (1992). Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp. Morgan Kaufmann.

Norvig, P. (2009). Natural language corpus data. In Segaran, T. and Hammerbacher, J. (Eds.), Beautiful Data. O’Reilly.

Nowick, S. M., Dean, M. E., Dill, D. L., and Horowitz, M. (1993). The design of a highperformance cache controller: A case study in asynchronous synthesis. Integration: The VLSI Journal, 15, 241–262.

Och, F. J. and Ney, H. (2003). A systematic comparison of various statistical alignment models. Computational Linguistics, 29, 19–51.

Och, F. J. and Ney, H. (2004). The alignment template approach to statistical machine translation. Computational Linguistics, 30, 417–449.

Och, F. J. and Ney, H. (2002). Discriminative training and maximum entropy models for statistical machine translation. In COLING-02.

Ogawa, S., Lee, T.-M., Kay, A. R., and Tank, D. W. (1990). Brain magnetic resonance imaging with contrast dependent on blood oxygenation. PNAS, 87, 9868–9872.

Oh, M.-S. and Berger, J. O. (1992). Adaptive importance sampling in Monte Carlo integration. Journal of Statistical Computation and Simulation, 41, 143–168.

Oh, S., Russell, S. J., and Sastry, S. (2009). Markov chain Monte Carlo data association for multi-target tracking. IEEE Transactions on Automatic Control, 54, 481–497.

Oizumi, M., Albantakis, L., and Tononi, G. (2014). From the phenomenology to the mechanisms of consciousness: Integrated information theory 3.0. PLoS Computational Biology, 10, e1003588.

Olesen, K. G. (1993). Causal probabilistic networks with both discrete and continuous variables. PAMI, 15, 275–279.

Oliver, N., Garg, A., and Horvitz, E. J. (2004). Layered representations for learning and inferring office activity from multiple sensory channels. Computer Vision and Image Understanding, 96, 163–180.

Oliver, R. M. and Smith, J. Q. (Eds.). (1990). Influence Diagrams, Belief Nets and Decision Analysis. Wiley.

Omohundro, S. (2008). The basic AI drives. In AGI-08 Workshop on the Sociocultural, Ethical and Futurological Implications of Artificial Intelligence.

O’Neil, C. (2017). Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Broadway Books.

O’Neil, C. and Schutt, R. (2013). Doing Data Science: Straight Talk from the Frontline. O’Reilly.

O’Reilly, U.-M. and Oppacher, F. (1994). Program search with a hierarchical variable length representation: Genetic programming, simulated annealing and hill climbing. In Proc. Third Conference on Parallel Problem Solving from Nature.

Osborne, M. J. (2004). An Introduction to Game Theory. Oxford University Pres.

Osborne, M. J. and Rubinstein, A. (1994). A Course in Game Theory. MIT Press.

Osherson, D. N., Stob, M., and Weinstein, S. (1986). Systems That Learn: An Introduction to Learning Theory for Cognitive and Computer Scientists. MIT Press.

Ostrom, E. (1990). Governing the Commons. Cambridge University Press.

Padgham, L. and Winikoff, M. (2004). Developing Intelligent Agent Systems: A Practical Guide. Wiley.

Paige, B. and Wood, F. (2014). A compilation target for probabilistic programming languages. In ICML-14.

Paige, B., Wood, F., Doucet, A., and Teh, Y. W. (2015). Asynchronous anytime sequential Monte Carlo. In NeurIPS 27.

Palacios, H. and Geffner, H. (2007). From conformant into classical planning: Efficient translations that may be complete too. In ICAPS-07.

Palmer, S. (1999). Vision Science: Photons to Phenomenology. MIT Press.

Papadimitriou, C. H. (1994). Computational Complexity. Addison-Wesley.

Papadimitriou, C. H. and Tsitsiklis, J. N. (1987). The complexity of Markov decision processes. Mathematics of Operations Research, 12, 441–450.

Papadimitriou, C. H. and Yannakakis, M. (1991). Shortest paths without a map. Theoretical Computer Science, 84, 127–150.

Papavassiliou, V. and Russell, S. J. (1999). Convergence of reinforcement learning with general function approximators. In IJCAI-99.

Parisi, G. (1988). Statistical Field Theory. Addison-Wesley.

Parisi, M. M. G. and Zecchina, R. (2002). Analytic and algorithmic solution of random satisfiability problems. Science, 297, 812–815.

Park, J. D. and Darwiche, A. (2004). Complexity results and approximation strategies for MAP explanations. JAIR, 21, 101–133.

Parker, A., Nau, D. S., and Subrahmanian, V. S. (2005). Game-tree search with combinatorially large belief states. In IJCAI-05.

Parker, D. B. (1985). Learning logic. Technical report, Center for Computational Research in Economics and Management Science, MIT.

Parker, L. E. (1996). On the design of behavior-based multi-robot teams. J. Advanced Robotics, 10, 547–578.

Parr, R. and Russell, S. J. (1998). Reinforcement learning with hierarchies of machines. In NeurIPS 10.

Parzen, E. (1962). On estimation of a probability density function and mode. Annals of Mathematical Statistics, 33, 1065–1076.

Pasca, M. and Harabagiu, S. M. (2001). High performance question/answering. In SIGIR-01.

Pasca, M., Lin, D., Bigham, J., Lifchits, A., and Jain, A. (2006). Organizing and searching the world wide web of facts—Step one: The one-million fact extraction challenge. In

AAAI-06.

Paskin, M. (2002). Maximum entropy probabilistic logic. Tech. report, UC Berkeley.

Pasula, H., Marthi, B., Milch, B., Russell, S. J., and Shpitser, I. (2003). Identity uncertainty and citation matching. In NeurIPS 15.

Pasula, H., Russell, S. J., Ostland, M., and Ritov, Y. (1999). Tracking many objects with many sensors. In IJCAI-99.

Patel-Schneider, P. (2014). Analyzing schema.org. In Proc. International Semantic Web Conference.

Patrick, B. G., Almulla, M., and Newborn, M. (1992). An upper bound on the time complexity of iterative-deepening-A*. AIJ, 5, 265–278.

Paul, R. P. (1981). Robot Manipulators: Mathematics, Programming, and Control. MIT Press.

Pauls, A. and Klein, D. (2009). K-best A* parsing. In ACL-09.

Peano, G. (1889). Arithmetices principia, nova methodo exposita. Fratres Bocca, Turin.

Pearce, J., Tambe, M., and Maheswaran, R. (2008). Solving multiagent networks using distributed constraint optimization. AIMag, 29, 47–62.

Pearl, J. (1982a). Reverend Bayes on inference engines: A distributed hierarchical approach. In AAAI-82.

Pearl, J. (1982b). The solution for the branching factor of the alpha–beta pruning algorithm and its optimality. CACM, 25, 559–564.

Pearl, J. (1984). Heuristics: Intelligent Search Strategies for Computer Problem Solving. AddisonWesley.

Pearl, J. (1985). Bayesian networks: A model of self-activated memory for evidential reasoning. In COGSCI-85.

Pearl, J. (1986). Fusion, propagation, and structuring in belief networks. AIJ, 29, 241–288.

Pearl, J. (1987). Evidential reasoning using stochastic simulation of causal models. AIJ, 32, 247–257.

Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann.

Pearl, J. (2000). Causality: Models, Reasoning, and Inference. Cambridge University Press.

Pearl, J. and McKenzie, D. (2018). The Book of Why. Basic Books.

Pearl, J. and Verma, T. (1991). A theory of inferred causation. In KR-91.

Pearson, K. (1895). Contributions to the mathematical theory of evolution, II: Skew variation in homogeneous material. Phil. Trans. Roy. Soc., 186, 343–414.

Pearson, K. (1901). On lines and planes of closest fit to systems of points in space. Philosophical Magazine, 2, 559–572.

Pease, A. and Niles, I. (2002). IEEE standard upper ontology: A progress report. Knowledge Engineering Review, 17, 65–70.

Pednault, E. P. D. (1986). Formulating multiagent, dynamic-world problems in the classical planning framework. In Reasoning About Actions and Plans: Proc. 1986 Workshop.

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011). Scikit-learn: Machine learning in Python. JMLR, 12, 2825–2830.

Peirce, C. S. (1870). Description of a notation for the logic of relatives, resulting from an amplification of the conceptions of Boole’s calculus of logic. Memoirs of the American Academy of Arts and Sciences, 9, 317–378.

Peirce, C. S. (1883). A theory of probable inference. Note B. The logic of relatives. In Peirce, C. S. (Ed.), Studies in Logic, Little, Brown.

Peirce, C. S. (1909). Existential graphs. Unpublished manuscript; reprinted in (Buchler 1955).

Peleg, B. and Sudholter, P. (2002). Introduction to the Theory of Cooperative Games (2nd edition). Springer-Verlag.

Pelikan, M., Goldberg, D. E., and Cantu-Paz, E. (1999). BOA: The Bayesian optimization algorithm. In GECCO-99.

Pemberton, J. C. and Korf, R. E. (1992). Incremental planning on graphs with cycles. In AIPS-92.

Penberthy, J. S. and Weld, D. S. (1992). UCPOP: A sound, complete, partial order planner for ADL. In KR-92.

Peng, J. and Williams, R. J. (1993). Efficient learning and planning within the Dyna framework. Adaptive Behavior, 2, 437–454.

Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation. In EMNLP-14.

Penrose, R. (1989). The Emperor’s New Mind. Oxford University Press.

Penrose, R. (1994). Shadows of the Mind. Oxford University Press.

Peot, M. and Smith, D. E. (1992). Conditional nonlinear planning. In ICAPS-92.

Pereira, F. and Schabes, Y. (1992). Inside-outside reestimation from partially bracketed corpora. In ACL-92.

Pereira, F. and Warren, D. H. D. (1980). Definite clause grammars for language analysis: A survey of the formalism and a comparison with augmented transition networks. AIJ, 13, 231–278.

Peters, J. and Schaal, S. (2008). Reinforcement learning of motor skills with policy gradients. Neural Networks, 21, 682–697.

Peters, J., Janzing, D., and Schölkopf, B. (2017). Elements of Causal Inference: Foundations and Learning Algorithms. MIT press.

Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv:1802.05365.

Peterson, C. and Anderson, J. R. (1987). A mean field theory learning algorithm for neural networks. Complex Systems, 1, 995–1019.

Petosa, N. and Balch, T. (2019). Multiplayer AlphaZero. arXiv:1910.13012.

Pfeffer, A. (2001). IBAL: A probabilistic rational programming language. In IJCAI-01.

Pfeffer, A., Koller, D., Milch, B., and Takusagawa, K. T. (1999). SPOOK: A system for probabilistic object-oriented knowledge representation. In UAI-99.

Pfeffer, A. (2016). Practical Probabilistic Programming. Manning.

Pfeffer, A. (2000). Probabilistic Reasoning for Complex Systems. Ph.D. thesis, Stanford University.

Pfeffer, A. (2007). The design and implementation of IBAL: A general-purpose probabilistic language. In Getoor, L. and Taskar, B. (Eds.), Introduction to Statistical Relational Learning. MIT Press.

Pfeifer, R., Bongard, J., Brooks, R. A., and Iwasawa, S. (2006). How the Body Shapes the Way We Think: A New View of Intelligence. Bradford.

Pham, H., Guan, M. Y., Zoph, B., Le, Q. V., and Dean, J. (2018). Efficient neural architecture search via parameter sharing. arXiv:1802.03268.

Pineau, J., Gordon, G., and Thrun, S. (2003). Point-based value iteration: An anytime algorithm for POMDPs. In IJCAI-03.

Pinedo, M. (2008). Scheduling: Theory, Algorithms, and Systems. Springer Verlag.

Pinkas, G. and Dechter, R. (1995). Improving connectionist energy minimization. JAIR, 3, 223–248.

Pinker, S. (1995). Language acquisition. In Gleitman, L. R., Liberman, M., and Osherson, D. N. (Eds.), An Invitation to Cognitive Science (2nd edition). MIT Press.

Pinker, S. (2003). The Blank Slate: The Modern Denial of Human Nature. Penguin.

Pinto, D., McCallum, A., Wei, X., and Croft, W. B. (2003). Table extraction using conditional random fields. In SIGIR-03.

Pinto, L. and Gupta, A. (2016). Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In ICRA-16.

Platt, J. (1999). Fast training of support vector machines using sequential minimal optimization. In Advances in Kernel Methods: Support Vector Learning. MIT Press.

Plotkin, G. (1972). Building-in equational theories. In Meltzer, B. and Michie, D. (Eds.), Machine Intelligence 7. Edinburgh University Press.

Plummer, M. (2003). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In Proc. Third Int’l Workshop on Distributed Statistical Computing.

Pnueli, A. (1977). The temporal logic of programs. In FOCS-77.

Pohl, I. (1971). Bi-directional search. In Meltzer, B. and Michie, D. (Eds.), Machine Intelligence 6. Edinburgh University Press.

Pohl, I. (1973). The avoidance of (relative) catastrophe, heuristic competence, genuine dynamic weighting and computational issues in heuristic problem solving. In IJCAI-73.

Pohl, I. (1977). Practical and theoretical considerations in heuristic search algorithms. In Elcock, E. W. and Michie, D. (Eds.), Machine Intelligence 8. Ellis Horwood.

Pohl, I. (1970). Heuristic search viewed as path finding in a graph. AIJ, 1, 193–204.

Poli, R., Langdon, W., and McPhee, N. (2008). A Field Guide to Genetic Programming. Lulu.com.

Pomerleau, D. A. (1993). Neural Network Perception for Mobile Robot Guidance. Kluwer.

Poole, B., Lahiri, S., Raghu, M., Sohl-Dickstein, J., and Ganguli, S. (2017). Exponential expressivity in deep neural networks through transient chaos. In NeurIPS 29.

Poole, D. (1993). Probabilistic Horn abduction and Bayesian networks. AIJ, 64, 81–129.

Poole, D. (2003). First-order probabilistic inference. In IJCAI-03.

Poole, D. and Mackworth, A. K. (2017). Artificial Intelligence: Foundations of Computational Agents (2 edition). Cambridge University Press.

Poppe, R. (2010). A survey on vision-based human action recognition. Image Vision Comput., 28, 976–990.

Popper, K. R. (1959). The Logic of Scientific Discovery. Basic Books.

Popper, K. R. (1962). Conjectures and Refutations: The Growth of Scientific Knowledge. Basic Books.

Portner, P. and Partee, B. H. (2002). Formal Semantics: The Essential Readings. WileyBlackwell.

Post, E. L. (1921). Introduction to a general theory of elementary propositions. American Journal of Mathematics, 43, 163–185.

Poulton, C. and Watts, M. (2016). MIT and DARPA pack Lidar sensor onto single chip. IEEE Spectrum, August 4.

Poundstone, W. (1993). Prisoner’s Dilemma. Anchor.

Pourret, O., Naïm, P., and Marcot, B. (2008). Bayesian Networks: A Practical Guide to Applications. Wiley.

Pradhan, M., Provan, G. M., Middleton, B., and Henrion, M. (1994). Knowledge engineering for large belief networks. In UAI-94.

Prawitz, D. (1960). An improved proof procedure. Theoria, 26, 102–139.

Press, W. H., Teukolsky, S. A., Vetterling, W. T., and Flannery, B. P. (2007). Numerical Recipes: The Art of Scientific Computing (3rd edition). Cambridge University Press.

Preston, J. and Bishop, M. (2002). Views into the Chinese Room: New Essays on Searle and Artificial Intelligence. Oxford University Press.

Prieditis, A. E. (1993). Machine discovery of effective admissible heuristics. Machine Learning, 12, 117–141.

Prosser, P. (1993). Hybrid algorithms for constraint satisfaction problems. Computational Intelligence, 9, 268–299.

Pullum, G. K. (1991). The Great Eskimo Vocabulary Hoax (and Other Irreverent Essays on the Study of Language). University of Chicago Press.

Pullum, G. K. (1996). Learnability, hyperlearning, and the poverty of the stimulus. In 22nd Annual Meeting of the Berkeley Linguistics Society.

Puterman, M. L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley.

Puterman, M. L. and Shin, M. C. (1978). Modified policy iteration algorithms for discounted Markov decision problems. Management Science, 24, 1127–1137.

Putnam, H. (1963). ‘Degree of confirmation’ and inductive logic. In Schilpp, P. A. (Ed.), The Philosophy of Rudolf Carnap. Open Court.

Quillian, M. R. (1961). A design for an understanding machine. Paper presented at a colloquium: Semantic Problems in Natural Language, King’s College, Cambridge, England.

Quine, W. V. (1953). Two dogmas of empiricism. In From a Logical Point of View. Harper and Row.

Quine, W. V. (1960). Word and Object. MIT Press.

Quine, W. V. (1982). Methods of Logic (4th edition). Harvard University Press.

Quinlan, J. R. (1979). Discovering rules from large collections of examples: A case study. In Michie, D. (Ed.), Expert Systems in the Microelectronic Age. Edinburgh University Press.

Quinlan, J. R. (1986). Induction of decision trees. Machine Learning, 1, 81–106.

Quinlan, J. R. (1993). C4.5: Programs for Machine Learning. Morgan Kaufmann.

Quinlan, S. and Khatib, O. (1993). Elastic bands: Connecting path planning and control. In ICRA-93.

Quirk, R., Greenbaum, S., Leech, G., and Svartvik, J. (1985). A Comprehensive Grammar of the English Language. Longman.

Rabani, Y., Rabinovich, Y., and Sinclair, A. (1998). A computational view of population genetics. Random Structures and Algorithms, 12, 313–334.

Rabiner, L. R. and Juang, B.-H. (1993). Fundamentals of Speech Recognition. Prentice-Hall.

Radford, A., Metz, L., and Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1511.06434.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1.

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv:1910.10683.

Rafferty, A. N., Brunskill, E., Griffiths, T. L., and Shafto, P. (2016). Faster teaching via POMDP planning. Cognitive Science, 40, 1290–1332.

Rahwan, T., Michalak, T. P., Wooldridge, M., and Jennings, N. R. (2015). Coalition structure generation: A survey. AIJ, 229, 139–174.

Raibert, M., Blankespoor, K., Nelson, G., and Playter, R. (2008). Bigdog, the rough-terrain quadruped robot. IFAC Proceedings Volumes, 41, 10822–10825.

Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. In EMNLP-16.

Ramsey, F. P. (1931). Truth and probability. In Braithwaite, R. B. (Ed.), The Foundations of Mathematics and Other Logical Essays. Harcourt Brace Jovanovich.

Ramsundar, B. and Zadeh, R. B. (2018). TensorFlow for Deep Learning: From Linear Regression to Reinforcement Learning. O’Reilly.

Rao, D. A. S. and Verweij, G. (2017). Sizing the prize: What’s the real value of AI for your business and how can you capitalise? PwC.

Raphael, B. (1976). The Thinking Computer: Mind Inside Matter. W. H. Freeman.

Raphson, J. (1690). Analysis aequationum universalis. Apud Abelem Swalle, London.

Raschka, S. (2015). Python Machine Learning. Packt.

Rashevsky, N. (1936). Physico-mathematical aspects of excitation and conduction in nerves. In Cold Springs Harbor Symposia on Quantitative Biology. IV: Excitation Phenomena.

Rashevsky, N. (1938). Mathematical Biophysics: Physico-Mathematical Foundations of Biology. University of Chicago Press.

Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. MIT Press.

Rassenti, S., Smith, V., and Bulfin, R. (1982). A combinatorial auction mechanism for airport time slot allocation. Bell Journal of Economics, 13, 402–417.

Ratliff, N., Bagnell, J. A., and Zinkevich, M. (2006). Maximum margin planning. In ICML-06.

Ratliff, N., Zucker, M., Bagnell, J. A., and Srinivasa, S. (2009). CHOMP: Gradient optimization techniques for efficient motion planning. In ICRA-09.

Ratnaparkhi, A. (1996). A maximum entropy model for part-of-speech tagging. In EMNLP96.

Ratner, D. and Warmuth, M. (1986). Finding a shortest solution for the n  ×  n extension of the 15-puzzle is intractable. In AAAI-86.

Rauch, H. E., Tung, F., and Striebel, C. T. (1965). Maximum likelihood estimates of linear dynamic systems. AIAA Journal, 3, 1445–1450.

Rayward-Smith, V., Osman, I., Reeves, C., and Smith, G. (Eds.). (1996). Modern Heuristic Search Methods. Wiley.

Real, E., Aggarwal, A., Huang, Y., and Le, Q. V. (2018). Regularized evolution for image classifier architecture search. arXiv:1802.01548.

Rechenberg, I. (1965). Cybernetic solution path of an experimental problem. Library translation, Royal Aircraft Establishment.

Regin, J. (1994). A filtering algorithm for constraints of difference in CSPs. In AAAI-94.

Reid, D. B. (1979). An algorithm for tracking multiple targets. IEEE Trans. Automatic Control, 24, 843–854.

Reif, J. (1979). Complexity of the mover’s problem and generalizations. In FOCS-79.

Reiter, R. (1980). A logic for default reasoning. AIJ, 13, 81–132.

Reiter, R. (1991). The frame problem in the situation calculus: A simple solution (sometimes) and a completeness result for goal regression. In Lifschitz, V. (Ed.), Artificial Intelligence and Mathematical Theory of Computation: Papers in Honor of John McCarthy. Academic Press.

Reiter, R. (2001). Knowledge in Action: Logical Foundations for Specifying and Implementing Dynamical Systems. MIT Press.

Renner, G. and Ekart, A. (2003). Genetic algorithms in computer aided design. Computer Aided Design, 35, 709–726.

Rényi, A. (1970). Probability Theory. Elsevier.

Resnick, P. and Varian, H. R. (1997). Recommender systems. CACM, 40, 56–58.

Rezende, D. J., Mohamed, S., and Wierstra, D. (2014). Stochastic backpropagation and approximate inference in deep generative models. In ICML-14.

Riazanov, A. and Voronkov, A. (2002). The design and implementation of VAMPIRE. AI Communications, 15, 91–110.

Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). Why should I trust you?: Explaining the predictions of any classifier. In KDD-16.

Richardson, M. and Domingos, P. (2006). Markov logic networks. Machine Learning, 62, 107–136.

Richter, S. and Helmert, M. (2009). Preferred operators and deferred evaluation in satisficing planning. In ICAPS-09.

Ridley, M. (2004). Evolution. Oxford Reader.

Riley, J. and Samuelson, W. (1981). Optimal auctions. American Economic Review, 71, 381– 392.

Riley, P. (2019). Three pitfalls to avoid in machine learning. Nature, 572, 27–29.

Riloff, E. (1993). Automatically constructing a dictionary for information extraction tasks. In AAAI-93.

Ringgaard, M., Gupta, R., and Pereira, F. (2017). SLING: A framework for frame semantic parsing. arXiv:1710.07032.

Rintanen, J. (1999). Improvements to the evaluation of quantified Boolean formulae. In IJCAI-99.

Rintanen, J. (2007). Asymptotically optimal encodings of conformant planning in QBF. In AAAI-07.

Rintanen, J. (2012). Planning as satisfiability: Heuristics. AIJ, 193, 45–86.

Rintanen, J. (2016). Computational complexity in automated planning and scheduling. In ICAPS-16.

Ripley, B. D. (1996). Pattern Recognition and Neural Networks. Cambridge University Press.

Rissanen, J. (1984). Universal coding, information, prediction, and estimation. IEEE Transactions on Information Theory, IT-30, 629–636.

Rissanen, J. (2007). Information and Complexity in Statistical Modeling. Springer.

Rivest, R. (1987). Learning decision lists. Machine Learning, 2, 229–246.

Robbins, H. (1952). Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58, 527–535.

Robbins, H. and Monro, S. (1951). A stochastic approximation method. Annals of Mathematical Statistics, 22, 400–407.

Roberts, L. G. (1963). Machine perception of three-dimensional solids. Technical report, MIT Lincoln Laboratory.

Robertson, N. and Seymour, P. D. (1986). Graph minors. II. Algorithmic aspects of treewidth. J. Algorithms, 7, 309–322.

Robertson, S. E. and Sparck Jones, K. (1976). Relevance weighting of search terms. J. American Society for Information Science, 27, 129–146.

Robins, J. (1986). A new approach to causal inference in mortality studies with a sustained exposure period: Application to control of the healthy worker survivor effect. Mathematical Modelling, 7, 1393–1512.

Robinson, A. and Voronkov, A. (Eds.). (2001). Handbook of Automated Reasoning. Elsevier.

Robinson, J. A. (1965). A machine-oriented logic based on the resolution principle. JACM, 12, 23–41.

Robinson, S. (2002). Computer scientists find unexpected depths in airfare search problem. SIAM News, 35(6).

Roche, E. and Schabes, Y. (Eds.). (1997). Finite-State Language Processing. Bradford Books.

Rock, I. (1984). Perception. W. H. Freeman.

Rokicki, T., Kociemba, H., Davidson, M., and Dethridge, J. (2014). The diameter of the Rubik’s Cube group is twenty. SIAM Review, 56, 645–670.

Rolf, D. (2006). Improved bound for the PPSZ/Schöning-algorithm for 3-SAT. Journal on Satisfiability, Boolean Modeling and Computation, 1, 111–122.

Rolnick, D., Donti, P. L., Kaack, L. H., et al. (2019). Tackling climate change with machine learning. arXiv:1906.05433.

Rolnick, D. and Tegmark, M. (2018). The power of deeper networks for expressing natural functions. In ICLR-18.

Romanovskii, I. (1962). Reduction of a game with complete memory to a matrix game. Soviet Mathematics, 3, 678–681.

Ros, G., Sellart, L., Materzynska, J., Vazquez, D., and Lopez, A. M. (2016). The SYNTHIA dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR-16.

Rosenblatt, F. (1957). The perceptron: A perceiving and recognizing automaton. Report, Project PARA, Cornell Aeronautical Laboratory.

Rosenblatt, F. (1960). On the convergence of reinforcement procedures in simple perceptrons. Report, Cornell Aeronautical Laboratory.

Rosenblatt, F. (1962). Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms. Spartan.

Rosenblatt, M. (1956). Remarks on some nonparametric estimates of a density function. Annals of Mathematical Statistics, 27, 832–837.

Rosenblueth, A., Wiener, N., and Bigelow, J. (1943). Behavior, purpose, and teleology. Philosophy of Science, 10, 18–24.

Rosenschein, J. S. and Zlotkin, G. (1994). Rules of Encounter. MIT Press.

Rosenschein, S. J. (1985). Formal theories of knowledge in AI and robotics. New Generation Computing, 3, 345–357.

Ross, G. (2012). Fisher and the millionaire: The statistician and the calculator. Significance, 9, 46–48.

Ross, S. (2015). A First Course in Probability (9th edition). Pearson.

Ross, S., Gordon, G., and Bagnell, D. (2011). A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS-11.

Rossi, F., van Beek, P., and Walsh, T. (2006). Handbook of Constraint Processing. Elsevier.

Roth, D. (1996). On the hardness of approximate reasoning. AIJ, 82, 273–302.

Roussel, P. (1975). Prolog: Manual de référence et d’utilization. Tech. rep., Groupe d’Intelligence Artificielle, Université d’Aix-Marseille.

Rowat, P. F. (1979). Representing the Spatial Experience and Solving Spatial Problems in a Simulated Robot Environment. Ph.D. thesis, University of British Columbia.

Roweis, S. T. and Ghahramani, Z. (1999). A unifying review of linear Gaussian models. Neural Computation, 11, 305–345.

Rowley, H., Baluja, S., and Kanade, T. (1998). Neural network-based face detection. PAMI, 20, 23–38.

Roy, N., Gordon, G., and Thrun, S. (2005). Finding approximate POMDP solutions through belief compression. JAIR, 23, 1–40.

Rubin, D. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688–701.

Rubin, D. (1988). Using the SIR algorithm to simulate posterior distributions. In Bernardo, J. M., de Groot, M. H., Lindley, D. V., and Smith, A. F. M. (Eds.), Bayesian Statistics 3. Oxford University Press.

Rubinstein, A. (1982). Perfect equilibrium in a bargaining model. Econometrica, 50, 97–109.

Rubinstein, A. (2003). Economics and psychology? The case of hyperbolic discounting. International Economic Review, 44, 1207–1216.

Ruder, S. (2018). NLP’s ImageNet moment has arrived. The Gradient, July 8.

Ruder, S., Peters, M. E., Swayamdipta, S., and Wolf, T. (2019). Transfer learning in natural language processing. In COLING-19.

Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536.

Rumelhart, D. E. and McClelland, J. L. (Eds.). (1986). Parallel Distributed Processing. MIT Press.

Rummery, G. A. and Niranjan, M. (1994). On-line

Q-learning using connectionist systems.

Tech. rep., Cambridge University Engineering Department.

Ruspini, E. H., Lowrance, J. D., and Strat, T. M. (1992). Understanding evidential reasoning. IJAR, 6, 401–424.

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. (2015). ImageNet large scale visual recognition challenge. IJCV, 115, 211–252.

Russell, J. G. B. (1990). Is screening for abdominal aortic aneurysm worthwhile? Clinical Radiology, 41, 182–184.

Russell, S. J. (1985). The compleat guide to MRS. Report, Computer Science Department, Stanford University.

Russell, S. J. (1992). Efficient memory-bounded search methods. In ECAI-92.

Russell, S. J. (1998). Learning agents for uncertain environments. In COLT-98.

Russell, S. J. (1999). Expressive probability models in science. In Proc. Second International Conference on Discovery Science.

Russell, S. J. (2019). Human Compatible. Penguin.

Russell, S. J., Binder, J., Koller, D., and Kanazawa, K. (1995). Local learning in probabilistic networks with hidden variables. In IJCAI-95.

Russell, S. J. and Norvig, P. (2003). Artificial Intelligence: A Modern Approach (2nd edition). Prentice-Hall.

Russell, S. J. and Subramanian, D. (1995). Provably bounded-optimal agents. JAIR, 3, 575– 609.

Russell, S. J. and Wefald, E. H. (1989). On optimal game-tree search using rational metareasoning. In IJCAI-89.

Russell, S. J. and Wefald, E. H. (1991). Do the Right Thing: Studies in Limited Rationality. MIT Press.

Russell, S. J. and Wolfe, J. (2005). Efficient belief-state AND-OR search, with applications to Kriegspiel. In IJCAI-05.

Russell, S. J. and Zimdars, A. (2003). Q-decomposition of reinforcement learning agents. In ICML-03.

Rustagi, J. S. (1976). Variational Methods in Statistics. Academic Press.

Saad, F. and Mansinghka, V. K. (2017). A probabilistic programming approach to probabilistic data analysis. In NeurIPS 29.

Sabin, D. and Freuder, E. C. (1994). Contradicting conventional wisdom in constraint satisfaction. In ECAI-94.

Sabri, K. E. (2015). Automated verification of role-based access control policies constraints using Prover9. arXiv:1503.07645.

Sacerdoti, E. D. (1974). Planning in a hierarchy of abstraction spaces. AIJ, 5, 115–135.

Sacerdoti, E. D. (1975). The nonlinear nature of plans. In IJCAI-75.

Sacerdoti, E. D. (1977). A Structure for Plans and Behavior. Elsevier.

Sadeghi, F. and Levine, S. (2016). CAD2RL: Real single-image flight without a single real image. arXiv:1611.04201.

Sadigh, D., Sastry, S., Seshia, S. A., and Dragan, A. D. (2016). Planning for autonomous cars that leverage effects on human actions. In Proc. Robotics: Science and Systems.

Sadler, M. and Regan, N. (2019). Game Changer. New in Chess.

Sadri, F. and Kowalski, R. (1995). Variants of the event calculus. In ICLP-95.

Sagae, K. and Lavie, A. (2006). A best-first probabilistic shift-reduce parser. In COLING-06.

Sahami, M., Hearst, M. A., and Saund, E. (1996). Applying the multiple cause mixture model to text categorization. In ICML-96.

Sahin, N. T., Pinker, S., Cash, S. S., Schomer, D., and Halgren, E. (2009). Sequential processing of lexical, grammatical, and phonological information within Broca’s area.

Science, 326, 445–449.

Sakuta, M. and Iida, H. (2002). AND/OR-tree search for solving problems with uncertainty: A case study using screen-shogi problems. Trans. Inf. Proc. Society of Japan, 43, 1–10.

Salomaa, A. (1969). Probabilistic and weighted grammars. Information and Control, 15, 529– 544.

Samadi, M., Felner, A., and Schaeffer, J. (2008). Learning from multiple heuristics. In AAAI08.

Samet, H. (2006). Foundations of Multidimensional and Metric Data Structures. Morgan Kaufmann.

Sammut, C., Hurst, S., Kedzier, D., and Michie, D. (1992). Learning to fly. In ICML-92.

Samuel, A. (1959). Some studies in machine learning using the game of checkers. IBM Journal of Research and Development, 3, 210–229.

Samuel, A. (1967). Some studies in machine learning using the game of checkers II—Recent progress. IBM Journal of Research and Development, 11, 601–617.

Sanchez-Lengeling, B., Wei, J. N., Lee, B. K., Gerkin, R. C., Aspuru-Guzik, A., and Wiltschko, A. B. (2019). Machine learning for scent: Learning generalizable perceptual representations of small molecules. arXiv:1910.10685.

Sandholm, T. (1999). Distributed rational decision making. In Weiß, G. (Ed.), Multiagent Systems. MIT Press.

Sandholm, T., Larson, K., Andersson, M., Shehory, O., and Tohmé, F. (1999). Coalition structure generation with worst case guarantees. AIJ, 111, 209–238.

Sandholm, T. (1993). An implementation of the contract net protocol based on marginal cost calculations. In AAAI-93.

Sang, T., Beame, P., and Kautz, H. (2005). Performing Bayesian inference by weighted model counting. In AAAI-05.

Sapir, E. (1921). Language: An Introduction to the Study of Speech. Harcourt Brace Jovanovich.

Sarawagi, S. (2007). Information extraction. Foundations and Trends in Databases, 1, 261–377.

Sargent, T. J. (1978). Estimation of dynamic labor demand schedules under rational expectations. J. Political Economy, 86, 1009–1044.

Sartre, J.-P. (1960). Critique de la Raison dialectique. Editions Gallimard.

Satia, J. K. and Lave, R. E. (1973). Markovian decision processes with probabilistic observation of states. Management Science, 20, 1–13.

Sato, T. and Kameya, Y. (1997). PRISM: A symbolic-statistical modeling language. In IJCAI97.

Saul, L. K., Jaakkola, T., and Jordan, M. I. (1996). Mean field theory for sigmoid belief networks. JAIR, 4, 61–76.

Saunders, W., Sastry, G., Stuhlmüller, A., and Evans, O. (2018). Trial without error: Towards safe reinforcement learning via human intervention. In AAMAS-18.

Savage, L. J. (1954). The Foundations of Statistics. Wiley.

Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., Parikh, D., and Batra, D. (2019). Habitat: A platform for embodied AI research. arXiv:1904.01201.

Sayre, K. (1993). Three more flaws in the computational model. Paper presented at the APA (Central Division) Annual Conference, Chicago, Illinois.

Schaeffer, J. (2008). One Jump Ahead: Computer Perfection at Checkers. Springer-Verlag.

Schaeffer, J., Burch, N., Bjornsson, Y., Kishimoto, A., Müller, M., Lake, R., Lu, P., and Sutphen, S. (2007). Checkers is solved. Science, 317, 1518–1522.

Schank, R. C. and Abelson, R. P. (1977). Scripts, Plans, Goals, and Understanding. Lawrence Erlbaum.

Schank, R. C. and Riesbeck, C. (1981). Inside Computer Understanding: Five Programs Plus Miniatures. Lawrence Erlbaum.

Schapire, R. E. and Singer, Y. (2000). Boostexter: A boosting-based system for text categorization. Machine Learning, 39, 135–168.

Schapire, R. E. (1990). The strength of weak learnability. Machine Learning, 5, 197–227.

Schapire, R. E. (2003). The boosting approach to machine learning: An overview. In Denison, D. D., Hansen, M. H., Holmes, C., Mallick, B., and Yu, B. (Eds.), Nonlinear Estimation and Classification. Springer.

Scharre, P. (2018). Army of None. W. W. Norton.

Schmid, C. and Mohr, R. (1996). Combining greyvalue invariants with local constraints for object recognition. In CVPR-96.

Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural Networks, 61, 85–117.

Schofield, M. and Thielscher, M. (2015). Lifting model sampling for general game playing to incomplete-information models. In AAAI-15.

Schölkopf, B. and Smola, A. J. (2002). Learning with Kernels. MIT Press.

Schöning, T. (1999). A probabilistic algorithm for k-SAT and constraint satisfaction problems. In FOCS-99.

Schoppers, M. J. (1989). In defense of reaction plans as caches. AIMag, 10, 51–60.

Schraudolph, N. N., Dayan, P., and Sejnowski, T. (1994). Temporal difference learning of position evaluation in the game of Go. In NeurIPS 6.

Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., Lillicrap, T., and Silver, D. (2019). Mastering Atari, Go, chess and shogi by planning with a learned model. arXiv:1911.08265.

Schröder, E. (1877). Der Operationskreis des Logikkalküls. B. G. Teubner, Leipzig.

Schulman, J., Ho, J., Lee, A. X., Awwal, I., Bradlow, H., and Abbeel, P. (2013). Finding locally optimal, collision-free trajectories with sequential convex optimization. In Proc. Robotics: Science and Systems.

Schulman, J., Levine, S., Abbeel, P., Jordan, M. I., and Moritz, P. (2015a). Trust region policy optimization. In ICML-15.

Schulman, J., Levine, S., Moritz, P., Jordan, M., and Abbeel, P. (2015b). Trust region policy optimization. In ICML-15.

Schultz, W., Dayan, P., and Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593.

Schulz, D., Burgard, W., Fox, D., and Cremers, A. B. (2003). People tracking with mobile robots using sample-based joint probabilistic data association filters. Int. J. Robotics Research, 22, 99–116.

Schulz, S. (2004). System Description: E 0.81. In Proc. International Joint Conference on Automated Reasoning, Vol. 3097 of LNAI.

Schulz, S. (2013). System description: E 1.8. In Proc. Int. Conf. on Logic for Programming Artificial Intelligence and Reasoning.

Schütze, H. (1995). Ambiguity in Language Learning: Computational and Cognitive Models. Ph.D. thesis, Stanford University. Also published by CSLI Press, 1997.

Schwartz, J. T., Scharir, M., and Hopcroft, J. (1987). Planning, Geometry and Complexity of Robot Motion. Ablex.

Schwartz, S. P. (Ed.). (1977). Naming, Necessity, and Natural Kinds. Cornell University Press.

Scott, D. and Krauss, P. (1966). Assigning probabilities to logical formulas. In Hintikka, J. and Suppes, P. (Eds.), Aspects of Inductive Logic. North-Holland.

Searle, J. R. (1980). Minds, brains, and programs. BBS, 3, 417–457.

Searle, J. R. (1990). Is the brain’s mind a computer program? Scientific American, 262, 26–31.

Searle, J. R. (1992). The Rediscovery of the Mind. MIT Press.

Sedgewick, R. and Wayne, K. (2011). Algorithms. Addison-Wesley.

Sefidgar, Y. S., Agarwal, P., and Cakmak, M. (2017). Situated tangible robot programming. In HRI-17.

Segaran, T. (2007). Programming Collective Intelligence: Building Smart Web 2.0 Applications. O’Reilly.

Seipp, J. and Röger, G. (2018). Fast downward stone soup 2018. IPC 2018 Classical Track.

Seipp, J., Sievers, S., Helmert, M., and Hutter, F. (2015). Automatic configuration of sequential planning portfolios. In AAAI-15.

Selman, B., Kautz, H., and Cohen, B. (1996). Local search strategies for satisfiability testing. In Johnson, D. S. and Trick, M. A. (Eds.), Cliques, Coloring, and Satisfiability. American Mathematical Society.

Selman, B. and Levesque, H. J. (1993). The complexity of path-based defeasible inheritance. AIJ, 62, 303–339.

Selman, B., Levesque, H. J., and Mitchell, D. (1992). A new method for solving hard satisfiability problems. In AAAI-92.

Seni, G. and Elder, J. F. (2010). Ensemble methods in data mining: Improving accuracy through combining predictions. Synthesis Lectures on Data Mining and Knowledge Discovery, 2, 1–126.

Seo, M., Kembhavi, A., Farhadi, A., and Hajishirzi, H. (2017). Bidirectional attention flow for machine comprehension. In ICLR-17.

Shachter, R. D. (1986). Evaluating influence diagrams. Operations Research, 34, 871–882.

Shachter, R. D. (1998). Bayes-ball: The rational pastime (for determining irrelevance and requisite information in belief networks and influence diagrams). In UAI-98.

Shachter, R. D., D’Ambrosio, B., and Del Favero, B. A. (1990). Symbolic probabilistic inference in belief networks. In AAAI-90.

Shachter, R. D. and Kenley, C. R. (1989). Gaussian influence diagrams. Management Science, 35, 527–550.

Shachter, R. D. and Peot, M. (1989). Simulation approaches to general probabilistic inference on belief networks. In UAI-98.

Shafer, G. (1976). A Mathematical Theory of Evidence. Princeton University Press.

Shanahan, M. (1997). Solving the Frame Problem. MIT Press.

Shanahan, M. (1999). The event calculus explained. In Wooldridge, M. J. and Veloso, M. (Eds.), Artificial Intelligence Today. Springer-Verlag.

Shanahan, M. (2015). The Technological Singularity. MIT Press.

Shani, G., Pineau, J., and Kaplow, R. (2013). A survey of point-based POMDP solvers. Autonomous Agents and Multi-Agent Systems, 27, 1–51.

Shankar, N. (1986). Proof-Checking Metamathematics. Ph.D. thesis, Computer Science Department, University of Texas at Austin.

Shannon, C. E. and Weaver, W. (1949). The Mathematical Theory of Communication. University of Illinois Press.

Shannon, C. E. (1950). Programming a computer for playing chess. Philosophical Magazine, 41, 256–275.

Shapley, L. S. (1953a). A value for n-person games. In Kuhn, H. W. and Tucker, A. W. (Eds.), Contributions to the Theory of Games. Princeton University Press.

Shapley, S. (1953b). Stochastic games. PNAS, 39, 1095–1100.

Sharan, R. V. and Moir, T. J. (2016). An overview of applications and advancements in automatic sound recognition. Neurocomputing, 200, 22–34.

Shatkay, H. and Kaelbling, L. P. (1997). Learning topological maps with weak local odometric information. In IJCAI-97.

Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. V., Hinton, G. E., and Dean, J. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv:1701.06538.

Shelley, M. (1818). Frankenstein: Or, the Modern Prometheus. Pickering and Chatto.

Sheppard, B. (2002). World-championship-caliber scrabble. AIJ, 134, 241–275.

Shi, J. and Malik, J. (2000). Normalized cuts and image segmentation. PAMI, 22, 888–905.

Shieber, S. (1994). Lessons from a restricted Turing test. CACM, 37, 70–78.

Shieber, S. (Ed.). (2004). The Turing Test. MIT Press.

Shimony, S. E. (1994). Finding MAPs for belief networks is NP-hard. AIJ, 68, 399–410.

Shoham, Y. (1993). Agent-oriented programming. AIJ, 60, 51–92.

Shoham, Y. (1994). Artificial Intelligence Techniques in Prolog. Morgan Kaufmann.

Shoham, Y. and Leyton-Brown, K. (2009). Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations. Cambridge Univ. Press.

Shoham, Y., Powers, R., and Grenager, T. (2004). If multi-agent learning is the answer, what is the question? In Proc. AAAI Fall Symposium on Artificial Multi-Agent Learning.

Shortliffe, E. H. (1976). Computer-Based Medical Consultations: MYCIN. Elsevier.

Siciliano, B. and Khatib, O. (Eds.). (2016). Springer Handbook of Robotics (2nd edition). Springer-Verlag.

Sigaud, O. and Buffet, O. (2010). Markov Decision Processes in Artificial Intelligence. Wiley.

Sigmund, K. (2017). Exact Thinking in Demented Times. Basic Books.

Silberstein, M., Weissbrod, O., Otten, L., Tzemach, A., Anisenia, A., Shtark, O., Tuberg, D., Galfrin, E., Gannon, I., Shalata, A., Borochowitz, Z. U., Dechter, R., Thompson, E., and Geiger, D. (2013). A system for exact and approximate genetic linkage analysis of SNP data in large pedigrees. Bioinformatics, 29, 197–205.

Silva, R., Melo, F. S., and Veloso, M. (2015). Towards table tennis with a quadrotor autonomous learning robot and onboard vision. In IROS-15.

Silver, D. and Veness, J. (2011). Monte-Carlo planning in large POMDPs. In NeurIPS 23.

Silver, D., Huang, A., Maddison, C. J., Guez, A., and Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529, 484–489.

Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2018). A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362, 1140–1144.

Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., van den Driessche, G., Graepel, T., and Hassabis, D. (2017). Mastering the game of Go without human knowledge. Nature, 550, 354–359.

Silverman, B. W. (1986). Density Estimation for Statistics and Data Analysis. Chapman and Hall.

Silverstein, C., Henzinger, M., Marais, H., and Moricz, M. (1998). Analysis of a very large AltaVista query log. Tech. rep., Digital Systems Research Center.

Simmons, R. and Koenig, S. (1995). Probabilistic robot navigation in partially observable environments. In IJCAI-95.

Simon, D. (2006). Optimal State Estimation: Kalman, H Infinity, and Nonlinear Approaches. Wiley.

Simon, H. A. (1947). Administrative Behavior. Macmillan.

Simon, H. A. (1963). Experiments with a heuristic compiler. JACM, 10, 493–506.

Simon, H. A. and Newell, A. (1958). Heuristic problem solving: The next advance in operations research. Operations Research, 6, 1–10.

Simon, J. C. and Dubois, O. (1989). Number of solutions to satisfiability instances— Applications to knowledge bases. AIJ, 3, 53–65.

Simonis, H. (2005). Sudoku as a constraint problem. In CP-05 Workshop on Modeling and Reformulating Constraint Satisfaction Problems.

Singer, P. W. (2009). Wired for War. Penguin Press.

Singh, P., Lin, T., Mueller, E. T., Lim, G., Perkins, T., and Zhu, W. L. (2002). Open mind common sense: Knowledge acquisition from the general public. In Proc. First International Conference on Ontologies, Databases, and Applications of Semantics for Large Scale Information Systems.

Sisbot, E. A., Marin-Urias, L. F., Alami, R., and Simeon, T. (2007). A human aware mobile robot motion planner. IEEE Transactions on Robotics, 23, 874–883.

Siskind, J. M. and Pearlmutter, B. A. (2016). Efficient implementation of a higher-order language with built-in AD. arXiv:1611.03416.

Sistla, A. P. and Godefroid, P. (2004). Symmetry and reduced symmetry in model checking. ACM Trans. Program. Lang. Syst., 26, 702–734.

Sittler, R. W. (1964). An optimal data association problem in surveillance theory. IEEE Transactions on Military Electronics, 8, 125–139.

Skolem, T. (1920). Logisch-kombinatorische Untersuchungen über die Erfüllbarkeit oder Beweisbarkeit mathematischer Sätze nebst einem Theoreme über die dichte Mengen. Videnskapsselskapets skrifter, I. Matematisk-naturvidenskabelig klasse, 4, 1–36.

Skolem, T. (1928). Über die mathematische Logik. Norsk matematisk tidsskrift, 10, 125–142.

Slagle, J. R. (1963). A heuristic program that solves symbolic integration problems in freshman calculus. JACM, 10.

Slate, D. J. and Atkin, L. R. (1977). CHESS 4.5— Northwestern University chess program. In Frey, P. W. (Ed.), Chess Skill in Man and Machine. Springer-Verlag.

Slater, E. (1950). Statistics for the chess computer and the factor of mobility. In Symposium on Information Theory. Ministry of Supply.

Slocum, J. and Sonneveld, D. (2006). The 15 Puzzle. Slocum Puzzle Foundation.

Smallwood, R. D. and Sondik, E. J. (1973). The optimal control of partially observable Markov processes over a finite horizon. Operations Research, 21, 1071–1088.

Smith, B. (2004). Ontology. In Floridi, L. (Ed.), The Blackwell Guide to the Philosophy of Computing and Information. Wiley-Blackwell.

Smith, B., Ashburner, M., Rosse, C., et al. (2007). The OBO Foundry: Coordinated evolution of ontologies to support biomedical data integration. Nature Biotechnology, 25, 1251– 1255.

Smith, D. E., Genesereth, M. R., and Ginsberg, M. L. (1986). Controlling recursive inference. AIJ, 30, 343–389.

Smith, D. A. and Eisner, J. (2008). Dependency parsing by belief propagation. In EMNLP-08.

Smith, D. E. and Weld, D. S. (1998). Conformant Graphplan. In AAAI-98.

Smith, J. Q. (1988). Decision Analysis. Chapman and Hall.

Smith, J. E. and Winkler, R. L. (2006). The optimizer’s curse: Skepticism and postdecision surprise in decision analysis. Management Science, 52, 311–322.

Smith, J. M. (1982). Evolution and the Theory of Games. Cambridge University Press.

Smith, J. M. and Szathmáry, E. (1999). The Origins of Life: From the Birth of Life to the Origin of Language. Oxford University Press.

Smith, M. K., Welty, C., and McGuinness, D. (2004). OWL web ontology language guide. Tech. rep., W3C.

Smith, R. G. (1980). A Framework for Distributed Problem Solving. UMI Research Press.

Smith, R. C. and Cheeseman, P. (1986). On the representation and estimation of spatial uncertainty. Int. J. Robotics Research, 5, 56–68.

Smith, S. J. J., Nau, D. S., and Throop, T. A. (1998). Success in spades: Using AI planning techniques to win the world championship of computer bridge. In AAAI-98.

Smith, W. E. (1956). Various optimizers for single-stage production. Naval Research Logistics Quarterly, 3, 59–66.

Smolensky, P. (1988). On the proper treatment of connectionism. BBS, 2, 1–74.

Smolensky, P. and Prince, A. (1993). Optimality theory: Constraint interaction in generative grammar. Tech. rep., Department of Computer Science, University of Colorado at Boulder.

Smullyan, R. M. (1995). First-Order Logic. Dover.

Smyth, P., Heckerman, D., and Jordan, M. I. (1997). Probabilistic independence networks for hidden Markov probability models. Neural Computation, 9, 227–269.

Snoek, J., Larochelle, H., and Adams, R. P. (2013). Practical Bayesian optimization of machine learning algorithms. In NeurIPS 25.

Solomonoff, R. J. (1964). A formal theory of inductive inference. Information and Control, 7, 1–22, 224–254.

Solomonoff, R. J. (2009). Algorithmic probability–theory and applications. In EmmertStreib, F. and Dehmer, M. (Eds.), Information Theory and Statitical Learning. Springer.

Sondik, E. J. (1971). The Optimal Control of Partially Observable Markov Decision Processes. Ph.D. thesis, Stanford University.

Sosic, R. and Gu, J. (1994). Efficient local search with conflict minimization: A case study of the n-queens problem. IEEE Transactions on Knowledge and Data Engineering, 6, 661– 668.

Sowa, J. (1999). Knowledge Representation: Logical, Philosophical, and Computational Foundations. Blackwell.

Spaan, M. T. J. and Vlassis, N. (2005). Perseus: Randomized point-based value iteration for POMDPs. JAIR, 24, 195–220.

Sparrow, R. (2004). The Turing triage test. Ethics and Information Technology, 6, 203–213.

Spiegelhalter, D. J., Dawid, A. P., Lauritzen, S., and Cowell, R. (1993). Bayesian analysis in expert systems. Statistical Science, 8, 219–282.

Spirtes, P., Glymour, C., and Scheines, R. (1993). Causation, Prediction, and Search. SpringerVerlag.

Spitkovsky, V. I., Alshawi, H., and Jurafsky, D. (2010a). From baby steps to leapfrog: How less is more in unsupervised dependency parsing. In NAACL HLT.

Spitkovsky, V. I., Jurafsky, D., and Alshawi, H. (2010b). Profiting from mark-up: Hyper-text annotations for guided parsing. In ACL-10.

Srivas, M. and Bickford, M. (1990). Formal verification of a pipelined microprocessor. IEEE Software, 7, 52–64.

Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014a). Dropout: A simple way to prevent neural networks from overfitting. JMLR, 15, 1929– 1958.

Srivastava, S., Russell, S. J., and Ruan, P. (2014b). First-order open-universe POMDPs. In UAI-14.

Staab, S. (2004). Handbook on Ontologies. Springer.

Stallman, R. M. and Sussman, G. J. (1977). Forward reasoning and dependency-directed backtracking in a system for computer-aided circuit analysis. AIJ, 9, 135–196.

Stanfill, C. and Waltz, D. (1986). Toward memory-based reasoning. CACM, 29, 1213–1228.

Stanislawska, K., Krawiec, K., and Vihma, T. (2015). Genetic programming for estimation of heat flux between the atmosphere and sea ice in polar regions. In GECCO-15.

Stefik, M. (1995). Introduction to Knowledge Systems. Morgan Kaufmann.

Steiner, D. F., MacDonald, R., Liu, Y., Truszkowski, P., Hipp, J. D., Gammage, C., Thng, F., Peng, L., and Stumpe, M. C. (2018). Impact of deep learning assistance on the histopathologic review of lymph nodes for metastatic breast cancer. Am. J. Surgical Pathology, 42, 1636–1646.

Steinruecken, C., Smith, E., Janz, D., Lloyd, J., and Ghahramani, Z. (2019). The Automatic Statistician. In Hutter, F., Kotthoff, L., and Vanschoren, J. (Eds.), Automated Machine Learning. Springer.

Stergiou, K. and Walsh, T. (1999). The difference all-difference makes. In IJCAI-99.

Stickel, M. E. (1992). A Prolog technology theorem prover: a new exposition and implementation in Prolog. Theoretical Computer Science, 104, 109–128.

Stiller, L. (1992). KQNKRR. J. International Computer Chess Association, 15, 16–18.

Stiller, L. (1996). Multilinear algebra and chess endgames. In Nowakowski, R. J. (Ed.), Games of No Chance, MSRI, 29, 1996. Mathematical Sciences Research Institute.

Stockman, G. (1979). A minimax algorithm better than alpha–beta? AIJ, 12, 179–196.

Stoffel, K., Taylor, M., and Hendler, J. (1997). Efficient management of very large ontologies. In AAAI-97.

Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. J. Royal Statistical Society, 36, 111–133.

Stone, P. (2000). Layered Learning in Multi-Agent Systems: A Winning Approach to Robotic Soccer. MIT Press.

Stone, P. (2003). Multiagent competitions and research: Lessons from RoboCup and TAC. In Lima, P. U. and Rojas, P. (Eds.), RoboCup-2002: Robot Soccer World Cup VI. Springer Verlag.

Stone, P. (2016). What’s hot at RoboCup. In AAAI-16.

Stone, P., Brooks, R. A., Brynjolfsson, E., Calo, R., Etzioni, O., Hager, G., Hirschberg, J., Kalyanakrishnan, S., Kamar, E., Kraus, S., et al. (2016). Artificial intelligence and life in 2030. Tech. rep., Stanford University One Hundred Year Study on Artificial Intelligence: Report of the 2015-2016 Study Panel.

Stone, P., Kaminka, G., and Rosenschein, J. S. (2009). Leading a best-response teammate in an ad hoc team. In AAMAS Workshop in Agent Mediated Electronic Commerce.

Stone, P., Sutton, R. S., and Kuhlmann, G. (2005). Reinforcement learning for robocup soccer keepaway. Adaptive Behavior, 13, 165–188.

Storvik, G. (2002). Particle filters for state-space models with the presence of unknown static parameters. IEEE Transactions on Signal Processing, 50, 281–289.

Strachey, C. (1952). Logical or non-mathematical programmes. In Proc. 1952 ACM National Meeting.

Stratonovich, R. L. (1959). Optimum nonlinear systems which bring about a separation of a signal with constant parameters from noise. Radiofizika, 2, 892–901.

Stratonovich, R. L. (1965). On value of information. Izvestiya of USSR Academy of Sciences, Technical Cybernetics, 5, 3–12.

Sturtevant, N. R. and Bulitko, V. (2016). Scrubbing during learning in real-time heuristic search. JAIR, 57, 307–343.

Subramanian, D. and Wang, E. (1994). Constraint-based kinematic synthesis. In Proc. International Conference on Qualitative Reasoning.

Suk, H.-I., Sin, B.-K., and Lee, S.-W. (2010). Hand gesture recognition based on dynamic Bayesian network framework. Pattern Recognition, 43, 3059–3072.

Sun, Y., Wang, S., Li, Y., Feng, S., Tian, H., Wu, H., and Wang, H. (2019). ERNIE 2.0: A continual pre-training framework for language understanding. arXiv:1907.12412.

Sussman, G. J. (1975). A Computer Model of Skill Acquisition. Elsevier.

Sutcliffe, G. (2016). The CADE ATP system competition - CASC. AIMag, 37, 99–101.

Sutcliffe, G. and Suttner, C. (1998). The TPTP Problem Library: CNF Release v1.2.1. JAR, 21, 177–203.

Sutcliffe, G., Schulz, S., Claessen, K., and Gelder, A. V. (2006). Using the TPTP language for writing derivations and finite interpretations. In Proc. International Joint Conference on Automated Reasoning.

Sutherland, I. (1963). Sketchpad: A man-machine graphical communication system. In Proc. Spring Joint Computer Conference.

Sutskever, I., Vinyals, O., and Le, Q. V. (2015). Sequence to sequence learning with neural networks. In NeurIPS 27.

Sutton, C. and McCallum, A. (2007). An introduction to conditional random fields for relational learning. In Getoor, L. and Taskar, B. (Eds.), Introduction to Statistical Relational Learning. MIT Press.

Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3, 9–44.

Sutton, R. S., McAllester, D. A., Singh, S., and Mansour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. In NeurIPS 12.

Sutton, R. S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In ICML-90.

Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd edition). MIT Press.

Swade, D. (2000). Difference Engine: Charles Babbage And The Quest To Build The First Computer. Diane Publishing Co.

Sweeney, L. (2000). Simple demographics often identify people uniquely. Health (San Francisco), 671, 1–34.

Sweeney, L. (2002a). Achieving k-anonymity privacy protection using generalization and suppression. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 10, 571–588.

Sweeney, L. (2002b). k-anonymity: A model for protecting privacy. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 10, 557–570.

Swerling, P. (1959). First order error propagation in a stagewise smoothing procedure for satellite observations. J. Astronautical Sciences, 6, 46–52.

Swift, T. and Warren, D. S. (1994). Analysis of SLG-WAM evaluation of definite programs. In Logic Programming: Proc. 1994 International Symposium.

Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2013). Intriguing properties of neural networks. arXiv:1312.6199.

Szeliski, R. (2011). Computer Vision: Algorithms and Applications. Springer-Verlag.

Szepesvari, C. (2010). Algorithms for reinforcement learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 4, 1–103.

Tadepalli, P., Givan, R., and Driessens, K. (2004). Relational reinforcement learning: An overview. In ICML-04.

Tait, P. G. (1880). Note on the theory of the “15 puzzle”. Proc. Royal Society of Edinburgh, 10, 664–665.

Tamaki, H. and Sato, T. (1986). OLD resolution with tabulation. In ICLP-86.

Tan, P., Steinbach, M., Karpatne, A., and Kumar, V. (2019). Introduction to Data Mining (2nd edition). Pearson.

Tang, E. (2018). A quantum-inspired classical algorithm for recommendation systems. arXiv:1807.04271.

Tarski, A. (1935). Die Wahrheitsbegriff in den formalisierten Sprachen. Studia Philosophica, 1, 261–405.

Tarski, A. (1941). Introduction to Logic and to the Methodology of Deductive Sciences. Dover.

Tarski, A. (1956). Logic, Semantics, Metamathematics: Papers from 1923 to 1938. Oxford University Press.

Tash, J. K. and Russell, S. J. (1994). Control strategies for a stochastic planner. In AAAI-94.

Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018). Deepmind control suite. arXiv:1801.00690.

Tate, A. (1975a). Interacting goals and their use. In IJCAI-75.

Tate, A. (1975b). Using Goal Structure to Direct Search in a Problem Solver. Ph.D. thesis, University of Edinburgh.

Tate, A. (1977). Generating project networks. In IJCAI-77.

Tate, A. and Whiter, A. M. (1984). Planning with multiple resource constraints and an application to a naval planning problem. In Proc. First Conference on AI Applications.

Tatman, J. A. and Shachter, R. D. (1990). Dynamic programming and influence diagrams. IEEE Transactions on Systems, Man and Cybernetics, 20, 365–379.

Tattersall, C. (1911). A Thousand End-Games: A Collection of Chess Positions That Can be Won or Drawn by the Best Play. British Chess Magazine.

Taylor, A. D. and Zwicker, W. S. (1999). Simple Games: Desirability Relations, Trading, Pseudoweightings. Princeton University Press.

Taylor, G., Stensrud, B., Eitelman, S., and Dunham, C. (2007). Towards automating airspace management. In Proc. Computational Intelligence for Security and Defense Applications (CISDA) Conference.

Taylor, P. (2009). Text-to-Speech Synthesis. Cambridge University Press.

Tedrake, R., Zhang, T. W., and Seung, H. S. (2004). Stochastic policy gradient reinforcement learning on a simple 3D biped. In IROS-04.

Tellex, S., Kollar, T., Dickerson, S., Walter, M. R., Banerjee, A., Teller, S., and Roy, N. (2011). Understanding natural language commands for robotic navigation and mobile manipulation. In AAAI-11.

Tenenbaum, J. B., Griffiths, T. L., and Niyogi, S. (2007). Intuitive theories as grammars for causal inference. In Gopnik, A. and Schulz, L. (Eds.), Causal Learning: Psychology, Philosophy, and Computation. Oxford University Press.

Tesauro, G. (1990). Neurogammon: A neural-network backgammon program. In IJCNN-90.

Tesauro, G. (1992). Practical issues in temporal difference learning. Machine Learning, 8, 257–277.

Tesauro, G. (1995). Temporal difference learning and TD-Gammon. CACM, 38, 58–68.

Tesauro, G. and Galperin, G. R. (1997). On-line policy improvement using Monte-Carlo search. In NeurIPS 9.

Tetlock, P. E. (2017). Expert Political Judgment: How Good Is It? How Can We Know? Princeton University Press.

Teyssier, M. and Koller, D. (2005). Ordering-based search: A simple and effective algorithm for learning Bayesian networks. In UAI-05.

Thaler, R. (1992). The Winner’s Curse: Paradoxes and Anomalies of Economic Life. Princeton University Press.

Thaler, R. and Sunstein, C. (2009). Nudge: Improving Decisions About Health, Wealth, and Happiness. Penguin.

Thayer, J. T., Dionne, A., and Ruml, W. (2011). Learning inadmissible heuristics during search. In ICAPS-11.

Theocharous, G., Murphy, K., and Kaelbling, L. P. (2004). Representing hierarchical POMDPs as DBNs for multi-scale robot localization. In ICRA-04.

Thiele, T. (1880). Om anvendelse af mindste kvadraters methode i nogle tilfælde, hvor en komplikation af visse slags uensartede tilfældige fejlkilder giver fejlene en ‘systematisk’ karakter. Vidensk. Selsk. Skr. 5. Rk., naturvid. og mat. Afd., 12, 381–408.

Thielscher, M. (1999). From situation calculus to fluent calculus: State update axioms as a solution to the inferential frame problem. AIJ, 111, 277–299.

Thomas, P. S., da Silva, B. C., Barto, A. G., and Brunskill, E. (2017). On ensuring that intelligent machines are well-behaved. arXiv:1708.05448.

Thomaz, A., Hoffman, G., Cakmak, M., et al. (2016). Computational human-robot interaction. Foundations and Trends in Robotics, 4, 105–223.

Thompson, K. (1986). Retrograde analysis of certain endgames. J. International Computer Chess Association, 9, 131–139.

Thompson, K. (1996). 6-piece endgames. J. International Computer Chess Association, 19, 215–226.

Thompson, W. R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25, 285–294.

Thorndike, E. (1911). Animal Intelligence. Macmillan.

Thornton, C., Hutter, F., Hoos, H. H., and Leyton-Brown, K. (2013). Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. In KDD-13.

Thrun, S., Burgard, W., and Fox, D. (2005). Probabilistic Robotics. MIT Press.

Thrun, S., Fox, D., and Burgard, W. (1998). A probabilistic approach to concurrent mapping and localization for mobile robots. Machine Learning, 31, 29–53.

Thrun, S. (2006). Stanley, the robot that won the DARPA Grand Challenge. J. Field Robotics, 23, 661–692.

Thrun, S. and Pratt, L. (2012). Learning to Learn. Springer.

Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273–286.

Tian, J., Paz, A., and Pearl, J. (1998). Finding a minimal d-separator. Tech. rep., UCLA Department of Computer Science.

Tikhonov, A. N. (1963). Solution of incorrectly formulated problems and the regularization method. Soviet Math. Dokl., 5, 1035–1038.

Tipping, M. E. and Bishop, C. M. (1999). Probabilistic principal component analysis. J. Royal Statistical Society, 61, 611–622.

Titterington, D. M., Smith, A. F. M., and Makov, U. E. (1985). Statistical Analysis of Finite Mixture Distributions. Wiley.

Toma, P. (1977). SYSTRAN as a multilingual machine translation system. In Proc. Third European Congress on Information Systems and Networks: Overcoming the Language Barrier.

Tomasi, C. and Kanade, T. (1992). Shape and motion from image streams under orthography: A factorization method. IJCV, 9, 137–154.

Topol, E. (2019). Deep Medicine: How Artificial Intelligence Can Make Healthcare Human Again. Basic Books.

Torralba, A., Fergus, R., and Weiss, Y. (2008). Small codes and large image databases for recognition. In CVPR.

Torralba, A., Linares López, C., and Borrajo, D. (2016). Abstraction heuristics for symbolic bidirectional search. In IJCAI-16.

Tramèr, F., Zhang, F., Juels, A., Reiter, M. K., and Ristenpart, T. (2016). Stealing machine learning models via prediction APIs. In USENIX Security Symposium.

Tran, D., Hoffman, M., Saurous, R. A., Brevdo, E., Murphy, K., and Blei, D. M. (2017). Deep probabilistic programming. In ICLR-17.

Trappenberg, T. (2010). Fundamentals of Computational Neuroscience (2nd edition). Oxford University Press.

Tsang, E. (1993). Foundations of Constraint Satisfaction. Academic Press.

Tshitoyan, V., Dagdelen, J., Weston, L., Dunn, A., Rong, Z., Kononova, O., Persson, K. A., Ceder, G., and Jain, A. (2019). Unsupervised word embeddings capture latent knowledge from materials science literature. Nature, 571, 95.

Tsitsiklis, J. N. and Van Roy, B. (1997). An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42, 674–690.

Tukey, J. W. (1977). Exploratory Data Analysis. Addison-Wesley.

Tumer, K. and Wolpert, D. (2000). Collective intelligence and Braess’ paradox. In AAAI-00.

Turian, J., Ratinov, L., and Bengio, Y. (2010). Word representations: a simple and general method for semi-supervised learning. In ACL-10.

Turing, A. (1936). On computable numbers, with an application to the Entscheidungsproblem. Proc. London Mathematical Society, 2nd series, 42, 230–265.

Turing, A. (1948). Intelligent machinery. Tech. rep., National Physical Laboratory. reprinted in (Ince, 1992).

Turing, A. (1950). Computing machinery and intelligence. Mind, 59, 433–460.

Turing, A., Strachey, C., Bates, M. A., and Bowden, B. V. (1953). Digital computers applied to games. In Bowden, B. V. (Ed.), Faster than Thought. Pitman.

Turing, A. (1947). Lecture to the London Mathematical Society on 20 February 1947.

Turing, A. (1996). Intelligent machinery, a heretical theory. Philosophia Mathematica, 4, 256–260. Originally written c. 1951.

Tversky, A. and Kahneman, D. (1982). Causal schemata in judgements under uncertainty. In Kahneman, D., Slovic, P., and Tversky, A. (Eds.), Judgement Under Uncertainty: Heuristics and Biases. Cambridge University Press.

Tygar, J. D. (2011). Adversarial machine learning. IEEE Internet Computing, 15, 4–6.

Ullman, J. D. (1985). Implementation of logical query languages for databases. ACM Transactions on Database Systems, 10, 289–321.

Ullman, S. (1979). The Interpretation of Visual Motion. MIT Press.

Urmson, C. and Whittaker, W. (2008). Self-driving cars and the Urban Challenge. IEEE Intelligent Systems, 23, 66–68.

Valiant, L. (1984). A theory of the learnable. CACM, 27, 1134–1142.

Vallati, M., Chrpa, L., and Kitchin, D. E. (2015). Portfolio-based planning: State of the art, common practice and open challenges. AI Commun., 28(4), 717–733.

van Beek, P. (2006). Backtracking search algorithms. In Rossi, F., van Beek, P., and Walsh, T. (Eds.), Handbook of Constraint Programming. Elsevier.

van Beek, P. and Chen, X. (1999). CPlan: A constraint programming approach to planning. In AAAI-99.

van Beek, P. and Manchak, D. (1996). The design and experimental analysis of algorithms for temporal reasoning. JAIR, 4, 1–18.

van Bentham, J. and ter Meulen, A. (1997). Handbook of Logic and Language. MIT Press.

van den Oord, A., Dieleman, S., and Schrauwen, B. (2014). Deep content-based music recommendation. In NeurIPS 26.

van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (2016a). WaveNet: A generative model for raw audio. arXiv:1609.03499.

van den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. (2016b). Pixel recurrent neural networks. arXiv:1601.06759.

van Harmelen, F., Lifschitz, V., and Porter, B. (2007). The Handbook of Knowledge Representation. Elsevier.

van Heijenoort, J. (Ed.). (1967). From Frege to Gödel: A Source Book in Mathematical Logic, 1879–1931. Harvard University Press.

Van Hentenryck, P., Saraswat, V., and Deville, Y. (1998). Design, implementation, and evaluation of the constraint language cc(FD). J. Logic Programming, 37, 139–164.

van Hoeve, W.-J. (2001). The alldifferent constraint: a survey. In 6th Annual Workshop of the ERCIM Working Group on Constraints.

van Hoeve, W.-J. and Katriel, I. (2006). Global constraints. In Rossi, F., van Beek, P., and Walsh, T. (Eds.), Handbook of Constraint Processing. Elsevier.

van Lambalgen, M. and Hamm, F. (2005). The Proper Treatment of Events. Wiley-Blackwell.

van Nunen, J. A. E. E. (1976). A set of successive approximation methods for discounted Markovian decision problems. Zeitschrift fur Operations Research, Serie A, 20, 203–208.

Van Roy, P. L. (1990). Can logic programming execute as fast as imperative programming? Report, Computer Science Division, UC Berkeley.

Vapnik, V. N. (1998). Statistical Learning Theory. Wiley.

Vapnik, V. N. and Chervonenkis, A. Y. (1971). On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and Its Applications, 16, 264–280.

Vardi, M. Y. (1996). An automata-theoretic approach to linear temporal logic. In Moller, F. and Birtwistle, G. (Eds.), Logics for Concurrency. Springer.

Varian, H. R. (1995). Economic mechanism design for computerized agents. In USENIX Workshop on Electronic Commerce.

Vasilache, N., Johnson, J., Mathieu, M., Chintala, S., Piantino, S., and LeCun, Y. (2014). Fast convolutional nets with fbfft: A GPU performance evaluation. arXiv:1412.7580.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2018). Attention is all you need. In NeurIPS 30.

Veach, E. and Guibas, L. J. (1995). Optimally combining sampling techniques for Monte Carlo rendering. In Proc. 22rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH).

Venkatesh, S. (2012). The Theory of Probability: Explorations and Applications. Cambridge University Press.

Vere, S. A. (1983). Planning in time: Windows and durations for activities and goals. PAMI, 5, 246–267.

Verma, S. and Rubin, J. (2018). Fairness definitions explained. In 2018 IEEE/ACM International Workshop on Software Fairness.

Verma, V., Gordon, G., Simmons, R., and Thrun, S. (2004). Particle filters for rover fault diagnosis. IEEE Robotics and Automation Magazine, June.

Vinge, V. (1993). The coming technological singularity: How to survive in the post-human era. In Proc. Vision-21: Interdisciplinary Science and Engineering in the Era of Cyberspace. NASA.

Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., Hassabis, D., Apps, C., and Silver, D. (2019).

Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575, 350–354.

Vinyals, O., Ewalds, T., Bartunov, S., and Georgiev, P. (2017a). StarCraft II: A new challenge for reinforcement learning. arXiv:1708.04782.

Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2017b). Show and tell: Lessons learned from the 2015 MSCOCO image captioning challenge. PAMI, 39, 652–663.

Viola, P. and Jones, M. (2004). Robust real-time face detection. IJCV, 57, 137–154.

Visser, U., Ribeiro, F., Ohashi, T., and Dellaert, F. (Eds.). (2008). RoboCup 2007: Robot Soccer World Cup XI. Springer.

Viterbi, A. J. (1967). Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory, 13, 260–269.

Vlassis, N. (2008). A Concise Introduction to Multiagent Systems and Distributed Artificial Intelligence. Morgan & Claypool.

von Mises, R. (1928). Wahrscheinlichkeit, Statistik und Wahrheit. J. Springer.

von Neumann, J. (1928). Zur Theorie der Gesellschaftsspiele. Mathematische Annalen, 100, 295–320.

von Neumann, J. and Morgenstern, O. (1944). Theory of Games and Economic Behavior (first edition). Princeton University Press.

von Winterfeldt, D. and Edwards, W. (1986). Decision Analysis and Behavioral Research. Cambridge University Press.

Vossen, T., Ball, M., Lotem, A., and Nau, D. S. (2001). Applying integer programming to AI planning. Knowledge Engineering Review, 16, 85–100.

Wainwright, M. and Jordan, M. I. (2008). Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1, 1–305.

Walker, G. (1931). On periodicity in series of related terms. Proc. Roy. Soc., A, 131, 518–532.

Walker, R. J. (1960). An enumerative technique for a class of combinatorial problems. In Proc. Sympos. Appl. Math., Vol. 10.

Wallace, A. R. (1858). On the tendency of varieties to depart indefinitely from the original type. Proc. Linnean Society of London, 3, 53–62.

Walpole, R. E., Myers, R. H., Myers, S. L., and Ye, K. E. (2016). Probability and Statistics for Engineers and Scientists (9th edition). Pearson.

Walsh, T. (2015). Turing’s red flag. arXiv:1510.09033.

Waltz, D. (1975). Understanding line drawings of scenes with shadows. In Winston, P. H. (Ed.), The Psychology of Computer Vision. McGraw-Hill.

Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. (2019). SuperGLUE: A stickier benchmark for general-purpose language understanding systems. arXiv:1905.00537.

Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. (2018a). GLUE: A multitask benchmark and analysis platform for natural language understanding. arXiv:1804.07461.

Wang, J., Zhu, T., Li, H., Hsueh, C.-H., and Wu, I.-C. (2018b). Belief-state Monte Carlo tree search for phantom Go. IEEE Transactions on Games, 10, 139–154.

Wanner, E. (1974). On Remembering, Forgetting and Understanding Sentences. Mouton.

Warren, D. H. D. (1974). WARPLAN: A System for Generating Plans. Department of Computational Logic Memo, University of Edinburgh.

Warren, D. H. D. (1983). An abstract Prolog instruction set. Technical note, SRI International.

Wasserman, L. (2004). All of Statistics. Springer.

Watkins, C. J. (1989). Models of Delayed Reinforcement Learning. Ph.D. thesis, Psychology Department, Cambridge University.

Watson, J. D. and Crick, F. (1953). A structure for deoxyribose nucleic acid. Nature, 171, 737.

Wattenberg, M., Viégas, F., and Johnson, I. (2016). How to use t-SNE effectively. Distill, 1.

Waugh, K., Schnizlein, D., Bowling, M., and Szafron, D. (2009). Abstraction pathologies in extensive games. In AAMAS-09.

Weibull, J. (1995). Evolutionary Game Theory. MIT Press.

Weidenbach, C. (2001). SPASS: Combining superposition, sorts and splitting. In Robinson, A. and Voronkov, A. (Eds.), Handbook of Automated Reasoning. MIT Press.

Weiss, G. (2000a). Multiagent Systems. MIT Press.

Weiss, Y. (2000b). Correctness of local probability propagation in graphical models with loops. Neural Computation, 12, 1–41.

Weiss, Y. and Freeman, W. (2001). Correctness of belief propagation in Gaussian graphical models of arbitrary topology. Neural Computation, 13, 2173–2200.

Weizenbaum, J. (1976). Computer Power and Human Reason. W. H. Freeman.

Weld, D. S. (1994). An introduction to least commitment planning. AIMag, 15, 27–61.

Weld, D. S. (1999). Recent advances in AI planning. AIMag, 20, 93–122.

Weld, D. S., Anderson, C. R., and Smith, D. E. (1998). Extending Graphplan to handle uncertainty and sensing actions. In AAAI-98.

Weld, D. S. and de Kleer, J. (1990). Readings in Qualitative Reasoning about Physical Systems. Morgan Kaufmann.

Weld, D. S. and Etzioni, O. (1994). The first law of robotics: A call to arms. In AAAI-94.

Wellman, M. P. (1985). Reasoning about preference models. Technical report, Laboratory for Computer Science, MIT.

Wellman, M. P. (1988). Formulation of Tradeoffs in Planning under Uncertainty. Ph.D. thesis, MIT.

Wellman, M. P. (1990a). Fundamental concepts of qualitative probabilistic networks. AIJ, 44, 257–303.

Wellman, M. P. (1990b). The STRIPS assumption for planning under uncertainty. In AAAI90.

Wellman, M. P., Breese, J. S., and Goldman, R. (1992). From knowledge bases to decision models. Knowledge Engineering Review, 7, 35–53.

Wellman, M. P. and Doyle, J. (1992). Modular utility representation for decision-theoretic planning. In ICAPS-92.

Wellman, M. P., Wurman, P., O’Malley, K., Bangera, R., Lin, S., Reeves, D., and Walsh, W. (2001). Designing the market game for a trading agent competition. IEEE Internet Computing, 5, 43–51.

Werbos, P. (1974). Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. Ph.D. thesis, Harvard University.

Werbos, P. (1990). Backpropagation through time: What it does and how to do it. Proc. IEEE, 78, 1550–1560.

Werbos, P. (1992). Approximate dynamic programming for real-time control and neural modeling. In White, D. A. and Sofge, D. A. (Eds.), Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches. Van Nostrand Reinhold.

Werbos, P. (1977). Advanced forecasting methods for global crisis warning and models of intelligence. General Systems Yearbook, 22, 25–38.

Wesley, M. A. and Lozano-Perez, T. (1979). An algorithm for planning collision-free paths among polyhedral objects. CACM, 22, 560–570.

West, D. M. (2018). The Future of Work: Robots, AI, and Automation. Brookings Institution Press.

West, S. M., Whittaker, M., and Crawford, K. (2019). Discriminating systems: Gender, race and power in AI. Tech. rep., AI Now Institute.

Wexler, Y. and Meek, C. (2009). MAS: A multiplicative approximation scheme for probabilistic inference. In NeurIPS 21.

Wheatstone, C. (1838). On some remarkable, and hitherto unresolved, phenomena of binocular vision. Phil. Trans. Roy. Soc., 2, 371–394.

White, C., Neiswanger, W., and Savani, Y. (2019). BANANAS: Bayesian optimization with neural architectures for neural architecture search. arXiv:1910.11858.

Whitehead, A. N. and Russell, B. (1910). Principia Mathematica. Cambridge University Press.

Whittle, P. (1979). Discussion of Dr Gittins’ paper. J. Royal Statistical Society, 41, 165.

Whorf, B. (1956). Language, Thought, and Reality. MIT Press.

Widrow, B. (1962). Generalization and information storage in networks of ADALINE “neurons”. In Yovits, M. C., Jacobi, G. T., and Goldstein, G. D. (Eds.), Self-Organizing Systems. Spartan.

Widrow, B. and Hoff, M. E. (1960). Adaptive switching circuits. In IRE WESCON Convention Record.

Wiedijk, F. (2003). Comparing mathematical provers. In Proc. 2nd Int. Conf. on Mathematical Knowledge Management.

Wiegley, J., Goldberg, K., Peshkin, M., and Brokowski, M. (1996). A complete algorithm for designing passive fences to orient parts. In ICRA-96.

Wiener, N. (1942). The extrapolation, interpolation, and smoothing of stationary time series. Tech. rep., Research Project DIC-6037, MIT.

Wiener, N. (1948). Cybernetics. Wiley.

Wiener, N. (1950). The Human Use of Human Beings. Houghton Mifflin.

Wiener, N. (1960). Some moral and technical consequences of automation. Science, 131, 1355–1358.

Wiener, N. (1964). God & Golem, Inc: A Comment on Certain Points Where Cybernetics Impinges on Religion. MIT Press.

Wilensky, R. (1978). Understanding Goal-Based Stories. Ph.D. thesis, Yale University.

Wilkins, D. E. (1988). Practical Planning: Extending the AI Planning Paradigm. Morgan Kaufmann.

Wilkins, D. E. (1990). Can AI planners solve practical problems? Computational Intelligence, 6, 232–246.

Wilks, Y. (2010). Close Engagements With Artificial Companions: Key Social, Psychological, Ethical and Design Issues. John Benjamins.

Wilks, Y. (2019). Artificial Intelligence: Modern Magic or Dangerous Future. Icon.

Williams, A., Nangia, N., and Bowman, S. (2018). A broad-coverage challenge corpus for sentence understanding through inference. In NAACL HLT.

Williams, B., Ingham, M., Chung, S., and Elliott, P. (2003). Model-based programming of intelligent embedded systems and robotic space explorers. Proc. IEEE, 91(212–237).

Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8, 229–256.

Williams, R. J. and Zipser, D. (1989). A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1, 270–280.

Williams, R. J. and Baird, L. C. I. (1993). Tight performance bounds on greedy policies based on imperfect value functions. Tech. rep., College of Computer Science, Northeastern University.

Wilson, D. H. (2011). Robopocalypse. Doubleday.

Wilson, R. A. and Keil, F. C. (Eds.). (1999). The MIT Encyclopedia of the Cognitive Sciences. MIT Press.

Wilson, R. (2004). Four Colors Suffice. Princeton University Press.

Wilt, C. M. and Ruml, W. (2014). Speedy versus greedy search. In Seventh Annual Symposium on Combinatorial Search.

Wilt, C. M. and Ruml, W. (2016). Effective heuristics for suboptimal best-first search. JAIR, 57, 273–306.

Wingate, D. and Seppi, K. D. (2005). Prioritization methods for accelerating MDP solvers. JMLR, 6, 851–881.

Wingate, D., Stuhlmüller, A., and Goodman, N. D. (2011). Lightweight implementations of probabilistic programming languages via transformational compilation. In AISTATS11.

Winograd, S. and Cowan, J. D. (1963). Reliable Computation in the Presence of Noise. MIT Press.

Winograd, T. (1972). Understanding natural language. Cognitive Psychology, 3, 1–191.

Winston, P. H. (1970). Learning structural descriptions from examples. Technical report, Department of Electrical Engineering and Computer Science, MIT.

Wintermute, S., Xu, J., and Laird, J. (2007). SORTS: A human-level approach to real-time strategy AI. In Proc. Third Artificial Intelligence and Interactive Digital Entertainment Conference.

Winternitz, L. (2017). Autonomous navigation above the GNSS constellations and beyond: GPS navigation for the magnetospheric multiscale mission and SEXTANT pulsar navigation demonstration. Tech. rep., NASA Goddard Space Flight Center.

Witten, I. H. (1977). An adaptive optimal controller for discrete-time Markov environments. Information and Control, 34, 286–295.

Witten, I. H. and Bell, T. C. (1991). The zero-frequency problem: Estimating the probabilities of novel events in adaptive text compression. IEEE Transactions on Information Theory, 37, 1085–1094.

Witten, I. H. and Frank, E. (2016). Data Mining: Practical Machine Learning Tools and Techniques (4th edition). Morgan Kaufmann.

Witten, I. H., Moffat, A., and Bell, T. C. (1999). Managing Gigabytes: Compressing and Indexing Documents and Images (2nd edition). Morgan Kaufmann.

Wittgenstein, L. (1922). Tractatus Logico-Philosophicus (2nd edition). Routledge and Kegan Paul. Reprinted 1971, edited by D. F. Pears and B. F. McGuinness.

Wittgenstein, L. (1953). Philosophical Investigations. Macmillan.

Wojciechowski, W. S. and Wojcik, A. S. (1983). Automated design of multiple-valued logic circuits by automated theorem proving techniques. IEEE Transactions on Computers, C32, 785–798.

Wolfe, J. and Russell, S. J. (2007). Exploiting belief state structure in graph search. In ICAPS Workshop on Planning in Games.

Wolpert, D. (2013). Ubiquity symposium: Evolutionary computation and the processes of life: what the no free lunch theorems really mean: how to improve search algorithms. Ubiquity, December, 1–15.

Wolpert, D. and Macready, W. G. (1997). No free lunch theorems for optimization. IEEE Trans. Evolutionary Computation, 1(1), 67–82.

Wong, C., Houlsby, N., Lu, Y., and Gesmundo, A. (2019). Transfer learning with neural AutoML. In NeurIPS 31.

Woods, W. A. (1973). Progress in natural language understanding: An application to lunar geology. In AFIPS Conference Proceedings.

Woods, W. A. (1975). What’s in a link? Foundations for semantic networks. In Bobrow, D. G. and Collins, A. M. (Eds.), Representation and Understanding: Studies in Cognitive Science. Academic Press.

Wooldridge, M. (2009). An Introduction to MultiAgent Systems (2nd edition). Wiley.

Wooldridge, M. and Rao, A. (Eds.). (1999). Foundations of Rational Agency. Kluwer.

Wos, L., Carson, D., and Robinson, G. (1964). The unit preference strategy in theorem proving. In Proc. Fall Joint Computer Conference.

Wos, L., Carson, D., and Robinson, G. (1965). Efficiency and completeness of the set-ofsupport strategy in theorem proving. JACM, 12, 536–541.

Wos, L., Overbeek, R., Lusk, E., and Boyle, J. (1992). Automated Reasoning: Introduction and Applications (2nd edition). McGraw-Hill.

Wos, L. and Robinson, G. (1968). Paramodulation and set of support. In Proc. IRIA Symposium on Automatic Demonstration.

Wos, L., Robinson, G., Carson, D., and Shalla, L. (1967). The concept of demodulation in theorem proving. JACM, 14, 698–704.

Wos, L. and Winker, S. (1983). Open questions solved with the assistance of AURA. In Bledsoe, W. W. and Loveland, D. (Eds.), Automated Theorem Proving: After 25 Years. American Mathematical Society.

Wos, L. and Pieper, G. (2003). Automated Reasoning and the Discovery of Missing and Elegant Proofs. Rinton Press.

Wray, R. E. and Jones, R. M. (2005). An introduction to Soar as an agent architecture. In Sun, R. (Ed.), Cognition and Multi-Agent Interaction: From Cognitive Modeling to Social Simulation. Cambridge University Press.

Wright, S. (1921). Correlation and causation. J. Agricultural Research, 20, 557–585.

Wright, S. (1931). Evolution in Mendelian populations. Genetics, 16, 97–159.

Wright, S. (1934). The method of path coefficients. Annals of Mathematical Statistics, 5, 161– 215.

Wu, F. and Weld, D. S. (2008). Automatically refining the Wikipedia infobox ontology. In 17th World Wide Web Conference (WWW2008).

Wu, Y., Li, L., and Russell, S. J. (2016a). SWIFT: Compiled inference for probabilistic programming languages. In IJCAI-16.

Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016b). Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144.

Wu, Y. and He, K. (2018). Group normalization. arXiv:1803.08494.

Xiong, W., Wu, L., Alleva, F., Droppo, J., Huang, X., and Stolcke, A. (2017). The Microsoft 2017 conversational speech recognition system. arXiv:1708.06073.

Yampolskiy, R. V. (2018). Artificial Intelligence Safety and Security. Chapman and Hall/CRC.

Yang, G., Lin, Y., and Bhattacharya, P. (2010). A driver fatigue recognition model based on information fusion and dynamic Bayesian network. Inf. Sci., 180, 1942–1954.

Yang, X.-S. (2009). Firefly algorithms for multimodal optimization. In International Symposium on Stochastic Algorithms.

Yang, X.-S. and Deb, S. (2014). Cuckoo search: Recent advances and applications. Neural Computing and Applications, 24, 169–174.

Yang, Z., Dai, Z., Yang, Y., Carbonell, J. G., Salakhutdinov, R., and Le, Q. V. (2019). XLNet: Generalized autoregressive pretraining for language understanding. arXiv:1906.08237.

Yarowsky, D. (1995). Unsupervised word sense disambiguation rivaling supervised methods. In ACL-95.

Ye, Y. (2011). The simplex and policy-iteration methods are strongly polynomial for the Markov decision problem with a fixed discount rate. Mathematics of Operations Research, 36, 593–784.

Yedidia, J., Freeman, W., and Weiss, Y. (2005). Constructing free-energy approximations and generalized belief propagation algorithms. IEEE Transactions on Information Theory, 51, 2282–2312.

Yeo, H.-S., Minami, R., Rodriguez, K., Shaker, G., and Quigley, A. (2018). Exploring tangible interactions with radar sensing. Proc. ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2, 1–25.

Ying, C., Kumar, S., Chen, D., Wang, T., and Cheng, Y. (2018). Image classification at supercomputer scale. arXiv:1811.06992.

Yip, K. M.-K. (1991). KAM: A System for Intelligently Guiding Numerical Experimentation by Computer. MIT Press.

Yngve, V. (1955). A model and an hypothesis for language structure. In Locke, W. N. and Booth, A. D. (Eds.), Machine Translation of Languages. MIT Press.

Yob, G. (1975). Hunt the wumpus! Creative Computing, Sep/Oct.

Yoshikawa, T. (1990). Foundations of Robotics: Analysis and Control. MIT Press.

You, Y., Pan, X., Wang, Z., and Lu, C. (2017). Virtual to real reinforcement learning for autonomous driving. arXiv:1704.03952.

Young, H. P. (2004). Strategic Learning and Its Limits. Oxford University Press.

Young, S., Gašić, M., Thompson, B., and Williams, J. (2013). POMDP-based statistical spoken dialog systems: A review. Proc. IEEE, 101, 1160–1179.

Younger, D. H. (1967). Recognition and parsing of context-free languages in time n3 . Information and Control, 10, 189–208.

Yu, D. and Deng, L. (2016). Automatic Speech Recognition. Springer-Verlag.

Yu, H.-F., Lo, H.-Y., Hsieh, H.-P., and Lou, J.-K. (2011). Feature engineering and classifier ensemble for KDD Cup 2010. In Proc. KDD Cup 2010 Workshop.

Yu, K., Sciuto, C., Jaggi, M., Musat, C., and Salzmann, M. (2019). Evaluating the search phase of neural architecture search. arXiv:1902.08142.

Yudkowsky, E. (2008). Artificial intelligence as a positive and negative factor in global risk. In Bostrom, N. and Cirkovic, M. (Eds.), Global Catastrophic Risk. Oxford University Press.

Yule, G. U. (1927). On a method of investigating periodicities in disturbed series, with special reference to Wolfer’s sunspot numbers. Phil. Trans. Roy. Soc., A, 226, 267–298.

Zadeh, L. A. (1965). Fuzzy sets. Information and Control, 8, 338–353.

Zadeh, L. A. (1978). Fuzzy sets as a basis for a theory of possibility. Fuzzy Sets and Systems, 1, 3–28.

Zaritskii, V. S., Svetnik, V. B., and Shimelevich, L. I. (1975). Monte-Carlo technique in problems of optimal information processing. Automation and Remote Control, 36, 2015– 22.

Zeckhauser, R. and Shepard, D. (1976). Where now for saving lives? Law and Contemporary Problems, 40, 5–45.

Zeeberg, A. (2017). D.I.Y. artificial intelligence comes to a Japanese family farm. New Yorker, August 10.

Zelle, J. and Mooney, R. (1996). Learning to parse database queries using inductive logic programming. In AAAI-96.

Zemel, R., Wu, Y., Swersky, K., Pitassi, T., and Dwork, C. (2013). Learning fair representations. In ICML-13.

Zemelman, B. V., Lee, G. A., Ng, M., and Miesenböck, G. (2002). Selective photostimulation of genetically chARGed neurons. Neuron, 33, 15–22.

Zermelo, E. (1913). Uber Eine Anwendung der Mengenlehre auf die Theorie des Schachspiels. In Proc. Fifth International Congress of Mathematicians.

Zermelo, E. (1976). An application of set theory to the theory of chess-playing. Firbush News, 6, 37–42. English translation of (Zermelo 1913).

Zettlemoyer, L. and Collins, M. (2005). Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In UAI-05.

Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2016). Understanding deep learning requires rethinking generalization. arXiv:1611.03530.

Zhang, H. and Stickel, M. E. (1996). An efficient algorithm for unit-propagation. In Proc. Fourth International Symposium on Artificial Intelligence and Mathematics.

Zhang, L., Pavlovic, V., Cantor, C. R., and Kasif, S. (2003). Human-mouse gene identification by comparative evidence integration and evolutionary analysis. Genome Research, 13, 1190–1202.

Zhang, N. L. and Poole, D. (1994). A simple approach to Bayesian network computations. In Proc. 10th Canadian Conference on Artificial Intelligence.

Zhang, S., Yao, L., and Sun, A. (2017). Deep learning based recommender system: A survey and new perspectives. arXiv:1707.07435.

Zhang, X., Zhao, J., and LeCun, Y. (2016). Character-level convolutional networks for text classification. In NeurIPS 28.

Zhang, Y., Pezeshki, M., Brakel, P., Zhang, S., Laurent, C., Bengio, Y., and Courville, A. (2017). Towards end-to-end speech recognition with deep convolutional neural networks. arXiv:1701.02720.

Zhao, K. and Huang, L. (2015). Type-driven incremental semantic parsing with polymorphism. In NAACL HLT.

Zhou, K., Doyle, J., and Glover, K. (1995). Robust and Optimal Control. Pearson.

Zhou, R. and Hansen, E. (2002). Memory-bounded A* graph search. In Proc. 15th International FLAIRS Conference.

Zhou, R. and Hansen, E. (2006). Breadth-first heuristic search. AIJ, 170, 385–408.

Zhu, B., Jiao, J., and Tse, D. (2019). Deconstructing generative adversarial networks. arXiv:1901.09465.

Zhu, D. J. and Latombe, J.-C. (1991). New heuristic algorithms for efficient hierarchical path planning. IEEE Transactions on Robotics and Automation, 7, 9–20.

Zhu, J.-Y., Park, T., Isola, P., and Efros, A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV-17.

Zhu, M., Zhang, Y., Chen, W., Zhang, M., and Zhu, J. (2013). Fast and accurate shift-reduce constituent parsing. In ACL-13.

Ziebart, B. D., Maas, A. L., Dey, A. K., and Bagnell, J. A. (2008). Navigate like a cabbie: Probabilistic reasoning from observed context-aware behavior. In Proc. 10th Int. Conf. on Ubiquitous Computing.

Ziebart, B. D., Ratliff, N., Gallagher, G., Mertz, C., Peterson, K., Bagnell, J. A., Hebert, M., Dey, A. K., and Srinivasa, S. (2009). Planning-based prediction for pedestrians. In IROS-09.

Zimmermann, H.-J. (Ed.). (1999). Practical Applications of Fuzzy Technologies. Kluwer.

Zimmermann, H.-J. (2001). Fuzzy Set Theory—And Its Applications (4th edition). Kluwer.

Zinkevich, M., Johanson, M., Bowling, M., and Piccione, C. (2008). Regret minimization in games with incomplete information. In NeurIPS 20.

Zipf, G. (1935). The Psychobiology of Language. Houghton Mifflin.

Zipf, G. (1949). Human Behavior and the Principle of Least Effort. Addison-Wesley.

Zobrist, A. L. (1970). Feature Extraction and Representation for Pattern Recognition and the Game of Go. Ph.D. thesis, University of Wisconsin.

Zollmann, A., Venugopal, A., Och, F. J., and Ponte, J. (2008). A systematic comparison of phrase-based, hierarchical and syntax-augmented statistical MT. In COLING-08.

Zoph, B. and Le, Q. V. (2016). Neural architecture search with reinforcement learning. arXiv:1611.01578.

Zuse, K. (1945). The Plankalkül. Report, Gesellschaft für Mathematik und Datenverarbeitung.

Zweig, G. and Russell, S. J. (1998). Speech recognition with dynamic Bayesian networks. In AAAI-98.

Index Page numbers in bold refer to definitions of terms and algorithms.

Page numbers in italics refer to items in the bibliography.

Symbols α (alpha) learning rate, 765 α (alpha) normalization constant, 400 ∧ (and), 217

χ2 (chi squared), 664 ⊢ (derives), 216 ⊨ (entailment), 214

ϵ (epsilon)-ball, 673 ∃ (there exists), 262 ∀ (for all), 261

γ (gamma) discount factor, 642 ⊔ (gap) in sentence, 846 | (given), 389 ⇔ (if and only if), 217 ⇒ (implies), 217 ∼ (indifferent), 530

λ (lambda)-expression, 259 ∇ (nabla) gradient, 120 ¬ (not), 217 ∨ (or), 217 ≻ (preferred), 530

σ (sigma) standard deviation, 1028

⊤ (matrix transpose), 1026

A A(s) (actions in a state), 562 A* search, 85–90 Aaronson, S., 984, 1034 Aarts, E., 206, 1034 Aarup, M., 384, 1034 Abbas, A., 559, 560, 1034 Abbeel, P., 62, 498, 561, 648, 719, 790, 817, 821, 822, 978, 979, 1004, 1034, 1040, 1044, 1046, 1048, 1051, 1052, 1061 Abbott, L. F., 788, 822, 1041 ABC computer, 14 Abdennadher, S., 207, 1044 abd James Andrew Bagnell, B. D. Z., 817, 1050 Abdolmaleki, A., 822, 1063 Abelson, R. P., 23, 1060 Abney, S., 838, 1034 Aboian, M. S., 30, 1042 Abramson, B., 176, 597, 1034 Abreu, D., 647, 1034 absolute independence, 398, 401 absorbing state, 799 abstraction, 66, 172, 622

abstraction hierarchy, 382 ABSTRIPS (planning system), 382 Abu-Hanna, A., 410, 1053 AC-3, 187 accessibility relation, 327 accountability, 711 accusative case, 841 Acharya, A., 109, 1039 Achlioptas, D., 249, 1034 Ackerman, E., 29, 1034 Ackerman, N., 526, 1034 Ackley, D. H., 143, 1034 acoustic model in disambiguation, 849 ACT (cognitive architecture), 292 acting rationally, 3 action, 36, 65, 105 egocentric, 67 high-level, 357 irreversible, 136, 799 joint, 603 monitoring, 372, 373 primitive, 357 rational, 7, 34

action-utility function, 545, 568 action-utility learning, 790 action cost function, 65, 105 Action Description Language (ADL), 380 action exclusion axiom, 245, 604 action monitoring, 372, 373 action schema, 345 action sequence, 53, 66 activation function, 752 active sensing, 881 actor, 601 actor model, 646 actuator, 36, 43, 929 electric, 929 AD-tree, 747 ADABOOST, 702 adaline, 21 Adams, A., 1016, 1053 Adams, J., 325 Adams, M., 313, 1046 Adams, R. P., 672, 1062 adaptive control theory, 819 adaptive dynamic programming (ADP), 793, 818 adaptive perception, 938

ADASYN (data generation system), 707, 995 add-one smoothing, 827 additive decomposition (of value functions), 810 add list, 345 Adida, B., 339, 1034 adjustment formula, 453 ADL (Action Description Language), 380 admissibility, 86 admissible heuristic, 86, 353 Adolph, K. E., 801, 1034 Adorf, H.-M., 384, 1049 ADP (adaptive dynamic programming), 793, 818 adversarial example, 770, 787 adversarial search, 146 adversarial training, 968 adversary argument, 136 Advice Taker, 19 AFSM (augmented finite state machine), 969 Agarwal, P., 967, 979, 990, 1054, 1061 agent, 3, 36, 60 active learning, 797 architecture of, 47, 1018 autonomous, 210 benevolent, 599

components, 1012–1018 decision-theoretic, 388, 528 function, 36, 37, 47, 564 goal-based, 53–54, 60, 61 greedy, 797 hybrid, 241 impatient, 642 intelligent, 34 knowledge-based, 13, 208–210 learning, 56–58, 62 logical, 237–246, 279 model-based, 52, 51–53 online planning, 379 personal, 1015 problem-solving, 63, 63–66 program, 37, 47, 48, 60 rational, 4, 3–4, 36, 39, 39–40, 55, 60, 61, 557 reflex, 49, 49–51, 60, 564, 790 situated, 982 software agent, 43 taxi-driving, 57, 1019 utility-based, 54–56, 60 vacuum, 39 wumpus, 212, 270, 881

Agerbeck, C., 205, 1034 Aggarwal, A., 787, 1059 Aggarwal, G., 637, 1034 aggregate querying, 991 aggregation, 376 Agha, G., 646, 1034 AGI (artificial general intelligence), 32 Agichtein, E., 855, 1034 Agmon, S., 785, 1034 Agostinelli, F., 106, 1034 Agrawal, P., 979, 1034 Agrawal, R., 1009, 1035 Agre, P. E., 383, 1034 Ai, D., 30, 1053 AI2 ARC (science test questions), 850 AI4People, 1008 AI FAIRNESS, 360, 996 AI for Humanitarian Action, 986 AI for Social Good, 986 AI Habitat (simulated environment), 822 AI Index, 27 Aila, T., 780, 1050 AI Now Institute, 995, 1008 Airborne Collision Avoidance System X (ACAS X), 598

aircraft carrier scheduling, 383 airport, driving to, 385 airport siting, 540, 545 AI safety, 1010 AI Safety Gridworlds, 822 AISB (Society for Artificial Intelligence and Simulation of Behaviour), 35 Aitken, S., 748, 1041 AI winter, 24, 27 Aizerman, M., 717, 1034 Akametalu, A. K., 821, 1034 Akgun, B., 979, 1034 al-Khwarizmi, M., 9 Alami, R., 979, 1062 Albantakis, L., 1007, 1057 Alberti, C., 853, 1034 Alberti, L., 920 Aldous, D., 142, 1034 ALE (Arcade Learning Environment), 822 Alemi, A. A., 312, 1034 Alexandria, 15 AlexNet (neural network system), 782 Algol-58, 852 algorithm, 9 algorithmic complexity, 716

Algorithmic Justice League, 995 Alhazen, 920 Alibaba, 1017 Allais, M., 538, 560, 1034 Allais paradox, 538, 560 Allan, J., 850, 1034 Alldiff constraint, 184 Allen, B., 383, 1044 Allen, C., 560, 1040 Allen, J. F., 324, 340, 383, 384, 1034 Allen, P., 1010, 1034 Allen-Zhu, Z., 786, 1034 Alleva, F., 29, 1066 alliance (in multiplayer games), 151 Almulla, M., 108, 1057 Alperin Resnick, L., 332, 1037 alpha–beta pruning, 152 alpha–beta search, 152–155, 174, 175 ALPHA-BETA-SEARCH, 154 ALPHAGO (Go program), ix, 19, 27, 30, 176, 177, 816 ALPHASTAR (game-playing program), 172, 179 ALPHAZERO (game-playing program), 30, 172, 174, 177 Alshawi, H., 840, 1062 Alterman, R., 382, 1034

alternating offers bargaining model, 641 Altman, A., 179, 1034 altruism, 387 Alvey report, 23 ALVINN (autonomous vehicle), 967 Amarel, S., 144, 338, 1034 Amazon, 29, 1017 ambient light, 886 ambiguity, 252, 847 lexical, 847 semantic, 847 syntactic, 847, 853 ambiguity aversion, 539, 560 Amir, E., 249, 527, 1034, 1041 Amit, Y., 718, 1034 Amodei, D., 15, 879, 1010, 1018, 1034, 1059 ANALOGY, 20 analysis of algorithms, 1023 Analytical Engine, 15 Anantharaman, T. S., 176, 1048 Anbulagan, 248, 1052 anchor box, 900 anchoring effect, 539 And-Elimination, 223

AND-OR-SEARCH, 125 AND–OR

graph, 230

AND–OR

tree, 123

Andersen, S. K., 455, 456, 1034 Anderson, C. R., 383, 1065 Anderson, J. A., 785, 1047 Anderson, J. R., 14, 292, 458, 1034, 1058 Anderson, K., 108, 822, 1034, 1036 Andersson, M., 648, 1060 AND

node, 123

Andoni, A., 717, 1034 Andor, D., 853, 1034 Andre, D., 143, 596, 820, 822, 1034, 1041, 1051 Andreae, P., 822, 1034 Andrew, G., 1009, 1055 Andrieu, C., 499, 1034 Andrychowicz, M., 959, 979, 1034 Aneja, J., 909, 1034 Angeli, G., 880, 1037 ANGELIC-SEARCH, 364 angelic semantics, 379 animatronics, 973 Anisenia, A., 456, 1061 answer set programming, 312

antecedent, 217 anthropomorphic robot, 926, 930 Antonoglou, I., 27, 30, 155, 174, 177, 178, 784, 790, 820, 822, 1055, 1061 anytime algorithm, 1019 Aoki, M., 597, 1034 aperture, 883 apparent motion, 893 Appel, K., 204, 1034 Appelt, D., 854, 1034 Apple, 1008 applicable, 65, 345 apprenticeship learning, 813, 1003, 1016 approval voting, 640 approximate near-neighbors, 689 Apps, C., 30, 179, 1064 Apt, K. R., 205, 207, 1034 Apté, C., 852, 1034 Arbuthnot, J., 408, 1034 Arcade Learning Environment (ALE), 822 arc consistency, 186 Archibald, C., 179, 1034 architecture agent, 47, 1018 AI, 1018

cognitive, 34, 292 computer, 652 for speech recognition, 25 network, 768, 770, 787, 922 reflective, 1019 RNN, 861 rule-based, 292 subsumption, 969 transformer, 868, 880 Arentoft, M. M., 384, 1034 Arfaee, S. J., 109, 1034, 1052 Argall, B. D., 979, 1034 argmax, 1026 argument from disability, 982–983 from informality, 981–982 Ariely, D., 538, 560, 1034 ARISTO (question-answering system), 875, 876, 880 Aristotle, ix, 3, 6, 7, 11, 60, 61, 247, 278, 338, 339, 341, 715, 920, 976 arity, 257, 288 Arkin, R., 980, 989, 1034 Armando, A., 250, 1034 Armstrong, S., 1002, 1034 Arnauld, A., 7, 10, 557, 1034

Arnoud, S., 769, 1046 Arora, J. S., 119, 1054 Arora, N. S., 512, 526, 1034 Arora, S., 107, 1034 Arous, G. B., 786, 1039 Arpit, D., 716, 1035 Arrow’s theorem, 640, 649 Arrow, K. J., 640, 649, 1035 artificial flight, 2 artificial general intelligence (AGI), 32 artificial intelligence, 1–1022 applications of, 27–30 conferences, 35 ethics, 986–1001 foundations, 5–17, 819 future of, 31–34, 1012–1022 goals of, 1020–1021 history of, 17–27 journals, 35 philosophy of, 981–1011 possibility of, 981–984 programming language, 19 provably beneficial, 5 real-time, 1019

risks, 31–34, 987–996 safety, 1001–1005 societies, 35 strong, 981, 1005, 1006 weak, 981, 1005, 1006 artificial intelligence (AI), 1 artificial life, 143 artificial superintelligence (ASI), 33 Arulampalam, M. S., 499, 1035 Arulkumaran, K., 820, 1035 Arunachalam, R., 649, 1035 arXiv.org, 27, 788, 1018 Asada, M., 976, 1050 asbestos removal, 533 Ashburner, M., 316, 1062 Ashby, W. R., 16, 1035 Asimov, I., 975, 1007, 1035 ASKMSR (question-answering system), 850 Aspuru-Guzik, A., 920, 1060 assertion (logical), 265 assertion (probabilistic), 388 assignment (in a CSP), 181 assistance, 561 assistance game, see game, assistance

assumption, 337 Astrom, K. J., 144, 597, 1035 asymmetry, 860 asymptotic analysis, 1024, 1023–1024 Atanasoff, J., 14 Atari video game, 816 Athalye, A., 787, 1039 Atkeson, C. G., 596, 820, 979, 1035, 1056 Atkin, L. R., 107, 1062 Atlas (robot), ix atom, 260 atomic representation, 59, 63 atomic sentence, 217, 260, 260, 264 attention (neural net), 865, 866, 880 attentional sequence-to-sequence model, 866 attribute, 59 AUC (area under ROC curve), 710 auction, 634 ascending-bid, 634 English, 634 first-price, 636 sealed-bid, 636 second-price, 636 truth-revealing, 635

Vickrey, 636 Audi, R., 1007, 1035 Auer, P., 597, 1035 Auer, S., 316, 339, 1037, 1052 augmentation, 851 augmented finite state machine (AFSM), 969 augmented grammar, 841 Aumann, R., 647, 1035 AURA (theorem prover), 309, 313 Auray, J. P., 559, 1036 Austerweil, J. L., 821, 1048 Australia, 181, 182, 193 author attribution, 826 AUTOCLASS (unsupervised learning algorithm), 748 autoencoder, 778 variational, 778 automata, 1006 automated machine learning (AutoML), 719 automated reasoning, 2 automated taxi, 42, 43, 57, 210, 385, 1019 automatic differentiation, 756 reverse mode, 756 Automatic Statistician, 719 AutoML, 719

automobile insurance, 539 Auton, L. D., 249, 1040 autonomatronics, 973 autonomic computing, 61 autonomous underwater vehicle (AUV), 927 autonomy, 42, 924, 971 autoregressive model, 779, 787 deep, 779 AUV (autonomous underwater vehicle), 927 average pooling, 863 average reward, 567 Awwal, I., 978, 1061 Axelrod, R., 647, 1035 axiom, 209, 267 action exclusion, 245, 604 decomposability, 531 domain-specific, 316 effect axiom, 239 frame axiom, 239 Kolmogorov’s, 393 of number theory, 268 of probability, 394 Peano, 268, 278, 289 precondition, 245

of probability, 393, 1027 of set theory, 269 successor-state, 240, 250 of utility theory, 531 wumpus world, 270 axon, 12