Machine Learning, Animated (Chapman & Hall/CRC Machine Learning & Pattern Recognition) 9781032462141, 9781032462134, 9781003441281, 9781003380580

Table of contents :
CHAPTER 1 ◾ Installing Anaconda and Jupyter Notebook

1.1 Why Python for Machine Learning?

1.1.1 The Rise of Python

1.1.2 Python for Machine Learning

1.2 Installing Anaconda

1.2.1 Installing Anaconda in Windows

1.2.2 Installing Anaconda in macOS

1.2.3 Installing Anaconda in Linux

1.2.4 Difference between Conda-install and Pip-install

1.3 Virtual Environment for this Book

1.3.1 Create the Virtual Environment MLA

1.3.2 Activate the Virtual Environment

1.3.3 De-activate the Virtual Environment

1.4 Set Up Jupyter Notebook in the Virtual Environment

1.4.1 Write Python in Jupyter Notebook

1.4.2 Issue Commands in Jupyter Notebook

1.5 File System for the Book

1.6 Glossary

1.7 Exercises

CHAPTER 2 ◾ Creating Animations

2.1 Create Plots with Matplotlib

2.1.1 A Single Line Plot

2.1.2 Multiple Lines in the Same Plot

2.2 Create Subplots

2.2.1 Create Individual Plots

2.2.2 Create Subplots

2.3 Create Animated Plots

2.3.1 Generate Annual Line Plots

2.3.2 Animate the Plots

2.4 Create Animated Bar Charts

2.4.1 Create a Horizontal Bar Chart

2.4.2 Generate Annual Bar Charts

2.4.3 Animate the Bar Charts

2.5 Put Bar Charts and Plots Side by Side

2.5.1 Combine a Bar Chart and a Plot

2.5.2 Create an Animation of the Combined Pictures

2.6 Animated Pie Charts

2.6.1 Create a Pie Chart

2.6.2 Generate Annual Pie Charts

2.6.3 Animate the Combined Pie Charts and Plots

2.7 Appendix: Download and Clean Up the GDP Data

2.8 Glossary

2.9 Exercises

SECTION II Machine Learning Basics

CHAPTER 3 ◾ Machine Learning: An Overview

3.1 ML: A New Paradigm for AI

3.1.1 What is AI?

3.1.2 Rule-Based AI

3.1.3 What is ML? Why ML Matters?

3.1.4 Why is ML so Popular?

3.2 Different Types of ML

3.2.1 Supervised Learning

3.2.2 Unsupervised Learning

3.2.3 Reinforcement Learning

3.3 Deep Reinforcement Learning

3.3.1 Deep Learning

3.3.2 Combine Deep Learning and Reinforcement Learning

3.4 Apply ML in the Real World

3.4.1 The Delivery Route Problem

3.4.2 Try the Problem before You Know the Answer

3.5 Glossary

3.6 Exercises

CHAPTER 4 ◾ Gradient Descent – Where Magic Happens

4.1 Optimization through Grid Search

4.1.1 How Grid Search Achieves Optimization

4.1.2 Curse of Dimensionality and Directional Grid Search

4.2 Gradient Descent

4.3 Use Tensorflow to Calculate Gradients

4.3.1 Install TensorFlow

4.3.2 Calculate Gradients Using TensorFlow

4.3.3 Gradient Descent Optimization with TensorFlow

4.3.4 Animate the Optimization Process

4.4 Choose the Right Learning Rate

4.4.1 When the Learning Rate is Too Large

4.4.2 When the Learning Rate is Too Small

4.5 Compare Learning Rates

4.5.1 Combine Animations

4.5.2 Subplots of Different Stages

4.6 Glossary

4.7 Exercises

CHAPTER 5 ◾ Introduction to Neural Networks

5.1 Anatomy of A Neural Network

5.1.1 Elements of a Neural Network

5.1.2 How Does a Neural Network Learn?

5.1.3 Make Predictions

5.2 Animate the Learning Process

5.2.1 Generate Graphs

5.2.2 Create Animation Based on Graphs

5.2.3 Subplots of Different Stages

5.3 Create A Neural Network with Keras

5.3.1 Construct the Model

5.3.2 Compile and Train the Model

5.3.3 Make Predictions

5.4 Customize Training with GradientTape

5.4.1 Construct the Model

5.4.2 Train the Model

5.4.3 Make Predictions

5.5 Glossary

5.6 Exercises

CHAPTER 6 ◾ Activation Functions

6.1 Why Do We Need Activation Functions?

6.1.1 Construct a Neural Network

6.1.2 Learn a Nonlinear Relation without Activation

6.2 The ReLU Activation Function

6.2.1 What is ReLU?

6.2.2 Animate the ReLU Function

6.2.3 Use ReLU to Model Nonlinearity

6.3 The Sigmoid Activation Function

6.3.1 Plot the Sigmoid Function

6.3.2 Animate the Sigmoid Function

6.3.3 Combine Animations

6.3.4 A Picture with Subplots of Different Stages

6.4 The Softmax Activation Function

6.4.1 What is the Softmax Function?

6.4.2 A Diagram of the Softmax Function

6.5 Glossary

6.6 Exercises

SECTION III Binary and Multi-Category Classifications

CHAPTER 7 ◾ Binary Classifications

7.1 What Is A Binary Classification Problem

7.1.1 Sigmoid Activation in Binary Classifications

7.1.2 The Binary Cross-Entropy Loss Function

7.2 Process Image Data

7.2.1 Download Data

7.2.2 Convert NumPy Arrays to Pictures and Back

7.2.3 Match Pictures with Labels

7.3 Binary Classification with A Logit Regression

7.3.1 Prepare the Data

7.3.2 Train the Logit Model

7.3.3 Predict Using the Logit Model

7.4 Binary Classification with A Simple Neural Network

7.4.1 Train and Test Using a Neural Network

7.4.2 Focus on Two Examples

7.4.3 Diagrams of the Network and Predictions

7.4.4 Animate the Training Process

7.4.5 Animate the Predictions for the Deer

7.5 Combine the Animations

7.5.1 Animate the Two Predictions

7.5.2 Subplots

7.6 Binary Classification with A Deep Neural Network

7.7 Appendix: Load CIFAR10 from Tensorflow Directly

7.8 Glossary

7.9 Exercises

CHAPTER 8 ◾ Convolutional Neural Networks

8.1 What Are Convolutional Neural Networks (CNNs)?

8.1.1 Our Running Example

8.1.2 A Horizontal Filter

8.2 Convolution Operations

8.2.1 Calculations in a Convolution Operation

8.2.2 Animate the Convolution Operations

8.2.3 Subplots

8.3 Stride and Padding

8.3.1 A Filter without Padding and a Stride of 2

8.3.2 Animate the Diagonal Filter

8.3.3 Animate the Diagonal Filter Convolution Operation

8.3.4 Subplots for Strides

8.4 Combine the Two Animations

8.4.1 Combine the Animations

8.5 Max Pooling

8.6 Binary Classifications with Convolutional Layers

8.7 Glossary

8.8 Exercises

CHAPTER 9 ◾ Multi-Category Image Classifications

9.1 Image Augmentations

9.1.1 The Keras Image Generator

9.1.2 Visualize Image Augmentations

9.2 What Is Multi-Category Classification?

9.2.1 One-Hot Encoder for Labels

9.2.2 The Activation and Loss Functions

9.3 Train the Multi-Category Classification Model

9.3.1 Load the Full Data Set

9.3.2 Convert Labels to One-Hot Variables

9.3.3 Train the Model

9.3.4 Evaluate the Model

9.4 Animate the Learning Process

9.4.1 Select Example Pictures

9.4.2 Animate Prediction Changes

9.4.3 Subplots of the Predictions on the Truck Image

9.4.4 Animate Predictions on the Frog Image

9.4.5 Subplots of the Predictions on the Frog Image

9.4.6 Combine the Animations

9.5 Glossary

9.6 Exercises

SECTION IV Developing Deep Learning Game Strategies

CHAPTER 10 ◾ Deep Learning Game Strategies

10.1 Get Started with the OpenAI Gym Environment

10.1.1 Basic Elements of a Game Environment

10.1.2 The Frozen Lake Game

10.1.3 Play the Frozen Lake Game Manually

10.2 Deep Learning Game Strategies: Generating Data

10.2.1 Summary of the Game Strategy

10.2.2 Simulate One Game

10.2.3 Simulate Many Games

10.3 Train the Deep Neural Network

10.3.1 Preprocess the Data

10.3.2 Train Deep Learning Game Strategies

10.4 Play Games with the Trained Model

10.4.1 Test One Game

10.4.2 Test the Efficacy of the Game Strategy

10.5 Animate the Decision-Making Process

10.5.1 Generate Figures

10.5.2 Create the Animation

10.5.3 Create a Figure with Subplots

10.6 Glossary

10.7 Exercises

CHAPTER 11 ◾ Deep Learning in the Cart Pole Game

11.1 Play the Cart Pole Game in OpenAI Gym

11.1.1 Features of the Cart Pole Game

11.1.2 Play a Full Game

11.2 Generate Data to Train the Model

11.2.1 How to Define Winning and Losing?

11.2.2 Prepare Data for the Neural Network

11.3 Train the Deep Neural Network

11.3.1 Preprocess the Data

11.3.2 Train the Deep Neural Network with Data

11.4 Play the Game with the Trained Model

11.4.1 A best_move() Function

11.4.2 Play One Cart Pole Game with the Trained Model

11.5 Compare Two Games

11.5.1 Record a Game with Random Moves

11.5.2 Combine Frames

11.5.3 Subplots of the Cart Pole Game Stages

11.6 Glossary

11.7 Exercises

CHAPTER 12 ◾ Deep Learning in Multi-Player Games

12.1 Create the Tic Tac Toe Game Environment

12.1.1 Use a Python Class to Represent the Environment

12.1.2 Create a Local Module for the Tic Tac Toe Game

12.1.3 Verify the Custom-Made Game Environment

12.1.4 Play a Game in the Tic Tac Toe Environment

12.2 Train A Deep Learning Game Strategy

12.2.1 A Blueprint of the Deep Learning Game Strategy

12.2.2 Simulate Tic Tac Toe Games

12.2.3 Train Your Tic Tac Toe Game Strategy

12.3 Use the Trained Model to Play Games

12.3.1 Best Moves Based on the Trained Model

12.3.2 Test a Game Using the Trained Model

12.3.3 Test the Efficacy of the Trained Model

12.4 Animate the Deep Learning Process

12.4.1 Probabilities of Winning for Each Hypothetical Move

12.4.2 Animate the Whole Game

12.4.3 Animate the Decision Making

12.4.4 Animate Board Positions and the Decision Making

12.4.5 Subplots of the Decision-Making Process

12.5 Glossary

12.6 Exercises

CHAPTER 13 ◾ Deep Learning in Connect Four

13.1 Create A Connect Four Game Environment

13.1.1 A Connect Four Game Environment

13.1.2 Verify the Connect Four Game Environment

13.1.3 Play a Connect Four Game

13.2 Train A Deep Neural Network

13.2.1 The Game Plan

13.2.2 Simulate Connect Four Games

13.2.3 Train the Connect Four Game Strategy

13.3 Use the Trained Model to Play Connect Four

13.3.1 Best Moves

13.3.2 Test Connect Four Deep Learning Game Strategies

13.4 Animate Deep Learning in Connect Four

13.4.1 Print Out Probabilities of Winning for Each Next Move

13.4.2 Animate a Complete Connect Four Game

13.4.3 Animate the Decision-Making Process

13.4.4 Combine Board Positions and Decision Making

13.4.5 Create Subplots of Deep Learning

13.5 Glossary

13.6 Exercises

SECTION V Reinforcement Learning

CHAPTER 14 ◾ Introduction to Reinforcement Learning

14.1 Basics of Reinforcement Learning

14.1.1 Basic Concepts

14.1.2 The Bellman Equation and Q-Learning

14.2 Use Q-Values to Play the Frozen Lake Game

14.2.1 The Logic Behind Q-Learning

14.2.2 A Q-Table to Win the Frozen Lake Game

14.3 Train the Q-Values

14.3.1 What is Q-Learning?

14.3.2 Let the Learning Begin

14.4 Q-Learning in A Self-Made Game Environment

14.4.1 A Self-Made Frozen Lake Game Environment

14.4.2 Use the Q-Table in the Self-Made Game Environment

14.5 Animate the Q-Learning Process

14.5.1 Highlight Values and Actions in the Q-Table

14.5.2 Animate the Use of the Q-Table

14.5.3 Game Board Positions and Best Actions

14.5.4 Subplots of the Q-Learning Process

14.6 Glossary

14.7 Exercises

CHAPTER 15 ◾ Q-Learning with Continuous States

15.1 The Mountain Car Game Environment

15.1.1 The Mountain Car Game

15.1.2 Convert a Continuous State into Discrete Values

15.1.3 The Reward Structure of the Game

15.2 Q-Learning in the Mountain Car Game

15.2.1 How to Train the Q-Table

15.2.2 Update the Q-Table

15.2.3 Train the Q-Table via Trial and Error

15.3 Test the Trained Q-Table

15.3.1 Define the Test_Q() Function

15.3.2 The Effectiveness of the Trained Q-Table

15.4 Animate the Game Before and After Q-Learning

15.4.1 The Mountain Car Game without Q-Learning

15.4.2 The Mountain Car Game with Q-Learning

15.4.3 The Mountain Car Game with and without Q-Learning

15.5 Glossary

15.6 Exercises

CHAPTER 16 ◾ Solving Real-World Problems with Machine Learning

16.1 Create A Delivery Route Game Environment

16.1.1 Draw Delivery Routes

16.1.2 Create a Game Environment

16.1.3 Use the Delivery Route Game Environment

16.2 Train A Q-Table between Any Two Positions

16.2.1 Create and Train A Q-table

16.2.2 Test the Trained Tabular Q-Values

16.3 Train the Q-Table for All Possible Routes

16.3.1 Train the Large Q-Table

16.3.2 Test the Large Q-Table

16.4 The Shortest Delivery Route to Eight Households

16.4.1 Find All Possible Permutations in Python

16.4.2 The Total Distance to Deliver to Eight Households

16.4.3 The Shortest Route

16.5 Animate the Delivery Route

16.5.1 Create a Graph at Each Stop

16.5.2 Animate the Shortest Route

16.5.3 Subplots of the Eight Deliveries

16.6 Glossary

16.7 Exercises

SECTION VI Deep Reinforcement Learning

CHAPTER 17 ◾ Deep Q-Learning

17.1 Deep Q-Learning for the Cart Pole Game

17.1.1 Create a Deep Q-Network

17.1.2 Train the Deep Q-Network

17.2 Test the Trained Deep Q-Network

17.2.1 Test and Record One Game

17.2.2 Test the Efficacy of the Deep Q-Network

17.3 Animate Deep Q-Learning

17.3.1 Draw the Current Game State and Q-Values

17.3.2 Create A Graph for Each Time Step

17.4 An Animation and A Picture with Subplots

17.5 Glossary

17.6 Exercises

CHAPTER 18 ◾ Policy-Based Deep Reinforcement Learning

18.1 Policy-Based Reinforcement Learning

18.1.1 What is a Policy?

18.1.2 What is the Policy Gradient Method?

18.2 Get Started with Atari Games

18.2.1 The Pong Game

18.2.2 Preprocess the Game Pictures

18.2.3 Use the Difference of Game Windows

18.3 Train the Policy Gradient Agent

18.3.1 Create a Policy Network

18.3.2 Train the Model

18.4 Test the Policy Gradient Agent

18.5 Animate the Pong Games

18.5.1 Record Games with Random Moves

18.5.2 Combine the Animations

18.5.3 Subplots of the Policy Gradient Agent

18.6 Glossary

18.7 Exercises

CHAPTER 19 ◾ The Policy Gradient Method in Breakout

19.1 Get Started with the Breakout Game

19.1.1 The Breakout Game

19.1.2 Preprocess the Game Frames

19.1.3 Obtain the Difference of Two Game Windows

19.2 Train the Policy Gradient Model in Breakout

19.2.1 Changes Needed

19.2.2 Create a Policy Network

19.2.3 Train the Policy Gradient Agent in Breakout

19.3 Test the Policy Gradient Agent in Breakout

19.3.1 Test the Trained Policy Gradient Agent

19.3.2 Search for Successful Episodes

19.4 Zero in on Interesting Time Steps

19.4.1 Animate Interesting Time Steps

19.4.2 Subplots of the Interesting Time Steps

19.5 Exercises

CHAPTER 20 ◾ Double Deep Q-Learning

20.1 Get Started with OpenAI Baselines

20.1.1 The Breakout Game with OpenAI Baselines

20.1.2 Preprocessed Frames from Baselines

20.1.3 Subplots of Preprocessed Frames

20.2 Train the Double Deep Q Agent

20.2.1 Create a Double Deep Q-Network

20.2.2 Train the Deep Q Network

20.3 Test the Trained Breakout Agent

20.3.1 Testing One Original Episode

20.3.2 Play Multiple Games and Test the Average Score

20.4 Animate Interesting Time Steps

20.4.1 Collect a Successful Episode

20.4.2 A Picture with Subplots

20.5 Glossary

20.6 Exercises

CHAPTER 21 ◾ Space Invaders with Double Deep Q-Learning

21.1 Getting Started with Space Invaders

21.1.1 Space Invaders in OpenAI Gym

21.1.2 Space Invaders with the Baselines Game Wrapper

21.1.3 Preprocessed Space Invaders Game Windows

21.2 Train the Double Deep Q-Network

21.2.1 The Same Double Deep Q-Network

21.2.2 The Same Training Process

21.3 Test the Trained Agent in Space Invaders

21.3.1 Testing One Full Original Episode

21.3.2 Average Performance of the Trained Model

21.4 Animate Space Invaders

21.4.1 Collect Space Invaders Episodes

21.4.2 Zero in on the Interesting Time Steps

21.4.3 Subplots of Space Invaders

21.5 Exercises

CHAPTER 22 ◾ Scaling Up Double Deep Q-Learning

22.1 Get Started with the Seaquest Game

22.1.1 The Seaquest Game in OpenAI Gym

22.1.2 Seaquest with the Baselines Game Wrapper

22.1.3 Preprocessed Seaquest Game Windows

22.1.4 Subplots of Seaquest Game Windows

22.2 Get Started with Beam Rider

22.2.1 Beam Rider without the Game Wrapper

22.2.2 Beam Rider with the Baselines Game Wrapper

22.2.3 Preprocessed Beam Rider Game Windows

22.2.4 Subplots of Beam Rider Game Windows

22.3 Scaling Up the Double Deep Q-Network

22.3.1 Differences among Atari Games

22.3.2 A Generic Double Deep Q-Network

22.3.3 The Training Process for any Atari Game

22.4 Try It on Seaquest

22.4.1 Train the Model in Seaquest

22.4.2 Test the Average Score in Seaquest

22.4.3 Animate a Successful Episode

22.5 Try It on Beam Rider

22.5.1 Train the Model in Beam Rider

22.5.2 The Average Score in Beam Rider

22.5.3 A Successful Episode in Beam Rider

22.6 Exercises

Bibliography

Index