▶What You Will Learn
- Develop an agent to play CartPole using the OpenAI Gym interface
- Discover the model-based reinforcement learning paradigm
- Solve the Frozen Lake problem with dynamic programming
- Explore Q-learning and SARSA with a view to playing a taxi game
- Apply Deep Q-Networks (DQNs) to Atari games using Gym
- Study policy gradient algorithms, including Actor-Critic and REINFORCE
- Understand and apply PPO and TRPO in continuous locomotion environments
- Get to grips with evolution strategies for solving the lunar lander problem
▶Key Features
- Learn, develop, and deploy advanced reinforcement learning algorithms to solve a variety of tasks
- Understand and develop model-free and model-based algorithms for building self-learning agents
- Work with advanced Reinforcement Learning concepts and algorithms such as imitation learning and evolution strategies
▶Who This Book Is For
If you are an AI researcher, deep learning user, or anyone who wants to learn reinforcement learning from scratch, this book is for you. You'll also find this reinforcement learning book useful if you want to learn about the advancements in the field. Working knowledge of Python is necessary.
▶What this book covers
- Chapter 1, The Landscape of Reinforcement Learning, gives you an insight into RL. It describes the problems that RL is good at solving and the applications where RL algorithms are already adopted. It also introduces the tools, the libraries, and the setup needed for the completion of the projects in the following chapters.
- Chapter 2, Implementing RL Cycle and OpenAI Gym, describes the main cycle of the RL algorithms, the toolkit used to develop the algorithms, and the different types of environments. You will be able to develop a random agent using the OpenAI Gym interface to play CartPole using random actions. You will also learn how to use the OpenAI Gym interface to run other environments.
- Chapter 3, Solving Problems with Dynamic Programming, introduces to you the core ideas, terminology, and approaches of RL. You will learn about the main blocks of RL and develop a general idea about how RL algorithms can be created to solve a problem. You will also learn the differences between model-based and model-free algorithms and the categorization of reinforcement learning algorithms. Dynamic programming will be used to solve the game FrozenLake.
- Chapter 4, Q-Learning and SARSA Applications, talks about value-based methods, in particular Q-learning and SARSA, two algorithms that differ from dynamic programming and scale well on large problems. To become confident with these algorithms, you will apply them to the FrozenLake game and study the differences from dynamic programming.
- Chapter 5, Deep Q-Networks, describes how neural networks and convolutional neural networks (CNNs) in particular are applied to Q-learning. You'll learn why the combination of Q-learning and neural networks produces incredible results and how its use can open the door to a much larger variety of problems. Furthermore, you'll apply the DQN to an Atari game using the OpenAI Gym interface.
- Chapter 6, Learning Stochastic and PG Optimization, introduces a new family of model-free algorithms: policy gradient methods. You will learn the differences between policy gradient and value-based methods, and you'll learn about their strengths and weaknesses. Then you will implement the REINFORCE and Actor-Critic algorithms to solve a new game called LunarLander.
- Chapter 7, TRPO and PPO Implementation, proposes a modification of policy gradient methods using new mechanisms to control the improvement of the policy. These mechanisms are used to improve the stability and convergence of the policy gradient algorithms. In particular you'll learn and implement two main policy gradient methods that use these techniques, namely TRPO and PPO. You will implement them on RoboSchool, an environment with a continuous action space.
- Chapter 8, DDPG and TD3 Applications, introduces a new category of algorithms called deterministic policy algorithms that combine both policy gradient and Q-learning. You will learn about the underlying concepts and implement DDPG and TD3, two deep deterministic algorithms, on a new environment.
- Chapter 9, Model-Based RL, illustrates RL algorithms that learn the model of the environment to plan future actions, or, to learn a policy. You will be taught how they work, their strengths, and why they are preferred in many situations. To master them, you will implement a model-based algorithm on Roboschool.
- Chapter 10, Imitation Learning with the DAgger Algorithm, explains how imitation learning works and how it can be applied and adapted to a problem. You will learn about the most well-known imitation learning algorithm, DAgger. To become confident with it, you will implement it to speed up the learning process of an agent on FlappyBird.
- Chapter 11, Understanding Black-Box Optimization Algorithms, explores evolutionary algorithms, a class of black-box optimization algorithms that don't rely on backpropagation. These algorithms are gaining interest because of their fast training and easy parallelization across hundreds or thousands of cores. This chapter provides a theoretical and practical background of these algorithms by focusing particularly on the Evolution Strategy algorithm, a type of evolutionary algorithm.
- Chapter 12, Developing ESBAS Algorithm, introduces the important exploration-exploitation dilemma, which is specific to RL. The dilemma is demonstrated using the multi-armed bandit problem and is solved using approaches such as UCB and UCB1. Then, you will learn about the problem of algorithm selection and develop a meta-algorithm called ESBAS. This algorithm uses UCB1 to select the most appropriate RL algorithm for each situation.
- Chapter 13, Practical Implementations to Resolve RL Challenges, takes a look at the major challenges in this field and explains some practices and methods to overcome them. You will also learn about some of the challenges of applying RL to real-world problems, future developments of deep RL, and their social impact in the world.
