Hands-On Vision and Behavior for Self-Driving Cars

A practical guide to learning visual perception for self-driving cars for computer vision and autonomous system engineers

▶Book Description
The visual perception capabilities of a self-driving car are powered by computer vision. The work relating to self-driving cars can be broadly classified into three components - robotics, computer vision, and machine learning. This book provides existing computer vision engineers and developers with the unique opportunity to be associated with this booming field.

You will learn about computer vision, deep learning, and depth perception applied to driverless cars. The book provides a structured and thorough introduction, as making a real self-driving car is a huge cross-functional effort. As you progress, you will cover relevant cases with working code, before going on to understand how to use OpenCV, TensorFlow and Keras to analyze video streaming from car cameras. Later, you will learn how to interpret and make the most of lidars (light detection and ranging) to identify obstacles and localize your position. You'll even be able to tackle core challenges in self-driving cars such as finding lanes, detecting pedestrian and crossing lights, performing semantic segmentation, and writing a PID controller.

By the end of this book, you'll be equipped with the skills you need to write code for a self-driving car running in a driverless car simulator, and be able to tackle various challenges faced by autonomous car engineers.

▶What You Will Learn
⦁ Understand how to perform camera calibration
⦁ Become well-versed with how lane detection works in self-driving cars using OpenCV
⦁ Explore behavioral cloning by self-driving in a video-game simulator
⦁ Get to grips with using lidars
⦁ Discover how to configure the controls for autonomous vehicles
⦁ Use object detection and semantic segmentation to locate lanes, cars, and pedestrians
⦁ Write a PID controller to control a self-driving car running in a simulator

▶Key Features
⦁Explore the building blocks of the visual perception system in self-driving cars
⦁Identify objects and lanes to define the boundary of driving surfaces using open-source tools like OpenCV and Python
⦁Improve the object detection and classification capabilities of systems with the help of neural networks

▶Who This Book Is For
This book is for software engineers who are interested in learning about technologies that drive the autonomous car revolution. Although basic knowledge of computer vision and Python programming is required, prior knowledge of advanced deep learning and how to use sensors (lidar) is not needed.

▶What this book covers
⦁ Chapter 1, OpenCV Basics and Camera Calibration, is an introduction to OpenCV and NumPy; you will learn how to manipulate images and videos, and how to detect pedestrians using OpenCV; in addition, it explains how a camera works and how OpenCV can be used to calibrate it.

⦁ Chapter 2, Understanding and Working with Signals, describes the different types of signals: serial, parallel, digital, analog, single-ended, and differential, and explains some very important protocols: CAN, Ethernet, TCP, and UDP.

⦁ Chapter 3, Lane Detection, teaches you everything you need to know to detect the lanes in a road using OpenCV. It covers color spaces, perspective correction, edge detection, histograms, the sliding window technique, and the filtering required to get the best detection.

⦁ Chapter 4, Deep Learning with Neural Networks, is a practical introduction to neural networks, designed to quickly teach how to write a neural network. It describes neural networks in general and convolutional neural networks in particular. It introduces Keras, a deep learning module, and it shows how to use it to detect handwritten digits and to classify some images.

⦁ Chapter 5, Deep Learning Workflow, ideally complements Chapter 4, Deep Learning with Neural Networks, as it describes the theory of neural networks and the steps required in a typical workflow: obtaining or creating a dataset, splitting it into training, validation, and test sets, data augmentation, the main layers used in a classifier, and how to train, do inference, and retrain. The chapter also covers underfitting and overfitting and explains how to visualize the activations of the convolutional layers.

⦁ Chapter 6, Improving Your Neural Network, explains how to optimize a neural network, reducing its parameters, and how to improve its accuracy using batch normalization, early stopping, data augmentation, and dropout.

⦁ Chapter 7, Detecting Pedestrians and Traffic Lights, introduces you to CARLA, a self-driving car simulator, which we will use to create a dataset of traffic lights. Using a pre-trained neural network called SSD, we will detect pedestrians, cars, and traffic lights, and we will use a powerful technique called transfer learning to train a neural network to classify the traffic lights according to their colors.

⦁ Chapter 8, Behavioral Cloning, explains how to train a neural network to drive CARLA. It explains what behavioral cloning is, how to build a driving dataset using CARLA, how to create a network that's suitable for this task, and how to train it. We will use saliency maps to get an understanding of what the network is learning, and we will integrate it with CARLA to help it self-drive!

⦁ Chapter 9, Semantic Segmentation, is the final and most advanced chapter about deep learning, and it explains what semantic segmentation is. It details an extremely interesting architecture called DenseNet, and it shows how to adapt it to semantic segmentation.

⦁ Chapter 10, Steering, Throttle, and Brake Control, is about controlling a self-driving car. It explains what a controller is, focusing on PID controllers and covering the basics of MPC controllers. Finally, we will implement a PID controller in CARLA.

⦁ Chapter 11, Mapping Our Environments, is the final chapter. It discusses maps, localization, and lidar, and it describes some open source mapping tools. You will learn what Simultaneous Localization and Mapping (SLAM) is and how to implement it using the Ouster lidar and Google Cartographer.