Learn how to write high-quality kernel module code, solve common Linux kernel programming issues, and understand the fundamentals of Linux kernel internals
▶Book Description
Linux Kernel Programming is a comprehensive introduction for those new to Linux kernel and module development. This easy-to-follow guide will have you up and running with writing kernel code in next-to-no time. This book uses the latest 5.4 Long-Term Support (LTS) Linux kernel, which will be maintained from November 2019 through to December 2025. By working with the 5.4 LTS kernel throughout the book, you can be confident that your knowledge will continue to be valid for years to come.
This Linux book begins by showing you how to build the kernel from the source. Next, you'll learn how to write your first kernel module using the powerful Loadable Kernel Module (LKM) framework. The book then covers key kernel internals topics including Linux kernel architecture, memory management, and CPU scheduling. Next, you'll delve into the fairly complex topic of concurrency within the kernel, understand the issues it can cause, and learn how they can be addressed with various locking technologies (mutexes, spinlocks, atomic, and refcount operators). You'll also benefit from more advanced material on cache effects, a primer on lock-free techniques within the kernel, deadlock avoidance (with lockdep), and kernel lock debugging techniques.
By the end of this kernel book, you'll have a detailed understanding of the fundamentals of writing Linux kernel module code for real-world projects and products.
▶What You Will Learn
-Write high-quality modular kernel code (LKM framework) for 5.x kernels
-Configure and build a kernel from source</li><li>Explore the Linux kernel architecture
-Get to grips with key internals regarding memory management within the kernel
-Understand and work with various dynamic kernel memory alloc/dealloc APIs
-Discover key internals aspects regarding CPU scheduling within the kernel
-Gain an understanding of kernel concurrency issues
-Find out how to work with key kernel synchronization primitives
▶Key Features
-Discover how to write kernel code using the Loadable Kernel Module framework
-Explore industry-grade techniques to perform efficient memory allocation and data synchronization within the kernel
-Understand the essentials of key internals topics such as kernel architecture, memory management, CPU scheduling, and kernel synchronization
▶Who This Book Is For
This book is for Linux programmers beginning to find their way with Linux kernel development. Linux kernel and driver developers looking to overcome frequent and common kernel development issues, as well as understand kernel internals, will benefit from this book. A basic understanding of Linux CLI and C programming is required.
▶What this book covers
- Chapter 1, Kernel Workspace Setup, guides you on setting up a full-fledged Linux kernel development workspace (typically, as a fully virtualized guest system). You will learn how to install all required software packages on it, including a cross toolchain. You will also learn about several other open source projects that will be useful on your journey to becoming a professional kernel/driver developer. Once this chapter is done, you will be ready to build a Linux kernel as well as to start writing and testing kernel code (via the loadable kernel module framework). In our view, it's very important for you to actually use this book in a hands-on fashion, trying out and experimenting with code. The best way to learn something is to do so empirically – not taking anyone's word on anything at all, but by trying it out and experiencing it for yourself.
- Chapter 2, Building the 5.x Linux Kernel from Source – Part 1, is the first part of explaining how to build the modern Linux kernel from scratch with source code. In this part, you will be given necessary background information – the version nomenclature, the different source trees, the layout of the kernel source – on the kernel source tree. Next, you will be shown in detail how exactly to download a stable vanilla Linux kernel source tree onto the VM. We shall then learn a little regarding the layout of the kernel source code, getting, in effect, a "10,000-foot view" of the kernel code base. The actual work of extracting and configuring the Linux kernel then follows. Creating and using a custom menu entry for kernel configuration is also shown.
- Chapter 3, Building the 5.x Linux Kernel from Source – Part 2, is the second part on performing kernel builds from source code. In this part, you will continue from the previous chapter, now actually building the kernel, installing kernel modules, understanding what exactly initramfs (initrd) is and how to generate it, as well as setting up the bootloader (for x86). Also, as a valuable add-on, this chapter then explains how to cross-compile the kernel for a typical embedded ARM target (using the popular Raspberry Pi as a target device). Several tips and tricks on kernel builds, and even kernel security (hardening), are mentioned as well.
- Chapter 4, Writing Your First Kernel Module – LKMs Part 1, is the first of two parts that cover a fundamental aspect of Linux kernel development – the LKM framework, and how it is to be understood and used by the "module user," by you – the kernel module or device driver programmer. It covers the basics of the Linux kernel architecture and then, in great detail, every step involved in writing a simple "Hello, world" kernel module, compiling, inserting, checking, and removing it from the kernel space. We also cover kernel logging via the ubiquitous printk API in detail.
- Chapter 5, Writing Your First Kernel Module – LKMs Part 2, is the second part that covers the LKM framework. Here, we begin with something critical – learning how to use a "better" Makefile, which will help you generate more robust code (having several code-checking, correction, static analysis targets, and so on). We then show in detail the steps to successfully cross-compile a kernel module for an alternate architecture, how to emulate "library-like" code in the kernel (via both the "linking" and the module-stacking approaches), defining and using passing parameters to your kernel module. Additional topics include the auto-loading of modules at boot, important security guidelines, and some information on the kernel documentation and how to access it. Several example kernel modules make the learning more interesting.
- Chapter 6, Kernel Internals Essentials – Processes and Threads, delves into some essential kernel internals topics. We begin with what is meant by execution in process and interrupt contexts, and minimal but required coverage of the process user virtual address space (VAS) layout. This sets the stage for you; you'll then learn about Linux kernel architecture in more depth, focusing on the organization of process/thread task structures and their corresponding stacks – user- and kernel-mode. We then show you more on the kernel task structure (a "root" data structure), how to practically glean information from it, and even iterate over various (task) lists. Several kernel modules make the topic come alive.
- Chapter 7, Memory Management Internals – Essentials, a key chapter, delves into essential internals of the Linux memory management subsystem, to the level of detail required for the typical module author or driver developer. This coverage is thus necessarily more theoretical in nature; nevertheless, the knowledge gained here is crucial to you, the kernel developer, both for deep understanding and usage of appropriate kernel memory APIs as well as for performing meaningful debugging at the level of the kernel. We cover the VM split (and how it is on various actual architectures), gaining deep insight into the user VAS (our procmap utility will be an eye-opener), as well as the kernel segment (or kernel VAS). We then briefly delve into the security technique of memory layout randomization ([K]ASLR), and end this chapter with a discussion on physical memory organization within Linux.
- Chapter 8, Kernel Memory Allocation for Module Authors Part 1, gets our hands dirty with the kernel memory allocation (and obviously, deallocation) APIs. You will first learn about the two allocation "layers" within Linux – the slab allocator that's layered above the kernel memory allocation "engine," and the page allocator (or BSA). We shall briefly learn about the underpinnings of the page allocator algorithm and its "freelist" data structure; this information is valuable when deciding which layer to use. Next, we dive straight into the hands-on work of learning about the usage of these key APIs. The ideas behind the slab allocator (or cache) and the primary kernel allocator APIs – the kzalloc/kfree – are covered. Importantly, the size limitations, downsides, and caveats when using these common APIs are covered in detail as well. Also, especially useful for driver authors, we cover the kernel's modern resourcemanaged memory allocation APIs (the devm_*() routines).
- Chapter 9, Kernel Memory Allocation for Module Authors Part 2, goes further, in a logical fashion, from the previous chapter. Here, you will learn how to create custom slab caches (useful for high-frequency (de)allocations for, say, a custom driver), along with some help regarding debugging memory allocations at the slab layer. Next, you'll understand and use the vmalloc() API (and friends). Very importantly, having covered many APIs for kernel memory (de)allocation, you will now learn how to pick and choose an appropriate API given the real-world situation you find yourself in. This chapter is rounded off with important coverage of the kernel's Out Of Memory (OOM) "killer" framework. Understanding it will also lead to a much deeper understanding of how user space memory allocation really works, via the demand paging technique.
- Chapter 10, The CPU Scheduler - Part 1, the first part of two chapters, covers a useful mix of theory and practice regarding CPU scheduling on the Linux OS. The minimal necessary theoretical background on the thread as the KSE and available kernel scheduling policies are topics initially covered. Next, sufficient kernel internal details on CPU scheduling are covered to have you understand how scheduling on the modern Linux OS works. Along the way, you will learn how to "visualize" PU scheduling with powerful tools such as perf; thread scheduling attributes (policy and real-time priority) are delved into as well.
- Chapter 11, The CPU Scheduler – Part 2, the second part on CPU scheduling, continues to cover the topic in more depth. Here, we cover further visualization tools for CPU scheduling (leveraging powerful software such as LTTng and the trace-cmd utility). Next, the CPU affinity mask and how to query/set it, controlling scheduling policy and priority on a per-thread basis – such a powerful feature! – are delved into. An overview of the meaning and importance of control groups (cgroups), along with an interesting example on CPU bandwidth allocation via cgroups v2 is seen. Can you run Linux as an RTOS? Indeed you can! The details on actually doing so are then shown. We round off this chapter with a discussion on (scheduling) latencies and how to measure them.
- Chapter 12, Kernel Synchronization – Part 1, first covers the key concepts regarding critical sections, atomicity, what a lock conceptually achieves and, very importantly, the why of all this. We then cover concurrency concerns when working within the Linux kernel; this moves us naturally on to important locking guidelines, what deadlock means, and key approaches to preventing deadlock. Two of the most popular kernel locking technologies – the mutex lock and the spinlock – are then discussed in depth along with several (driver) code examples.
- Chapter 13, Kernel Synchronization – Part 2, continues the journey on kernel synchronization. Here, you'll learn about key locking optimizations – using lightweight atomic and (the more recent) refcount operators to safely operate on integers, RMW bit operators to safely perform bit ops, and the usage of the readerwriter spinlock over the regular one. Inherent risks, such as cache "false sharing" are discussed as well. An overview of lock-free programming techniques (with an emphasis on per-CPU variables and their usage, along with examples) is then covered. A critical topic – lock debugging techniques, including the usage of the kernel's powerful "lockdep" lock validator, is then covered. The chapter is rounded off with a brief look at memory barriers (along with an example).
