▶Book Description
Mastering Distributed Tracing will equip you to operate and enhance your own tracing infrastructure. Through practical exercises and code examples, you will learn how end-to-end tracing can be used as a powerful application performance management and comprehension tool.
The rise of Internet-scale companies, like Google and Amazon, ushered in a new era of distributed systems operating on thousands of nodes across multiple data centers. Microservices increased that complexity, often exponentially. It is harder to debug these systems, track down failures, detect bottlenecks, or even simply understand what is going on. Distributed tracing focuses on solving these problems for complex distributed systems. Today, tracing standards have developed and we have much faster systems, making instrumentation less intrusive and data more valuable.
Yuri Shkuro, the creator of Jaeger, a popular open-source distributed tracing system, delivers end-to-end coverage of the field in Mastering Distributed Tracing. Review the history and theoretical foundations of tracing; solve the data gathering problem through code instrumentation, with open standards like OpenTracing, W3C Trace Context, and OpenCensus; and discuss the benefits and applications of a distributed tracing infrastructure for understanding, and profiling, complex systems.
▶What You Will Learn
⦁ How to get started with using a distributed tracing system
⦁ How to get the most value out of end-to-end tracing
⦁ Learn about open standards in the space
⦁ Learn about code instrumentation and operating a tracing infrastructure
⦁ Learn where distributed tracing fits into microservices as a core function
▶Key Features
⦁ A thorough conceptual introduction to distributed tracing
⦁ An exploration of the most important open standards in the space
⦁ A how-to guide for code instrumentation and operating a tracing infrastructure
▶Who This Book Is For
Any developer interested in testing large systems will find this book very revealing and in places, surprising. Every microservice architect and developer should have an insight into distributed tracing, and the book will help them on their way. System administrators with some development skills will also benefit. No particular programming language skills are required, although an ability to read Java, while non-essential, will help with the core chapters.
▶What this book covers
▷ Part I, Introduction, provides a general introduction to the area of distributed tracing.
⦁ Chapter 1, Why Distributed Tracing, frames the observability problem that distributed tracing aims to solve and explains why other monitoring tools fall short when it comes to troubleshooting pathological behavior in complex distributed systems. The chapter includes a brief history of my personal experience with tracing and an explanation of why I felt that writing this book would be a useful contribution to the industry.
⦁ Chapter 2, Take Tracing for a HotROD Ride, dives in with an easy to run, hands-on example used to illustrate the core features, benefits, and capabilities of distributed tracing, using Jaeger, an open source tracing platform, the OpenTracing instrumentation, and a demo application HotROD (Rides on Demand).
⦁ Chapter 3, Distributed Tracing Fundamentals, reviews the basic operating principles of end-to-end tracing, such as causality tracking and metadata propagation, and various design decisions historically taken by different implementations that affect the types of problems a given tracing architecture is able to solve. It introduces the reader to two different tracing models, the more expressive event model, and more popular span model.
▷ Part II, Data Gathering Problem, is dedicated to discussions about the different ways of getting tracing data out of the applications, through manual and automatic (agent-based) instrumentation, for both RPC-style and asynchronous (for example, using message queues) applications.
⦁ Chapter 4, Instrumentation Basics with OpenTracing, provides a step-by-step guide of manually instrumenting a simple "hello, world" style application for tracing, as it is being evolved from a monolith to a microservices-based system. Three parallel sets of examples are provided in popular programming languages: Go, Java, and Python. The chapter teaches the fundamentals of tracing instrumentation using the OpenTracing API; however, the general patterns are applicable to other
instrumentation APIs as well. In the final exercises the chapter introduces automatic (agent-based) instrumentation style that requires little, if any, actual code changes in the application.
⦁ Chapter 5, Instrumentation of Asynchronous Applications, continues the lessons from Chapter 4, and applies them to an "online chat" type of application built around asynchronous messaging using Apache Kafka.
⦁ Chapter 6, Tracing Standards and Ecosystem, explores the often confusing ecosystem of the tracing industry at large, including the emerging standards such as OpenTracing, W3C Trace Context, and OpenCensus. It provides a useful taxonomy of how to think about the different commercial and open source projects and their positions in relation to each other.
⦁ Chapter 7, Tracing with Service Mesh, uses the service mesh Istio, running on Kubernetes, to trace an application and compare the results with tracing an application that is natively instrumented for tracing via the OpenTracing API. It reviews the pros and cons of each approach.
⦁ Chapter 8, All About Sampling, explains why tracing platforms are often required to sample transactions and provides an in-depth review of different sampling techniques, from consistent head-based sampling strategies (probabilistic, rate limiting, adaptive, and so on) to the emerging favorite, tail-based sampling.
▷ Part III, Getting Value from Tracing, talks about the different ways engineers and organization can benefit from adopting a distributed tracing solution.
⦁ Chapter 9, Turning the Lights On, gives examples of the core value proposition of tracing, covering features that are commonly available in most tracing solutions; such as service graphs; critical path analysis; performance analysis with trace patterns; latency histograms and exemplars; and the long-term profiling techniques.
⦁ Chapter 10, Distributed Context Propagation, steps back to discuss context propagation, a technology that underpins most existing tracing infrastructures. It covers Tracing Plane from Brown University, which implements a general-purpose, tool-agnostic framework for context propagation, or "baggage," and covers a number of useful techniques and tools for observability and chaos engineering that have been built on top of context propagation and tracing.
⦁ Chapter 11, Integration with Metrics and Logs, shows how all is not lost for traditional monitoring tools, and how combining them with tracing infrastructure gives them new capabilities and makes them more useful in microservices environments.
⦁ Chapter 12, Gathering Insights with Data Mining, begins with the basics of data mining and feature extraction from tracing data, followed by a practical example involving the Jaeger backend, Apache Kafka, Elasticsearch, Kibana, an Apache Flink data mining job, and a microservices simulator, microsim. It ends with a discussion of further evolution of data mining techniques, such as inferring and observing trends, and historical and ad hoc data analysis.
▷ Part IV, Deploying and Operating Tracing Infrastructure, completes the book with an assortment of practical advice to the tracing teams about implementing and operating tracing platforms in large organizations.
⦁ Chapter 13, Implementing Tracing in Large Organizations, discusses how to overcome many technical and organizational challenges that often prevent wide adoption of distributed tracing in enterprises or full realization of its value.
⦁ Chapter 14, Under the Hood of a Distributed Tracing System, starts with a brief discussion of build versus buy considerations, then goes deep into many technical details of the architecture and deployment modes of a tracing platform, such as multi-tenancy, security, operation in multiple data centers, monitoring, and resiliency. The Jaeger project is used to illustrate many architectural decisions, yet overall the content is applicable to most tracing infrastructures.
