Solve real-world data problems and create data-driven workflows for easy data movement and processing at scale with Azure Data Factory
▶What You Will Learn
-Create an orchestration and transformation job in ADF
-Develop, execute, and monitor data flows using Azure Synapse
-Create big data pipelines using Azure Data Lake and ADF
-Build a machine learning app with Apache Spark and ADF
-Migrate on-premises SSIS jobs to ADF
-Integrate ADF with commonly used Azure services such as Azure ML, Azure Logic Apps, and Azure Functions
-Run big data compute jobs within HDInsight and Azure Databricks
-Copy data from AWS S3 and Google Cloud Storage to Azure Storage using ADF's built-in connectors
▶Key Features
-Learn how to load and transform data from various sources, both on-premises and on cloud
-Use Azure Data Factory's visual environment to build and manage hybrid ETL pipelines
-Discover how to prepare, transform, process, and enrich data to generate key insights
▶Who This Book Is For
This book is for ETL developers, data warehouse and ETL architects, software professionals, and anyone who wants to learn about the common and not-so-common challenges faced while developing traditional and hybrid ETL solutions using Microsoft's Azure Data Factory. You'll also find this book useful if you are looking for recipes to improve or enhance your existing ETL pipelines. Basic knowledge of data warehousing is expected.
▶What this book covers
- Chapter 1, Getting Started with ADF, will briefly show you the Azure data platform. In this chapter, you will learn about the ADF interface and options as well as common use cases. You will perform hands-on exercises in order to find ADF in the Azure portal and create your first job.
- Chapter 2, Orchestration and Control Flow, will introduce you to the building blocks of the data processing in Azure Data Factory. The chapter contains hands-on exercises which show you how to set up linked services and datasets for your data sources, use various types of activities, design data-processing workflows, and create triggers for the data transfers.
- Chapter 3, Setting up a Cloud Data Warehouse, covers key features and benefits of cloud data warehousing and Azure Synapse Analytics. You will learn how to connect and configure Azure Synapse Analytics, load data, build transformation processes, and operate data flows.
- Chapter 4, Working with Azure Data Lake, will go through the features of Azure Data Lake Storage Gen2. This is multi-modal cloud storage that is frequently used for big data analytics. We will load and manage the datasets that we will use for analytics in the next chapter.
- Chapter 5, Working with Big Data – HDInsight and Databricks, is where we will actively engage with analytical tools from the Azure data services. We will start with munging data with Azure Databricks, then train some models on big data, and analyze them to draw business insights. Also, we will go through Stream Analytics.
- Chapter 6, Integration with MS SSIS, covers using the Azure data platform and ADF on-premises. This chapter will help you leverage your on-premises infrastructure together with cloud-native tools to get relevant business insights.
- Chapter 7, Data Migration – Azure Data Factory and Other Cloud Services, explains how to use Azure Data factory to transfer data between Azure and other cloud providers, such as AWS or Google Cloud, using ADF built-in connectors. We also show how to integrate a provider not currently supported by a built-in ADF connector, using Dropbox as an example.
- Chapter 8, Working with Azure Services Integration, will cover how to do integrations of the most commonly used Azure services into ADF. You will also learn how Azure services can be useful in designing ETL pipelines.
- Chapter 9, Managing Deployment Processes with Azure DevOps, will cover the key features of Azure DevOps. You will learn how to build CI/CD processes and continuous monitoring with Microsoft Azure. You will create a platform for application deployment and integrate it with ADF.
- Chapter 10, Monitoring and Troubleshooting Data Pipelines, will teach readers how to use the Azure Data Factory Monitor interface to evaluate the progress of your data transfers, how to understand error messages and set up alerts for the pipelines. This chapter contains hands-on recipes highlighting the debugging capabilities of ADF.
