Azure Data Factory

Module 1: Introduction to Azure Data Factory

• Introduction to ADF:
• What is ADF?
• Key components (activities, pipelines, datasets, linked services)
• Use cases and benefits

• ADF Architecture:
• Overview of the ADF service
• Integration runtime
• Data flow

• Creating and Managing ADF Resources:
• Creating an ADF instance
• Managing pipelines, datasets, and linked services
• Using the ADF UI and PowerShell

Module 2: Core Concepts

• Pipelines:
• Representing data sources and sinks
• Supported data formats (JSON, CSV, Parquet, etc.)
• Linking datasets to linked services

• Linked Services:
• Connecting to data sources (on-premises, cloud)
• Authentication and authorization
• Self-hosted integration runtime

Module 3: Data Integration Patterns

• Extract, Transform, Load (ETL):
• Extracting data from source systems
• Transforming data using data flows or activities
• Loading data into target systems

• Extract, Load, Transform (ELT):
• Extracting data from source systems
• Loading data into a data warehouse or data lake
• Transforming data using data flows

• Data Replication:
• Replicating data between systems in real-time or batch

• Data Migration:
• Migrating data from one system to another

• Data Warehousing::
• Building and maintaining data warehouses using ADF

Module 4: Data Transformation

• Data Flows:
• Creating and managing data flows
• Using data flow activities (source, sink, transformations)
• Mapping data between columns

• • Transformations:
• Derived columns
• Aggregations
• Joins
• Unions
• Filtering
• Sorting
• Lookups

Module 5: Advanced Topics

• Control Flow:
• Conditional statements
• Loops

• Error Handling:
• Try-catch blocks
• Retry policies

• Monitoring and Debugging:
• Using ADF monitoring tools
• Debugging pipelines and data flows

• Performance Optimization:
• Optimizing pipeline execution
• Improving data flow performance

• Integration with Other Azure Services:
• Azure Data Lake Storage
• Azure Synapse Analytics
• Azure Machine Learning
• Azure Logic Apps

Module 6: Hands-on Exercises

• Hands-on Projects:
• Creating and running pipelines
• Building data flows
• Integrating with various data sources and sinks
• Implementing ETL and ELT patterns
• Troubleshooting and optimizing ADF workflows

Book Now