In this instructional video, we explore how to use Dagster to automate and orchestrate your data pipeline. We demonstrate extracting and loading data into Snowflake using DLT and transforming it using DBT. Previously, this process was done manually, but with the help of Dagster, we automate scheduling and triggering jobs based on dependencies. The video serves as a demo that introduces core concepts and steps involved in setting up and running a Dagster orchestration project. Follow along as we set up assets, jobs, schedules, and sensors, and watch the entire data pipeline come to life in the Dagster UI.
Code repo
Previous video
#dagster #dlt #dataengineering #dbt
00:00 Introduction to Dagster and Video Overview
00:19 Manual Process vs. dagster Automation
01:08 Future Dagster Videos and Demo Focus
01:38 Theory and Code Explanation
02:31 Introduction to Dagster Core Concepts
04:35 Simplified Dagster Setup
05:37 Understanding Assets, Jobs, and Schedules
07:14 Setting Up the Development Environment
09:37 Modifying the DLT Script for Dagster
13:44 Data Transformation with DBT
15:47 Connecting DBT to Dagster
20:13 Configuring Dagster for Orchestration
25:18 Introduction to DLT Load
26:12 Understanding Dependency Injection in Dagster
28:05 Setting Up DBT Project Directory
29:38 Creating DBT Project Instance
30:40 Running DBT CLI Commands
32:37 Creating DBT Assets
34:33 Defining Jobs and Schedules
40:36 Launching Dagster Services
41:59 Exploring Dagster Web Server
43:06 Viewing Data Lineage in Dagster
44:11 Materializing Data in Snowflake
48:20 Conclusion and Future Videos