Switching from Snowflake to DuckDB: Automating Data Pipelines with Dagster, dlt, and dbt
In this video, we demonstrate how to transition from using Snowflake to a local DuckDB memory database for orchestrating data pipelines with dagster, dlt, and dbt. We cover the configuration changes required to switch data warehouses and show how to automate the entire process using dagster. We begin by modifying the code from our previous video to point from Snowflake to DuckDB and walk through setting up the necessary configurations in Visual Studio Code. Additionally, we explore how to schedule automated data pipeline runs with dagster. By the end of the video, you will learn how to run data extractions, transformations, and load them into DuckDB, ensuring a cost-effective and flexible data management solution. Stay tuned for future tutorials on containerizing and deploying these projects in Azure.
Code repo
Previous video
00:00 Introduction and Overview
01:02 Setting Up the Environment
03:01 Configuring the Pipeline
04:34 Modifying Configuration Files
09:44 Running the Pipeline
11:33 Scheduling and Automation
15:34 Conclusion and Next Steps