Create an account to use AI

Create a free account to unlock all features:

Ask questions about videos with AI
Use smart search (AI)
Save your favorite videos
Bookmark your favorite categories
Organize your favorite subcategories
Access your personalized favorites page
Access your conversation history
Personalize your homepage with your favorites

Create an account

Already have an account? Log in

Dagster - Data Orchestration and Pipelines with Python & DAGs

BugBytes

🙏 Join our channel to get access to perks:

https://www.youtube.com/channel/UCTwxaBjziKfy6y_uWu30orA/join

☕️ 𝗕𝘂𝘆 𝗺𝗲 𝗮 𝗰𝗼𝗳𝗳𝗲𝗲:

To support the channel and encourage new videos, please consider buying a coffee here:

https://ko-fi.com/bugbytes

In this video, we'll dive into Dagster for Python task orchestration and data engineering. We will look at core Dagster concepts, including Assets, Definitions, Dagster UI, Schedules and Jobs.

📌 𝗖𝗵𝗮𝗽𝘁𝗲𝗿𝘀:

00:00 Intro

01:11 Data Pipeline & DAG overview

01:52 Installing Dagster with uv

03:41 Defining Dagster Asset

05:10 Definitions object

07:59 Real-World example with Polars & DuckDB

08:58 Car dataset

09:41 Dagster asset to download CSV file

13:24 Loading aggregated data into DuckDB database

20:10 Scheduling jobs in Dagster

26:20 Sensors in Dagster

𝗦𝗼𝗰𝗶𝗮𝗹 𝗠𝗲𝗱𝗶𝗮:

📖 Blog: https://bugbytes.io/posts/

👾 Github: https://github.com/bugbytes-io/

📚 𝗙𝘂𝗿𝘁𝗵𝗲𝗿 𝗿𝗲𝗮𝗱𝗶𝗻𝗴 𝗮𝗻𝗱 𝗶𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻:

Docker: https://www.docker.com/

Postgres DockerHub: https://hub.docker.com/_/postgres

#dagster #dataengineering #python

Dagster - Data Orchestration and Pipelines with Python & DAGs | Knoovi