In this video we will revisit dagster. We will talk about changes to this workflow orchestration system due to recent updates (update from version 0.15 to 1.3.1)
Dagster is an orchestrator that's designed for developing and maintaining data assets, such as tables, data sets, machine learning models, and reports.
We will cover Software Defined Assets as dagster is pushing towards the Software Defined Assets. By default our pipeline outputs are stored as a pickle file in the dagster home folder. What if we want to store the outputs in a database table, or in a readable file such as a csv or parquet file. Dagster provides us with Input and Output managers (IO managers) that enable reading and writing data to storage systems. Using Store IO managers we can save the outputs in a file system or store our data as tables in a database. We will define file csv/parquet and database IO Managers.
Get started with Dagster in just three quick steps:
Install Dagster, Define assets and Materialize assets.
Create a virtual environment: python -m venv env
Activate the virtual environment: env\Scripts\activate
To install Dagster into an existing Python environment, run:
pip install dagster dagit
Command to create a new project
dagster project scaffold --name my-dagster-project
Additional libraries required: Pandas, psycopg2
To run dagster issue following command:
dagit
dagster-daemon run
💥Subscribe to our channel:
📌 Links
-----------------------------------------
#️⃣ Follow me on social media! #️⃣
-----------------------------------------
#Python #ETL #Dagster
Topics covered in this video:
==================================
0:00 - Introduction to Dagster
2:11 - Dagster create new project
3:03 - Dagster Project Structure
4:18 - Software Defined Assets
5:35 - Install Required Libraries
5:58 - Source DB Connection
6:27 - Source Asset
10:05 - File IO Manager
14:16 - Second Asset
16:19 - Parquet IO Manager
16:26 - Database IO Manager
19:05 - Materialize Assets