Skip to main content

Migrating an Airflow BashOperator (dbt) to Dagster

In this page, we'll explain migrating an Airflow BashOperator that runs a dbt command to Dagster.

About the Airflow BashOperator​

In Airflow, you might have a BashOperator that runs a dbt command. For example, you might have a task that runs dbt run to build your dbt models.

from airflow.operators.bash import BashOperator

run_dbt_model = BashOperator(task_id="build_dbt_models", bash_command="dbt run")

Dagster equivalent​

The Dagster equivalent is to instead use the dagster-dbt library to run commands against your dbt project. Here would be the equivalent code in Dagster:

from dagster_dbt import DbtCliResource, DbtProject, dbt_assets

from dagster import AssetExecutionContext

project = DbtProject(project_dir="path/to/dbt_project")


@dbt_assets(manifest=project.manifest_path)
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["run"], context=context).stream()

Migrating the operator​

Migrating the operator breaks down into a few steps:

  1. Making the dbt project available to both your Airflow and Dagster deployments.
  2. Writing a @dbt_asset-decorated function which runs your dbt commands.
  3. Using dagster-airlift to proxy execution of the original task to Dagster.

Step 1: Making the dbt project available & building manifest​

First, you'll need to make the dbt project available to the Dagster runtime and build the manifest.

  • If you're building your Dagster deployment in a monorepo alongside your dbt and Airflow projects, you can follow this guide: Monorepo setup.
  • If you're deploying within a separate repository, you can follow this guide: Separate repository setup. */}

Step 2: Writing a @dbt_asset-decorated function​

Once your dbt project is available, you can write a function that runs your dbt commands using the dbt_assets decorator and DbtCliResource. Most dbt CLI commands and flags are supported - to learn more about using @dbt_assets, check out the dagster-dbt quickstart and reference.

Step 3: Using dagster-airlift to proxy execution​

Finally, you can use dagster-airlift to proxy the execution of the original task to Dagster. For more information, see "Migrating from Airflow to Dagster.