Can we use dbt to prepare data for ML models and LLM features?

dbt can absolutely be used to prepare data for ML models and LLM features, and it has become a common choice for the transformation layer specifically because it applies software engineering discipline, version control, testing and documentation, to SQL transformations that used to live in scattered, undocumented scripts. For classical ML, dbt models can compute engineered features directly in the warehouse, such as rolling aggregates and time-windowed statistics, with tests enforcing that a feature never contains unexpected nulls or out-of-range values before reaching a training pipeline or feature store. For LLM applications, dbt is useful upstream of retrieval, cleaning and structuring source data, deduplicating records, and joining metadata like access permissions before that data gets chunked and embedded, though dbt itself does not handle chunking or embedding since those need a Python pipeline outside its SQL-only execution model. dbt's built-in lineage graph also shows which upstream tables feed a given feature, supporting both debugging and the lineage requirements increasingly expected under AI governance. The main limitation is that dbt operates entirely within the warehouse, so any step requiring an external API call or unstructured document parsing needs a separate tool orchestrated alongside it. Nanobase AI integrates dbt as the transformation layer feeding both ML feature pipelines and LLM ingestion pipelines where a client already has a dbt-based warehouse.

Draw the boundary before building the pipeline

dbt earns its place in an ML or LLM pipeline for a specific reason: it brings software engineering discipline, version control, testing, documentation, lineage, to transformations that used to live in scattered, undocumented SQL scripts. The mistake is not using dbt for ML data preparation; it is trying to make dbt do work that requires calling an external API or processing unstructured content, which sits outside what a SQL-only execution model can do.

A concrete feature model

A rolling aggregate feature, common in classical ML, fits naturally as a dbt model with tests enforcing its validity before it ever reaches a training pipeline:

-- models/features/customer_30d_activity.sql
select
    customer_id,
    count(*) as event_count_30d,
    sum(amount) as total_amount_30d
from {{ ref('stg_events') }}
where event_ts >= dateadd('day', -30, current_date)
group by customer_id

A corresponding schema test enforces that customer_id is never null and event_count_30d is never negative, so a broken upstream join fails the dbt run rather than silently producing a corrupted feature that a training pipeline consumes without complaint.

Testing features the same way code gets tested

1. Add not-null and range tests to every feature column, catching upstream data issues before they reach a training job or feature store.
2. Add freshness tests on source tables, so a stalled upstream pipeline is caught as a dbt failure rather than discovered when a model trains on outdated data.
3. Document expected value ranges in the dbt model itself, so the lineage graph doubles as documentation an ML engineer can trust without asking the data team what a column actually means.
4. Run dbt tests in CI on every pull request touching a feature model, the same discipline applied to CI/CD for the model code itself.

The handoff point to feature stores and embedding pipelines

For classical ML, a validated dbt model is typically the direct input to a feature store, which then serves those features to both training and real-time inference. For LLM applications, dbt's role stops earlier in the chain: it cleans, deduplicates and joins metadata like access permissions onto source documents, and a separate Python-based pipeline takes that cleaned output, chunks it, calls an embedding model, and writes to a vector index. Treating that handoff point explicitly, rather than trying to extend dbt past it, keeps each tool doing what it is actually good at.

Frequently asked questions

Can dbt call an embedding API directly?

Not natively; dbt's execution model is SQL against a warehouse, so generating embeddings requires a separate Python step, typically orchestrated by a tool like Airflow or Dagster, that consumes dbt's cleaned output as its input.

Does dbt's lineage graph help with AI governance requirements?

Yes, dbt's built-in lineage showing which upstream tables feed a given feature or dataset supports the documentation and traceability increasingly expected under AI governance frameworks, though it typically needs to be paired with model-level lineage tracking to cover the full pipeline.

Is dbt worth adopting if we do not already have a warehouse-centric data stack?

Probably not as the first tool to introduce; dbt's value comes from applying engineering discipline to transformations that already run in a warehouse, so a team without a warehouse-centric stack should build that foundation first rather than adopting dbt in isolation.

How Nanobase AI helps

Nanobase AI integrates dbt as the transformation layer feeding both ML feature pipelines and LLM ingestion pipelines where a client already has a dbt-based warehouse, drawing a clear boundary with the Airflow, Dagster or Prefect orchestration handling everything dbt cannot.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.