THE FEDERATION ENGINE
You've written an f_tablemodel that joins tables living on different engines, and you'd like to know what actually happens when you run it. Fair question — here's the answer: six steps, no magic, each one inspectable.
DVT is a wrapper around stock dbt-core: dbt compiles your project and runs everything it can; DVT picks up the federation models dbt can't express and runs them through this pipeline.
We'll follow one model through the pipeline — two engines, one join:
{{ config(materialized='f_table') }}
select o.order_id, o.amount, c.region
from {{ source('oracle_crm', 'orders') }} o -- Oracle
join {{ source('sf_finance', 'customers') }} c -- Snowflake
on o.customer_id = c.customer_idCOMPILE — DBT DOES WHAT DBT DOES
Your project compiles with dbt itself: Jinja renders, ref() and source() resolve, the manifest is built.
DVT reads the compiled SQL from the manifest. It never reimplements dbt's compilation — which is exactly why every dbt feature keeps working.
DECOMPOSE — FIND WHAT LIVES WHERE
The compiled query is parsed with SQLGlot. Every table reference is matched to a connection from your profiles.yml (via each source's meta.connection in sources.yml).
Now DVT knows: orders is on Oracle, customers is on Snowflake, and that ref() over there is a table my last run materialized on PostgreSQL.
TRANSPILE & PUSH DOWN — PER SOURCE, IN ITS OWN DIALECT
For each source engine, DVT builds the smallest possible extraction query in that engine's SQL dialect: only the columns the model actually uses, with every filter that can legally move to the source pushed into it. The Snowflake source gets Snowflake SQL, the Oracle source gets PL/SQL-flavored syntax — all generated from your one DuckDB-dialect model.
This is the single biggest performance lever — see the predicate pushdown deep dive.
EXTRACT — SLING MOVES DATA AS PARQUET
Each extraction query runs through Sling, landing results as Parquet files — columnar, compressed, fast. Extractions for different sources run in parallel.
For f_incremental models, a watermark column limits extraction to new rows only, and previous extracts persist in the cache between runs.
COMPUTE — DUCKDB JOINS IT ALL LOCALLY
The Parquet files are ingested into a local DuckDB database (.dvt/cache.duckdb), and your model's SQL — the actual JOIN / GROUP BY logic — executes there, on one of the fastest analytical engines that exists, using your machine's cores and memory.
Two details worth knowing: files are ingested one at a time through the single cache connection — deliberately staying out of DuckDB's single-writer lock, while each individual ingest is still multi-threaded by DuckDB — and every table is namespaced per model, so parallel models never collide.
LOAD — RESULTS LAND ON THE TARGET
The computed result goes back out through Sling to wherever the model materializes: the default target, a per-model config(target=...) override, or a cloud bucket as Parquet/CSV. Incremental strategies (append, merge, delete+insert) apply on the target.
The output prints dbt-style — START / OK lines, rows, timings — because your tooling already understands that format.
THE SHORTCUT: SLING DIRECT (HOMOGENEOUS MODELS)
When every source in an f_table model lives on oneconnection and the target is a SQL engine, steps 4–6 collapse into a single hop. The whole query — joins included — is transpiled to that engine's dialect and handed to Sling as a custom-SQL stream, loaded straight to the target. No Parquet. No DuckDB.
The check is automatic, and any failure falls back to the standard pipeline. Bucket targets and file-based sources always take the standard pipeline; heterogeneous models do too. The same model switches between paths purely based on where its sources live.
WHY THIS DESIGN WINS
- ▸Source engines only ever run simple filtered SELECTs — your OLTP databases are barely touched.
- ▸The heavy lifting happens in DuckDB on your hardware — no warehouse compute billed for the join.
- ▸Parallel where it pays: extractions and model waves run concurrently; ingest is serialized through DuckDB's single writer on purpose.
- ▸Models stay in one dialect (DuckDB) no matter how many engines they read from.