THE FEDERATION ENGINE
This is exactly what happens when you run a f_table model that joins tables living on different engines. No magic, no black box — six steps, each one inspectable.
DVT is a wrapper around stock dbt-core: dbt compiles your project and runs everything it can; DVT picks up the federation models dbt can't express and runs them through this pipeline.
COMPILE — DBT DOES WHAT DBT DOES
Your project compiles with dbt itself: Jinja renders, ref() and source()resolve, the manifest is built. DVT reads the compiled SQL from the manifest — it never reimplements dbt's compilation, which is why every dbt feature keeps working.
DECOMPOSE — FIND WHAT LIVES WHERE
The compiled query is parsed with SQLGlot. Every table reference is matched to a connection from your profiles.yml (via each source's meta.connection). Now DVT knows: these two tables are on MySQL, that one is on Snowflake, this ref() is a table my last run materialized on PostgreSQL.
TRANSPILE & PUSH DOWN — PER SOURCE, IN ITS OWN DIALECT
For each source engine, DVT builds the smallest possible extraction query in that engine's SQL dialect: only the columns the model actually uses, with every filter that can legally move to the source pushed into it. A Snowflake source gets Snowflake SQL, an Oracle source gets PL/SQL-flavored syntax — all generated from your one DuckDB-dialect model.
This is the single biggest performance lever — see the predicate pushdown deep dive.
EXTRACT — SLING MOVES DATA AS PARQUET
Each extraction query runs through Sling, landing results as Parquet files — columnar, compressed, fast. Extractions for different sources run in parallel. For f_incremental models, a watermark column limits extraction to new rows only, and previous extracts persist in the cache between runs.
COMPUTE — DUCKDB JOINS IT ALL LOCALLY
The Parquet files are then ingested into a local DuckDB database (.dvt/cache.duckdb) — deliberately one file at a time through the single cache connection, staying out of DuckDB's single-writer lock, with each ingest itself multi-threaded by DuckDB — each table namespaced per model so parallel models never collide. Your model's SQL — the actual JOIN/GROUP BY logic — executes here, on one of the fastest analytical engines that exists, using your machine's cores and memory.
LOAD — RESULTS LAND ON THE TARGET
The computed result goes back out through Sling to wherever the model materializes: the default target, a per-model config(target=...) override, or a cloud bucket as Parquet/CSV. Incremental strategies (append, merge, delete+insert) apply on the target. The output prints dbt-style — START / OK lines, rows, timings — because your tooling already understands that format.
THE SHORTCUT: SLING DIRECT (HOMOGENEOUS MODELS)
When every source in an f_table model lives on oneconnection and the target is a SQL engine, steps 4–6 collapse into a single hop: the whole query — joins included — is transpiled to that engine's dialect and submitted to Sling as a custom-SQL stream loaded straight to the target. No Parquet, no DuckDB. The check is automatic, and any failure falls back to the standard pipeline. Bucket targets and file-based sources always take the standard pipeline; heterogeneous models do too. The same model switches between paths purely based on where its sources live.
WHY THIS DESIGN WINS
- ▸Source engines only ever run simple filtered SELECTs — your OLTP databases are barely touched.
- ▸The heavy lifting happens in DuckDB on your hardware — no warehouse compute billed for the join.
- ▸Parallel where it pays: extractions and model waves run concurrently; ingest is serialized through DuckDB's single writer on purpose.
- ▸Models stay in one dialect (DuckDB) no matter how many engines they read from.