SOURCES.YML

ONE EXTRA LINE: META.CONNECTION

dbt assumes every source lives on the one warehouse it's connected to, so its sources.yml never has to say where data is. DVT talks to many engines at once, so it has to ask. connection: is the answer: it names the output in profiles.yml where this data actually lives.

It sits under the source's meta:block because that is dbt's sanctioned home for custom keys — plain dbt reads the file happily and simply ignores it.

A FULLER EXAMPLE

One file can mix external engines, bucket storages, and tables on your default target — each source simply says where it lives:

# models/staging/sources.yml
version: 2

sources:
  # ─── External Sources (need connection:) ───

  - name: crm
    description: "CRM data from Oracle operational database"
    meta:
      connection: oracle_ops              # ← REQUIRED: points to profiles.yml
    tables:
      - name: customers
        description: "Customer master data"
      - name: orders
        description: "Sales orders"
      - name: products

  - name: warehouse
    description: "ERP data from Snowflake"
    meta:
      connection: sf_warehouse           # ← REQUIRED: points to profiles.yml
    database: PROD_DB
    schema: RAW
    tables:
      - name: invoices
      - name: shipments
      - name: vendors

  - name: data_lake
    description: "Raw files from S3"
    meta:
      connection: data_lake              # ← works with bucket storages too
    tables:
      - name: web_events
      - name: app_logs

  # ─── Default Target Sources ───

  - name: analytics
    description: "Tables on the default PostgreSQL target"
    schema: public
    meta:
      connection: pg_dev                 # ← recommended even here: dbt ignores
    tables:                              #   it, and a future target switch is free
      - name: seed_categories
      - name: dim_dates
      - name: historical_metrics

HOW DVT READS SOURCES

How a model runs is decided by the model — its materialized config — never by the sources it reads. What meta.connection decides is where a federated model extracts each source from:

The rule of thumb: a standard model should only read sources that live on the default target. The moment a model needs data from another engine, declare it federated yourself — DVT will coerce it for you rather than fail, but an explicit materialization states your intent and silences the warning. See f_table on the federated materializations page.

COMMON PATTERNS

Three shapes cover most projects.

Multi-source extraction model:

-- models/marts/cross_engine_report.sql
-- crm is on Oracle, warehouse is on Snowflake
-- Both get extracted → DuckDB joins them → result loads to default target

{{ config(materialized='f_table') }}

SELECT
    c.customer_name,
    i.invoice_date,
    i.total_amount
FROM {{ source('crm', 'customers') }} c          -- Oracle (extracted)
JOIN {{ source('warehouse', 'invoices') }} i      -- Snowflake (extracted)
    ON c.customer_id = i.customer_id

Default-target pushdown model:

-- models/staging/stg_categories.sql
-- analytics source lives on the default target → pushdown, no extraction
-- SQL dialect: PostgreSQL (native)

{{ config(materialized='view') }}

SELECT id, name, parent_id
FROM {{ source('analytics', 'seed_categories') }}

Bucket storage source:

-- models/staging/stg_web_events.sql
-- data_lake source points to S3 → extracted via Sling

{{ config(materialized='f_table') }}

SELECT event_id, user_id, event_type, event_timestamp
FROM {{ source('data_lake', 'web_events') }}

LET DVT WRITE IT — DVT GENERATE-SOURCES

Nobody enjoys transcribing an information schema by hand. dvt generate-sources <connection>reads the connection's live catalog and writes <model-root>/<connection>_sources.yml in exactly the canonical shape shown above — meta.connection on every source. No AI anywhere: it is a metadata query and a renderer.

dvt generate-sources                        # no argument: lists your connections
dvt generate-sources pg_docker              # one connection → pg_docker_sources.yml
dvt generate-sources --all                  # sweep the whole profile, one yml each
dvt generate-sources sf_dev -s analytics.*              # only these schemas/tables
dvt generate-sources sf_dev -s sales '*.orders' -x '*.tmp_*'
dvt generate-sources oracle_docker --print              # stdout only, write nothing
dvt generate-sources dbx_dev -o models/staging/dbx.yml
dvt generate-sources s3_lake -s 'raw/*.parquet'         # buckets work too

Bucket connections (S3, GCS, Azure, SFTP) are first-class here too. They have no information schema, so DVT lists the connection's objects and declares its data files as file sources instead. The -s/-x patterns match the object path (raw/*.csv, **/*.parquet) rather than schema.table.

Every run also records what it learned about each table — columns, types, constraints — into the project's source-metadata store. The dvt metadata family reads and maintains that store, and the data catalog is its visual sibling. The full command walkthrough lives on the dvt generate-sources page.

FROM SOURCES TO BRONZE — DVT GENERATE-BASE-VIEWS

Once sources are declared and captured, dvt generate-base-views takes the next step for you: one bronze model per source table, written to models/bronze/<connection>/bronze_<table>.sql. Each model selects the exact columns the source-metadata store recorded, in the engine's own spelling — zero engine traffic and zero AI.

dvt generate-base-views                       # everything the store knows
dvt generate-base-views --select-connection pg_docker
dvt generate-base-views --select-table orders customers
dvt generate-base-views --dry-run             # print the plan, write nothing

The materialization is chosen by physics, not preference: a live view when the source lives on the default target, federated_table (the long spelling of f_table) when it doesn't — a plain view cannot cross engines. The generated header states which was chosen and why.

Nothing is overwritten silently: existing files are reported as skipped, and --force replaces them while keeping a timestamped .bak. The generated models reference declared sources — run dvt parse to check them, then dvt run --select bronze_* to build.

COMMON MISTAKES

sources:
  - name: analytics
    connection: pg_dev           # ← top level: dbt REJECTS this
    tables:
      - name: dim_dates

sources:
  - name: analytics
    meta:
      connection: pg_dev         # ← under meta: dbt allows, DVT reads
    tables:
      - name: dim_dates

sources:
  - name: crm
    # Missing connection!
    tables:
      - name: customers        # lives on Oracle

sources:
  - name: crm
    meta:
      connection: oracle_ops      # external source
    tables:
      - name: customers

REFERENCE: SOURCE PROPERTIES

This page is the working guide. For the full file anatomy — every accepted key, with dbt-compatibility notes — see the sources.yml reference.