DVT GENERATE-SOURCES

POINT IT AT A CONNECTION

Name any output from your profiles.ymland run it. Here's a Postgres connection called pg_docker:

$ dvt generate-sources pg_docker
pg_docker (postgres): 12 table(s) across 3 schema(s)
excluded 4 dvt-managed relation(s) (2 from ledger)
excluded 0 DVT staging leftover(s) — recognized by DVT's own marker, recorded in the metadata store, never dropped
   new version detected in postgres on pg_docker in dvt.public.customers: first capture
   new version detected in postgres on pg_docker in dvt.public.orders: first capture
   new version detected in postgres on pg_docker in dvt.sales.invoices: first capture
   ...
Wrote models/pg_docker_sources.yml

Four things just happened, and every one of them is stated. DVT walked the connection's information schema and found 12 tables across 3 schemas. It excluded 4 relations that DVT itself materialized — your models are not your sources, so they never re-enter the file (the count is printed even when it's zero; DVT never does this silently). It checked for its own abandoned staging leftovers and found none. And the indented lines are a bonus: the introspection you just paid for is kept— column shapes and types land in the project's metadata store (.dvt/metadata.duckdb), one line per table on a first capture.

Don't remember what your connections are called? Run it bare and it lists them:

$ dvt generate-sources
Pick a connection:
  minio_bucket                 s3
  pg_docker                    postgres
  sf_dev                       snowflake

Usage: dvt generate-sources <connection> [-s PATTERN ...] (or --all)

This works on every engine DVT connects to — postgres, mysql, sqlserver, oracle, snowflake, bigquery, databricks, redshift, clickhouse, trino, athena, vertica, duckdb, sqlite — and on bucket connections too, which get their own treatment below.

THE FILE IT WRITES

models/pg_docker_sources.yml is canonical DVT sources — grouped by schema, sorted, meta.connection on every source (one rule, no exceptions), and a header that repeats what the run told you:

# generated by /generate-sources from connection 'pg_docker' (postgres)
# 12 table(s) across 3 schema(s) — prune what you don't need
# excluded 4 dvt-managed relation(s) (2 from ledger) — dvt materializations never re-enter as sources (ruling 88)
# excluded 0 DVT staging leftover(s) — recognized by DVT's own opaque marker, recorded in the metadata store, never dropped (--select one to declare it anyway)
version: 2
sources:
- name: pg_docker__ops
  meta:
    connection: pg_docker
  database: dvt
  schema: ops
  tables:
  - name: shipments
  - name: warehouses
- name: pg_docker__public
  meta:
    connection: pg_docker
  database: dvt
  schema: public
  tables:
  - name: customers
  - name: orders
  - name: payments
- name: pg_docker__sales
  meta:
    connection: pg_docker
  database: dvt
  schema: sales
  tables:
  - name: invoices
  - name: invoice_lines

Source names are prefixed with the connection — pg_docker__public, not public— so generated names can never collide with sources you already have. From here it's ordinary DVT: {{ source('pg_docker__sales', 'invoices') }} in any model, on any target. The header says it and we'll say it too: prune what you don't need. The file is yours now.

RUN IT AGAIN — NOTHING IS EVER CLOBBERED

The warehouse grew? Run the same command again. The file is regenerated from a fresh walk, and the previous version is kept beside it as a timestamped .bak — re-running is always safe:

$ dvt generate-sources pg_docker
pg_docker (postgres): 13 table(s) across 3 schema(s)
excluded 4 dvt-managed relation(s) (2 from ledger)
excluded 0 DVT staging leftover(s) — recognized by DVT's own marker, recorded in the metadata store, never dropped
   new version detected in postgres on pg_docker in dvt.sales.refunds: first capture
Wrote models/pg_docker_sources.yml
Previous version kept at models/pg_docker_sources.yml.20260803094012.bak

Notice the metadata lines earn their keep on re-runs: only refunds is new, so only refunds gets a line. A run where nothing changed says exactly that — no source-metadata change on pg_docker (13 table(s) checked) — because a zero-change run states its zero too.

WHAT NEVER ENTERS THE FILE

The exclusion line is doing real work. DVT keeps a materialization ledger — every relation DVT itself has landed, through either federation lane, the dbt phase, seeds, even test-failure tables — and generate-sourceschecks every discovered table against it, plus your manifest's model, seed and snapshot names and aliases, plus seed file stems on disk. Anything DVT made is dropped from the proposal, on every engine, with the count stated:

excluded 4 dvt-managed relation(s) (2 from ledger)

Why so strict? Because a model that re-enters as a source is a circle: you'd be declaring your own output as an input, and one sweep later your DAG eats its tail. DVT also recognizes its own abandoned staging tables by an opaque marker it wrote — those are excluded and recorded, never dropped, and if you genuinely want one declared, naming it with --select overrides the exclusion. If everything on a connection turns out to be dvt-managed, the command refuses with the full count rather than writing an empty file.

NARROWING THE SWEEP WITH -S AND -X

You rarely want all 200 tables. -s/--select keeps only what matches; -x/--exclude drops what matches. A pattern covers a schema or a schema.table pair, case-insensitively, with shell-style wildcards — sales, sales.*, *.orders:

$ dvt generate-sources sf_dev -s analytics.*
sf_dev (snowflake): 23 table(s) across 1 schema(s)
excluded 0 dvt-managed relation(s) (0 from ledger)
excluded 0 DVT staging leftover(s) — recognized by DVT's own marker, recorded in the metadata store, never dropped
   ...
Wrote models/sf_dev_sources.yml

Both flags take several patterns, and they compose — keep two shapes of thing, then carve out the junk:

$ dvt generate-sources sf_dev -s sales '*.orders' -x '*.tmp_*'

A selection that matches nothing is an error that teaches, not an empty file:

generate-sources: the selection matched no tables on 'sf_dev' — patterns cover a schema or schema.table (wildcards ok: sales, sales.*, *.orders)

LOOK BEFORE YOU WRITE: --PRINT AND -O

--print sends the YAML to stdout and writes nothing — the counts and exclusions print exactly as before, then the file content follows. Good for a first look, or for piping into whatever you like:

$ dvt generate-sources oracle_docker --print

-o/--output writes somewhere other than the default <model-root>/<connection>_sources.yml. The path must stay inside your project's model-paths — writes are confined there, no escapes:

$ dvt generate-sources dbx_dev -o models/staging/dbx.yml

--ALL: THE WHOLE PROFILE IN ONE SWEEP

--all runs the same generation for every connection in the profile — one yml per connection, one ledger line per connection, and a failure on one never aborts the rest:

$ dvt generate-sources --all
✅ minio_bucket                 7 table(s) across 1 schema(s) · excluded 1 dvt-managed relation(s) (1 from ledger), 0 DVT staging leftover(s) → models/minio_bucket_sources.yml
✅ pg_docker                    13 table(s) across 3 schema(s) · excluded 4 dvt-managed relation(s) (2 from ledger), 0 DVT staging leftover(s) → models/pg_docker_sources.yml (previous kept at models/pg_docker_sources.yml.20260803094012.bak)
❌ sf_dev                       metadata query failed: incorrect username or password was specified

1 connection(s) failed — see the ledger above.

Database connections walk their information schema; bucket connections list their objects and declare the data files. The exit code is honest: nonzero if any connection failed, zero only when the whole profile generated. -s and -x apply across the sweep; -o doesn't compose with --all (one path can't hold many files) and the command says so instead of guessing.

BUCKETS BECOME FILE SOURCES

A bucket connection — s3, gcs, azure, sftp, or a local folder — has no information schema. So generate-sources lists its objects recursively, keeps the data files, and declares each one as a source table whose name is its path, relative to the connection's root. That's exactly the shape DVT's federation engine streams — no load step, no external-table ceremony:

$ dvt generate-sources minio_bucket
minio_bucket (s3): 7 table(s) across 1 schema(s)
excluded 1 dvt-managed relation(s) (1 from ledger)
excluded 0 DVT staging leftover(s) — recognized by DVT's own marker, recorded in the metadata store, never dropped
   ...
Wrote models/minio_bucket_sources.yml

The generated file declares one source, tables named by object path:

# generated by /generate-sources from connection 'minio_bucket' (s3)
# 7 data file(s) discovered, 2 non-data object(s) ignored — table names are object paths, streamed live
# excluded 1 dvt-managed relation(s) (1 from ledger) — dvt materializations never re-enter as sources (ruling 88)
version: 2
sources:
- name: minio_bucket__files
  meta:
    connection: minio_bucket
  tables:
  - name: raw/customers.csv
    quoting:
      identifier: true
  - name: raw/events/2026-07.jsonl
    quoting:
      identifier: true
  - name: raw/orders.parquet
    quoting:
      identifier: true

The quotingblock is deliberate: a path isn't a plain SQL identifier, and quoting it makes the generated source compile the same on every engine instead of depending on which adapter is your default. Selecting from one is ordinary: {{ source('minio_bucket__files', 'raw/orders.parquet') }} — the object streams live at run time. On buckets, -s/-x patterns match the object path instead of schema.table:

$ dvt generate-sources s3_lake -s 'raw/*.parquet'

File formats are DVT's four and exactly four: csv, parquet, json, jsonl (ndjson is an accepted spelling of jsonl). Objects in any other tabular format are refused by namein the file's header — a # REFUSED, unsupported file format: line listing them — never silently skipped; everything else (readmes, manifests) is just counted as non-data and ignored. And ruling-88 exclusion applies here too: a model or seed DVT landed as an object never re-enters as a source file.

REFERENCE — EVERY FLAG

One positional argument, six flags. Patterns are shell-style (fnmatch) wildcards, matched case-insensitively.