DVT METADATA

THE WALKTHROUGH — A COLUMN LANDS UPSTREAM

That 🔔 above is step one of a three-step story. Step two: ask status what the store now knows, any time, from any terminal — it reads the local store and touches no engine at all:

$ dvt metadata status
Source metadata — /Users/you/shop/.dvt/metadata.duckdb
  73 table(s) across 2 connection(s), 38 version(s) recorded
  lake_s3                      s3             61 table(s)   last seen 2026-08-03 09:12:44.183921
  pg_docker                    postgres       12 table(s)   last seen 2026-08-03 09:14:02.412881

1 table(s) changed since dvt last saw them:
  shop.public.orders → v2: +1 column

`dvt metadata propagate` plans what carrying these through would rebuild.

The change is pending: recorded, named, and not yet incorporated into your project. Step three is propagate — and propagate is strictly plan first, consent second. The plan pass costs one DuckDB read and one manifest load, no engine is touched, so --dry-run is free and safe on any project:

$ dvt metadata propagate --dry-run
Propagation plan
────────────────────────────────────────────────────────────────────
1 table(s) changed upstream:
  postgres on pg_docker in shop.public.orders → v2: +1 column

sources.yml edits (1):
  ✎ models/sources/pg_docker_sources.yml: pg_docker.orders — +1 column(s) (discount) — from metadata store v2

downstream models affected via the DAG (2):
  · stg_orders
  · fct_daily_revenue   [FULL REFRESH]

⚠ 1 INCREMENTAL model(s) will be FULLY REFRESHED — their existing target tables are rebuilt from scratch: fct_daily_revenue

would run: dvt run --select stg_orders fct_daily_revenue --full-refresh
would mark 1 version(s) incorporated (their highlights clear).
────────────────────────────────────────────────────────────────────
--dry-run: nothing was changed. Re-run without it to be asked for consent.

Everything propagate would do, before it does any of it: which declaration entries change, which downstream models are affected via the DAG, and — in capitals, because it is the one destructive part — which incremental models will be fully refreshed, their target tables rebuilt from scratch. That warning is the whole reason consent is required. Re-run without --dry-run and DVT prints the same plan, then asks:

$ dvt metadata propagate
Propagation plan
────────────────────────────────────────────────────────────────────
  … the same plan prints again …
────────────────────────────────────────────────────────────────────
Each edited yml is backed up beside itself as a timestamped .bak. The file is re-rendered from its parsed contents, so comments below the header do not survive — the .bak has the original.
Carry these changes through — rewrite the declarations, FULLY REFRESH the incremental model(s) listed and rebuild? [y/N] y

  … dvt run streams its own output here …

Propagation result
────────────────────────────────────────────────────────────────────
wrote models/sources/pg_docker_sources.yml (backup pg_docker_sources.yml.20260803092144.bak)
dvt run --select stg_orders fct_daily_revenue --full-refresh — succeeded
1 version(s) marked incorporated — their highlights are cleared

Order is deliberate: declarations first, then the rebuild, then incorporation. If the rebuild fails, nothing is marked incorporated — the next dvt metadata status still lists the change as pending, so a half-done propagation can never look finished. Answer n at the prompt and DVT says Aborted — nothing was changed, and the versions stay pending too.

REFRESH IN DETAIL — KNOWLEDGE, NEVER PROPOSALS

Run bare, refresh lists your connections and stops — it never guesses which one you meant:

$ dvt metadata refresh
Pick a connection:
  lake_s3                      s3
  pg_docker                    postgres

Usage: dvt metadata refresh <connection>   (or --all)

--all sweeps every connection in the profile. Each line leads with a marker: 🔔 means the store recorded something new (a version, a dropped table), ✅ means it checked and found nothing — and it says so out loud, because a run that found nothing is still an answer:

$ dvt metadata refresh pg_docker
✅ pg_docker                    3 catalog queries, 12 table(s)
   no source-metadata change on pg_docker (12 table(s) checked)

sources.yml was not touched — `dvt metadata propagate` carries changes into your declarations (with consent).

Refresh is the same capture that dvt generate-sources performs as a side product of walking a connection — minus the proposal. generate-sources writes declarations; refresh only writes knowledge. That split is the point of having two commands: checking on upstream drift should never be able to rewrite a file you maintain.

BUCKETS — PROBE UNDER A STATED BUDGET

A bucket connection has no catalog to query. The only way to learn an object's shape is to read the object — one round-trip per file — so refresh probes up to a stated budget (default 25 objects, raise it with --probe N) and always says exactly how many of how many it opened:

$ dvt metadata refresh lake_s3 --probe 40
✅ lake_s3                      40 object(s) read, 61 object(s) known
   no source-metadata change on lake_s3 (61 table(s) checked)
   21 relation(s) on lake_s3 recorded by name only — no column shape was read for them, so none is claimed
   probed 40 of 61 object(s) on lake_s3 (budget 40, raise it with --probe N or DVT_METADATA_PROBE_BUDGET); 21 not opened this run

sources.yml was not touched — `dvt metadata propagate` carries changes into your declarations (with consent).

What the budget did not cover is recorded by name only, never guessed at — an object without a probed shape has no version at all in the store, so nothing downstream can read a shape that was never learned. The probe order is deterministic, so "the first N" means the same objects on every run. --probe 0 records every object by name and opens none; the DVT_METADATA_PROBE_BUDGET environment variable sets the budget where there is no flag to pass. The very first sweep of a bucket prints one first capture version line per object it opens — that is the store learning, not the bucket changing.

The probing itself goes through DVT's own file read lane, so only the four file formats DVT reads at all — csv, parquet, json, jsonl — are ever opened. An object in any other format is recorded by name, honestly, as something DVT does not read.

STATUS IN DETAIL — THE STORE ANSWERS, LOCALLY

status never touches an engine: it reads .dvt/metadata.duckdb and reports. On a project that has never captured anything, that too is an answer:

$ dvt metadata status
This project has no source-metadata store yet.
Build it with `dvt metadata refresh --all` (or any `dvt generate-sources` run).

Pass a connection name (dvt metadata status pg_docker) to filter the per-connection lines and the pending list to just that connection. When every version the store knows has been carried through, status says No pending changes — every version dvt knows is incorporated.

One more thing status watches for: DVT's own abandoned staging tables — leftovers from an interrupted run. They are flagged from a marker DVT itself minted, recorded and named, and never dropped automatically:

1 DVT staging leftover(s) flagged (recorded, never dropped — dropping needs the intent journal and a dead owner):
  shop.public.dvt_stg_9f3c21ab04de — DVT's own staging marker (dvt_stg_) in public — DVT names its staging opaquely, so this is a load of ours that did not clean up after itself; dropping it needs the intent journal and a dead owner, never the name alone

PROPAGATE IN DETAIL — CONSENT, REFUSAL, AND HONEST EDGES

Two flags narrow the plan: a connection argument scopes it to one connection's changes, and --select takes table patterns — a schema, a schema.table, or a bare table name, wildcards welcome (sales, sales.*, *.orders). --yes skips the consent prompt for scripts — the plan still prints, so your CI log shows exactly what was agreed to.

The sources.yml edits deserve a word. Generated declarations are minimal — names only — so on those the honest plan line is sources.yml edits: none: there is nothing in them to update, and the store carries the column knowledge. But a source you chose to document with a columns: list is a claim about the table, and propagate brings that claim back in line — adding what appeared, dropping what is gone, refreshing data_typewhere the engine's own word changed. Your descriptions and tests are kept: propagate carries the shape, never your documentation. Every edited file gets a timestamped .bak beside it first.

Now the hard case. Suppose the upstream change was a dropped column that one of your models still reads. Propagate does not rebuild a model it knows the change broke — it refuses it, names it, and shows its evidence:

$ dvt metadata propagate pg_docker --dry-run
Propagation plan
────────────────────────────────────────────────────────────────────
1 table(s) changed upstream:
  postgres on pg_docker in shop.public.orders → v3: -1 column

sources.yml edits (1):
  ✎ models/sources/pg_docker_sources.yml: pg_docker.orders — -1 column(s) (coupon_code) — from metadata store v3

downstream models affected via the DAG (2):
  · stg_orders   [REFUSED]
  · fct_daily_revenue   [REFUSED]

✖ 1 model(s) this change BREAKS — refused, never rebuilt:
  ✖ stg_orders reads 'coupon_code' from pg_docker:shop.public.orders, which was DROPPED upstream — the model's own SQL names it (word-boundary match on raw_code — a text match, not a parse)
  ✖ also refused because they sit downstream of a broken model: fct_daily_revenue
  → fix them first (Martin proposes the SQL repair behind its own APPLY), then run propagate again.

would run: nothing — no model is rebuildable from this change.
would LEAVE 1 version(s) pending — a model they feed was refused, so the change was not carried through and the highlight stays.
────────────────────────────────────────────────────────────────────
--dry-run: nothing was changed. Re-run without it to be asked for consent.

Three things to notice. The refusal cascades: a model downstream of a broken one is refused too, because rebuilding it would run the broken one. The evidence is stated as exactly what it is — a word-boundary text match on the model's SQL, not a parse — so a false positive is arguable rather than mysterious. And a version that feeds a refused model is left pending: the change was not carried through, so the store keeps saying so until it truly is. Fix the model (Martin, DVT's assistant, proposes the SQL repair behind its own consent gate), run propagate again, and the versions are still there waiting.

Two more honest edges. If the pending changes touch tables no model reads, propagate says Nothing to execute — the changes stay recorded and status keeps listing them until a model depends on them. And if you edited model files since the last parse, propagate re-parses first — Model files changed since the last parse — re-parsing so the plan describes the project as it is now — because a plan built on a stale manifest would name the wrong models. A project that does not parse gets a refusal, not a guess.

EXPORT — THE CATALOG AS A REPORT

export writes the store out for humans: csv, xlsx or json, four datasets in each — Sources, Columns, Constraints and Changes. CSV writes four files into a directory; XLSX writes one workbook with four sheets; JSON writes one document keyed connection → schema → table:

$ dvt metadata export -f xlsx -o catalog.xlsx
wrote /Users/you/shop/catalog.xlsx
  Sources: 73 row(s), Columns: 214 row(s), Constraints: 22 row(s), Changes: 19 row(s)
  scope: the whole store, current version per table (--from/--to widen it to a version range, --since bounds the changes)

A word on those three formats, because the wording matters: they are reportformats — "give me the catalog as a spreadsheet" — not part of DVT's data-file lane. DVT moves data in exactly four file formats (csv, parquet, json, jsonl); the export is a local file a person opens, it cannot be pointed at a bucket, and DVT never reads an export back in. XLSX exists here precisely because it is something a human opens, and nowhere else in DVT.

Without flags you get the whole store at the current version per table, into a timestamped default location:

$ dvt metadata export
wrote /Users/you/shop/exports/metadata_20260803093012/sources.csv
wrote /Users/you/shop/exports/metadata_20260803093012/columns.csv
wrote /Users/you/shop/exports/metadata_20260803093012/constraints.csv
wrote /Users/you/shop/exports/metadata_20260803093012/changes.csv
  Sources: 73 row(s), Columns: 214 row(s), Constraints: 22 row(s), Changes: 19 row(s)
  scope: the whole store, current version per table (--from/--to widen it to a version range, --since bounds the changes)

Because the store keeps versions, the export can answer "what changed between v1 and v4" in one file: --from/--to bound the version numbers per table, --since bounds the changes by detection time, and --connection/--table narrow the scope. The scope is always printed back, so the file says what it contains:

$ dvt metadata export -f json --from 1 --to 4 --connection pg_docker
wrote /Users/you/shop/exports/metadata_20260803093155.json
  Sources: 12 row(s), Columns: 96 row(s), Constraints: 22 row(s), Changes: 19 row(s)
  scope: connection=pg_docker; versions v1..v4 per table

An empty store is an answer too: the export is still written, headers and no rows, and DVT says the metadata store is EMPTY — the export was written with headers and no rows, so it is a truthful report of an empty store rather than a missing file. A date --since cannot read is refused by name (YYYY-MM-DD or YYYY-MM-DD HH:MM:SS) — never silently treated as "everything".

REFERENCE — EVERY VERB AND FLAG

All four verbs accept --project-dir (default .); refresh, status and propagate also accept --profiles-dir. Everything else: