DATABRICKS

A COMPLETE PROFILE

Here is a complete, working output using a personal access token — the auth method that works everywhere in DVT. Copy it, change the values, and dvt debug --all will tell you the moment it connects:

# ~/.dbt/profiles.yml
my_project:
  target: databricks
  outputs:
    databricks:
      type: databricks
      host: adb-1234567890123456.7.azuredatabricks.net   # no https://
      http_path: /sql/1.0/warehouses/abc123def456
      token: "{{ env_var('DATABRICKS_TOKEN') }}"
      catalog: main
      schema: analytics
      threads: 4

All three workspace values are a click away. host and http_path live under SQL Warehouses → your warehouse → Connection details — use the bare hostname, no scheme. The token comes from User Settings → Developer → Access tokens.

http_path may point at a SQL warehouse or an all-purpose cluster; a serverless SQL warehouse is the comfortable default — it wakes fastest and needs no cluster babysitting.

OAUTH — HONESTLY

dbt-databricks also speaks OAuth (auth_type: oauth, or a service principal's client_id and client_secret). In DVT that works when Databricks is your default target — the dbt lane handles it — but the data-movement lane cannot federate with it. An OAuth-only output is refused up front, with the fix in the message:

databricks: OAuth works in the dbt lane, but Sling federation
requires a PAT token — set token (a personal access token)
on this output

The practical guidance is simple: give the output a personal access token and every part of DVT works with it. For service accounts, Databricks lets a service principal own a PAT too — so CI can stay password-less without OAuth.

THREE LEVELS, ON PURPOSE

Unity Catalog addresses everything as catalog.schema.table — three levels, where most engines have two. DVT leans into that: the profile names the catalog and the schema separately, and both are required. Plain dbt would quietly fall back to hive_metastore when the catalog is missing; DVT asks you to state it, because in federation the three-part address is load-bearing — it is exactly how your tables are found.

READING DATABRICKS IN FEDERATION

To read Databricks inside federated models, point a source at the output via meta.connection. Sources spell the catalog as database — the standard dbt shape for three-part engines:

# models/sources.yml
sources:
  - name: lakehouse_bronze
    meta:
      connection: databricks    # the profiles.yml output above
    database: main              # the catalog
    schema: bronze
    tables:
      - name: raw_events
      - name: raw_users

The token's owner needs USE CATALOG and USE SCHEMA on whatever your sources name, plus SELECT on the tables — the usual Unity Catalog trio.

COLUMN NAMES DELTA WOULD REFUSE — HANDLED, NOT RENAMED

Delta tables reject column names containing spaces or special characters outright — a plain CREATE TABLE with a column literally named column 1 fails with DELTA_INVALID_CHARACTERS_IN_COLUMN_NAMES. DVT's rule is that the data's names are the data's names — renaming per engine is not on the table — so when a federated load carries such columns to Databricks, DVT routes it through a Parquet file in a Unity Catalog volume and creates the table with Delta column mappingenabled. The names land verbatim; the same model with the same column names works on Databricks exactly as it does everywhere else, and numeric precision is floored to what Delta's decimals actually hold rather than left to overflow.

WAREHOUSES THAT SLEEP — THE WAKE CLOCK

A SQL warehouse with auto-stop enabled suspends itself between runs, and the first requests inside its wake window come back as transient request errors — a wake takes minutes, not seconds. DVT knows the difference between that and a real failure: an operation against a cloud warehouse that fails with a transient request error retries on a warehouse clock — 60 seconds, then 120 — instead of giving up inside the wake window, while genuine engine refusals still surface on the first try. Local engines never take this branch; their errors reproduce and are reported immediately.

For long builds there's still a cheaper answer than retrying through wakes: raise the warehouse's auto-stop timeout while the work runs, and put it back after.

INCREMENTAL MODELS — THE FILE FORMAT DECIDES

Databricks accepts five strategies by name — append, merge, insert_overwrite, replace_where and delete+insert — plus microbatch when the installed dbt is 1.9.0b1 or newer, which is a version gate in the adapter, not a Databricks one. Leave incremental_strategy unset and you get merge: it is the adapter's explicit default, not a fallback that resolves elsewhere. Writing incremental_strategy='default' lands on merge too.

Which of those five you may use depends on file_format. merge needs delta or hudi. replace_where, microbatch and delete+insert need delta and nothing else. append and insert_overwrite run on any accepted format — text, csv, json, jdbc, parquet, orc, hive, delta, libsvm, hudi. Out of the box none of this constrains you: file_format defaults to delta for both catalog types — Unity Catalog and Hive metastore — so every strategy is legal until you set the format yourself. And table_format: iceberg narrows nothing either — it pins file_format to delta, refusing any other value with When table_format is 'iceberg', cannot set file_format to other than delta.

The key rule is the asymmetric one. delete+insert requires unique_key — checked at validation and required again at execution. merge does not require it, and that is the trap: with no unique_key the merge join predicate becomes the literal FALSE, so no row ever matches and every row is inserted. A keyless merge on Databricks is an append wearing a merge's name.

{{ config(
    materialized='incremental',
    incremental_strategy='merge',   -- append | merge | insert_overwrite
                                    -- | replace_where | delete+insert | microbatch
    file_format='delta',            -- the default; merge also allows hudi
    unique_key='order_id'           -- required by delete+insert;
                                    -- optional for merge, but a keyless merge
                                    -- matches nothing and inserts everything
) }}

select order_id, status, updated_at
from {{ ref('stg_orders') }}
{% if is_incremental() %}
where updated_at > (select max(updated_at) from {{ this }})
{% endif %}

Databricks Runtime version changes the SQL underneath a strategy without changing its name. delete+insert uses an efficient REPLACE ON only on DBR 17.1 and newer; below that it falls back to a DELETE followed by an INSERT, and that legacy INSERT only uses by name on DBR 12.2 and newer. replace_where — and therefore microbatch, which is replace_where underneath and inherits its Delta-only restriction — emits INSERT INTO ... BY NAME only on DBR 18.0 and newer, because emitting it on older clusters fails to parse. insert_overwrite sets dynamic partition-overwrite mode only when partition_by is configured, and only on a cluster; on a SQL warehouse it warns on every run: insert_overwrite is supported on SQL warehouses with DBR 17.1+. On older DBR versions, this strategy would be equivalent to using the table materialization. It warns — it does not refuse.

merge carries the widest config surface of any strategy on any DVT engine: target_alias, source_alias, merge_update_columns, merge_exclude_columns, merge_with_schema_evolution, skip_matched_step, skip_not_matched_step, matched_condition, not_matched_condition, and not_matched_by_source_action with its not_matched_by_source_condition — the last of which is only emitted when the action is delete or starts with update. Setting merge_update_columns and merge_exclude_columns together is refused: Model cannot specify merge_update_columns and merge_exclude_columns. Please update model to use only one config. Incremental models here support Python as well as SQL.

Databricks handles an unrecognised strategy name unusually — it warns rather than fails, because a custom strategy macro is a legitimate thing to ship:

WARNING - You are using an unsupported incremental strategy: upsert
You can ignore this warning if you are using a custom incremental strategy

The run then continues and, if no macro of that name exists, dies at lookup instead: dbt could not find an incremental strategy macro with the name "get_incremental_upsert_sql" in my_project. The refusals that genuinely stop a run are the two gates above — You can only choose this strategy when file_format is set to 'delta' and This strategy requires 'unique_key' to be configured.

When Databricks is the target of a federated f_incremental, the strategy no longer compiles into Databricks SQL — DVT computes the model in DuckDB and lands rows through its data-movement layer, so the strategy selects a load mode. Only append, merge and delete+insert mean anything there. insert_overwrite and microbatch are refused by name — DVT025, "has no Sling equivalent — f_incremental supports append, merge and delete+insert" — and replace_where, which no other DVT engine declares, isn't refused so much as unrecognised: it falls through as an unknown strategy and the model is rebuilt in full, with a warning. merge and delete+insert collapse into the same upsert on unique_key, and a missing unique_key quietly becomes a full refresh rather than an error.

One federated refusal is specific to this engine: a keyless f_incremental — one relying on a watermark instead of a unique_key — is rejected with DVT026 on Databricks, because the data-movement connector has not been verified to honour an update-key-only load here. That verification exists today for Postgres, Redshift, MySQL, DuckDB, SQLite, SQL Server, Oracle, ClickHouse and Trino; Databricks is still refused alongside StarRocks, Snowflake, BigQuery, Fabric and Athena. Give the model a unique_keyand it runs. What DVT will never do on Databricks is rewrite your config: the substitution map that swaps one strategy for another has exactly one entry in it, and it is ClickHouse's.

DRIVERS — ONE COMMAND

dvt sync reads your profiles.yml and installs what each output needs — for Databricks, the official dbt adapter and the databricks-sql-connector driver. Then dvt debug --all proves both lanes: the dbt connection and the federation lane, per output.

REFERENCE — EVERY FIELD

Everything a Databricks output can carry. The OAuth fields are listed for completeness — they work in the dbt lane only.