[v2 Bug] BigQuery: a failed dataset listing is cached as empty and every incremental model in the run is recreated with `create or replace table`

[matang-odd]

Is this a new bug in dbt v2.x compared to the latest version of dbt 1.x?

• I believe this is a new bug in dbt v2.x
• I have searched the existing issues and could not find a duplicate

Current Behavior

Two dbt runs sharing a BigQuery dataset are enough to trigger it. Run B drops one of its __dbt_tmp tables while run A's relation-cache pre-warm is listing the dataset. The ADBC driver lists the tables and then fetches each table's metadata; the dropped table returns 404, the driver fails the whole listing, Fusion classifies the error as not-found and caches the dataset as empty and complete. is_incremental() is then false for every incremental model in run A, and each one is recreated with create or replace table <model> from its full sources. No merge, no warning, exit code 0.

Where, on dbt-labs/dbt main as of 2026-09-15:

1. hydrate_relation_cache (crates/dbt-adapter/src/adapter/mod.rs:369-391) keeps only the Ok listings (filter_map, commented "XXX: Warnings are not shown right now") and insert_schema stores each as is_complete: true (cache.rs:158-175). This is the "downloading relations" phase.
2. The driver's GetTablesForDBSchema (go/connection.go:141-172) calls Table.Metadata() for every listed table and returns an error for the whole schema when one returns 404. Upstream skips only a 403 (feat(go): skip tables in GetObjects that 403 adbc-drivers/bigquery#296); the dbt Labs fork skips nothing. errToAdbcErr renders the 404 as ... 404 Not Found: Not found: Table ... with StatusNotFound.
3. list_relations_in_parallel_inner (crates/dbt-adapter/src/metadata/bigquery/mod.rs:1427-1465) maps any error accepted by is_bigquery_not_found_error (:1677) to Ok(Vec::new()). The predicate's doc comment names this exact driver error as one it is meant to match. verify_empty_adbc_listing (:166-184) re-checks an empty listing only when is_cross_project.
4. Each model's get_relation misses in the complete-but-empty schema, runs the routines fallback (one INFORMATION_SCHEMA.ROUTINES query per model when the project has functions) and returns none. The incremental materialization takes the existing_relation is none branch.

Expected Behavior

• A failed listing is not cached as a complete, empty schema. Re-verify through INFORMATION_SCHEMA.TABLES as the cross-project path does, retry, or leave the schema out of the pre-warm so get_relation does its own lookup.
• A warning when an incremental model's table is missing from the cache, and no create or replace without --full-refresh unless the absence is confirmed ([Feature] Safer incrementals/snapshots (don't allow create or replace ...) #15953).
• Driver: skip a table whose metadata get returns 404, as feat(go): skip tables in GetObjects that 403 adbc-drivers/bigquery#296 does for 403, or fail with an error scoped to the table rather than the dataset. Filed separately at adbc-drivers/bigquery.

Steps To Reproduce

1. profiles.yml: a bigquery target, method: oauth, a dev dataset.
2. One table and four identical incremental models:

-- models/source_rows.sql
{{ config(materialized="table") }}
select id, current_timestamp() as loaded_at from unnest(generate_array(1, 100)) as id

-- models/inc_a.sql (inc_b, inc_c, inc_d identical)
{{ config(materialized="incremental", unique_key="id", incremental_strategy="merge") }}
select id, loaded_at from {{ ref("source_rows") }}
{% if is_incremental() %}
    where id > (select coalesce(max(id), 0) - 10 from {{ this }})
{% endif %}

3. dbt build --target dev, then dbt run --target dev --select inc_a inc_b inc_c inc_d: four merges, no CTAS.
4. In a second shell, create and drop tmp-shaped tables in the same dataset as fast as the API allows:

import itertools
from google.cloud import bigquery
client = bigquery.Client(project="<project>")
schema = [bigquery.SchemaField("id", "INTEGER")]
for i in itertools.count():
    name = f"<project>.<dataset>.churn_{i % 6}__dbt_tmp{i:06d}"
    client.create_table(bigquery.Table(name, schema=schema), exists_ok=True)
    client.delete_table(name, not_found_ok=True)

5. Repeat the dbt run from step 3 twenty to twenty-five times with --log-level debug.

Results, one row per invocation in INFORMATION_SCHEMA.JOBS_BY_USER:

engine	runs	recreated all four (create or replace table inc_*, 0 merges)
dbt-fusion 2.0.0-preview.218	25	6
dbt-fusion 2.0.0-preview.218, fresh dataset	25	12
dbt 2.0.2	20	12
dbt-core 1.12.4 + dbt-bigquery 1.12.0 (control, same churn)	15	0

Every hit recreated all four models, one listing for the run. Every run exited 0. The hit rate depends on where the dropped table sorts, since the driver fetches metadata in tables.list order.

Relevant log output

# A good run and a bad run print the same at --log-level debug:
  Started downloading relations (1 items)
  Finished [  2.24s] downloading relations (1 items)
# A bad run then emits, per model, with no merge:
create or replace table `<project>`.`<dataset>`.`inc_c`
create or replace table `<project>`.`<dataset>`.`inc_d`
create or replace table `<project>`.`<dataset>`.`inc_a`
create or replace table `<project>`.`<dataset>`.`inc_b`

Environment

- OS: macOS 26.6 (reproduction); Linux x86_64 in an ECS container (production)
- CPU: ARM (reproduction), x86 (production)
- dbt distribution and version: `dbt-fusion 2.0.0-preview.218` in production; reproduces on `dbt 2.0.2`

Which database adapter are you using?

bigquery

Is this a discrepancy vs. dbt 1.x?

• Yes — this works in dbt 1.x but not in dbt v2.x

Additional Context

• Impact: 12 recreations of a six-model chain in two weeks, about 870 GiB and $5.3 each. Fingerprint in JOBS: a burst of INFORMATION_SCHEMA.ROUTINES queries against the models' dataset right before the CTAS statements, where a normal run issues one.
• Since d26a967 (2026-08-04), which moved list_relations from a single INFORMATION_SCHEMA.TABLES query to the driver's per-table GetObjects. The not-found-to-empty mapping predates it and was written for a dataset that does not exist; the migration is what first lets one table's 404 reach it. 653c19b (2026-08-10) added the INFORMATION_SCHEMA.TABLES re-check of an empty listing, but only for cross-project datasets, where GetObjects always returns nothing because it exposes the connection project alone.
• Same symptom, no cause: [Feature] Safer incrementals/snapshots (don't allow create or replace ...) #15953 (comment) (preview.212, two models recreated, rows lost). Same defect class on Snowflake: Catalog-linked databases: swallowed SHOW OBJECTS error empties the relation cache, breaking every multi-model run #15923.

[jeremyyeo]

https://github.com/jeremyyeo/dbt-16331-repro