Skip to content

fix: accept dbt 2.0 capitalized source freshness statuses (sources v3) - #113

Closed
dev-punia-altimate wants to merge 1 commit into
mainfrom
fix/sources-v3-status-case
Closed

fix: accept dbt 2.0 capitalized source freshness statuses (sources v3)#113
dev-punia-altimate wants to merge 1 commit into
mainfrom
fix/sources-v3-status-case

Conversation

@dev-punia-altimate

@dev-punia-altimate dev-punia-altimate commented Jul 29, 2026

Copy link
Copy Markdown

What broke

dbt 2.0.0-preview.202 emits capitalized source-freshness statuses (Pass, Error) where every earlier dbt emitted lowercase. The sources v3 parser's Status1 enum only knows lowercase, so every freshness result raises ValidationError and the ingestion worker silently drops the entire sources.json.

Live impact (dbt artifact monitor, 2026-07-29): harvestgroup — 50 runs/day across 5 environments, results[].Results.status ×2600: Input should be 'runtime error'. Receipt: real artifact run_500657745/sources.json has {'Pass': 45, 'Error': 5} and fails to parse on current main.

The fix — same pattern as #106 (AI-7435, run_results reused)

  • Status1 (freshness arm): str, Enum + _missing_ — known statuses canonicalize by case to their lowercase members ('Pass'Status1.pass_, .value == 'pass', so downstream stored values stay canonical); truly unknown future statuses surface as dynamic members instead of dropping the file.
  • Status (runtime-error union arm): case-canonicalization only, stays narrow — it must not swallow freshness rows into the wrong Results variant (the concern fix: [AI-7435] accept dbt 2.0 reused run status in run_results v6 parser #106 solved with union reordering).

Why the runtime-error handling is safe (verified against dbt-fusion source)

Two facts close the "what if a capitalized Error is actually a runtime error?" question:

  1. The 5 'Error' rows in the real artifact are freshness failures, not runtime errors. Every one carries the full freshness shape — max_loaded_at (~2026-07-09), snapshotted_at (2026-07-28), criteria.error_after: 48h — i.e. data ~19 days stale against a 48-hour threshold.
  2. dbt 2.0 (Fusion) has no runtime-error variant in sources.json at all. crates/dbt-schemas/src/schemas/common.rs: enum FreshnessStatus { Pass, Warn, Error } — serde serializes the variant names as-is, which is exactly the capitalization seen in the artifact. And sources.rs defines results: Vec<FreshnessResultsNode> — a single result shape; the runtime-error row type doesn't exist in Fusion's output. So the narrow Status arm only ever serves classic dbt 1.x artifacts, which emit lowercase "runtime error" verbatim.

New test test_union_arm_routing_mixed_artifact locks the routing: a classic runtime-error row parses into Results, capitalized freshness rows (including 'Error') parse into Results1 — asserted on the model types, not just status values.

Receipts

  • Real harvestgroup artifact parses post-fix: {'error': 5, 'pass': 45}, all 50 rows in the freshness arm (Results1)
  • Red-before-green: same artifact raises ValidationError on unpatched code
  • tests/test_vendor/41 passed (5 new in test_sources_v3.py: capitalized, lowercase-unchanged, unknown-future, narrow runtime-error arm, union arm routing)
  • Version 0.3.5 → 0.3.6 (same 4 files as fix: [AI-7435] accept dbt 2.0 reused run status in run_results v6 parser #106)

After merge

Release 0.3.6 → bump altimate-datapilot-cli in altimate-backend + the Airflow worker pin in altimate-infra (same rollout as #106 / backend #5881 / infra #750).

Reviewer: @sourabh — same defect class you fixed in #106; flagging per the agreed automation-proposes/you-verify flow.

🤖 Generated with Claude Code

…v3 parser

dbt 2.0.0-preview.202 emits 'Pass'/'Error' where earlier dbt emitted
lowercase. Every freshness result then failed validation and the whole
sources.json was silently dropped (live: harvestgroup, 50 runs/day).
Same class + same fix pattern as run_results_v6 'reused' (AI-7435/#106):
str Enum + _missing_. Known statuses canonicalize by case to their
lowercase members; unknown future statuses become dynamic members. The
runtime-error union arm canonicalizes case only and stays narrow so it
cannot swallow freshness rows. Version 0.3.5 -> 0.3.6.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant