fix: accept dbt 2.0 capitalized source freshness statuses (sources v3) - #113
Closed
dev-punia-altimate wants to merge 1 commit into
Closed
fix: accept dbt 2.0 capitalized source freshness statuses (sources v3)#113dev-punia-altimate wants to merge 1 commit into
dev-punia-altimate wants to merge 1 commit into
Conversation
…v3 parser dbt 2.0.0-preview.202 emits 'Pass'/'Error' where earlier dbt emitted lowercase. Every freshness result then failed validation and the whole sources.json was silently dropped (live: harvestgroup, 50 runs/day). Same class + same fix pattern as run_results_v6 'reused' (AI-7435/#106): str Enum + _missing_. Known statuses canonicalize by case to their lowercase members; unknown future statuses become dynamic members. The runtime-error union arm canonicalizes case only and stays narrow so it cannot swallow freshness rows. Version 0.3.5 -> 0.3.6. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
dbt 2.0.0-preview.202 emits capitalized source-freshness statuses (
Pass,Error) where every earlier dbt emitted lowercase. The sources v3 parser'sStatus1enum only knows lowercase, so every freshness result raisesValidationErrorand the ingestion worker silently drops the entiresources.json.Live impact (dbt artifact monitor, 2026-07-29): harvestgroup — 50 runs/day across 5 environments,
results[].Results.status ×2600: Input should be 'runtime error'. Receipt: real artifactrun_500657745/sources.jsonhas{'Pass': 45, 'Error': 5}and fails to parse on current main.The fix — same pattern as #106 (AI-7435, run_results
reused)Status1(freshness arm):str, Enum+_missing_— known statuses canonicalize by case to their lowercase members ('Pass'→Status1.pass_,.value == 'pass', so downstream stored values stay canonical); truly unknown future statuses surface as dynamic members instead of dropping the file.Status(runtime-error union arm): case-canonicalization only, stays narrow — it must not swallow freshness rows into the wrongResultsvariant (the concern fix: [AI-7435] accept dbt 2.0reusedrun status in run_results v6 parser #106 solved with union reordering).Why the runtime-error handling is safe (verified against dbt-fusion source)
Two facts close the "what if a capitalized
Erroris actually a runtime error?" question:'Error'rows in the real artifact are freshness failures, not runtime errors. Every one carries the full freshness shape —max_loaded_at(~2026-07-09),snapshotted_at(2026-07-28),criteria.error_after: 48h— i.e. data ~19 days stale against a 48-hour threshold.crates/dbt-schemas/src/schemas/common.rs:enum FreshnessStatus { Pass, Warn, Error }— serde serializes the variant names as-is, which is exactly the capitalization seen in the artifact. Andsources.rsdefinesresults: Vec<FreshnessResultsNode>— a single result shape; the runtime-error row type doesn't exist in Fusion's output. So the narrowStatusarm only ever serves classic dbt 1.x artifacts, which emit lowercase"runtime error"verbatim.New test
test_union_arm_routing_mixed_artifactlocks the routing: a classic runtime-error row parses intoResults, capitalized freshness rows (including'Error') parse intoResults1— asserted on the model types, not just status values.Receipts
{'error': 5, 'pass': 45}, all 50 rows in the freshness arm (Results1)ValidationErroron unpatched codetests/test_vendor/→ 41 passed (5 new intest_sources_v3.py: capitalized, lowercase-unchanged, unknown-future, narrow runtime-error arm, union arm routing)reusedrun status in run_results v6 parser #106)After merge
Release 0.3.6 → bump
altimate-datapilot-cliin altimate-backend + the Airflow worker pin in altimate-infra (same rollout as #106 / backend #5881 / infra #750).Reviewer: @sourabh — same defect class you fixed in #106; flagging per the agreed automation-proposes/you-verify flow.
🤖 Generated with Claude Code