[codex] Land core platform migration slices - #11
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR consolidates the “core platform migration” surfaces by standardizing adapter-backed loading (DataContext.from_adapters(...) + ctx.load(...)), promoting temporal semantics into alphaforge.time, and expanding PIT’s ref-period, panel, and explainability contracts with accompanying docs, examples, benchmarks, and contract tests.
Changes:
- Introduces canonical temporal semantics under
alphaforge.time(release rules, missingness, ref-period normalization) with PIT compatibility shims and top-level re-exports. - Expands PIT’s public contract (typed ref queries, snapshot/panel metadata, lineage/explainability) and adds contract tests + benchmark harness.
- Refactors public-web loader foundations (shared base/finalization/schema/tabular/archive helpers) and updates docs/examples to reflect canonical vs legacy boundaries.
Reviewed changes
Copilot reviewed 124 out of 124 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_temporal_semantics_api.py | Verifies canonical exports and missingness classification behavior. |
| tests/test_release_rules.py | Adds coverage for weekday-aware weekly release expectations. |
| tests/test_ref_period.py | Adds parsing/normalization tests for pandas Period/Timestamp ref inputs and anchor enforcement. |
| tests/test_public_web_registry_exports.py | Validates root-level default public-web registry exports and class re-exports. |
| tests/test_pit_union_vintages_snapshot_panel.py | Extends PIT snapshot/panel tests to assert source_asof_utc and ref-anchor spec support. |
| tests/test_pit_release_helpers.py | Confirms PIT release helpers accept pandas Period refs. |
| tests/test_pit_ref_queries.py | Adds tests for canonical typed ref snapshot/revision query exports and behavior. |
| tests/test_pit_gdp_derived.py | Updates snapshot multi assertions to account for new source_asof_utc metadata. |
| tests/test_pit_explainability.py | Adds lineage + explainability contract tests for transforms and expression graphs. |
| tests/test_pit_accessor.py | Adds coverage for PITAccessor.open(...) bootstrap behavior. |
| tests/test_pipeline_health.py | Adds release-aware health semantics tests and report-building assertions. |
| tests/test_first_rate_futures_loader.py | Migrates usage from ctx.fetch(Query(...)) to canonical ctx.load(...). |
| tests/test_first_rate_bars.py | Migrates usage from ctx.fetch(Query(...)) to canonical ctx.load(...). |
| tests/test_dataset_spec_composition.py | Adds tests for FeatureRequestGroup composition, tags, keys, and override merge behavior. |
| tests/public_web/test_source_test_mapping.py | Updates non-source module/test-file lists for new public-web helper modules. |
| tests/fixtures/public_web/cftc_cot/sample_disagg.csv | Adds a fixture for disaggregated CoT parsing/coverage. |
| tests/contracts/test_volatility_dataset_contract.py | Adds adapter-backed volatility recipe contract coverage using canonical load path. |
| tests/contracts/test_pit_benchmarks.py | Smoke-tests the PIT benchmark harness contract. |
| tests/contracts/test_operations_contract.py | Adds contract coverage for health reporting and archive planning helpers. |
| tests/contracts/test_nowcast_pit_contract.py | Adds nowcast-style PIT ref-query + panel-builder contract coverage. |
| mkdocs.yml | Updates MkDocs nav to include new guides and API pages. |
| examples/volatility_dataset_recipe.py | Adds runnable volatility recipe example using canonical adapter-backed loading. |
| examples/short_rate_kim_dataset.py | Clarifies legacy/raw-loader usage boundary in example wording/comments. |
| examples/public_web_macro_derivs.py | Documents that the example exercises the raw-loader/DataSource path. |
| examples/pit_splice_importance.py | Switches to PITAccessor.open(...) for simpler PIT store bootstrap. |
| examples/first_rate_futures_continuous.py | Migrates example to canonical ctx.load(...) calls. |
| examples/first_rate_cross_asset_5m.py | Migrates example to canonical ctx.load(...) calls. |
| docs/guides/temporal-semantics.md | Documents the new temporal semantics layer and canonical import paths. |
| docs/guides/source-operations.md | Documents release-aware health reports and deterministic archive planning helpers. |
| docs/guides/research-recipes.md | Documents notebook-oriented volatility recipe patterns supported by the platform. |
| docs/guides/public-web-source-authoring.md | Adds contributor guidance for authoring/refactoring public-web sources. |
| docs/guides/pit.md | Updates PIT guide with ref-period query objects, panel metadata, lineage diagnostics, and open(...). |
| docs/guides/pit-api-contract.md | Expands PIT API contract docs to cover ref queries, explainability, and panel-long semantics. |
| docs/guides/development.md | Adds documented regression gates for core-platform surface changes. |
| docs/guides/dataset-spec.md | Documents FeatureRequestGroup composition rules and built-in market templates. |
| docs/guides/data-sources.md | Reframes DataContext routing around adapters + load/fetch_many/prefetch as canonical. |
| docs/guides/core-platform-migration.md | Adds migration guidance for downstream users targeting canonical surfaces. |
| docs/guides/core-platform-architecture.md | Documents canonical layer boundaries and compatibility-only surfaces. |
| docs/guides/contracts-and-benchmarks.md | Documents the contract suite and PIT benchmark harness as regression gates. |
| docs/getting-started/quickstart-public-web.md | Updates public-web quickstart to use default registry and clarifies compatibility boundary. |
| docs/api/time-ref-period.md | Adds API doc stub for alphaforge.time.ref_period. |
| docs/api/source-adapters.md | Updates adapter docs with DTCC family split and migration notes. |
| docs/api/public-web.md | Adds a detailed public-web API overview and helper-module map. |
| docs/api/pit-release-rules.md | Repoints release rule docs to canonical alphaforge.time.release_rules. |
| docs/api/pit-queries.md | Adds API doc stub for alphaforge.pit.queries. |
| docs/api/pit-missingness.md | Repoints missingness docs to canonical alphaforge.time.missingness. |
| docs/api/pit-accessor.md | Adds documentation note for PITAccessor.open(...) bootstrap. |
| docs/api/market-templates.md | Adds API doc stub for built-in market templates. |
| docs/api/dataset-spec.md | Updates dataset-spec API doc intro to reflect canonical composition surfaces. |
| docs/api/data-context.md | Updates DataContext API doc intro for canonical adapter-first loading and routing. |
| benchmarks/pit.py | Adds PIT benchmark harness for snapshot_ref + panel-long contract performance tracking. |
| benchmarks/init.py | Exposes the benchmark harness via a lightweight module attribute loader. |
| alphaforge/time/missingness.py | Introduces canonical missingness taxonomy and classifier in alphaforge.time. |
| alphaforge/time/init.py | Exposes canonical temporal semantics types/functions from alphaforge.time. |
| alphaforge/pit/target.py | Switches quarter ref parsing to coerce_ref_period(...) for unified ref-period handling. |
| alphaforge/pit/panel.py | Refactors wide panel build to use get_snapshot_multi(...) batch API. |
| alphaforge/pit/observation.py | Updates type-checking import path for ReleaseRule to alphaforge.time. |
| alphaforge/pit/models.py | Extends SnapshotSeriesSpec coercion to accept ref inputs + freq/anchor semantics. |
| alphaforge/pit/missingness.py | Replaces PIT missingness implementation with a compatibility shim to alphaforge.time.missingness. |
| alphaforge/pit/catalog.py | Repoints ReleaseRule import to canonical alphaforge.time.release_rules. |
| alphaforge/pit/adapters/alphaforge_layer.py | Normalizes ref inputs via coerce_ref_period(...) in layer methods. |
| alphaforge/pit/init.py | Re-exports typed ref query objects and coercion helpers from PIT package root. |
| alphaforge/pipeline/tracker.py | Records overdue_days and adds report(...) helper using build_health_report. |
| alphaforge/pipeline/init.py | Exports health report helpers alongside existing health APIs. |
| alphaforge/futures/loader.py | Minor import ordering cleanup. |
| alphaforge/features/dataset_spec.py | Adds FeatureRequestGroup, override merging, request flattening, and policy validation. |
| alphaforge/features/dataset_builder.py | Switches to flattened feature requests and annotates catalog with request/template metadata. |
| alphaforge/features/init.py | Re-exports built-in market templates from the features package. |
| alphaforge/data/transforms/dtcc_pit.py | Makes DTCC PIT transform configurable for key prefix and lineage source naming. |
| alphaforge/data/transforms/cot_pit.py | Makes CoT PIT transform configurable for key prefix and lineage source naming. |
| alphaforge/data/sources/cftc.py | Extends CFTC adapter to support multiple datasets via per-dataset raw fetchers and routing. |
| alphaforge/data/source.py | Clarifies DataSource as a legacy/compatibility protocol in module/class docs. |
| alphaforge/data/public_web/tabular.py | Introduces shared helpers and a TabularDocumentSourceBase for tabular public-web loaders. |
| alphaforge/data/public_web/schema_helpers.py | Adds helpers to build consistent TableSchema definitions for common shapes. |
| alphaforge/data/public_web/registry.py | Refactors default registry construction via a factory list and adds disaggregated CoT source. |
| alphaforge/data/public_web/registry_api.py | Adds a registry-backed API base class for registry-driven public-web sources. |
| alphaforge/data/public_web/philadelphia_spf.py | Migrates to PublicWebSourceBase and shared finalize/schema helpers. |
| alphaforge/data/public_web/mof_jgb.py | Migrates to PublicWebSourceBase and shared finalize/schema helpers. |
| alphaforge/data/public_web/lch_cdsclear_daily.py | Migrates to tabular helper base and shared schema/finalization helpers. |
| alphaforge/data/public_web/ibge_sidra.py | Migrates to registry API base and shared schema/finalization helpers. |
| alphaforge/data/public_web/frb_term_structure.py | Migrates to PublicWebSourceBase and shared finalize/schema helpers. |
| alphaforge/data/public_web/finalize.py | Adds shared schema-aware finalization utilities for projection/filtering/sorting/empties. |
| alphaforge/data/public_web/ezoic_adrevenue_daily.py | Migrates to tabular helper base and shared schema/finalization helpers. |
| alphaforge/data/public_web/eurostat.py | Migrates to registry API base and shared schema/finalization helpers. |
| alphaforge/data/public_web/eurex_stats_daily.py | Migrates to tabular helper base and shared schema/finalization helpers. |
| alphaforge/data/public_web/eurex_refdata_contracts.py | Migrates to PublicWebSourceBase and shared finalize/schema helpers. |
| alphaforge/data/public_web/eia.py | Migrates to registry API base and shared schema/finalization helpers. |
| alphaforge/data/public_web/ecb_sdmx.py | Migrates to registry API base and shared schema/finalization helpers. |
| alphaforge/data/public_web/ec_weekly_oil_bulletin.py | Migrates to tabular helper base and shared schema/finalization helpers. |
| alphaforge/data/public_web/dtcc_ppd.py | Migrates DTCC PPD source to PublicWebSourceBase + schema/finalize helpers. |
| alphaforge/data/public_web/destatis_genesis.py | Migrates to registry API base and shared schema/finalization helpers. |
| alphaforge/data/public_web/cme_productslate_reference.py | Migrates to tabular helper base and shared schema/finalization helpers. |
| alphaforge/data/public_web/cftc_swaps_weekly.py | Adds archive planning + ZIP handling and migrates to PublicWebSourceBase + finalize. |
| alphaforge/data/public_web/bls.py | Migrates to PublicWebSourceBase and shared schema/finalization helpers. |
| alphaforge/data/public_web/bea.py | Migrates to registry API base and shared schema/finalization helpers. |
| alphaforge/data/public_web/bcb_sgs.py | Migrates to PublicWebSourceBase and shared schema/finalization helpers. |
| alphaforge/data/public_web/base.py | Adds PublicWebSourceBase providing shared HTTP/asof/empty/finalize utilities. |
| alphaforge/data/public_web/b3_historical_quotes.py | Migrates to archive planning helpers and shared finalize/schema helpers. |
| alphaforge/data/public_web/archive.py | Adds deterministic archive fetch planning and ZIP-member read helpers. |
| alphaforge/data/public_web/init.py | Re-exports added public-web sources (incl. disaggregated CoT) from package. |
| alphaforge/data/pit_source.py | Clarifies PITDataSource as a legacy/raw-loader bridge in docstring. |
| alphaforge/data/fred_source.py | Clarifies FREDDataSource as legacy/raw-loader with preferred adapter path. |
| alphaforge/init.py | Updates package-root exports to include canonical time semantics, PIT queries, templates, and public-web registry helpers. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
119
to
127
| if all_cached and cached_frames: | ||
| combined = pd.concat(cached_frames, ignore_index=True) | ||
| return FetchResult( | ||
| data=combined, | ||
| source=self.source_name, | ||
| dataset=query.table, | ||
| is_pit=True, | ||
| cached_at=cached_at, | ||
| ) |
There was a problem hiding this comment.
CFTCAdapter returns different columns on cache hit vs cache miss: CacheLayer.lookup drops the source column, so the cached path returns only series_key/obs_date/asof_utc/value. The non-cached path returns PIT-format rows including source (e.g. "cftc_cot_disagg"), and some tests/consumers expect that lineage column. Consider restoring a deterministic source column in the cache-hit branch (based on dataset -> lineage mapping) or adjusting the caching layer so PIT rows preserve source consistently.
Contributor
There was a problem hiding this comment.
Fixed in 8c06c07. In CFTCAdapter.fetch, the cache-hit branch now restores the lineage source column using _DATASET_SOURCE_NAMES.get(dataset, "cftc_cot") after concatenating cached frames — giving the same deterministic value (e.g. "cftc_cot_disagg") that cot_to_pit_observations sets on a cache miss.
Two regression tests added in TestCFTCAdapterBulkCache:
test_cache_hit_preserves_source_column— verifies"cftc_cot"is present on both cache-miss and cache-hit forcot.tfftest_cache_hit_preserves_disagg_source_column— verifies"cftc_cot_disagg"is present on both paths forcot.disagg
Comment on lines
+95
to
+103
| if ext.endswith(".zip"): | ||
| member = read_first_zip_member(payload, suffixes=(".csv", ".xlsx", ".xls")) | ||
| if member is None: | ||
| return pd.DataFrame() | ||
| member_name, member_payload = member | ||
| member_ext = member_name.lower() | ||
| if member_ext.endswith(".csv"): | ||
| return parse_csv_bytes(member_payload) | ||
| return parse_xlsx_bytes(member_payload) |
There was a problem hiding this comment.
When a planned archive URL is a ZIP, _read_file returns an empty DataFrame if no member with the expected suffix is found. That can silently skip a requested archive year/file and yield partial history, which is risky operationally. Consider raising an explicit error (including the ZIP URL/artifact name and expected suffixes) so broken archive structure changes fail fast instead of being treated as an empty fetch.
…OURCE_NAMES When all requested entities are served from cache, the combined DataFrame was missing the lineage 'source' column because CacheLayer.lookup drops internal columns before returning. Restore the deterministic source value (e.g. 'cftc_cot_disagg') from _DATASET_SOURCE_NAMES in the cache-hit branch, matching the schema produced by cot_to_pit_observations on a cache miss. Add two regression tests: - test_cache_hit_preserves_source_column: verifies source column on cache-hit equals cache-miss for cot.tff - test_cache_hit_preserves_disagg_source_column: verifies 'cftc_cot_disagg' is present and correct on cache-hit for cot.disagg Agent-Logs-Url: https://github.com/steveya/alphaforge/sessions/c996d980-e2e2-4541-a6f0-4271aa518e9a Co-authored-by: steveya <7390489+steveya@users.noreply.github.com>
…ue assertions Agent-Logs-Url: https://github.com/steveya/alphaforge/sessions/c996d980-e2e2-4541-a6f0-4271aa518e9a Co-authored-by: steveya <7390489+steveya@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CacheLayer.lookupdropssourcecolumn; cache-hit path inCFTCAdapter.fetchreturns DataFrame withoutsourceCFTCAdapter.fetchto restore thesourcecolumn in the cache-hit branch using_DATASET_SOURCE_NAMEScot.tff(cftc_cot) andcot.disagg(cftc_cot_disagg) on both cache-miss and cache-hit paths