feat(connector): network-as-subcatalog architecture, STAC compliance fixes, DCAT mapping migration - #204
Open
Vishmayraj wants to merge 50 commits into
Open
feat(connector): network-as-subcatalog architecture, STAC compliance fixes, DCAT mapping migration#204Vishmayraj wants to merge 50 commits into
Vishmayraj wants to merge 50 commits into
Conversation
… currently just HARVEST_INTERVAL_MINUTES
…iting a base harvester error class
…we have dataclass in harvester.py
…catalog from initial HTTP arch
… and including them in main api.py
… with a /connector endpoint
…ce transformation time
…but using a seperate connection for lock and harvest
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR extends the STAC 1.0 connector introduced in #195 with a network-as-subcatalog architecture, resolves a persistent dummy-data/schema issue that was blocking test verification, and migrates the DCAT-AP 3.0 mapping reference into the main repo ahead of
dcat_transformer.pyimplementation.What's in this PR
Network-as-subcatalog architecture
harvester.pynow exposes two separate harvest queries, one for deployments using theNetworkentity, one without, so non-network deployments aren't forced through unnecessary grouping logic.build_stac_catalog_with_networksand its supporting helpers, restructuring the STAC output so each network is served as its own subcatalog rather than a single flattened top-level catalog.cache.pygained a per-network caching path, so network-scoped catalogs are written and invalidated independently of the base catalog.scheduler.pyupdated to call the new network-aware build functions._item_nav_links, which still had hardcoded orphan-collection paths from before the network architecture existed and was producing broken navigation links for items outside any network.STAC compliance fixes
licenseandconformancefields for full STAC 1.0 compliance.config.py.Testing/dummy-data bug -> now resolved
generator.py), and brought the database init scripts in line with the latest upstream schema.DCAT-AP 3.0 mapping migration started
docs/, reconciled against the current harvesting layer and network architecture.dcat_transformer.pyimplementation is scoped as the next PR.Notes
dcat_transformer.pyremains a stub, as noted infeat(connector):STAC 1.0 connector - harvesting, transformation, scheduling, caching, and API layer #195.