Skip to content

feat(connector): network-as-subcatalog architecture, STAC compliance fixes, DCAT mapping migration - #204

Open
Vishmayraj wants to merge 50 commits into
istSOS:STACfrom
Vishmayraj:gsoc-stac-next
Open

feat(connector): network-as-subcatalog architecture, STAC compliance fixes, DCAT mapping migration#204
Vishmayraj wants to merge 50 commits into
istSOS:STACfrom
Vishmayraj:gsoc-stac-next

Conversation

@Vishmayraj

Copy link
Copy Markdown
Contributor

Summary

This PR extends the STAC 1.0 connector introduced in #195 with a network-as-subcatalog architecture, resolves a persistent dummy-data/schema issue that was blocking test verification, and migrates the DCAT-AP 3.0 mapping reference into the main repo ahead of dcat_transformer.py implementation.


What's in this PR

Network-as-subcatalog architecture

  • harvester.py now exposes two separate harvest queries, one for deployments using the Network entity, one without, so non-network deployments aren't forced through unnecessary grouping logic.
  • Added build_stac_catalog_with_networks and its supporting helpers, restructuring the STAC output so each network is served as its own subcatalog rather than a single flattened top-level catalog.
  • cache.py gained a per-network caching path, so network-scoped catalogs are written and invalidated independently of the base catalog.
  • scheduler.py updated to call the new network-aware build functions.
  • Fixed _item_nav_links, which still had hardcoded orphan-collection paths from before the network architecture existed and was producing broken navigation links for items outside any network.

STAC compliance fixes

  • Transformer output now includes license and conformance fields for full STAC 1.0 compliance.
  • Removed hardcoded configuration values from the connector; these are now sourced from config.py.

Testing/dummy-data bug -> now resolved

  • Fixed a parameter crash, an environment variable default, and a delete-user issue in the dummy-data generation path (generator.py), and brought the database init scripts in line with the latest upstream schema.
  • Root cause was a stale branch, and a fix for the underlying issue had already landed upstream. This had been blocking verification of the network architecture above via the testing infrastructure.

DCAT-AP 3.0 mapping migration started

  • Migrated the DCAT-AP 3.0 field mapping reference doc from the pre-GSoC standalone research repo into docs/, reconciled against the current harvesting layer and network architecture.
  • dcat_transformer.py implementation is scoped as the next PR.

Notes


…but using a seperate connection for lock and harvest
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant