feat(schema): publish versioned schema artifacts and a manifest (#74)

[blushi]

Closes #74 (WP1-07). Stacked on #90 (base feat/73-attestation-evidence-c06).
It generates the published artifacts from the composed schema, which needs #90's schemas and its shared QuantityValue (two classes no longer share the class IRI qudt:QuantityValue).

Implements ADR: ADR 0001 — "Standards boundaries already established" at 0cfe1c52 (#56, Proposed): claims use the schema-generated JSON-LD context from this repository, under a pinned context. This PR publishes that context per schema version, and its check verifies the RDF examples and conversion against a pinned schema revision, which the ADR lists under "Consequences and acceptance".

Differences from ADR (if any) and why: the published context is generated (gen-jsonld-context --xsd-anyuri-as-iri) and then corrected, because the generated context does not give the same RDF as LinkML's Turtle output (see Known limitations). The corrections are made by schema-artifacts.py, and every example and playground fixture is checked against its Turtle.

Usage, the versioning rule and the generator limitations are documented in schema/README.md, "Published schema artifacts".

Changes

• schema/versions/<version>/, new. Generated from src/schema.yaml and its imports, for the version it declares (version: 0.2.0):

  • context.jsonld: the JSON-LD context, corrected;
  • json-schema.json: gen-json-schema --include-range-class-descendants, the options linkml-validate uses. The entry point for a class is #/$defs/<class>;
  • shacl.ttl: gen-shacl, unchanged except that it is serialized canonically so that regeneration gives the same bytes;
  • linkml.yaml: the schema with every import merged, including linkml:types, so that generation and validation need no import lookups;
  • examples/: the JSON-LD examples with that version's context.

• schema/versions/manifest.json, new. For each version it records:

  • source: the git tree hash of schema/src, the blob hash of each module (which pins the imported base and shared modules), and the commit where known;
  • generators: the linkml and linkml-runtime versions;
  • files: each file's path and SHA-256;
  • entryPoints: Claim and each concrete subclass, with its class IRI, JSON Schema location and SHACL shape;
  • examples: each example's entry point, whether it is valid, and, if not, the only slots it may violate.

  The tree hash is what git rev-parse <commit>:schema/src prints. It survives a squash merge, and both recorded hashes match git's.

• Two versions.

  • 0.1.0 is the base Claim of #85 as upgraded by #87 (commit ab39066, tree f4c11af), with one entry point (Claim) and two examples. feat(schema): refactor Claim.yaml into the reusable base Claim #85's own head (8d817c7) was not used: its wasRevisionOf has range Resource, so its own SHACL rejects its revision example ("Value does not have class rfs:Resource"), and build(schema): upgrade LinkML to 1.11.1 #87 fixed that.
  • 0.2.0 is this stack (tree c9c0f57), with nine entry points and eleven examples.

• scripts/schema-artifacts.py, new. generate writes the artifacts, bundle copies the examples and writes the manifest entry, and check validates the bundle. --source, --version and --commit generate an older version from an archive of its commit.

• Makefile. New targets:

  • gen-context, gen-json-schema and gen-shacl, for one artifact at a time;
  • gen-schema-artifacts, which builds the artifacts, then the examples, then the bundle;
  • check-schema-artifacts [BASE=<ref>], which all now includes.

• check-schema-artifacts blocks network access, then checks:

  • that every listed file exists and has its digest;
  • that every entry point exists in the JSON Schema and the SHACL shapes;
  • that every example of every version is accepted by both validators, or, if invalid, rejected by both and only on its listed slots;
  • for the current version, that src/ is the recorded source and that regenerating gives the same bytes;
  • with BASE, that every version published in that ref is unchanged.

• CI (sparql-playground.yaml) runs the check. On a pull request it fetches the base branch and passes it as BASE.

• scripts/claim-examples.py.

  • Builds the examples with the published context of the current version, instead of one context per module.
  • Validates them with the published json-schema.json, instead of linkml-validate, and the published shacl.ttl.
  • Adds nested @type wherever LinkML inlines an object, including classes without an identifier.
  • Before the switch, the published JSON Schema gave the same verdicts and messages as linkml-validate on all 20 invalid examples and all 68 playground fixtures.
  • A new playground mode writes each fixture's JSON-LD and checks it against the fixture's Turtle.

• scripts/gen-rdf.sh.

  • Writes the playground JSON-LD with the published context: linkml-convert's own context wrote every uriorcurie value as an xsd:anyURI literal.
  • Checks that the JSON-LD and Turtle give the same graph, comparing numbers by value.
  • The playground step always uses data/playground, even when DATA_DIR is overridden.

• c06-mvp-claim-domain-invalid.jsonld, new, generated from c06-mvp-claim.INVALID-unknown-practice.yaml.

  • It is c06-mvp-claim.jsonld with one practice, CONTINUOUS_TILLAGE, that is not an ActivityType term. Its base Claim fields are those of the valid example.
  • Both validators reject it for the entry point C06SiteClaim, and the check asserts that every violation is on practices.

• src/schema.yaml and src/SDG.yaml: two prefix fixes, needed for the JSON-LD and Turtle to agree. Both bugs are already on main.

  • geo: was not declared in the root schema, so the Turtle output had IRIs such as <geo:asWKT> and <geo:Feature> in every project dataset. update-graph pushed those to the graph store.
  • sdgs: (https://sdgs.un.org/goals/) was not declared, so LinkML dropped the SDG enum's meanings, which are written as full IRIs, and supportsSDG was a string. It is not named sdg: LinkML's generated Python would name that namespace SDG and clash with the enum.

• Examples. All ten JSON-LD examples now carry the composed context, so each grows from about 6 KB to about 33 KB.

• Docs.

  • schema/README.md gets the new section.
  • docs/claim-base.md and docs/attestation-and-evidence.md now describe the published validators and list the domain-invalid example.

#74 checklist

Item	Status
gen-context, gen-json-schema, gen-shacl, gen-schema-artifacts targets	Done
Generate from schema/src/schema.yaml and its imports as a whole	Done
Record source revision and generator versions; document output locations	Done: manifest source and generators; README
Apply generated contexts to WP1-05/06 examples; run valid and invalid examples through the generated validators; record what a generator cannot express	Done: the examples use the published context and validators; limitations in the README
Commands, artifact links and limitations in schema/README.md	Done
No service hashing or conformance runner (WP3)	Respected
c06-mvp-claim.jsonld through the composed schema, with imported and inherited constraints	Done: entry point C06SiteClaim, closed shapes including the base Claim's
c06-mvp-claim-domain-invalid.jsonld; entry point documented; imported definitions included for offline validation	Done: linkml.yaml merges every import; the check blocks network access
Define and populate schema/versions/manifest.json	Done
Map each version to source revision, context, JSON Schema and SHACL, entry points and pinned dependencies	Done
An older supported version; changed definitions need a new version; mappings immutable	Done: 0.1.0; check-schema-artifacts BASE=<ref>, run in CI against the base branch
Check bundle completeness and offline example validation	Done
Publish the artifacts and manifest together	Committed together under schema/versions/. They are not served on the docs site
WP3-02 loads the bundle in the service	claims#17

Validation

Run locally with the pinned toolchain (requirements.txt: linkml 1.11.1, pyshacl 0.40.1, Python 3.10), on the tree committed as aa130d2:

Check	Result
make -C schema lint	No problems
make -C schema check-claim-examples	Passes: 71 checks, plus the CS-4 rule example, rejected by JSON Schema and, as marked, accepted by SHACL
make -C schema gen-rdf	68 of 68 Turtle conversions; 68 of 68 playground JSON-LD files give the same graph as their Turtle
make -C schema check-schema-artifacts	Passes: 2 versions, 13 examples (12 accepted, 1 rejected on practices only), entry points and digests
make -C schema gen-doc	Succeeds

Failure cases, tested with temporary inputs:

• a base ref whose manifest differs for 0.1.0 fails with "must not change: declare a new version";
• a base ref without a manifest is skipped;
• editing src/ without regenerating fails;
• parsing a remote @context with the network blocked raises an error instead of fetching.

Not run: update-graph (needs a graph store) and the site build.

Known limitations and open points

• Generator limitations, LinkML 1.11.1, verified this session and detailed in the README:
  • the generated context needs the corrections above;
  • rules are not translated to SHACL, so CS-4 is enforced by JSON Schema only (linkml/linkml#2464);
  • gen-shacl drops a class-level any_of. A slot-level any_of over enums gets no meaning mapping in the context, so SHACL rejects valid values, and linkml-convert fails on it;
  • a JSON-LD number in a decimal slot fails SHACL, while a string passes;
  • a subclass node fails sh:class without the class hierarchy;
  • a uriorcurie slot cannot be narrowed to an enum;
  • gen-shacl's output order varies between runs.
• Enum values without a meaning. In an enum where only some values have a meaning (ImpactType's UNKNOWN), a value without one expands against @vocab (rfs:) in JSON-LD, while the Turtle output has a string. No fixture uses it.
• Every change to src/ after a version is published needs a new version, including a description-only change, because the source is identified by its tree hash. Upgrading LinkML also changes the artifacts, so it needs a new version too.
• Source commits after the squash merge. 0.1.0's commit ab39066 will be orphaned when build(schema): upgrade LinkML to 1.11.1 #87 is squash-merged; GitHub keeps it under build(schema): upgrade LinkML to 1.11.1 #87's pull ref, and the tree hash identifies the source either way. 0.2.0 records no commit, because its merge commit does not exist yet.
• Example size. The composed context is inline in every example. The alternative is to reference a published context URL and resolve it offline, which needs a decision on that URL.
• Open for review: whether the two prefix fixes should move to their own PR against main, and whether the bundle should also be served on the docs site.